ADHD-200 Data Corrections and Test-Set Update Posted By: Michael Milham - Jun 24, 2011Tool/Resource: 1000 Functional Connectomes Project Dear all, I am writing with two updates. I’ll start with the bad news (and apologies), then follow with the good. On the bad news front, while preparing the protocol for the ADHD-200 test dataset release, Maarten Mennes unearthed an error in the primary data release. Specifically, he found that, despite our best efforts to ensure the accuracy of the data, the final phenotypic data contained several errors with respect to IQ measures at the KKI site. The errors originated in the initial translation of site-specific measures to the common phenotypic set. They went undetected in our final quality control procedures, which involve inspection of the data by two independent raters, due to deviations from the specified protocol. This is highly unfortunate, and we are deeply embarrassed by the incident. The end result is that for the KKI site, 38 of the datasets have incorrect IQ scores. After uncovering the error, Maarten conducted an exhaustive dataset-wide quality control check, using an automated protocol written by our new research programmer (Sharad Sikka). This automated check was followed with manual inspection of any suspicious values. The search verified the integrity of the primary ADHD-200 dataset. Across all 776 datasets, the only additional discrepancies were that: 1) the sex of 4 of the 222 NYU participants was incorrectly specified, due to a conflict between information entered at the scanner (and preserved in the DICOM) and that entered by the clinical assessor (chart review was employed to reconcile these discrepancies), 2) 4 of the 79 OHSU participants were noted to have discrepancies – specifically, the sex of one participant was incorrectly specified, while for 3 participants, age was incorrectly specified, and 3) sex was incorrectly specified for 1 of 194 Peking/Beijing participants. Updated ADHD-200 site release files and a quick-fix .csv file will be available on 6/27. Once again, we sincerely apologize for these errors, and any inconvenience they have caused. We hope that the accuracy checking steps described above will alleviate any concerns about the presence of further errors in the data. 2) We are on target for the July 1 release of the test dataset. Data from all but one site have been received, yielding a total of 254 potential test datasets, with 175 passing both visual inspection and the requirement for a maximum of 4mm movement in Euclidean space. Our team will use the automated phenotypic data-check system described above, followed by manual verification of all data-values by two independent raters, to ensure an error-free test-set data release. Realizing the potential inconvenience resulting from errors in the training dataset, we will extend the due date for final entries for the diagnostic classification portion of the competition to August 30, 2011. Sincerely, Mike |
| Latest News |
|