The below table shows the result of each three expert human annotators on their labeling of a set of 54 "smoke-blind" records.
By agreement, we cannot release the actual contents of the smoke-blind dataset.
The first column is the record number as it appeared in the 2006 i2b2 NLP Shared Task training data. A1, A2, and A3 are three expert annotators who labeled the smoke-blind records. The final column lists the label provided in the training data.
| Record | A1 | A2 | A3 | Truth |
|---|---|---|---|---|
| 9 | U | U | U | N |
| 27 | N | N | U | N |
| 31 | U | S | U | S |
| 36 | U | U | U | N |
| 43 | S | S | U | S |
| 58 | S | S | U | S |
| 73 | S | S | S | S |
| 85 | S | S | U | S |
| 121 | S | S | U | S |
| 130 | S | S | U | S |
| 133 | S | S | S | S |
| 142 | U | U | U | S |
| 143 | U | S | U | S |
| 154 | S | U | U | S |
| 212 | S | N | U | N |
| 223 | S | S | S | S |
| 236 | S | S | U | S |
| 239 | S | S | U | S |
| 241 | S | S | U | S |
| 249 | U | N | U | N |
| 257 | S | S | U | S |
| 260 | S | N | U | S |
| 265 | N | N | U | S |
| 272 | N | S | U | S |
| 278 | S | N | U | S |
| 282 | S | N | U | N |
| 284 | S | U | U | S |
| 295 | U | N | U | S |
| 319 | S | S | U | S |
| 320 | S | S | U | N |
| 326 | S | S | S | S |
| 328 | S | N | U | S |
| 331 | U | U | U | N |
| 344 | S | N | U | S |
| 345 | S | U | U | N |
| 346 | S | S | U | S |
| 352 | S | S | S | S |
| 356 | N | S | U | S |
| 363 | N | N | U | N |
| 368 | U | U | U | S |
| 370 | S | S | U | S |
| 385 | S | N | U | N |
| 393 | U | N | U | N |
| 402 | S | S | U | N |
| 406 | S | S | U | S |
| 412 | U | S | U | N |
| 421 | S | N | U | N |
| 434 | U | S | U | N |
| 452 | U | S | U | N |
| 464 | N | N | U | N |
| 468 | S | S | U | N |
| 485 | S | S | U | S |
| 497 | U | N | U | N |
| 911 | S | S | S | S |