Purpose: Availability of resuscitation episode datasets with cardiac rhythm annotations would improve the understanding of the interplay between therapies and patient response (rhythm state), contributing to enhance interventions. Resuscitation rhythm annotation is challenging for reasons like ambiguous rhythm definitions, annotator biases, borderline rhythms or artefacts. The objective of this study was to identify challenges for clinicians when annotating resuscitation rhythms, and to define procedures to obtain quality controlled datasets with rhythm annotations.
Materials and methods: Data from automated external defibrillators (AEDs) used between 2013 and 2015 by basic life support personnel in the Basque autonomous region (Spain) were analysed. The AED’s analysis intervals were used to annotate rhythms, 4375 intervals from 855 out-of-hospital cardiac arrest files were annotated. Following strict rhythm definition charts, six clinicians independently annotated rhythms into four categories: asystole (AS, rate<12bpm, amplitude<100V), organized rhythm (ORG), ventricular tachycardia (VT), ventricular fibrillation (VF, amplitude>200V). An unclear category was added for intermediate rhythms (fine VF, slow VT), rhythm transitions, unclear rhythms or artefacts. Reviewer consensus was measured using Fleiss’ kappa () and a modified kappa (1) in which cases with only one disagreement were considered perfect agreement (systematic annotation errors and/or annotation lapses). 
Results: The table summarizes the agreement among reviewers for all analyses (A) and excluding intermediate rhythms and artefacts (B). Analyses were repeated for all reviewers and for the 5 and 4 reviewers with best agreement. Disagreements among 4 reviewers (n=455) included: borderline rhythms AS/ORG (n=149), VF/VT (n=17), ORG/VT (n=9); artefacts (n=31 chest compressions, n=35 other); intermediate rhythms (n=44); and annotation errors (170). 


Conclusions: Agreement was very high (>0.8), and increased considerably after excluding reviewers that systematically deviated from rhythm chart definitions (1 case), applying inclusion criteria (B) and removing isolated disagreements (1). The main causes of disagreement were AS/ORG borderline rhythms and isolated annotation errors.

Challenges for clinicians in ECG based retrospective resuscitation rhythm annotation