How do annotators label short texts? Toward understanding the temporal dynamics of tweet labeling

As case study we investigate a hierarchical tweet labeling task, distinguishing first between relevant and irrelevant tweets, before classifying the relevant ones into factual and non-factual, and further splitting the non-factual ones into positive and negative. As indicator of learning we use the annotation time, i.e. the elapsed time for the inspection of a tweet before the labels across the hierarchy are assigned to it. We show that this annotation time drops as an annotator proceeds through the set of tweets she has to process. We report on our results on identifying the learning phase and its follow-up exploitation phase, and on the differences in annotator behavior during each phase.