Validity of altmetrics data for measuring societal impact: A study using data from Altmetric and F1000Prime

• Can altmetric data be validly used for the measurement of societal impact?
• The study is based on data from F1000, Altmetric, and an in-house database based on Web of Science.
• In the F1000 peer review system, experts assess scientific papers and attach particular tags.
• With better experts’ recommendation scores, higher total altmetric and citation counts are to be expected.
• In contrast to citation counts, papers with the tag “good for teaching” achieve higher altmetric counts than papers without this tag.

Can altmetric data be validly used for the measurement of societal impact? The current study seeks to answer this question with a comprehensive dataset (about 100,000 records) from very disparate sources (F1000, Altmetric, and an in-house database based on Web of Science). In the F1000 peer review system, experts attach particular tags to scientific papers which indicate whether a paper could be of interest for science or rather for other segments of society. The results show that papers with the tag “good for teaching” do achieve higher altmetric counts than papers without this tag – if the quality of the papers is controlled. At the same time, a higher citation count is shown especially by papers with a tag that is specifically scientifically oriented (“new finding”). The findings indicate that papers tailored for a readership outside the area of research should lead to societal impact.If altmetric data is to be used for the measurement of societal impact, the question arises of its normalization. In bibliometrics, citations are normalized for the papers’ subject area and publication year. This study has taken a second analytic step involving a possible normalization of altmetric data. As the results show there are particular scientific topics which are of especial interest for a wide audience. Since these more or less interesting topics are not completely reflected in Thomson Reuters’ journal sets, a normalization of altmetric data should not be based on the level of subject categories, but on the level of topics.