Bibliometrics research has used citation data to empirically investigate and characterize scientific and academic fields (Ardanuy, 2013; Börner et al., 2003; Cronin, 2008), trace interdisciplinarity (Yu & Pan, 2021), map research trends (Xiao et al., 2021), among others. Many of these works have relied on databases like Web of Science, Scopus (Zhu & Liu, 2020) and Google Scholar (Harzing & Wal, 2008) despite the fact that research has shown the errors and biases that can be induced when using these sources (Franceschini et al., 2016; Gutiérrez et al., 2022; Zhu et al., 2019). Moreover, although dissertations are perhaps the most important academic publication of junior researchers these outputs are not indexed in Web of Science or Scopus and only a few university repositories are indexed by Google Scholar (Kousha & Thelwall, 2019), thus these works are almost a neglected source.
In this presentation we propose a two-folded contribution, on the one side, we will use reference mining (RM) to unveil an untapped source of data: Humanities and Social Sciences (HSS) Electronic Thesis and Dissertations (ETDs) using a state-of-the-art tool (Grennan et al., 2019; Rodrigues Alves et al., 2018): Grobid (Lopez, 2012/2019). On the other side, we will offer an understanding of the RM pipeline via an exploratory study to understand factors that could lead to disparities in machine learning based reference extraction results. We expect our study to yield practical lessons for future bibliometricians who seek to make use of reference data, mined from research literature to understand scholarly communities.
We will present the lessons learned by running Grobid across a full text corpus of 1,132 HSS digitally born dissertations (2011-2020) in Spanish and English from the same institution: El Colegio de México, a university of higher learning in Mexico City (Mexico).
Fig. 1 Examp le of reference string components (Rodrigues Alves et al., 2018)
Given that there is no “gold standard” reference for this type of corpora, we have used a proxy for faulty references: publication year (see fig. 1). That is, by identifying all reference strings that do not follow the YYYY format, of which fig. 2 is one example, we have created a binary classifier that distinguishes reference strings that contain at least one mistake.
Fig. 2 Examp le of a typical error in date segmentation using GROBID
We will present a typification of errors both at the token level and at the metadata field level. For the first level, we will identify tokens (“words”) that can help predict a faulty reference. Our intention is to find tokens that are more relevant for the classification task of faulty references using a logistic regression method following previous experiments with this method (Zeng & Acuna, 2020). For instance, in Fig. 2 “ProQuest” is a token that has a significant logistic regression coefficient in the string where the date was wrong (“302841123”), and could be a candidate to predict that strings with this token in the title segmentation are faulty extracted references. As for the metadata level, we will show which metadata fields (such as discipline, language or year) significantly correlate with more mistakes using random forests as done in another experiment in which we determined metadata factors that could correlate with more citations (Gutiérrez De la Torre & Equihua Benítez, 2019).
References
Ardanuy, J. (2013). Sixty years of citation analysis studies in the humanities (1951–2010). Journal of the American Society for Information Science and Technology, 64(8), 1751–1755. https://doi.org/10.1002/asi.22835
Börner, K., Chen, C., & Boyack, K. W. (2003). Visualizing knowledge domains. Annual Review of Information Science and Technology, 37(1), 179–255. https://doi.org/10.1002/aris.1440370106
Cronin, B. (2008). Journal citation among heterodox economists 1995‐2007: Dynamics of community emergence. On the Horizon, 16(4), 226–240. https://doi.org/10.1108/10748120810912556
Franceschini, F., Maisano, D., & Mastrogiacomo, L. (2016). Empirical analysis and classification of database errors in Scopus and Web of Science. Journal of Informetrics, 10(4), 933–953. https://doi.org/10.1016/j.joi.2016.07.003
Grennan, M., Schibel, M., Collins, A., & Beel, J. (2019). GIANT: The 1-Billion Annotated Synthetic Bibliographic-Reference-String Dataset for Deep Citation Parsing [Data] [Data set]. Harvard Dataverse. https://doi.org/10.7910/DVN/LXQXAO
Gutiérrez De la Torre, S. E., & Equihua Benítez, J. A. (2019, May). Las rutas del idioma: Circulación de las ciencias sociales y humanidades. Mesa Redonda, El Colegio Nacional. https://youtu.be/rd8-Rgr5bZE?t=5320
Gutiérrez, S., Ortiz, V., Escobar, I., Bocanegra, T., Cid, V., Escobar, C., Quiroa, L., & Romero, C. (2022). Datos bibliométricos para las Ciencias Sociales y las Humanidades: Un método para el acopio, validación y análisis con herramientas de acceso gratuito. Memorias de La Conferencia Internacional BIREDIAL-ISTEC.
Harzing, A.-W. K., & Wal, R. van der. (2008). Google Scholar as a new source for citation analysis. Ethics in Science and Environmental Politics, 8(1), 61–73. https://doi.org/10.3354/esep00076
Kousha, K., & Thelwall, M. (2019). Can Google Scholar and Mendeley help to assess the scholarly impacts of dissertations? Journal of Informetrics, 13(2), 467–484. https://doi.org/10.1016/j.joi.2019.02.009
Lopez, P. (2019). Grobid [Java]. https://github.com/kermitt2/grobid (Original work published 2012)
Rodrigues Alves, D., Colavizza, G., & Kaplan, F. (2018). Deep Reference Mining From Scholarly Literature in the Arts and Humanities. Frontiers in Research Metrics and Analytics, 3. https://doi.org/10.3389/frma.2018.00021
Rovira, C., Codina, L., & Lopezosa, C. (2021). Language Bias in the Google Scholar Ranking Algorithm. Future Internet, 13(2), 31. https://doi.org/10.3390/fi13020031
Xiao, Y., Wu, H., Wang, G., & Mei, H. (2021). Mapping the Worldwide Trends on Energy Poverty Research: A Bibliometric Analysis (1999–2019). International Journal of Environmental Research and Public Health, 18(4), 1764. https://doi.org/10.3390/ijerph18041764
Yu, D., & Pan, T. (2021). Tracing the main path of interdisciplinary research considering citation preference: A case from blockchain domain. Journal of Informetrics, 15(2), 101136. https://doi.org/10.1016/j.joi.2021.101136
Zeng, T., & Acuna, D. E. (2020). Modeling citation worthiness by using attention-based bidirectional long short-term memory networks and interpretable models. Scientometrics, 124(1), 399–428. https://doi.org/10.1007/s11192-020-03421-9
Zhu, J., Hu, G., & Liu, W. (2019). DOI errors and possible solutions for Web of Science. Scientometrics, 118(2), 709–718. https://doi.org/10.1007/s11192-018-2980-7
Zhu, J., & Liu, W. (2020). A tale of two databases: The use of Web of Science and Scopus in academic papers. Scientometrics, 123(1), 321–335. https://doi.org/10.1007/s11192-020-03387-8

Changing the bibliometrics game: Reference mining Social Sciences and Humanities ETDs and detecting errors for the common good

Join us for ETD2021 (Electronic Theses and Dissertations) Conference in November 2021 in the United Arab Emirates.

Our theme, ‘Open scholarship in a post-pandemic world’ addresses the enhanced need for discoverability of research findings in a virtual world.

Since the global impact of the COVID-19 pandemic has constrained international travel and mobility, we have all come to rely on working in the virtual world. These developments have merely accelerated the existing trends towards more electronic and digital repositories.

Since 1991, the Networked Digital Libraries of Theses and Dissertations (NDLTD) have contributed to research collaboration by identifying and aggregating digital repositories of theses and dissertations as a previously fugitive class of research resources.

 ETD 2021 Conference: Open scholarship in a post-pandemic world

United Arab Emirates University

Copyright Policies and Guidelines for Thesis in Universities

Building ladders to the shoulders of giants: How partnering with ProQuest is enabling Lancaster University to realise its aim to be a globally significant University for research in a post-pandemic world