Skip to main content

Tracing the digital social impact of a research project

Getting cited by other researchers is a welcome indication that your work has an impact on your field. Increasingly, however, researchers in the social sciences and humanities are asked how their work impacts society in broader ways and beyond academia.

Last week, I finished a project on the wider societal impact of a large research project carried out by my colleagues at the Centre for Youth Research (CeFU). The centre is committed to an increased focus on how their research makes a difference for the young people they study and society more generally. But how do you measure that?

Digital media come with new opportunities for data collection and analysis. One way of exploiting these opportunities is to deploy various digital methods - a growing interdisciplinary field associated with science and technology studies (STS) and media research. In the words of Richard Rogers of the Digital Methods Initiative at the University of Amsterdam (Rogers 2013:1), digital methods can be understood as
a proposal to reorient the field of internet-related research by studying and repurposing ... the methods of the medium. 
The hyperlink is an example of how this might work. Links are abundant on the Internet. They are one of the core 'methods of the medium'. Following the links among a given set of actors (relevant organizations, institutions, blogs, etc.) can help paint a picture of who are prominent within a given field and who are allied with whom. In this sense, digital methods can assist in answering age-old questions in social science. Might digital methods also assist in measuring the social impact of research?

The research agenda

The central theme of the CeFU project was young people's motivation for learning. CeFU's aim was to develop new knowledge and new experiments for increasing young people's inclination to learn. Funded by the Egmont Foundation, the project was carried out by several CeFU researchers over the last five years. Among the results are more than 85 presentations, 5 books, and 4 conferences organized by CeFU. A lot of work, in other words. What impact did it all have?

In approaching the question, we took two lines of inquiry:
  1. A semantic analysis of the discourse around youth and motivations for learning
  2. A hyperlink analysis focusing on relevant actors and their 'outreach'
In the rest of the blog post, I will briefly describe each approach and our results. It is worth emphasising that no defined method existed for us to follow - we had to improvise along the way. This is very much in the spirit of digital methods, which is an exploratory approach that insists on situating the methods in relation to the questions at hand and the data that is available.

Semantic analysis: Did the motivation discourse shift?

Digital media make is possible to assemble large corpora of text with relative ease. Once assembled, such large chunks of text can be examined with a technique called co-word analysis, which uses various statistical measures to draw a network of words (or phrases) that tend to occur together. Such a network is useful because it can serve as a map of the discourse present in the text: What ideas and actors are central here? What are they associated with?

In relation to digital social impact, co-word analysis can help establish shifts in the discourse around ie. the motivation of young people when it comes to learning. If the discourse shifted between the beginning and the end of the CeFU project, it might indicate research impact.

Following this idea, we assembled one text corpus from 2010, the year before the launch of CeFU's project, and another corpus from 2016, the last complete year available to us. Several text sources could be relevant here, but we decided to go for a collection of all news stories written by major Danish newspapers that included the words "unge, uddannelse, motiv*" (youth, education, motiv*). The asterix after motiv* allowed us to capture all variations of the word motivation (such as motivate, motivating, motivations, etc.).

On a methodological note, this search string also returned some crime stories (as in 'motive for the crime'), which have been cleaned out to some extent, but only partly (in order to allow for exploration of whether they were relevant). If you encounter crime related words in the visualisations, that might be the explanation. Such words could be related to youth rehabilitation, so they are not necessarily irrelevant to questions about motivations for learning.

The search results were as follows:
  • 2010: 409 articles
  • 2016: 433 articles
So the discussion around motivation and youth education has been alive and well in both years, but has it changed? To examine this, we employed the research tool Textometrica, which is developed by researchers at the HUMlab, Umeå University. 

Textometrica makes it possible for us to automatically examine what words occur within the same sentence across all the articles from a given year. If youth and motivation appear in the same sentence, this counts as one co-occurrence. Textometrica not only counts all words and their co-occurences with other words, it also calculates which words have a higher-than-average tendency to occur together. This makes it possible to draw a map of words that are especially associated with each other in 2010 and 2016 respectively and thus profile and compare the discourse around youth education and motivation.

Results: Two co-word maps

Below is first the map of the 2010 discourse around youth education and motivation, then follows the 2016 equivalent. Both maps are in Danish.

The 2010 map - download full size here

One thing to note in the visualisation above are the two large nodes at the bottom, which are only connected to each other. They say 'succes' and 'praktikpladser' (internships). The theme of internships in relation to young education and motivation is not present in the 2016 map below, which suggest that one characteristic of the 2010 discourse is a focus on internship success.

The 2016 map - download full size here

The main finding in relation to the CeFU project, however, is related to how 'motivation' placed in the two maps. In 2010, motivation was only associated with 'unge' (youth):

In 2016, motivation is not just associated with 'unge' but also 'talent', 'krav' (demands), and 'læring' (learning), which is again associated with 'lyst' (inclination):

This is an indication that the discourse around youth education and motivation has shifted between 2010 and 2016 towards highlighting motivation as more multi-facetted and potentially connected to learning, and questions about what creates the inclination to learn, which is related to the issues raised by the CeFU project on motivation. Here is an indication that CeFU's research has had an impact on how we talk about motivation in relation to youth education.

Other relations have shifted in interesting ways between 2010 and 2016. For instance, 'kompetencer' (competencies) are now associated with 'digital', which was not the case in 2010. As another example, 'samarbejde' (cooperation) is now connected with community, which was also not the case in 2010. This is related to CeFU's interest in shifting our understanding of motivation from something carried by individuals to something that is created through social interaction and communities.

If you would like to explore the two maps on your own, the files can be downloaded here:

Final words

The other part of the project used hyperlinks to investigate what actors have engaged with the outputs of CeFU's research project. Given the length of this post, I will discuss the second part of the project in a future blog post.

The digital social impact project was a collaborative effort involving TANTlab intern Pelle Israelsson, and Ida Maria Bjelke Bjarløv from CeFU, aside from myself. We wish to thank the Department of Learning and Philosophy, Aalborg University, for making the project possible.

If you have any questions or comments, I can be reached at