SOUTH AFRICA
bookmark

The research subsidy may penalise high-citation articles

Yolande X Harley, Esmari Huysamen, Carlette Hlungwani and Tania Douglas  10 June 2016
South African universities are awarded an annual subsidy from the Department of Higher Education and Training, or DHET, founded on research publication output – a significant proportion of which is composed of journal articles.

The journal article subsidy is based on the number of journal output units generated by the university, calculated from the number of research publications in DHET-approved journals and the proportional contribution of authors from the university.

This subsidy provides financial incentive to increase research output.

Given that the DHET subsidy rewards and intends to stimulate research for the benefit of the country, the manner of awarding subsidy should align with strategies to maintain or improve the impact of South Africa’s research.

Ideally, university researchers would publish high-quality research that makes an impact in the scientific field and, where appropriate, work collaboratively with other groups to add value to studies and aid further development and translation of the research.

What policies say

The goals of South Africa’s 2008 National Research Foundation Strategic Plan, NRF Vision 2015, incorporate not only research output but also ‘citation intensity’, emphasising the importance of the impact of the country’s research (not only the volume). The NRF system of rating researchers is also based primarily on the quality and impact of their outputs.

The 2014 South African Medical Research Council – SAMRC – Strategic Plan for 2014-15 to 2018-19 highlights the need to publish in high-impact journals, and includes the number of articles published in the ‘top four’ journals – The New England Journal of Medicine, The Lancet, Science and Nature – as an indicator. The SAMRC also encourages scientists to work collaboratively, as “no single group can respond alone to the priorities”.

This sentiment is echoed in the NRF Strategic Plan, which advocates “promoting and enhancing international networks and partnerships”, and by the Department of Science and Technology’s Innovation Towards a Knowledge-Based Economy: Ten-year plan for South Africa (2008-2018), which states that greater networking and collaboration – domestic and international – is needed for the country’s biotechnology industry to grow.

Providing disincentives?

However, the current subsidy model does not factor in research quality or impact – other than specifying that journals must be DHET accredited. In addition, the greater the co-authorship with other institutions (domestic or international), the lower the subsidy received by a university. This may result in a disincentive to collaboration.

It has been argued in the Report of the Ministerial Committee for the Review of the Funding of Universities that the current system may lead to “non-virtuous practices in research”, such as writing short ‘salami-sliced’ papers, targeting low-tier journals with high acceptance rates, and avoiding collaboration to increase subsidy.

Cautioning that the drive to increase research volume had come at the expense of the pursuit of excellence, the 2014 ministerial funding report argued that it was time to change the funding framework. Yet the new Research Outputs Policy published in March 2015 did not make any changes to the journal article subsidy formula to address these concerns.

Inter-institutional collaboration may affect the scope and quality of research as well as the impact of the resulting publications in the scientific field, as indicated by the number of citations received. Indeed, it has been shown that research that is more collaborative is associated with higher citation rates (Khor et al 2015 and Sooryamoorthy 2009).

The inverse relationship between DHET subsidy units received by an institution for a paper and the proportion of authors from outside that university may therefore lead to greater subsidy being awarded to articles of lower citation impact than those of higher citation impact.

Materials and methods

We hypothesised that greater subsidy (as a result of fewer ‘outside’ authors) would in fact be associated with lower citation impact.

We analysed a set of journal articles published in 2011 by authors from the University of Cape Town, or UCT, faculty of health sciences that were audited and approved for subsidy by DHET, to determine if there was a significant relationship between subsidy units received and (1) the number of citations received (citation count) and (2) the field-weighted citation impact.

The latter measure is the ratio of citations received by a publication and the average number of citations received by all other similar publications (Colledge 2014) – that is, with the same publication year, publication type and discipline – and so takes into account differences in citation patterns across disciplines or publication types.

For each article, we identified the subsidy units assigned by DHET to UCT and the proportion of non-UCT authors. For each article, we extracted the citation count (total number of citations received) and the field-weighted citation impact for 2011-15 using data drawn from Scopus accessed via SciVal. Articles not listed on the Scopus database were excluded.

The relationship between subsidy units assigned to articles and their citation count, as well as between subsidy units and the articles’ field-weighted citation impact, were examined using a Pearson correlation, with a two-tailed p-value and a 95% confidence interval.

Hypothesis confirmed

Following the exclusion of 38 articles not listed on the Scopus database, 812 articles were included in the analysis, with a mean subsidy unit assignment of 0.53±0.35 units. Of these, 589 (72.5%) articles had non-UCT co-authorship.

Both citation count and field-weighted citation impact were negatively correlated with subsidy units. While the shared variance was small, the correlation was significant in both cases.

DHET subsidy units assigned to the University of Cape Town for journal articles published by the faculty of health sciences in 2011 had a significant inverse relationship with both citation count and field-weighted citation impact.

This finding implies that subsidy allocation is smaller for articles receiving a greater number of citations than it is for those receiving a lower number of citations, whether in absolute terms (citation count) or when compared with the average number of citations received by similar publications.

Greater collaboration is associated with greater citation rates and, author affiliation aside, citation count tends to increase with the number of authors – with self-citation likely to play only a minor role (Aksnes 2003 and Van Raan 1998). It is therefore not unexpected that lower subsidy-earning publications, which will have been more collaborative, are more highly cited.

The analysis has confirmed our hypothesis.

By directly relating subsidy to the proportional contribution of authors from a university, and therefore penalising universities for collaborative research, the annual subsidy may also be inadvertently penalising high-citation, ‘high-impact’ publication.

Need to encourage collaboration

Given the financial benefit of subsidy unit assignment, the existing model discourages inter-institutional collaboration.

This situation seems particularly punitive regarding international collaboration. Universities only receive subsidy for the proportional contribution of authors from that university, whether the external authors are based at other South African institutions or international ones.

The annual research publication output subsidy distributed by DHET is currently valued at about ZAR1.6 billion (US$108 million). Such large-scale funding should align with the strategic goals of the government – it should incentivise collaborative research that may be associated with high-impact science.

Collaborative publishing has increased in recent years in African institutions, and Pouris and Ho (2014) suggest that the large proportion of inter-institutional articles from South African universities indicates that factors encouraging collaboration outweigh the adverse impact of the funding model.

Indeed, we found 72.5% of the publications in our analysis had non-UCT co-authorship. That noted, universities have different methods of internally allocating subsidy received, and this could influence researchers’ publishing behaviour. Our findings are representative of health sciences articles from a research-intensive university that does not directly allocate publication subsidy to researchers.

Funding framework should be revised

Our findings support the recommendation of the 2014 universities funding review that the funding framework be revised. The report proposed that subsidy units be divided only among South African authors of articles, so the model no longer actively discourages international collaboration.

This revision would better align the DHET model with the Department of Science and Technology’s Innovation Plan, the SAMRC Strategic Plan and the NRF Strategic Plan, which encourage collaboration.

The DHET research output subsidy is a means of distributing government funding in a way that factors productivity. Research output has been steadily increasing in South Africa over the last decade and the subsidy system is thought by the funding review and Pouris (2012) to have contributed to this increase.

However, quantity should not be emphasised at the expense of quality (Ioannidis et al 2014). It is perhaps notable that while South Africa’s medical publication output increased from 1996 to 2010, the number of citations per document declined (Pillay 2013).

The 2014 universities funding review also recommends that the quality and scientific impact of publications be directly factored in the model; preferential weighting of journals with higher impact factors was suggested.

The notion of a journal’s impact factor being a measure of the quality of papers published in it has been contested, and deficiencies of this measure as a tool for research assessment have been highlighted in the 2012 San Francisco Declaration on Research Assessment.

The impact factor may have some utility in the funding framework as an indicator of the quality of the journals in which universities publish, especially given that research output is aggregated at an institutional level. This is in line with the original intention of the impact factor, namely as a measure of journal (rather than article) quality (Alberts 2013).

Revising the subsidy model to include a weighting for research impact would potentially better align it with the NRF Strategic Plan and SAMRC Strategic Plan, which advocate for targeting impact. As bibliometric measures of quality and impact evolve, their utility in directing funding allocation should be evaluated on a regular basis.

In summary, the annual subsidy awarded by DHET to South African universities for research publication output may be inadvertently penalising high-citation publication. Revising the funding model to address this effect would better align DHET funding allocation with government strategic plans and may better support publication of greater impact research.

Yolande X Harley, Esmari Huysamen, Carlette Hlungwani and Tania Douglas are in the faculty of health sciences at the University of Cape Town, South Africa.

This article, “Does the DHET research output subsidy model penalise high-citation publication? A case study” was published by the South African Journal of Science, Volume 112, Number 5/6, May-June 2016. Read the full article here.

References