Research Performance Evaluation in China: A Big Data Analysis

China’s scientific achievement has received considerable international attention due to a large amount of research and development (R&D) spending. This article aims to study the performance of China’s R&D expenditures (in the form of research funding) by examining the research performance of individual researchers based on bibliometric measures. This study concludes that research practice is not merely determined by capital possessed. Besides, international collaboration primarily accounts for research performance of scholars, whereas research funding and publishing in Chinese-based journals do not impact research performance significantly.


Conceptualization and Operationalization of Research Performance
Performance is a multidimensional concept and has been evaluated by a variety of criteria (Ford & Schellenberg, 1982). Therefore, evaluating any type of performance is always controversial, and conceptualizing and assessing research performance is not an exception (Daniel & Fisch, 1990;Froghi et al., 2012;Jauch & Glueck, 1975;Moed, 2002). Bazeley (2010) subdivides the construct of research performance into two components: "research activity" and 'performance'-the outcome of research practice made visible and passed on to others. In a more straightforward form of the conceptualization (Bazeley, 2010), research performance is defined as the anticipated research outcomes of researchers in concrete products (e.g., publications), academic standing, personal understanding, and benefits to the community. This theorizing is illuminating in that it has encompassed all the essential elements of research performance, but its operationalization is far more contentious. Many (e.g., García & Sanz-Menéndez, 2005;Hicks et al., 2015;Potì & Reale, 2007;Roessner, 2002) suggest that measures of research performance may include bibliometric measures, research funding, awards, academy memberships, royalty income, activity measures, mid-term impact measures, long-term measures, and other indicators of increased competitiveness.
Bibliometric measures may include both quantitative and qualitative indicators that measure the performance of a journal, researcher, or research group (Durieux & Gevenois, 2010). Quantity may include the number of publications and number of publications in top-ranked journals, while quality may include the journal's impact factor (IF), 5-year journal IF, immediacy index, cited half-life, Crown indicator, H-index, and other indicators (Durieux & Gevenois, 2010). In spite of the prevalence, many scholars (Kurmis, 2003;Milesi et al., 2014;Moed, 2002;Seglen, 1997) argue that bibliometric indicators do not reflect scientific quality and only provide useful supplementary tools for evaluating academic research (for an overview, see D. R. Smith, 2012). Nature (2002) contended that comparing IFs of journals is meaningless if the journals serve different disciplines (see Moed, 2006). A similar argument is shared by many other scholars (Amin & Mabe, 2000;Durieux & Gevenois, 2010;Hicks et al., 2015;Kurmis, 2003;Seglen, 1997). In addition, certain scholars (Bordons et al., 2002;Kurmis, 2003) note that IFs may be easily Source. NSF, National Center for Science and Engineering Statistics, National Patterns of R&D Resources; Organization for Economic Co-operation and Development, Main Science and Technology Indicators (2017/1); United Nations Educational, Scientific and Cultural Organization, Institute for Statistics database, NSFC, and NSSFC. Note. The vertical axis is logarithmic, but the mark labels on the lines are original Purchasing Power Parity dollars (million). The amount of CN_grant equals the sum of the amount of NSFC and NSSFC. In addition, NSF is only one of six major federal funders in the U.S., but NSFC in China accounts for the majority of the research funding. R&D = research and development; NSF = National Science Foundation; NSFC = National Natural Science Foundation of China; NSSFC = National Social Science Foundation of China. manipulated and may also be limited and biased. Consequently, many (Daniel & Fisch, 1990;García & Sanz-Menéndez, 2005;Hicks et al., 2015;"The Maze of Impact Metrics," 2013;Roessner, 2002) strongly advocate for non-bibliometric measures. Nevertheless, bibliometric measures are still the most popular in practice, primarily because other measures could also backfire (see "The Maze of Impact Metrics," 2013), and bibliometric measures are readily accessible (Bordons et al., 2002) and are defended by numerous scholars (e.g., Bornmann & Williams, 2017;Durieux & Gevenois, 2010;Garfield, 1999Garfield, , 2001Ketcham, 2007;Moed, 2002).

Theoretical Framework Explaining Research Performance
Credibility cycle model. Among many conceptual frameworks regarding research performance evaluation, the credibility cycle model (Latour & Woolgar, 1982) may be the most notable perspective. Latour and Woolgar (1982) explain how struggles for reputation influence the behavior of individual scientists whose primary motivation is the quest for credibility. Latour and Woolgar (1982) argue that the research process is a repetitive cycle involving conversions between funding, staff, data, arguments, articles, and recognition. The credibility cycle model has its origin in the Matthew Effect in Science (Merton, 1968a;Zuckerman, 1977), and has been widely applied in research evaluation Hessels & van Lente, 2011;Leisyte et al., 2008). However, scientists' behavior and motivations are hardly homogeneous, especially considering different scientific systems. Noting the limitations of Latour and Woolgar (1982), Horta and Santos (2016) further added the indicator of Drive to Publish or Perish in the ambition factor, in addition to a few separate factors, for example, collaboration, mentor influence, and dispositional propensities.
Bourdieu's capital theory. As a well-known sociologist of science, Bourdieu (1975) believes that every scientific choice is "a political investment strategy directed toward maximization of strictly scientific profit." Consistent with his other famous works (Bourdieu, 1984(Bourdieu, , 1986, Bourdieu (1975) held that the extent to which they are motivated to fulfill their self-interests depends on the amount of scientific resources (capital) they possess, specifically, the economic, cultural and social network resources, and the symbolic capital 1 (see Bourdieu, 1986), and the power that the resources can bring. Researchers occupying the varying amounts of resources have discrepant research capabilities and varying levels of productivity (Bourdieu, 1975). All this ends up with the social distinction of researchers in a field to which they belong.
According to Bourdieu (1984Bourdieu ( , 1986, in addition to resources or capital, other factors are at play to cause scholars to have very distinctive practices, all of which are represented in the following formula: capital + field = practices.

( ) ( ) ×
In a nutshell, habitus, which is a set of preconscious dispositions including tastes, translates agents' different social positions in social space specified by different forms of capital into observable practices or behavior in a particular field (Bourdieu, 1984; also see Feng et al., 2019). That is, the practices that habitus produces vary according to agents' position in social space (Weininger, 2005). Social space is the very structural factor. Scientists' activities occur in the context of a "research system," that is, filed, consisting of universities, research institutes, funding agencies, governmental organizations, and firms, among others Rip & Van der Meulen, 1996). Capital is distributed in particular types, and habitus operates in a delimiting structure in this institutional environment (Bourdieu, 1984), which also provides research organizations and researchers with incentives and constraints to conduct research (Bourdieu, 1975;. Consequently, scholars have drastically distinctive research practices and performance under the varied structure of the scientific system. Scholars are motivated to have power, and they strive for opportunities to acquire power (Bourdieu, 1975). Those who have resources, such as social networks, and the ability to engage in political struggle, will choose more prestigious high-impact journals. They will also be more likely to succeed in taking power, which in turn brings more resources to them, eventually facilitating the cycle of capital (Latour & Woolgar, 1982). In contrast, those who do not enjoy power for its own sake and who are less competitive in terms of academic ability due to a lack of resources will choose domestic and low-impact international journals. Scholars who work in diverse circles are rewarded with different kinds of power and capital (Latour & Woolgar, 1982).
The conceptual model. Based on the literature review and data availability, a theoretical model is subsequently proposed (see Figure 2), in which the effects of various types of resources or capital on scholars' research performance (measured by the number of citations) are mediated by the research practice or strategy adopted (measured by choice of different tiers of journals). Specifically, the amount of a grant, the number of foreign coauthors, the number of China-based international, referring in particular to Sciences Citation Index (SCI)/Social Sciences Citation Index (SSCI)/Arts & Humanities Citation Index (A&HCI) listed journals, publications (for comparing research performance evaluation between international and China's domestic indices, see Liang (2003)), and the number of collaborative national grants, indicate the economic, social, and cultural capital, respectively.

Hypotheses
Science in China, like other sectors of the Chinese system, has been reigned in under a Soviet-style of top-down planned control, although decentralization of control exists to the extent that the government feels politically safe (see Yang et al., 2007). Both scientists and universities are subject to direct orders from higher entities on the one hand, whereas research institutions compete for resources and power among each other, on the other hand (Yang & Welch, 2012). Such a structural idiosyncrasy of China's science system influences agency without doubt. Therefore, the accumulation of competitive advantage of scholars in China is not merely determined by academic ability or any single one capital but the totality of all forms of abilities to capitalize on available resources.
Hypothesis 1. Among all the forms of capital, social capital has been widely studied with respect to its antecedents as well as impact in a society (e.g., Lawson et al., 2008). Bourdieu (1986, p. 51) defined social capital as "the aggregate of the actual or potential resources which are linked to possession of a durable network of more or less institutionalized relationships of mutual acquaintance and recognition." In the context of the research community, Luukkonen (1990) discovered that social capital indicated by social relations in the scientific community affects researchers' citations received.
Social capital could take on various forms, but the number of collaborators could be one of the crucial indicators. Some scholars (Garfield, 1999;Leimu & Koricheva, 2005) have reported that the number of authors primarily affects citation rates rather than IFs. In the context of this study, the number of authors is indicated by the number of grants reported in the paper in addition to the principal investigator's (PI's) grant. As a result, the number of collaborative grants has a positive effect on the choice of tiers of journals (H1).
Hypothesis 2. In addition, international collaboration is another form of valuable social capital. Numerous scholars (Adams, 2013;Aksnes, 2003;Arunachalam et al., 1994;Glänzel & de Lange, 2002;Tang, 2013;Wagner et al., 2017) have discovered that highly cited articles are typically authored by a large number of researchers and often involve international collaboration. This social network capital is particularly true for Chinese scholars because their first language is not English, and most high-impact journals published overseas are produced in English (also see Li et al., 2015;Tang, 2013). Although the effect of the number of authors on IFs is debatable, the presence of foreign collaborators in authorship may be influential in affecting the publishing probability in high-impact journals. Therefore, collaboration with foreign scholars may have a positive effect on the choice of tiers of journals (H2).
Hypothesis 3. More than 100 international journals that are listed in the Science Citation Index (SCI) are based in China (Wagner & Wong, 2012). Publishing in China-based international journals is relatively easier for Chinese scholars because of professional connections and other non-academic cultural factors (Jiang, 2007). Nonetheless, networking with these journals needs not only social capital and sometimes economic capital, but also cultural capital. Chinese scholars are familiar with the habitus as well as favored research objects of these journals, and even endowed with institutionalized privileges (these are embodied, objectified, and institutionalized cultural capital, according to Bourdieu (1986)). However, such cultural capital is a double-edged sword. That is, scholars who put considerable effort into securing such capital will instead choose lower impact journals in general (H3).
Hypothesis 4. Economic capital is generally considered as the root of other forms of capital (Bourdieu, 1986). Research funding is one of the most essential and precious forms of economic capital. Although its effect on IFs has been well explored, findings are mixed. Gaughan and Bozeman (2002) and Campbell et al. (2010) maintained that research productivity might not have a direct relationship to the magnitude of funding (also see Larivière et al., 2010). Nevertheless, despite the fact that amounts of grants are often related to field-and discipline-specific dynamics, such as the expense of equipment, certain scholars (Defazio et al., 2009;Lee & Bozeman, 2005;Milesi et al., 2014) reported that a relationship exists between the amount of the grant and publication productivity. A consensus has not yet been reached; however, it is hypothesized that a positive relationship exists between the amount of funding and the choice of tiers of journals (H4).
Hypothesis 5. As suggested by the theories reviewed above, practices or strategies mediate the effects of other predictors on performance. Although citation rates mathematically determine IFs, the research practices are quite divergent. For instance, Seglen (1997) found that some authors select journals based on their IFs prior to submission of an article. Papers published in journals with larger IFs are generally cited more often (Tang, 2013), albeit this has been challenged by Finardi (2013). Consequently, the choice of tiers of journals should mediate the effects of resource (capital) factors, such as the total number of collaborative grants (Hypothesis 5a), number of foreign coauthors (Hypothesis 5b), amount of grant funds (Hypothesis 5c), and number of publications in China-based SCI/SSCI/A&HCI journals (Hypothesis 5d), on research performance, specifically the average citation rate of articles (ACA hereinafter) generated by a grant (Callaham et al., 2002).
In addition to the hypotheses proposed above, two research questions were raised according to the suggestions of the editor and an anonymous reviewer:

Measures
The predictors, the mediator, and the outcome variable are measured by mostly bibliometric measures.
Dependent variables. The dependent variables include research performance and research strategies, which are measured by the two primary bibliometric measures, that is, ACA and average impact factor (AIF), respectively.
AIF. Because yearly IFs of some journals have changed dramatically, the 5-year IF better captures the stable quality of journals (Amin & Mabe, 2000). Journal Citation Reports (JCR) accessed through the WoS database provides these data. Moreover, because the grant is the unit of analysis, IFs of all journal publications funded by a single grant are simply averaged to form the AIF.
ACA. The number of citations is recorded for each journal article in the database of Web of Science. The number of citations of all articles funded by a grant is simply averaged to derive the ACA, which reflects the average actual impact or performance of a grant or the PI. Independent variables. Independent variables are three forms of capital mentioned above, that is, social capital, cultural capital, and economic capital, which are respectively measured by the number of foreign coauthors and number of collaborative national grants, number of publications in China-based SCI (Sciences Citation Index)/ SSCI (Social Sciences Citation Index)/A&HCI (Arts & Humanities Citation Index) journals, and the amount of grant. All measures but the amount of grant (data were from the NSFC and NSSFC) used the data provided by the WoS database.

Descriptive Analysis
Before testing hypotheses, a series of descriptive analyses were conducted.
The characteristics of Chinese scholars' publications. China's social scientists account for only 1.60% (the sum of the number of SSCI and A&HCI listed articles divided by the total number of SSCI, A&HCI, and SCI listed articles of Chinese scholars) of all of the publications authored by Chinese scholars. In total, 45.83% of Chinese scholars' articles are published in journals whose IFs are in the first quantile. The AIF of these journals is 2.93 (SD = 2.76), and the maximum is 54.39. The ACA is 6.56 (SD = 18.10), with a maximum of 2,610, and 76.49% of the articles have been cited at least 10 times. Besides, 20.6% are collaborations with foreign authors, and 1.81% are published in China-based international journals, which have an AIF of 0.64.

How China's major funding initiatives have distributed grants.
The NSFC (representing science) and NSSFC (representing social sciences) have distributed over 30.24 billion U.S. dollars over the past 20 years, of which the NSFC accounts for 94.35%. Universities included in "Project 985" have received half of the grants. Provinces with more key universities (listed in either "Project 985" or "Project 211," two of which aimed to strengthen the competitiveness of certain selected universities in China (see Huang, 2005;Yang & Welch, 2012), but have been replaced with the "Double First-Class" program in September 2017) and are more affluent (based on gross domestic product [GDP] per capita) have secured more grants, and nearly half of the total number of grants (46.90%) have been distributed in Beijing, Shanghai, Jiangsu, Guangdong, and Hubei (also see Chu & Li, 2000;Tang, 2013). R 2 s are above .70 for the correlations between the number of grants and the number of key universities, and R 2 is .524 for the correlation between the number of grants and GDP per capita. NSFC grants were primarily distributed to universities and research institutes, while approximately 10% of NSSFC grants 2 were distributed to non-research-related organizations such as governments, the subordinate organizations of the Communist Party of China, media, and even commercial firms. 3

Hypothesis Testing
Multiple-group analysis of structural invariance. To examine whether AIF mediates the effects on ACA (average citation of articles) and whether the grant type (NSFC vs. NSSFC) moderates the structural relationships, a multi-group mediation analysis was performed. The multiple-group analysis of structural invariance is used to test whether differences observed in the structural parameters across groups are statistically significant. The analysis begins by fitting a model to the data for each group considered separately, with none of the parameters constrained to be equal across groups. This unconstrained model serves as a baseline model and is compared to constrained models, in which constraints are placed by specifying parameters of interest to be equal across groups (Koufteros & Marcoulides, 2006). In light of the principles stated above, two models (i.e., the unconstrained baseline model, and the constrained model) in which all of the path coefficients were constrained to be invariant between the two groups, were compared. Given the complexity of the data, the unconstrained and constrained structural models were compared by conducting chi-square tests for nested models based on Log-likelihood values and scaling correction factors obtained by the MLR estimator (see Asparouhov & Muthén, 2006). A significant χ 2 difference represents a rejection of the null hypothesis (that the parameters are equal), whereas a non-significant χ 2 indicates otherwise.
Following the above-mentioned procedures, the multigroup mediation analysis with a bias-corrected bootstrap procedure (5,000 resamples) was performed, and we determined that the constrained model was insignificant compared to the baseline model, χ 2 difference (df = 13) = 10.925, p = .617. The model with equal path coefficients across the groups (NSFC vs. NSSFC) fits the data better, χ 2 (df =13, N = 143,026) = 21.849, p = .06; standardized root mean squared residual (SRMR) = .002; root mean square error of approximation (RMSEA) = .003; comparative fit index (CFI) = 1; Tucker-Lewis index (TLI) = 1, with the addition of a few predictions beyond the proposed theoretical model by means of modification indices. Table 1, both direct effects and indirect effects of all exogenous variables on ACA are significant. AIF, therefore, only partially mediates the effects of exogenous variables. R squared values of citation rates in the group NSFC and NSSFC are .525 and .306, respectively, which indicates that overall, the model has satisfactory explanatory power. In addition, although all of the effects of predictors on AIF were significant, the positive effects of the number of collaborative grants (Hypothesis 1, β = − < . , . . ) 011 001 p were supported (see Table 1 and Figure 3).

Results of research questions. A multiple regression analysis
was performed to answer the two research questions, in which the amount of grant was the dependent variable, and ROI and the number of publications were the independent variables. The higher ROI is, the lower the amount of grant is (β = − < . , . ). 121 001 p The higher the number of publications is, the greater the amount of grant is (β = < . , . ). 207 001 p There was a negative interaction effect of ROI and the number of publications (β = − < . , . ). 096 001 p That is, when ROI is higher, the higher the number of publications, the lower the amount of grants. Corresponding to the questions raised above, scholars with higher ROI will not dominate research funding, but scholars' publication records were linked to succeeding funding.  Note. AIF = average impact factor; ACA = average citation rate of articles.

Capital Discount
There are some quite counterintuitive findings concerning the effects of the number of collaborative grants (Hypothesis 1) and the amount of grant (Hypothesis 4) on the choice of tiers of journals. Although significant, their effects were opposite to what hypothesized. That is, both the effects of the number of collaborative grants and the amount of grant had negative effects. Furthermore, given the negative effect of cultural capital on the choice of tiers of journals (Hypothesis 3), such a result could be attributable to the phenomenon of so-called "capital discount," a term inspired by the concept of cultural discount (see Hoskins & Mirus, 1988). That is, the presumed positive effect of capital on the choice of high-impact journals is discounted by the decreased motivation due to the excessive possession of resources. The reason why the motivation is decreased when possessing excessive resources may be that needed resources exceed the threshold level, and this causes the satiation effect (see Mojzisch & Schulz-Hardt, 2007;Reutskaja & Hogarth, 2009). In addition, such a finding may confirm the totality of the Bourdieusian view of research practice, which is determined by habitus in addition to capital, and field, in which habitus is formed and capital is distributed, that is,, the structural factor (Bourdieu, 1984).

Positive Effect of Social Capital
As expected, Hypothesis 2 (a positive effect of foreign coauthorship) was supported. In general, foreign collaboration has been effective in terms of producing high-impact publications (Glänzel & de Lange, 2002;Glänzel & Schubert, 2001). Chinese research institutions have attached greater importance to high-impact publications, but only the negligible portion of prestigious international journals are based in China. In view of the importance and effectiveness, these institutions, orchestrated by the Chinese government, who launched a series of initiatives (such as Thousand Talents Program, or Qianren Jihua) to lure foreign top scholars to collaborate with the Chinese counterparts, have been craving for international collaboration (Li et al., 2015;Lundh, 2011).

Indirect Effects of the Resource Predictors on Research Performance
According to the results of mediation analysis (hypothesis 5), the mediator, that is, AIF, partially mediates the effects of exogenous variables on ACA in that all the resource predictors have significant effects on research performance (measured by ACA). Moreover, AIF strengthens the positive effect of foreign co-authorship, whereas it weakens the effects of the amount of grant, the number of grants collaborated on, and publication in China-based SSCI/SCI/A&HCI journals.
In addition, collaboration with foreign scholars has been beneficial in achieving both AIF and ACA.

Total Effects of the Resource Predictors on Research Performance
Due to mostly the competing role that the mediation effects play, some of the total effects of the resource variables on research performance have changed compared to the direct effects. As seen in Table 1 with respect to the total effects of the predictors on research performance, we conclude that all the forms of capital but cultural capital (measured by publishing in China-based SCI/SSCI/A&HCI journals) contribute to the research performance of scholars. This finding generally confirms the applicability of the proposed theoretical framework and is consistent with the classic capital theory of Bourdieu (1975Bourdieu ( , 1984Bourdieu ( , 1986. However, not all the effects of capital are equally important. The social capital, that is, the number of collaborative grants and foreign coauthors, play the most critical role, and yet the economic capital, that is, the amount of grant, and the cultural capital, that is, the number of publications in China-based SCI/SSCI/ A&HCI journals, exert the minor or inconsequential role in affecting research performance.

Implications of the Findings
Theoretical implications. The author has tried to transform some grand theories from the humanistic paradigm (e.g., Bourdieu's capital theory) into an empirical-based middlerange theory (Merton, 1968b) so that the study can propose a theoretical model that explains research performance. This might have made a theoretical contribution to the literature.
Practical implications. This study also has practical implications. In light of the findings on the unequal importance of capital factors, the university's management should encourage international collaboration and help faculty expand their social networks. Moreover, submission to international journals instead of domestic English journals should be given greater priority, and yet research funding should not be stressed in tenure and promotion (T&P) review practices of assessing faculty's performance (cf. García & Sanz-Menéndez, 2005). Besides, considering the above-mentioned finding of "capital discount," funding agencies should decide on the amount of funds of each grant proposal based on needs case by case, rather than evenly distribute funds among successful applicants, which has been a long-time practice in China, particularly in the area of the social sciences.
Methodological implications. As previously stated, China's scientific achievement has received significant international attention, but few studies have thoroughly analyzed and reported on specific outcomes. This study scraped big data related to China's grant information and publications funded by these grants. We have not only described the data but also presented them interactively in a web application (https:// public.tableau.com/profile/publish/my_submission/ Story1#!). Data were further analyzed using innovative modeling techniques. This work is the first study to adopt multigroup mediation analysis in the context of research evaluation, as far as the author is aware. Consequently, this study has made methodological contributions.

Limitations
This study has limitations. Some factors in the Bourdieusian view of practice, for example, habitus, were not tested due to the problem of data availability. Likewise, the concept of field (Bourdieu, 1984) is confined to the context of the Chinese science system. Consequently, this proposed theoretical framework is not meant to be applied to any science systems, but only applicable in non-autonomous or semiautonomous science systems similar to China's. In addition, the unit of analysis is the grant; data were retrieved from two major national funding initiatives, the NSFC (concerning science) and NSSFC (concerning social sciences). Although these two agencies are the primary funders of research, grants are also available from certain ministries and local governments. Hence, it is possible to compare the differences in certain bibliometric measures among studies funded by the aforementioned grants and non-funded studies by expanding the publication search in WoS and available grant data in China.

Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study is supported by a grant of National Social Science Fund of China (grant NO.: 18BXW082).

ORCID iD
Guangchao Charles Feng https://orcid.org/0000-0003-0563-9885 Notes 1. Symbolic capital is not a different form of capital, but rather should be seen as the legitimated, recognized form of the other capitals (Lawler, 2011). 2. In all, 57.44% of NSSFC funded projects are book related; this percentage increases to 84.12% if reports submitted to authorities are included in this figure.