Evaluating the Research Performance of Islamabad-Based Higher Education Institutes

Research discovers new knowledge, ideas, and technologies essential in driving the future of society and humanity. Without research, a relevant and modern country cannot exist. Similarly, the research output of universities and degree-awarding institutes (DAIs) plays a significant role in higher education and development of any country. Research fosters professional excellence in faculty, important for delivering outstanding student education and training. Therefore, this research is an evaluation of the research output of higher education DAIs in the capital of Pakistan. This research was conducted in four steps: (a) data collection: where the data were extracted from Scopus for the years 2008–2017; (b) data cleansing and labeling: which included removal of nonrequired contents and labeling of nonnumeric data to meaningful classes; (c) feature selection: useful features according to proposed research questions were selected; and (d) data analysis: data were analyzed according to research questions and results obtained from experiments with the help of statistical tools. It is concluded that with only 1% of country’s population and around 11% of Higher Education Commission’s (HEC) recognized universities geographically located at Islamabad Capital Territory, Islamabad-based universities and DAIs contributed 34% of the total publications of Pakistan produced during the study time period. This shows Islamabad-based universities and DAIs standing at the national level. The public sector universities shared the major portion of total publications of the studied universities. Private sector universities and DAIs shared 8% of the total publications.


Introduction
Research in universities and higher education institutions in Pakistan has witnessed growth in recent years due to the funding from country's research and funding agencies, internal projects, establishments of research centers at universities, and universities' desire for better regional and international rankings. Pakistan started to reform its higher education system in 2002. This reform was designed to invest in academic staff and to revive declining research creativeness. This resulted in greater research visibility and a higher number of research publications (Osama et al., 2009). However, this growth is not substantial comparing the neighboring countries India, China, and Iran. Mufti (2003) observed in his study that only 90 persons are engaged in research and development (R&D) per million of the population in Pakistan, compared with 4,100 in Japan. The share of Pakistani researchers is a mere 0.04% compared with India's 20% in the world's research publications. Only 2.6% of 17-to 23-year-olds were enrolled in higher education institutions compared with 6.2% in India and 12.7% in Iran (Task Force on Higher Education & Society, 2000).
In the First Education Conference in 1947, founder of the nation provided basic guidelines for the future of educational developments in his newly established country. Identifying scientific research was one of the priorities set at the conference. Research remained in the recommendations of almost all the National Educational Policies. The change of the successive governments affected the implementation of these recommendations (Mahmood et al., 2015). The spending on higher education in Pakistan was increased from Rs. 3.9 billion in 2001-2002to Rs. 33.7 billion in 2006-2007. The average expenditure for every university student was increased from around Rs. 30,000 in 2001-2002to Rs. 135,000 in 2006-2007(Hoodbhoy, 2009). This shows the government's shifted focus on promoting higher education and research.
One of the assigned roles for teachers in higher education institutions in Pakistan is to conduct research. The research productivity of faculty is considered one of the criteria for performance evaluation at many universities (Khan et al., 2018). As research is a performance indicator of higher education institutions, there is a need to continuously assess performance and standing in the scientific community so that appropriate corrective actions may be taken in the right direction (Mushtaq et al., 2012). Country-level bibliometric analyses produce the evaluation of subject areas as a whole, for example, how good is the research performance of a particular country in mathematics; therefore, research performance analysis of universities and larger organizations should be on meso-level (van Raan, 2003).
In this article, a detailed evaluation of the research output of Higher Education Commission (HEC) of Pakistan's recognized universities and degree-awarding institutes (DAIs) geographically located in Islamabad, the federal capital of Pakistan, is made. Bibliometric indicators are used to measure the research performance of these institutions. Bibliometrics is a common research tool used for systematic and quantitative analysis of published scientific and academic literature. Bibliometrics provides an attractive source of data for research performance (Adams, 2009). Researchers have been using this method to evaluate the publication output and research trends of institutions, countries, journals, and subject categories (Maharana & Das, 2013).
The objective of this study is to examine and evaluate the research productivity and publication patterns of public and private universities and DAIs in Islamabad. The results of this study can help the policy and decision-making bodies to formulate their research policies accordingly. The results will also help to determine where these institutions stand with their research output within the country and at the worldwide scientific landscape. The results can help in setting research priorities, allocation of funds, establishing local and international research collaborations, providing incentives to faculty to do more research, and awarding the most prolific researchers.

Literature Review
A number of bibliometric studies have already been published to measure the research performance of universities, countries, subjects, journals, and authors, whereas only a few are carried out in Pakistan. Studies in Pakistan are conducted to measure the research output in specific subject fields, journals, and authors (e.g., Anwar & Saeed, 1999;Bajwa & Yaldram, 2013;Baladi & Umedani, 2017;Jan, 2013;Kousar & Mahmood, 2010;Naseer & Mahmood, 2009;Qayyum & Naseer, 2013;Warraich & Ahmad, 2011). There are some country-level comparisons of Pakistan's research output with neighboring China and South Asian Association for Regional Cooperation (SAARC) countries (e.g., Javed & Liu, 2018;Mahbuba & Rousseau, 2010;Uddin & Singh, 2014). Only a few studies focused on measuring the research output of universities and other institutions of higher learning in Pakistan (e.g., Farooq et al., 2019;Kumari, n.d.;Mushtaq et al., 2012). As per our best knowledge, there is no comprehensive study conducted to evaluate the research performance of the universities in Islamabad Capital Territory (ICT). One possible reason for this may be the lack of use of bibliometric methods by researchers in Pakistan as indicated by Naseer and Mahmood (2009) for the field of Library and Information Science. In neighboring countries, there are several examples of studies on measuring research performance of institutions and universities (e.g., Abolghassemi Fakhree & Jouyban, 2011; Daim et al., 2006;Gourikeremath et al., 2015;Maharana & Das, 2013;Meera & Sahu, 2014;Zhu et al., 2014). Zhu et al. (2014) discussed the research performance of Chinese universities using the bibliometric method. In their study, they analyzed the internationalization of Chinese universities' research, considering their international collaborations and international citations. The authors evaluated the quality of Chinese universities' publication using Source-Normalized Impact per Paper (SNIP) value and h-index. Authors proposed a score called "research performance point" to evaluate the overall research performance with total points for each subject category of Chinese universities. Howaidi et al. (2017) conducted a study to perform bibliometric analysis to evaluate and showcase the research output of government hospitals in Riyadh. They concluded that there is a numerical increase in research publication output in government hospitals in Riyadh, Saudi Arabia. Moed et al. (1985) analyzed the research performance of two faculties of the University of Leiden. Authors were concerned about output and impact aspects of research performance of the academic staff of the University of Leiden. The number of publications that appeared in the international literature were considered as the indicators of research performance. Data for research were retrieved from the Science Citation Index. According to the authors, an insight into the impact of research can be gained by counting the number of times the publications are cited. The research process is imperative for the development of society, and research results are valuable for the benefit of the general public (Naseer & Mahmood, 2009). Exchange of research results is a driving force in science. Publications are important elements in this knowledge exchange process, and for this reason, scientists must publish their research results in easily accessible international scientific literature. These publications form the basis of bibliometric studies of scientific development. The scientific analysis of publications carries interesting things-"For instance, how specific work is used, and particularly, how, when and where it is cited by other researchers" (van Raan, 2003).
As Pakistan is a developing country, some studies from developing countries are reviewed (Abbas, 2016;Dakik et al., 2006;Jokić et al., 2005;Pandita et al., 2014;Sweileh et al., 2014). Okafor (2011) argued that researchers from developing countries do not have a fair share of representation in databases and citation indexes. Huang et al. (2006) provided a brief comparison of peer review and bibliometric methods in research evaluation. According to the authors, subjectivity, dependency, and cost are the factors of criticism of peer review system. Stating the merits of bibliometric system of research, the authors observed that the method presents the simple calculations of research publications. The authors collected the data using The Insititute for Scientific Information (ISI's) Essential Science Indicators in their study to examine the research evaluation of research-oriented universities in Taiwan. The study revealed that the studied universities performed well in terms of quantity compared with quality.
Among a very few studies conducted to measure the research performance of educational institutes of Pakistan Mushtaq et al. (2012) prepared a study to determine the change in research culture at Pakistani medical institutions from 2007 to 2010 and to assess and compare the status of research activity in different medical institutions of the country. Twenty-four medical universities and institutions selected to evaluate their publications appeared in impact factor journals. These publications were analyzed for their year-wise and city-wise distributions as well as their distribution within each studied institution. They identified that there is an increase in the number of publications in the years 2007-2010. The study focused on the degree of the evolvement of research pattern in medical education institutions in Pakistan.

Research Questions
Research Question 1: To measure the research output of HEC-recognized Islamabad-based higher education institutes. Research Question 2: To measure the research productivity of private and public sector universities separately. Research Question 3: To know the researchers' preferred type of publication to publish their research. Research Question 4: To know the top preferred journals/publications of each studied university.

Research Method
The data for this research are retrieved from Scopus. Scopus claims to be the largest citation and abstract database of peerreviewed literature (Elsevier Solutions, n.d.). It ensures data validity from highly recognized sources. This research is based on public and private chartered universities in ICT of Pakistan. At the time of the study, a total of 21 universities and DAIs were recognized by the HEC. Out of these, 17 were selected as the rest (4) do not have their presence in Scopus. They are either new or teaching universities only. Of the 17 universities, 12 are from the public sector and the rest (5) from the private sector.

Search Criteria
The data are retrieved from Scopus database using the following search criteria: 1. Affiliation is registered with each of the selected universities; 2. The time span of 10 years (2008-2017); 3. Subject area: all.

Process
To conduct the research presented, the following steps were followed: 1. Data collection; 2. Data cleansing and labeling; 3. Feature selection; 4. Data Analysis.

Data Collection
The data are collected for all 17 universities listed in Table 1 using the search criteria mentioned above. Names of HECrecognized universities and DAIs were retrieved from the HEC website (www.hec.gov.pk).
The data were collected using export option in Scopus into Microsoft Excel where data were exported into Microsoft Access for easy search and retrieval of records as well as the provision of running text searching queries. The total data collected consisted of 36,577 records.

Data Cleansing and Data Labeling
It is one of the most important steps in data analysis. To ensure data are symmetric, clean, and can be used in selected algorithms, data cleansing was performed with the following steps: 1. Removal of garbage in terms of incomplete data, missing author names, and missing titles. 2. Data were labeled as 0 if the publication was published in closed-access publications and 1 for openaccess publications. 3. Removal of special characters in author names and publication titles was performed to use the data inside data mining tools.

Feature Selection
The collected raw data consisted of 18 fields: Authors, Author IDs, Title, Year, Source, Volume, Issue, Art No., Page start, Page end, Page count, Cited by, DOI, Link, Document Type, Access Type, Source, and EID.
1. Only fields that did not contribute to the research questions are evaluated in this process. 2. First, empty data extracted such as Page count were removed. 3. Second, the fields that had very scarce data were removed, so it resulted in the removal of Page start and Page end. 4. The fields that had the same data were also removed, such as Source. 5. The fields that had real-time changing data, which means each field has a different value, were removed as Sr. No. is maintained through Microsoft Access. This step resulted in the removal of Author IDs, DOI, and EID.

Data Analysis
First, the selected universities were divided into four categories: universities with (a) high producers, (b) medium producers, (c) low producers, and (d) very low producers in terms of research. Second, the comparison of research has been made according to publication types in terms of public and private universities. To measure the joint publication or co-authorship, this research finds the number of authors per publication per selected university for both public and private categories. Third, the research impact is calculated and analyzed using citation count per university, average citations per publication, and noncited publications per university. Preferred journal per university is calculated by selecting the top three preferred publication sources in which maximum articles were published for the selected study period. Publication sources ranking are also provided using SCImago Journal Rank (SJR) to highlight the quality of research. Figure 1 shows each university's number of publications and publication types from 2008 to 2017. The studied universities are grouped into four categories per their volume of research output: CUI and QU in high category, each with more than 25% contribution; NUST in medium category, with more than 15% output volume; IIU, PIAS, BU, NU-FAST, RIU, CUST, AU, IST, AIOU, and PIDE in low category, with 1% to 2% contribution; and PIMS, STMU, FU, and FUUAST in very low category, with less than 1% of share in overall research output of Islamabadbased universities (see the appendix). Results show that CUI is the top productive university with 28% of the total publications of selected universities, followed by QU (27%) crossing the 10,000 publications mark during the period of study. CUI and QU contribute 55% share of the total research output of studied universities. NUST ranked third with 19% share.

Results and Discussion
Analysis of document types helps to identify the preferred type of publication that researchers opt to communicate research results in. Researchers in these universities published in 13 different types of documents. Journal articles (74.5%) are the most preferred document type for researchers to publish their research followed by conference papers (18.5%). Results show that researchers are not inclined to write books and conference reviews. Only four papers are retracted, which may be considered an indicator of the quality of the work of authors associated with these universities. The lion's share of publications (92%) is with public sector universities, with a meager proportion of 8% with private universities. Publishing articles in journals is preferable in public and private universities with 75% and 62%, respectively. No publication from private sector universities is retracted (Figure 2).

Co-Authorship Patterns
Co-authorship may increase scientific productivity and impact (Noruzi & Abdekhoda, 2014). This is reflected in the case of these studied universities. The ratio of a number of authors to write a publication is high in three top performing universities: QU, CUI and NUST, respectively. Collaboration level is high in public sector universities compared with the ones in the private sector. In all, 983,875 authors produce 33,796 publications with an average of 29 authors per publication and 0.03 publications per author in public sector universities. In total, 11,972 authors produce 2,765 publications with an average of four authors per publication and 0.23 publications per author in private sector universities. Highest and abnormal level of collaboration is recorded for QU with 70 authors per document, followed by COMSATS with 22 authors per document. It is observed that the highest number of authors for a single publication is in QU, where maximum authors go up to 1,009. Approximately two authors contribute to writing a publication for FUUAST and PIDE, indicating the lowest level of collaboration in the studied universities ( Figure 3).

Citation Analysis
Based on the argument of Gonzalez-Pereira et al. (2009) that publication citation has been regarded as an indicator of research impact and of Garfield (2006) that it has flourished over decades, the citation analysis of ICT-based universities was conducted. It was analyzed that QU-affiliated documents received the highest number of citations. QU is the only university that received citations in six digits (exact number = 112,859) with the highest h-index of 99 and with an average of 11 citations per document in the group. The second highest number is 87,000 for CUI for its documents with an h index of 87 and eight citations per document. The h-index of documents associated with NUST, IIU, and PIEAS is 63, 43 and 41, respectively. PIEAS and NU-FAST received an average of seven citations per document, the third highest in the group considering the average citation per document. The lowest h-index was recorded for FUUAST, FU, and PIDE with 1, 10, and 12, respectively. The lowest average of fewer than three citations per document was recorded for FUUAST, PIDE, and BU for their publications.
Seventy-four percent of publications of the studied universities are cited at least one or more times. The publications of PIDE are least cited with 61% noncited documents, followed by BU (47%) and FU (43%). QU, STMU, PIEAS, NU-FAST, FUUAST, CUI, and CUST each have a higher percentage of 70% of documents that are one or more times cited. There are almost 95% of citations received on documents of public sector universities, whereas it was 5% on private sector universities. On average, every document published by public sector universities received eight citations. Private sector universities received five citations on average for their documents. Among private sector universities, NU-FAST has the highest average of seven citations per document, whereas FU has the lowest with three citations on average for its documents. QU leads with 11 citations per publication, whereas FUUAST and PIDE share the bottom with two citations per publication among public sector universities (Figure 4).
Journal articles received the highest share of citations (86%) in document types, followed by conference papers and reviews with 6% each. The journal articles published by public sector universities attracted higher number of citations (87%) compared with the ones published by private sector universities (71%) within the citation share of each sector received. A similar trend of receiving citations on other types of documents in public and private sectors is observed except for the conference papers of private sector universities. Conference papers published by private sector universities attracted a better share of citations (22%) compared with the same kind of document in public sector universities that received around 6% share of citations ( Figure 5).
The studied universities received the highest number of citations on their documents of around 14% and 13% of the studied time in 2013 and 2014, respectively. The lowest citation count was recorded in the last and first year of studied time, respectively. The same trend of highest and lowest citation counts is followed by public sector universities in the group; however, in private universities, the highest number of citations was received in 2016 and 2010, respectively, whereas the years in which lowest number of citations was received were the same as of overall and public sector universities. Table 2 shows the list of top three preferred journals with journal metrics, number, and percentage of total publications of each university.

Preferred Journals
A mixed trend of publishing in local and international journals was observed in the studied universities. Authors associated with PIDE preferred (67% of their publications) to publish in journals of their own institution, that is, Pakistan Development Review and PIDE -Working Papers. Pakistan Journal of Botany and Acta Crystallographica Section E: Structure Reports Online published the highest number of  publications (0.01%) cumulatively for the studied universities. CiteScore and other journal metrics of preferred journals, listed in Table 2, are taken of the latest available at the time of the study ( Figure 6).
Islamabad-based universities and DAIs contributed 34% of the total publications of Pakistan produced during the study time period.

Conclusion and Recommendations
It is concluded that with only 1% of country's population and around 11% of HEC-recognized universities geographically located at ICT, Islamabad-based universities and DAIs contributed 34% of total publications of Pakistan published during the study time period. This shows the contribution and research commitment of Islamabad-based universities and DAIs at national and international levels. The public sector universities shared the major portion of total publications of studied universities. A correlation in co-authorship and research output has been analyzed. The universities with better performance have a higher number of authors per publication compared with low-performing universities. It was observed that publications of public sector universities received a higher number of citations compared with private sector universities.
Although Islamabad-based universities and DAIs have their fair share of research publications at the national level, these institutions need to align their research priorities with HEC and other international institutions to compete with peer universities and institutions in neighboring countries such as China, India, and Iran. Private sector universities have to enhance their focus on their research activities along with teaching to increase their research output and to match with public sector universities. To improve the visibility of   research output, the indexing of local journals in Scopus and Web of Science is strongly recommended.
Overall, higher education institutes and universities in Pakistan are working hard to align themselves to be researchbased institutes and competing in the world's best 1,000 universities. It is observed that the HEC played an important role in the research output of the universities by providing large and seed project grants. HEC also supports researchers to showcase the research output at national and international levels by providing conference funding grants. There is always room for improvement, so HEC needs to do more to achieve and create high-quality DAIs and universities in Pakistan.

Significance of the Study
This study evaluates the research performance of the universities, and literature shows such studies are not frequently conducted in Pakistan. This study helps the policy and decision-making bodies to formulate and design their research policies. It also helps in setting research priorities, allocation of funds, establishing local and international research collaborations, and providing incentives to faculty to do more research.

Research Limitations
1. This study is limited only to the universities and DAIs that are geographically located at ICT and are recognized by HEC of Pakistan. 2. The study covers the time span from 2008 to 2017, and the focus is on Scopus database. Note. DAIs = degree-awarding institutes.