Psychometric Properties of an Instrument to Evaluate Students’ Perception of Learning Objects in Statistics

Digital Learning Resources (DLRs) have the function of facilitating learning; however, the evaluation regarding their functionality is limited since there are few tools for this purpose. Of these, the instrument that stands out is “Perception of the use of Digital Learning Resources in Statistics” (IPDLR-S), which allows for evaluating the perception of undergraduate university students about the functionality of the LDRs. With the objective of providing a tool with scientific quality, the IPDLR-S instrument underwent to an analysis of construct validity and reliability. The instrument was applied to 1,148 university students, the construct validity analysis was performed using an exploratory (EFA) and confirmatory (CFA) factor analysis, and the reliability was determined by ordinal Cronbach’s alpha. According to the EFA, it was found that all the items are represented in the factorial model; however, differences were found for the theoretical model since a fourth dimension emerges (Cooperative Learning). On the other hand, using the CFA, the resulting factorial model showed a good fit (GFI = 0.999, RMSEA = 0.064, CFI = 0.999, and TLI = 0.999). Overall reliability, as well as by factor showed an optimal value (ordinal Cronbach’s alpha > .98). The IPDLR-S instrument has adequate psychometric properties.


Introduction
Higher education institutions have had to generate changes in teaching-learning processes, not only in methodologies but also by integrating Information and Communication Technologies (ICT) into them (Adedokun-Shittu & Shittu, 2015).The integration of ICT into classrooms not only implies changes both in the curriculum and in their facilities, but also an invitation to pedagogical innovation (Sa´nchez, 2019).In the educational context, ICTs are important for students and should be used effectively in learning, as well as, for the teachers who must integrate them into the classroom (Deshpande & Shesh, 2021); this is why the integration of ICT in the university is favorable for learning and teaching (Adedokun-Shittu & Shittu, 2015).
Given the integration of ICT, the use of Learning Objects (LO) stands out, which have been considered functional to improve students learning (Kay & Knaack, 2009).In this sense, LOs are referred to as a digital representation that allows uses in different educational contexts, use different modalities of media (and often interactivity) to represent data, information, concepts, and ideas and also are designed to allow educational reuse (Churchill, 2007).From this perspective, Churchill (2007) refers that, for teachers, an LO allows its use in a variety of foreseen and unforeseen circumstances; in this sense, LO can be used as: 1. a component in the direct instruction for online delivery, 2. a mediating instrument in a problem-solving activity, 3. the object of an investigation, 4. digital resources so that students can use them in their studies, assignments and independent projects, and 5. models and means that the LO designers can use as a base for the design of other's LO.The LO can be delivered through various digital devices; this is how those who design an LO may perceive it as applicable to a specific educational use.
Within the LO, there is the Digital Learning Resources (DLR), which consist of practice LO and are designed based on the development of practical exercises with feedback, and include graphs and other resources such as tables (Churchill, 2007).In the educational context, there are various types of DLR as those provided by Salajan et al. (2009) designed for the learning of dentistry, Hortense et al. (2018) designed and aimed at patients in cancer treatment, and by Basuhail (2019) designed for teaching programming to computer science students.
It has been indicated that digital resources and specifically LO, can enhance or improve student learning (Kay & Knaack, 2009); therefore, evidence of their effectiveness from the perspective of the student for learning support becomes especially important (Chiu & Churchill, 2016;Garay Ruiz et al., 2017).However, there are few instruments to assess the students' perceptions of the LO, only including Kay and Knaack's (2009), which is aimed at high school students, and the one from Monge-Rogel et al. (2022), which is aimed at students of higher education.
In this regard, the contribution of Monge-Rogel et al. ( 2022) is called ''Instrument of Perception of the use of Digital Learning Resources in Statistics'' (IPDLR-S), consists of 32 items and 3 dimensions: 1. Functionality and Accessibility of the DLR; 2. Design and Structure of the DLR; 3. Learning statistics.In this order, these dimensions are relevant since, respectively they are intended to obtain information about the ease of access and navigation, interaction in a context applied to the professional field, and the contribution to the learning of statistics based on challenging and motivating activities, oriented to the interaction between peers and teachers, in a constructive alignment plane (Barattucci, 2017).
An aspect to highlight of the IPDLR-S instrument is that after its construction, it was subjected to the analysis of the psychometric property of content validity (Monge-Rogel et al., 2022), through which it was demonstrated that the dimensions and items that make it up are relevant, pertinent and representatives of the construct to be evaluated (Koller et al., 2017;Jua´rez-Herna´ndez & Tobo´n, 2018).This is relevant since, as Carvajal et al. (2011) refer to, the evaluation of the psychometric properties of an instrument is an essential criterion to determine the quality of its measurement.
While content validity is a relevant psychometric property, it is emphasized that the main type of validity is the construct (Furr, 2020), which is conceptualized as the degree to which inferences can be made legitimately from operationalizations in their study to the theoretical constructions on which those operationalizations were based.Its importance lies in minimizing the problems that arise from the poor quality of the measurement and in proving that the real internal structure is the one it should possess (Furr, 2020).
Therefore, this study's objective was to assess the psychometric properties of construct validity and reliability of the IPDLR-S instrument to collect valid and reliable evidence on the perception of the use of DLRs in the learning of statistics.

Type of Study
According to the classification of Ato et al. (2013), in this work, an instrumental study was carried out to analyze the construct validity and reliability of the instrument to evaluate the perception of students about learning objects in statistics.

Instrument
The purpose of the IPDLR-S instrument (Monge-Rogel et al., 2022) is to evaluate the perception of undergraduate students about the use of DLR in a statistics subject.The instrument consists of 32 items grouped in three dimensions: Functionality and Accessibility of the DLR (7 items), Design and Structure of the DLR (13 items), and Learning of the Statistics (12 items); and for the answers, it uses the Likert scale of six options (1 = strongly disagree, 2 = disagree, 3 = moderately disagree, 4 = moderately agree, 5 = agree, and 6 = strongly agree).
As it was indicated, after its construction, the instrument was submitted to a trial of 31 experts, through which its validity of content was determined (Monge-Rogel et al., 2022).Once this phase was concluded, a pilot was carried out with 50 undergraduate students of a statistics subject, through which the reliability of the instrument and the degree of affordability were analyzed, resulting in high reliability (ordinal Cronbach's alpha = .963)and a high degree of acceptance and satisfaction with the instrument (Monge-Rogel et al., 2022).

Selection of a Sample of the Population for the Application of the Instrument
In this study, a sample of 1,148 students of the Veterinary Medicine, Nursing, and Agronomy careers of a Chilean university were considered, who took a subject of statistics and used the DLR.Participants were selected by intentional sampling according to the purpose of the study.The intentional sampling method is considered a non-probabilistic sampling whose main purpose is to produce a sample that can be assumed to be logically representative of the population (Bhardwaj, 2019;Jamalzadeh et al., 2021).
The participants were informed of the availability of the instrument, which was applied in Spanish, and it was placed in the virtual classroom of the course and was available for a month; the student could access the instrument through a link enabled in the same virtual classroom.The sociodemographic data of the participants are shown in Table 1.
This study had the approval of the Ethics Committee of the institution, complying with all safety and reliability protocols, in addition to obtaining the informed consent of all participants who were included in this study.

Construct Validity and Reliability Analysis
Item Analysis.Initially, the adjustment of the items to the univariate normal distribution was analyzed by calculating the asymmetry and kurtosis, considering that if any item had a value greater than 6 2 it was subject to elimination (Bollen & Long, 1993).Likewise, the analysis of multivariate normality was considered using the Mardia coefficient (Mardia, 1970) and under the criteria of Bollen (1989).The item-test correlation was examined to verify whether the items revealed a value less than .2or a value higher than .9,which led to their elimination (Kline, 1986;Tabachnick & Fidell, 2001).In addition, ordinal Cronbach's alpha was calculated to evaluate the reliability of the instrument, and an item-by-item contribution analysis was performed on ordinal Cronbach's alpha; to verify that there were no increases in the value of trustworthiness when not considering one of the items.
Factor Analysis.According to the sample size, a crossvalidation process was carried out, for which the sample was divided into two equal parts (n 1 = n 2 = 574), the first of which was analyzed by the Exploratory Factor Analysis (EFA) and the second using the Confirmatory Factor Analysis (CFA; Brown, 2015).The mechanism for the division of the sample was through a randomization process, which is used to avoid any bias or pattern, using random numbers through the statistical R software.
Exploratory Factor Analysis.In order to perform the EFA, first, the relevance of the data was verified through the degree of correlation between items, the value of the determinant, KMO index, and Barlett test (Jua´rez-Herna´ndez, 2018;Thomson, 2004).When it was confirmed that the items were adjusted to the normal distribution, the EFA was carried out by selecting the Maximum likelihood estimation method (Howard & Jayne, 2015;Thomson, 2004;Yong & Pearce, 2013).However, if the absence of normality was verified, it was chosen to work with the polychoric correlation matrix, and the chosen extraction method was the unweighted least square (ULS; Xia & Yang, 2019).The determination of the number of factors to be retained was based on the technique of maximum consensus among 23 methods used (Gutman-Kaiser rule, sedimentation graph, parallel analysis, optimal coordinates, and variance explained, among others; L€ udecke et al., 2020).Following the analysis of the factorial matrix, if factorial complexity was presented, the rotation of the matrix was carried out using the algorithm of greater convenience (Jua´rez-Herna´ndez, 2018;Lloret et al., 2017;Thompson, 2004).These analyses were performed with the statistical R software and the Parameters library version 0.14.0.1 (L€ udecke et al., 2020).
Confirmatory Factor Analysis.According to the factorial structure obtained through the EFA, with the second subsample, the CFA was performed using the Robust Unweighted Least Square method.Specifically, the goodness of fit of the model was evaluated using chi-square, ratio, and adjustment index (x 2 /gl Goodness of Fit Index (GFI); Mean Square Error of approximation (RMSEA), Root Mean Square Residual (RMSR); Comparative Fit Index (CFI); Index de Tucker-Lewis (TLI)), according to the criteria proposed by Blunch (2012), Schumacker and Lomax (2016), Xia andYang (2019), andRa´czova´et al. (2021).This analysis was performed with the statistical R software and Laavan (Rosseel, 2012) and psych (Revelle, 2017) library.
Divergent and Discriminant Validity.To evaluate whether the behavior of a factor is strongly explained by the items that compose it, convergent validity was explored through the Extracted Mean-Variance (AVE), specifying that an AVE greater than or equal to .5 is considered adequate according to the criterion of Fornell and Larcker (1981).On the other hand, to evaluate the ability to discriminate different constructs, the discriminant validity was analyzed by comparing the correlation squared (r 2 ) with the AVE of each factor; when r 2 was lesser than the AVE of each factor, the discriminant validity was verified (Fornell & Larcker, 1981).
Instrument Reliability and Composite Reliability.The internal consistency of the instrument was evaluated through the ordinal Cronbach's alpha reliability, as well as the composite reliability (CR), considering a threshold level greater than .70 as the minimum acceptable value (DeVellis, 2016).

Instrument Satisfaction Analysis
Finally, the degree of understanding and satisfaction with respect to the instructions, items, and descriptors of the instrument was evaluated through a satisfaction questionnaire (Centro Universitario, 2018).

Construct Validity and Reliability Analysis
Construct Validity and Reliability Analysis.Although the values of the asymmetry and kurtosis statisticians could suggest that the items are distributed in a normal way (Table 2), the evaluation of multivariate normality was not satisfactory according to the Mardia test (Kurtosis p \ .05;Asymmetry p \ .05),so robust unweighted least squares were used as the extraction method in the EFA.According to the item-test correlation indicator and the ordinal Cronbach's alpha, it was not necessary to remove any items (Table 2).
According to the EFA, it was found that all the items presented adequate communalities (Table 3).The matrix revealed the complexity of a series of items, so a rotation was made, clarifying the factor loadings (FL; Table 3).Specifically, it is reported that the resulting factorial model differs from the proposed theoretical model since four factors were denoted, which explained more than 83% of the variance, which is highly representative, and in which all items with significant factorial loads were integrated (FL .0.50; Table 3).In this sense, the first factor explained more than 35% of the variance and integrated items 8 to 20 (Table 3), corresponding to the theoretical dimension of LDR Design and Structure.Factor two explained 8% of the variance and integrated items 23, 29, and 30 (Table 3), which address the aspects of realization of the LDR by students, either with the help of their peers or the teacher, and emphasized that these items belonged to the theoretical dimension of Learning Statistics.According to the conjunction of these items, the new factor was called ''Cooperative Learning.''Factor three explained 18% of the variance and integrated items 1 to 7 (Table 3); this series of items corresponded to the theoretical dimension LDR Functionality and Accessibility.
Factor four explained 22% of the variance and integrated items 21, 22, 24, to 28 along with 31 and 32 (Table 3); this series of items corresponded to the theoretical dimension of Learning Statistics.
Confirmatory Factor Analysis.The adjustment measures of the CFA showed a good fit for the model (Table 4).More precisely, the ratio x 2 /gl, the absolute indices (GFI and RMSEA; Table 4), and the incremental indices (CFI and TLI), and the RMSR showed optimal values (Table 4).The final model is shown in Figure 1.
The resulting factorial model can be seen in Figure 1.As shown, all the items presented high factor loadings (FL .0.50).Correlations between latent variables are shown with two-way arrows.The IPDLR-S consists of the latent variables: Functionality and Accessibility of the RDR (FA), RDR Design and Structure (DE), Learning Statistics (LS), and Cooperative Learning (CL).
Convergent and discriminant validity.It was found that the four dimensions presented convergent validity since the AVE were greater than .50AVE FA = 0:7999; AVE DE ð = 0:8375; AVE AE = 0:8488; AVE AC = 0:8099Þ.Likewise, it is stated that the existence of discriminant validity was denoted since the AVE of each dimension is greater than their corresponding correlations with the other dimensions (Table 5).
Instrument Reliability and Composite Reliability.Regarding the reliability of the instrument, an optimal value was obtained (ordinal Cronbach's alpha: .98).Also, all four factors demonstrate reliability well above the minimum acceptable (CR ..7;Table 6).

Instrument satisfaction analysis
Finally, the analysis of satisfaction with the instrument (Table 7) revealed a good degree of understanding of the instructions and items by the students.

Discussion
The COVID-19 pandemic has forced the implementation of new teaching methods strongly supported by new information technologies.Therefore, emergency teaching (Kohnke et al., 2021;Xu et al., 2021) involves the use of LO in online classes.
As indicated, the DLRs (as practice LOs) have been applied in this new pandemic context; therefore, it is extremely important to investigate how they influence learning processes; therefore, it is necessary to know the perception of students regarding the significance of DLRs in the learning of statistics.As already indicated, validated instruments to assess students' perception of the significance of DLRs in learning statistics are scarce.
That is why Monge-Rogel et al. ( 2022) proposed the IPDLR-S instrument, which allows measuring the perception of students regarding the use of DLRs in the learning of statistics.An aspect to highlight of the instrument is that it was subject to a process of content validation, meaning that its items have a high degree of relevance and relevance to the objective of the construct (Haynes et al., 1995;Jua´rez-Herna´ndez & Tobo´n, 2018;Koller et al., 2017).According to the above, as referred to by Carvajal et al. (2011), the analysis of the psychometric properties of an instrument is a continuous process, and the greater the number of properties analyzed, the quality of the instrument is strengthened.In this sense, several authors consider that construct validity is the property of greatest relevance since it allows to confirm or adapt the theoretical construct of the instrument and provides reliability for the realization of new predictions (Brown, 2015;Houston, 2004;Wilson, 2010).In this sense, the construct validity analysis is carried out to guarantee the validity of the measurement; it relates the structure and demolishes that the theoretical and emerging relationships are appropriate (Lagunes-Co´rdoba, 2017;Mavrou, 2015).That is, construct validity guarantees that the instrument measures what it wants to measure since it uses valid measures of the studied constructs (Houston, 2004).
Therefore, in the present study, the construct validity of the IPDLR-S instrument was analyzed, the above through a cross-validation process, which is referred to as the process of greatest precision (Brown, 2015;Fokkema & Greiff, 2017;Schmitt, 2018).
In this regard, it is necessary to indicate that to perform the EFA, a polychoric matrix was used since the data originate from a Likert scale, which corresponds to   variables that do not conform to a normal distribution (Watkins, 2020).According to Flora and Curran (2004), polychoric correlations tend to be robust to data violations.
According to the EFA, in the first instance, it is denoted that all the items are represented within the factorial model, which shows that the items represent the proposed construct (Lagunes-Co´rdoba, 2017;Mavrou, 2015).The foregoing reveals the importance of the content validation phase to which the instrument was submitted (Monge-Rogel et al., 2022), since content validity is an important step in construct validity because it contributes evidence regarding the degree to which the items of an instrument are relevant and pertinent to the construct (Haynes et al., 1995;Jua´rez-Herna´ndez & Tobo´n, 2018).
The factorial model obtained through the EFA showed differences concerning the theoretical model since it went from three dimensions to four dimensions.This can be explained because the perception that students have regarding the functionality of DLRs when they ask for help or develop it with others (student or teacher) forms a new dimension.Regarding the migration or movements of items, it is specified that items 23, 29, and 30 were moved to the new Cooperative Learning dimension, which would be explained by what has already been mentioned around the cooperation with others regarding the use of the DLRs.According to the aspects addressed by these items, and given the current context of the COVID-19 pandemic, collaborative learning takes on a special connotation that, for researchers was not initially considered.This is explained since the DLRs were designed to complement student learning in statistics subject in collaboration with peers and teachers, learning situations that changed in emergency teaching (Kohnke et al., 2021;Xu et al., 2021).It is important to note that altogether the resulting factorial model explained 83% of the variance, which according to Hair et al. (2010) and Rietveld and van Hout (1993), is considered highly acceptable.
The evaluation of the resulting factorial model through the CFA revealed an optimal adjustment of the same, which is verified by the different indices and adjustment indicators used (GFI = 0.999, RMSEA = 0.064, CFI = 0.999, and TLI = 0.999).In this regard, through this analysis the empirical sustainability of the model (Hair et al., 2010) can be indicated meaning that the 4-factor model fits the instrument appropriately to measure students' perception of the use of DLRs and their influencing factors.
Regarding the analysis of convergent validity of the factorial model, according to the indicators indicated (AVE .0.50), it can be indicated that the constructs that are expected to be related, in fact, are (Fornell & Larcker, 1981).Regarding the discriminant validity, it was found that the correlations squared are less than the AVE of each corresponding factor; therefore, the model can discriminate between different constructs (Hair et al., 2010).
The global reliability (ordinal Cronbach's alpha ..98), as well as by factors (ordinal Cronbach's Alpha ..92), was optimal, meaning that the reliability of both the instrument and the factors is excellent (George & Mallery, 2003), that is, the instrument will produce consistent results, free of errors, always measuring the perception of the students in the use and functionality of the DLRs in the four dimensions that compose it.
Finally, it is important to highlight what refers to the affordability of the instrument to the target population; the understanding of the instructions and items was between good and excellent degree.The analysis of this characteristic is fundamental since it allows to ensure that the instrument and its items are understandable by the students and possible alterations of their answers by aspects of redaction and, at the same time, reduce possible problems with the items that may affect evaluative capacity (Carrillo-Avalos et al., 2020;Carvajal et al., 2011).
Although the IPDLR-S instrument applies to the DLRs that are used for teaching statistics, it is important to note that this instrument can be used for any subject.The foregoing is justified considering that the structure of the instrument allows making minimal changes in its wording to adapt the instrument to apply it to other subjects in science, technology, mathematics, or engineering.
One of the limitations of this study is that the designed instrument only covers higher education.Likewise, the validated instrument only allows evaluation of the perception of university students in the field of practical learning objects (Churchill, 2007).

Conclusions
It can be concluded that the IPDLR-S instrument (Monge-Rogel et al., 2022) has adequate psychometric properties, which is why it constitutes an innovative tool that will make it possible to reliably and validly evaluate the functionality of the RDAs from the student's perspective.

Figure 1 .
Figure 1.Confirmatory factor analysis model of the IPDLR-S instrument.

Table 3 .
Communalities and Factor Loadings by Items.

Table 4 .
Adjustment of the Factorial Model.

Table 5 .
Discriminant Validity of IPDLR-S With the Second Sample (n 2 = 574).Note.The values in bold correspond to the AVE, and the non-bold correspond to the correlation between the dimensions (r 2 ).

Table 6 .
Composite Reliability Analysis for IPDLR-S Dimensions With the Second Sample (n 2 = 574).

Table 7 .
Frequency Distribution of Satisfaction With the IPDLR-S Instrument (n = 1,148).