The Development of a Critical–Creative Reading Assessment Based on Problem Solving

The ability to think critically and creatively is essential for students to help them thrive in the 21st century. Creative and critical thinking can be measured through problem solving because the assessment contains tasks that require students to find problems, analyze and evaluate problems, and work out the solutions. Therefore, this study was designed to develop a critical and creative reading assessment model based on problem solving and test the validity and reliability of the assessment. The assessment contained problem-solving essay tests that were presented in the form of stories. The material stimulus was news item texts that focused on social problems. The data collected were in the form of mean scores of expert validity testing and practitioner inter-rater reliability testing. The assessment was tried out to 36 students. The validity of the instrument was analyzed descriptively, while the reliability of the instrument was analyzed using analysis of variance (ANOVA) and intraclass correlation coefficients (ICCs) with SPSS 26. The results of the study indicated that the mean score of the expert validity test was categorized high (3.74), and the result of the reliability test showed that there was no difference in the inter-rater assessment because p > .05 and the ICC obtained was excellent (rxx = .716). It can be concluded that the assessment developed in this study was valid and reliable, and thus can be used as an assessment to improve the critical–creative reading skills which are the essential skills needed to survive and thrive in the 21st century.


Introduction
Critical-creative reading is a reading activity that involves high-level thinking skills, that is, critical and creative thinking (Wallace, 2010). Both thinking activities have a different focus; critical reading is focused on assessing, while creative reading is focused on a process of making or producing (Beghetto & Sriraman, 2017;Paul & Elder, 2008). The critical-creative reading ability is required by everyone in today's global age to respond critically and creatively to new, irregular situations that are rapidly changing and developing (World Economic Forum, 2015).
The critical-creative reading ability can be measured through problem-solving-based assessments because the assessment contains tasks that require students to find problems, analyze and evaluate them, and then work out the solutions (Jonassen, 2010). Problem-solving-based assessments involve a cognitive operation directed to discover the unknown (Anderson, 2010). The problem-solving skills signified by two important skills, namely, problem finding skill and its problem context and the skills describing the problems' structure, knowledge about the problem, as well as activities for problem solving (Jonassen, 2010). In the problem-solving activities, learners are also demanded to actively develop various alternative models of problem solving and test the accuracy of the developed model (Jonassen, 2010).
Why is a problem-based assessment needed? In daily life, we are never free from big or small, simple or complex problems. The success of our lives is actually marked by our success in solving various problems in this life. Therefore, learners need to be equipped with problem-solving skills. In professional context, people are paid to solve problems, not complete exams (Organisation for Economic Co-Operation and Development [OECD], 2014). Therefore, the problemsolving skill needs to be trained so that learners can be actively engaged in solving their own problem and the problems faced by society (Griffin & Care, 2015). In general, the problem-solving skill is oftentimes neglected by educators in the teaching and learning as well as in the evaluation (Beghetto & Sriraman, 2017;Jonassen, 2010). Indonesia is one example of a country whose national examination test for Indonesian subjects is all in the form of multiple choice (Priyatni & Martutik, 2019). The level of thinking measured in Indonesian national test is dominated by basic reading skills (Harsiati, 2017b).
Multiple-choice assessment cannot be used to measure higher order thinking skill needed to live in the era of the 21st century, particularly critical-creative reading skills because the competence needed to answer the multiple-choice objective questions is to choose one answer from the provided options (OECD, 2014). Assessment practiced in Indonesia is likely focused on what is known instead of what can be performed by the students (Harsiati, 2017a). The teachers mostly concentrate on evaluating lower order thinking skills rather than emphasizing on developing the skills to think at a higher order level, such as critical thinking, reasoning, and problem solving, which in fact constitute skills and abilities needed in the 21st century (Priyatni & Martutik, 2019). Therefore, educators, especially teachers, need to transform the way they teach and evaluate the students.
The problem-solving-based critical and creative reading assessment proposed in this study requires students to respond critically as well as simultaneously construct a product rather than simply select from a set of predefined alternatives or answers. Therefore, this assessment is important to be developed to strengthen or complement multiple-choice assessments that only select the already provided answers.

Literature Review
There are two essential topics discussed in this section, namely, the element of critical-creative reading and the elements of the proposed assessment.

The Element of Critical-Creative Reading
Critical reading is an activity to express ideas, draw conclusions, predict goals, analyze arguments, find different points of view, and evaluate ideas from written texts (Priyatni & Nurhadi, 2017;Zintz & Maggart, 1984). Critical reading contains not only the activity of understanding a text but also the activity to determine whether the information in the text is true or false and to assess the knowledge and opinions contained in the text (Sultan et al., 2017b(Sultan et al., , 2018. The main objectives of critical reading activities are to evaluate information from various points of view, including logical, rhetorical, historical, ethical, social, and personal perspectives (Wheeler, 2013), and discover new interpretations of texts (Asilioglu, 2008).
Critical reading is a reading activity that provides the reader with the skills to analyze, synthesize, and evaluate what is being read (Hudson, 2010). Some experts argue that critical reading is characterized by reasoning and reflecting activities (Costa, 1991). Being reasonable means showing a reason or sound judgment, and being reflective means to think about what we have read or heard seriously and involves any activity to assess the validity of the information and opinions we encounter (Sultan et al., 2017a). Based on the aforementioned information, it can be concluded that the essential elements of critical reading include analysis and evaluation of what is being read, examination of what is good and what is not, and "why." Unlike critical reading, creative reading is a reading activity that involves creative thinking skills. Creative thinking skill is identified with the emerging of new ideas or the combination of several ideas in the form of new ideas (Marin & Halpern, 2011). Creative thinking also involves divergent thinking, a cognitive aspect that is characterized by fluency, flexibility, elaboration, and originality (Nakano & Wechsler, 2018).
Creative reading is strongly associated with imagination, innovation, originality, and excellence (Jonassen, 2010;Swann et al., 2011). As the aforementioned definitions, it can be summarized that creative reading is reading to produce creative writing based on the response to what has been read. Creative reading is the key to productive thinking because creative thinking is defined as a mental activity used to build ideas (Syahrin et al., 2019). In conclusion, critical-creative reading consists of two essential elements that are the skills to make a decision and the ability to develop a new idea based on the result of the decision-making process.

The Existing Critical-Creative Reading Assessment and the Proposed Critical-Creative Reading Assessment
Program for International Student Assessment (PISA) included the assessment based on problem solving as an additional assessment. The problem-solving assessment was first carried out separately in PISA 2003 which focused on individual problem solving capabilities and static assignments. Students must choose several alternatives, and student also need to analyze systems and identify relationships between parts of a system. The stimulus used for problemsolving-based assessment in PISA 2003 is a robot cleaner unit (OECD, 2014).
In addition, a problem-solving assessment has also become a domain in PISA 2012, which focused on complex and computer-based issues (OECD, 2014). PISA 2012 also used a creative problem-solving assessment to assess an individual's ability to solve real-life problems of a general nature. PISA 2012 also utilized an assessment based on problem solving as an additional assessment in general and based on the interactive computer (OECD, 2014).
Problem-based assessment developed by OECD (2013) is designed to solve general problems that are problems related to technology. This article proposes an assessment model to evaluate students' abilities to solve social issues that emerge in their daily lives. This assessment was also designed to measure the students' critical-creative reading skills in learning the Indonesian language (Bahasa Indonesia). This assessment model is modified from the Story Problem Solving Model developed by Jonassen (2010), with three focused activities, namely, (a) finding the problem and the root of the problem, (b) planning and executing solutions, and (c) expressing the best solution in the form of creative writing. Jonassen's model assessment is very simple and in line with the problem-solving critical-creative reading framework.
The validity and reliability of the proposed assessment model were tested. The result was then submitted to complement objective tests that had been national domination assessment in Indonesia. The problem-solving critical-creative reading assessment requires students to respond critically as well as simultaneously construct a product rather than simply select from a set of predefined alternatives or answers. Therefore, this critical-creative reading with a problem-solving-based assessment is important to be developed to strengthen multiple-choice assessments that only select the already provided answers.
The assessment proposed in this study was adapted from Jonassen (2010). Jonassen (2010) suggests four steps in solving a problem, namely, (a) problem presentation, (b) problem articulation, (c) alternative solution finding, and (d) argumentation/justification. The problem presented is the analysis of the problem and its context, problem articulation is to formulate the problem and find the root of the problem, alternative solution finding includes considering several alternative solutions, and argumentation/justification is to determine the best solution along with the reason. The four steps were adapted into three steps, namely, (a) finding the problem and determining the root of the problem, (b) planning a solution, and (c) expressing the solution in the form of creative writing.
Problems that are to be solved by the students are those that are found in their everyday lives. These problems are problems with an unknown solution. If someone has set a goal and does not know how to reach that goal, that is what is called an unknown solution (Jonassen, 2010). Problems solved in the assessment are simple problems because the assessment is going to be used to measure senior high school students' problem-solving skills. Problems can be sorted into two types, namely, simple problems and complicated problems (Jonassen, 2010). A simple problem is in small-scale, disconnected from other problems, has no great consequences, the solution is not too complicated, and can be solved individually.
Problems developed in the assessment are well-structured problems. Problems can be categorized into two, namely, a well-structured problem and ill-structured problems. A problem can be categorized as a well-structured when all elements of the problem are clearly described, requiring convergent answers and prescribing solution process. A problem is defined as an ill-structured when all elements of the problem are not clearly described and require the learners to secrete an opinion and personal belief to determine the solution of the problem.
The proposed assessment was in the form of problemsolving essays that allow students to discover problems and determine the root of the problems, plan solutions to the problems, and express the solutions in the form of creative writing. This assessment targets various aspects, including creative and critical thinking.

Research Design
The current study aimed to develop a product in the form of a problem-solving-based critical-creative assessment. The design of the study was adopted from O' Malley and Pierce (1996), who suggests eight stages of product development, namely, (a) build a team, (b) determine the purpose, (c) specify objectives, (d) conduct professional development, (e) collect sample assessments, (f) adapt existing assessments, (g) try out the assessments, and (h) review of the assessments.
The first step was to build a team. The team consisted of researchers, experts, and practitioners (teachers) who collaborated in developing the assessment. The second step was to determine the aim of the assessment, which was to improve students' problem-solving, critical thinking, and creative thinking skills. Next, the third step involved deciding which basic competence to be measured. Students' critical-creative thinking skills were finally determined as the basic competence to be measured in the assessment.
The fourth step, conducting professional development, was to inform all team members about the design of the prepared assessment. The fifth step was finding appropriate examples or models of assessment. The assessment model was adopted from Jonassen (2010). Jonassen (2010) proposes three steps in solving problems, namely, (a) choosing the problem to be solved; (b) understanding the structure of the problem, including recognizing the structure that underlies the problem, producing an appropriate presentation to find a solution, and drawing a conclusion on the problem presentation; and (c) determining the solution.
The sixth step was to adapt the assessment developed by Jonassen (2010). The three problem-solving steps proposed were adapted into the following steps: adding solutions (a minimum of three solutions), selecting the best solution and the reason why choosing the solution, and developing the solution in the form of creative writing. The assessment contains three main tasks, namely, (a) finding the problem and the root of the problem, (b) planning and executing at least three alternative solutions, and (c) expressing the best solution in the form of creative texts. The main features of the critical-creative reading assessment based on problem solving can be seen in Table 1.
The seventh step was to carry out validity and reliability tests. Validity was measured from two aspects, namely, construct validity and content validity. A construct validity test was done to see the compatibility of the items with the theoretical foundation of the problem-solving-based critical-creative reading assessment. A content validity test was carried out to see the appropriateness of the test items with the direction or objectives and aspects measured in the assessment pack. The validity of the assessment was tested by a team of reading and writing experts.
In addition, the reliability of the assessment tool was also tested. Inter-rater reliability was conducted because the assessment contained essay questions. The reliability test was performed using the following procedures: (a) the assessment was tested on 36 students; (b) subsequently, four practitioners were invited to rate student work using the compiled assessment rubric, with the highest score of 4 and the lowest score of 1; (c) the assessment data obtained from four assessors were inserted into a table; and (d) the data were analyzed using SPSS 26. The eighth step, product revising, was conducted to make adjustments or changes in the assessment according to the validity and reliability test results.

Data and Data Sources
The data of this study consisted of two categories, namely, quantitative and qualitative data. The quantitative data were collected in the form of validity test scores from a group of experts and practitioners. The qualitative data were in the form of responses, suggestions, or criticisms on the validity of the contents and constructs of instruments from expert teams. The sources of research data were four experts, divided into two experts in the evaluation and two experts in reading and writing learning, who hold a doctoral degree qualification. The sources of data for reliability testing were four practitioners and 36 students.

Instrument
An assessment review guide was used to collect the data on assessment validity. The guide contained an assessment table which measured three aspects, namely, (a) construct and content validity, (b) aspect description, and (c) scores scaled from 1 to 4 (1 = not appropriate, 2 = less appropriate, 3 = appropriate, and 4 = highly appropriate) (for complete instrument, see Supplemental Appendix 1).
The instrument used to collect the inter-rater reliability data was the problem-solving-based critical-creative reading assessment and its scoring guidelines. The reliability test was conducted by (a) instructing the students, as the tryout subjects, to read the test guideline, stimulus, and the essay questions; (b) asking the students to understand the scoring guidelines and telling them that their test answers would be scored using that guideline; (c) asking the students to complete the essay tests; (d) asking the students to review their work based on the guideline; and (e) evaluating the students' work and providing feedback to their work.
The full scoring guidelines are listed in Table 2.

Data Analysis
The result of the validity test was analyzed descriptively and interpreted based on the criteria of assessment validity, such as presented in Table 3. The reliability test result was analyzed using analysis of variance (ANOVA) and intraclass correlation coefficients (ICCs).

The Main Features of the Critical-Creative Reading Assessments Based on Problem Solving
The product developed in this research is a problem-solving assessment model used to measure the ability of high school students in critical-creative reading. The assessments contain test models and scoring guidelines. The objectives of the test focus on three skills, namely, (a) finding the problem and the root of the problem, (b) planning and executing solutions, and (c) expressing the best solution in the form of creative texts. The first skill consists of two sub-skills, namely, the skill to find problems and root problems based on information in story problem solving. The second skill contains two sub-skills, namely, the ability to design at least three different alternative problem-solving solutions and determine the best solution completed with reasons. The third skill contains two sub-skills, namely, the ability to create a text framework and develop the text framework into a creative writing text. The type of test was an essay-story problem solving. The test package contained six items that constituted a complete overall ability which consisted of the ability to find the problem, the root of the problem, finding three different solutions, finding the best solution with the reasons, designing the text framework to explain the best solution, and developing the text framework into a full text. The time allocation for the essay test was 120 min. The material stimulus used to develop essay tests originated from news texts that address social issues so that students can provide solutions to social problems faced by the environment/nation. The examples of social problems are the rise of hate speech in social media, the clash of supporters, rampant fraud, early marriage, or drug abuse. The writing style of the material stimulus is narrative. The length of the text is 450 to 600 words. The main features of the criticalcreative reading assessment based on problem solving were used to develop the blueprint of the test and also as a reference for developing the test. In addition to the tests, the scoring guideline is also developed. The score is ranged from 1 to 4, the best score is 4 and the worst score is 1. There are six major sub-skills assessed, which are finding the problem, determining the root of the problem, finding three significantly different solutions, determining the best solution with the reasons, planning and executing solutions, and expressing the best solution in the form of creative texts.

The Validity of the Test
The mean of the validity test score was 3.74. This suggests that the validity of the assessment can be categorized as high. The expert judgment on construct and content test validity also showed high validity of the assessment because the four experts stated that all items were in line with the theoretical critical-creative reading assessment and the framework of problem-solving skills. The result of the validity test can be seen in Table 4.
There were some important suggestions from the expert team that serves as a reference for refining these developed assessments. The suggestions from the expert team are presented in Table 5.
Feedback from the experts has been considered to improve the assessment (see Supplemental Appendix 2).

The Reliability of the Test
The inter-rater reliability test was performed by asking four practitioners to score the work results of the 36 students, using the scoring guideline (see Table 2). The result of the reliability test was analyzed using ANOVA and ICC (see Tables 6 and 7).
The result of the ANOVA showed that there was no difference in the inter-rater scores because p > .05 and the result of the ICC analysis showed a satisfactory score, which is r xx = .716.

Discussion
Based on the result of the test validity, proved by studying the construct and content validity, it can be concluded that the assessment developed in this study had high validity to be used to measure critically creative reading skills. This is because the assessment involves three cognitive operations, which are as follows: (a) finding the problem and the root of the problem, (b) planning and executing the solution, and (c) expressing the best solution in the form of creative texts. Finding the problem and the root of the problem is a critical reading activity done first by understanding the story problem solving, identifying the problem contained in it, analyzing it, and then determining the root of the problem. This is in harmony with the critical thinking feature that involves bringing the outside knowledge and values to deliver and evaluate the presentation and decide what to ultimately accept as true (Facione, 2015). The essence of critical reading activity is an activity to analyze and evaluate what we read, working out what is good and what is not, and why.
Finding three significantly different solutions is one form of creative reading skills. This is because students not only understand the text but also respond to the text content. Students are also asked to express the best solution in essay form. The essay contains at least four or five full paragraphs containing one introductory paragraph, two or three paragraphs detailing the original ideas offered for problem solving, and a closing paragraph. The creativity in reading involves an output which is the result of a process, and all processes require some form of input. The input of the creative result is an idea derived from critical reading; the idea is then processed into a creative work. The basic information processing model of creative reading is usually represented as Input → Process → Output. Creative skills are also characterized by original ideas relating to solutions to solve problems, the use of complex sentence structures with effective grammatical structures, appropriate and varied use of vocabulary and conjunctions, and minimal spelling errors. This is in line with the characteristics of creativity, which is associated with imagination, innovation, originality, and invention (Pope, 2005). The assessment tools in the form of story problem-solving essay tests were taken from the news texts. From the news text, it can be found real problems, such as social problems faced by the community around the students, so that students can play an active role in helping solve the problems faced by society or themselves. This is in line with the opinion of Harrington (2018), which states that the characteristics of creativity are new, useful, or valuable. Useful refers to the fact that creative "thing" has to make sense or be useful for a particular context. This is also in line with the opinion which states that the problem-solving-based assessment requires students to respond critically and simultaneously construct a product rather than simply select from a set of predefined alternatives or answers. The item is appropriate to measure critical reading skills, that is, the ability to assess or judge

High
The item is suitable to measure creative reading skills, which are the skill to make or produce texts

High
The item is appropriate to measure problem-solving ability 4.00 High The item is appropriate to measure students' sensitivity to the social issues in their surroundings 3.40 High

Content validity
The appropriateness of the test items with the framework of critical-creative reading assessment based on problem solving The test item is appropriate to measure the ability to find the problem

High
The test item is appropriate to measure the ability to find the root of the problem

High
The item is appropriate to measure the ability to plan and execute solutions

High
The item is appropriate to measure the ability to express the best solution in the form of creative text

High
Average 3.74 High The root of the problem should be added to the synonym to avoid misinterpretation For students to distinguish between the problem and the root of the problem, the item needs to be elaborated with the task of creating a diagram showing the root due to the discovery of the problem The task of finding the root of this problem is important so that students are accustomed to solving problems based on the root of the problem rather than on surface-level issues 3.
Scoring guideline/rubric The scoring guideline needs to be weighted for each item because the level of difficulty is different The test results of the reliability test, which were performed using an inter-rater reliability test to measure the reliability of essay tests and analyzed using SPSS 26, indicated that the developed essay test proved to be reliable for measuring critical-creative reading ability. This means that this assessment can produce a constant, consistent, and unchangeable or reliable measurement result (O'Malley & Pierce, 1996).
The assessment developed in this study can be categorized as an authentic assessment seen from the context, students, and assignments given (Herrington & Herrington, 2006). In relation to the context, this assessment uses a real-world context. The next criterion is the students. An authentic assessment must contain problem-solving and higher order thinking skills. In the context of the task, what is required is to stimulate various active responses which involve complexity in decision making and require assessment that is integrated with student activities (Mueller, 2005).

Conclusions and Recommendations
From the above description, it can be concluded that the critical-creative reading assessments based on problem solving had high validity, as seen from the construct and content validity. This means that this assessment can be used to measure students' critical-creative reading skills, especially for high school students. In addition to being valid, this assessment is also reliable, which means the critical-creative reading assessments based on problem solving can produce trusted or reliable measurement results. Furthermore, this assessment was also comprehended that students could measure the usefulness of the Indonesian language in daily life and play an active role in society.
Based on the results, it is recommended to implement this problem-solving-based critical-creative reading assessment to complement the multiple-choice tests that have been used in national examinations, school exams, or formative examinations. In summary, this assessment proves to be valid and reliable to be used as an assessment to improve critical-creative reading skills, which are the essential skills needed to survive and thrive in the 21st century.

Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was funded by the Directorate of Research and Community Service, the Directorate of Higher Education, the Ministry of Research, Technology, and Higher Education of the Republic of Indonesia under contract number: 21.8.23/UN32.14/LT/2017.  Note. ICC = intraclass correlation coefficient.