Exploring Issues of Implementation, Equity, and Student Achievement With Educational Software in the DC Public Schools

This research is part of a growing number of studies that examine the relationship between technology integration and student achievement in K–12 schools. Current initiatives use terms such as blended learning, which describes the ability for teachers to use face-to-face practices and offload some of the learning tasks to digital curriculum, games, or other learning software (Staker & Horn, 2012). Recent studies suggest that integrating digital tools into classroom settings may be beneficial for student learning outcomes. For example, a 2010 meta-analysis on online and blended learning found that students in blended learning conditions generally performed better than students in fully online or fully face-to-face conditions (Means, Toyama, Murphy, Bakia, & Jones, 2010). The authors of the study warn that positive achievement outcomes are not caused by technology alone and that other factors—namely, increased time spent on learning activities—may be the direct links to student outcomes.

Most studies of blended learning situations tend to examine higher education contexts, but research in the K–12 arena is steadily increasing. The results are mixed when different uses of educational software and student learning outcomes are examined (Barrow, Markman, & Rouse, 2008; D. B. Clark et al., 2015; Figlio, Rush, & Yin, 2013; Hong, Tsai, Ho, Hwang, & Wu, 2013; Jia et al., 2013; Pane, Griffin, McCaffrey, & Karam, 2014; Shin et al., 2012; Smith & Suzuki, 2015; Wijekumar, Meyer, & Lei, 2012). Most studies examine the efficacy or average effects of blended environments, as compared with a face-to-face or fully online treatment, and find positive benefits on student achievement. However, one study of 10,000 students and their usage of several software products found no significant difference between blended learning and fully face-to-face learning during the first year of implementation. The researchers found mixed results (some positive and some negative) during the second year of the study (Campuzano, Dynarski, Agodini, & Rall, 2009).

Prior research suggests that introducing computing, software, and games into formal instruction is associated with positive gains in student achievement, but there are open questions about how to explain these relationships, for whom, and under what conditions. For example, Means et al. (2010) suggest that learning gains are explained, not by technology, but by the added time spent with instruction or with the learning tasks that technology may enable. Furthermore, certain types of technology-driven blended learning situations may benefit subgroups of students more than others. For example, Shin et al. (2012) found that White male students benefited the most from playing math games on the GameBoy system. These various findings point to the need to better parse out the role of technology in learning outcomes and the complex ways that implementation affects this relationship.

At the most basic level, students need access to a new technology or curricular tool and must readily use the tool to expect any learning benefits. Thus, we focus our analyses on a commonly collected metric in digital learning platforms—the time that a learner spent in a given platform. Through our RPP with DCPS, we were able to obtain detailed log data from the vendor of FIM. These data included variables that described when students logged into the platform and how much time they spent playing the math games in FIM throughout the school year. We recognize some of the limitations related to using time-on-task measures in the Discussion section (Kovanovic, Gašević, Dawson, Joksimovic, & Baker, 2016). However, taking into account these considerations, we note that having detailed access to time spent in FIM at the student level allows us to delve deeper into how software use was distributed across this urban school district and to explore the correlation between use and achievement more directly than what was possible in past research.

Past studies on technology implementation and equity in schools also inform our conceptual framework and data analyses. Education scholars know that there are many complex organizational and social processes that have to be aligned for technology to be used in effective ways (Fishman, Marx, Blumenfeld, Krajcik, & Soloway, 2004). School districts must make particular decisions about what technologies to purchase, and they must provide organizational support to maintain access to these resources. For effective adoption to take place within a given school, school leaders and teachers need to create environments such as peer advice networks, technical support staff, and organizational schedules conducive to experimenting with new technology (Cuban et al., 2001; Penuel et al., 2010; Warschauer et al., 2011; Zhao & Frank, 2003). Teachers need to be able to learn how to use technologies but, more important, learn how to effectively integrate tools to create better learning environments (Ertmer, 2005; Mishra & Koehler, 2006). Finally, personal factors—such as students’ motivations, goals, prior learning histories, and life contexts—influence how they take up new technologies and for what means, such as play, socializing, and studying (Ahn et al., 2014; Azevedo, 2011; Ito et al., 2010; Ito et al., 2013; Polman, 2006). Thus, many factors may come into play in the interpretation of a simple metric such as time spent in the FIM platform.

In the following study, our RPP team focused on two research questions that examine issues of implementation, equity, and student achievement. First, we were interested to document and track which students were spending time in the FIM platform throughout the school year. Past research in technology implementation suggests a stark inequality in terms of which students have access to and take up new technology (Warschauer & Matuchniak, 2010). DCPS structured its technology initiative to give hardware and software access to all schools in the district. However, there may still be inequitable uptake of FIM among students in different schools that is important to examine.

Second, an important question from the district perspective was to understand whether using FIM was correlated with improved student achievement at the end-of-year standardized assessments. We examined this question directly by exploring whether students’ time spent in the software was related to student achievement measures.

DCPS is a large urban school district that, in the 2015–2016 school year, operated 113 schools serving P–12. The district enrolls >49,000 students. Approximately 88% of students identify with an ethnic minority group; 10% are English language learners; and 16% are in special education programs. Finally, >75% of students in the district qualify for free and reduced-price meals, which is a common indicator of students who fall under the poverty line. DCPS reported that in 2014 approximately 51% of its students scored “proficient” or “advanced” on the standardized math exam (“DCPS at a Glance,” n.d.). However, the district, much like other urban school systems in the United States, sees stark gaps in educational achievement across indicators such as race and gender. For example, in 2014, the district reported that >90% of White students (female and male) in the district scored proficient in the math exam. Comparatively, only 45% and 37% of African American girls and boys respectively scored proficient on this assessment (“Achievement Gap, Race and Equity,” n.d.).

In 2012, DCPS began to implement a mathematics program in Grades K–8 called FIM: a web-based series of mathematics games where students practice mathematics computation skills such as addition, subtraction, multiplication, and division, as well as concepts such as fractions, decimals, and integers. The activities focus on skill-building practice with the various mathematics concepts. It is important to note that the game tasks are largely akin to drilling math computation. For example, students may be presented with different multiplication problems and, in a timed exercise, asked to fill in the answers to those problems, which are scored on correctness. Students can earn points and badges as they practice more. The program attempts to motivate students to engage through gamification elements, such as rankings and leaderboards (tracking a school’s progress versus other schools), opportunities to win “stickers” that mark achievement, and chances to unlock bonus games.

It is also important to note the limitations of FIM’s design. From a critical perspective, the program is not designed to teach deeper conceptual thinking or mathematical practice; instead, it is a program that presents drills and rote practice for students (Gee, 2005). From a learning perspective, the program can largely be viewed as a supplemental tool where students can develop skills through repeated practice, and we corroborated with our partners in DCPS that teachers largely used the program as extra practice activities for learners. DCPS offered the program to all schools, using an opt-in model of implementation rather than a top-down mandate.

This study offers several implications for RPPs and research about implementation of educational software at scale in public education systems. We demonstrate the potential to combine student-level data from a school district, with usage data from digital platforms (e.g., FIM), to delve deeper into issues of technology implementation, student use, and relationships with learning outcomes of interest to educators and district leaders. Our analyses have been valuable for our RPP team, compelling new insights and further questioning that informs our future work together. This iterative inquiry-based process is core to RPPs (Penuel et al., 2011), and the availability of new data sources helps to facilitate this work. For example, our findings about students’ time spent in FIM help to validate district decision making. We show that efforts of DCPS to provide computing, Internet, and technical infrastructure, as well as to make software such as FIM available to all schools, may have some positive benefits. Students of color show more time spent with FIM, and it appears that schools that serve lower-achieving students are providing students more access to the program (e.g., students in lower-achieving school environments are spending more time in FIM). Using FIM during the school year is also significantly correlated with higher performance on the DC-CAS at the end of the year. Thus, some amount of usage seems to be worth the district’s efforts.

However, our analyses also suggest areas that require further inquiry for DCPS. For example, female students are spending less time in FIM, and students who are already higher achieving spend substantially more time in the program. These patterns compel further questions about those students who are not using FIM. Our data cannot tell us why these patterns of nonuse are present, but prior research provides some frameworks for potential avenues of future work in the district. Perhaps school leadership and organizational routines in different schools are shaping how FIM is presented and given to students for use (or nonuse). It is possible that the activities in FIM (drill and practice) are enjoyable for students who already have higher achievement but less so for students struggling to understand the math content and skills presented to them. Teachers could be conceptualizing their classroom practice in ways that may align or misalign with their understanding of how to use FIM with students. Our analyses are a first step in identifying potential inequities in implementation and spurring further questions for our RPP to understand these patterns in the data. We argue that this iterative inquiry—and a focus on equitable implementation patterns—would benefit school districts across the United States that are also implementing new digital platforms.

The findings suggest that spending time using FIM is correlated with improved performance on the DC-CAS at the end of the school year, but they also provide some nuance to how useful FIM may be and for which students. We know that FIM is limited in design, as largely a drill-and-practice application, and students spent relatively very little time over the course of the school year using it. However, even a small amount of drill and practice, such as 20 hr total over the entire school year, could potentially relate to higher performance on math assessments in the district (0.14-SD improvement). Our findings add to other studies showing that some computer-aided instructional programs could be related to improved academic achievement in math classrooms. For example, Barrow and colleagues (2008) found that a computer-aided math instruction platform—I Can Learn—was related to 0.17 SD of higher performance for middle and high school students.

Finally, we observed an interesting finding where spending time using FIM was more positive for students in lower-performing school environments. Taken together, these findings suggest some potential strategies for our RPP. It may be very reasonable for the district to frame FIM as a beneficial tool for students to practice basic skills, a little at a time and as a supplement, and focus the rest of its mathematics instruction on more ambitious learning activities. Past research in educational technology suggests that underserved and lower-performing schools are more likely to engage students in drill-and-practice forms of technology, with all of the negative connotations associated with that finding. However, our analyses suggest that a limited amount of drill and practice in FIM, for students in lower-performing school environments, appears to be beneficial for promoting student achievement on standardized assessments.

However, the limitations of our data also compel future questions for our RPP and general research on technology implementation in schools. First, time on task in a software environment is a potentially problematic measure for interpretation (Kovanovic et al., 2016). Our data cannot tell us how students spent time logged into the FIM platform—that is, whether they were progressing, distracted, logged in but not doing any tasks, and so on. We checked our time-on-task data with other indicators in the FIM log data to check for robustness. For example, we examined the relationship between the stickers/badges that players earned in FIM (for completing tasks) and their time spent in the program, and we found a largely linear relationship. This finding suggested to us that time also was a good indicator for progress (earning stickers/badges) in our FIM data set, and it gave us confidence to move forward with this measure in our models. However, in other districts or settings, this pattern is not a given, as time can be used in many different ways. From a theoretical perspective, we also cannot make claims about how students used their time in FIM beyond rote practice tasks. Did they speak with their peers, do activities together, or ask questions of their teachers? Our indicators for time provide analytics and patterns of interest to observe, but they are limited in delving into microlearning processes that are important.

Second, as with many field studies, we cannot control for the myriad of other programs that students are exposed to in the school district that are implemented simultaneously with FIM. For example, it may be possible that students with zero time on task in FIM or little time spent in the program are benefiting from other similar programs. Without data on those programs, it is difficult to fully pinpoint the impact of FIM on achievement. However, we previously noted that we examined models with school fixed effects and found similar results for time on task. In those models, we control for any programs adopted at the school level, and it would be likely that students within the same school would be exposed to similar sets of programs. That we find similar results for time on task in these models suggests some robustness of our findings.

Finally, we note that correlation does not equal causation. Our models take advantage of time, using strong controls such as prior achievement and modeling time spent in FIM to predict future academic performance. Although our correlational analyses provide compelling evidence that time spent in FIM was related to improved academic achievement in small but significant ways, we cannot make any causal claims that time spent in FIM caused better performance. It may be plausible that students who used FIM more also had better teachers, more motivation, academic supports, or other situations that explain the relationship between their time spent in FIM and academic achievement. From a theoretical perspective, we cannot rule out these alternative possibilities. However, from an analytics perspective, time spent in FIM may serve as a useful indicator that signals a need to look deeper into these students. Qualitative understanding about which students spend substantial time in the program and, relatedly, their peers who decide to be nonusers will be beneficial for the district to understand how to better support diverse learners over time. In this way, delving deeper into multiple data streams and analytics of digital trace data in educational software can play an integral role in school district decision making, educator practice, and improvement over time.