Assessing the Value of the OpenOrbiter Program’s Research Experience for Undergraduates

This article presents an assessment of the benefits gained by undergraduate students who participated in the OpenOrbiter Small Spacecraft Development Initiative. It provides an overview of the program and its learning objectives, as they apply to undergraduate students. It compares the learning impact between students who participated and those who assumed leadership roles. Qualitative assessment with regard to benefits is also discussed. The article extrapolates from these results to identify program elements that were particularly instrumental in delivering the positive benefits discussed. Finally, future work is discussed.


Introduction
The OpenOrbiter Small Spacecraft Development Initiative at the University of North Dakota provides a project-based experiential learning experience for students involved with the program. Participants have included the full range of university students: from freshmen to PhD students. These students have served in a variety of roles and have performed work spanning numerous disciplines of study. Many students worked on topics related to their major; a few pursued topics of interest that did not align with their field of academic study. Numerous teams were instantiated; each was led by a student team leader and mentored by a faculty member. In some disciplines (e.g., computer science and electrical engineering), student group leaders managed the interaction between multiple teams (each of which had its own team lead). Participants were able to learn new skills and apply existing skills to a real-world challenge. They also gained skills related to working with students in different disciplines: They learned the vernacular of these different fields as well as the working styles of their practitioners.
Some students participated in the project to satisfy a project component of a course or for independent or directed study credit. Many, however, participated as volunteers to gain experience in this real-world project which operated, in many ways, like an industry project. While the project had established learning goals (and delivered, based on participant feedback, some unanticipated learning benefits), it did not incorporate formal test-based assessment techniques, as the learning experience and topics varied by individual areas of participation. Student participants, instead, were asked to respond to an experience assessment survey, which asked them to characterize their competence in key learning focus areas prior to and after program participation. This article presents and analyzes the responses from a limited number of undergraduate participants in the program.

Background
The OpenOrbiter program draws on several different areas of prior work. It is working to design, build, and validate a CubeSat-class spacecraft, drawing (as would be expected) heavily on prior work in small spacecraft design. As an experiential or project-based research experience, it also draws on prior work in the design and operation of these techniques. Both these topics are now expanded upon. use of the spacecraft in educational settings, allowing students to, prospectively, be involved in the complete spacecraft design, construction, and validation process during their academic career. CubeSats have been successfully developed by numerous institutions (Klofas, 2011;Klofas, Anderson, & Leveque, 2008;M. Swartwout, 2012;. Many (M. Swartwout, 2004; served educational purposes; more recently, CubeSats have been used for bona fide research, communications, and other mission types. CubeSats were designed to cost a fraction of the price of larger spacecraft (Straub, 2012); recent work  has demonstrated that this cost can be driven lower through the use of publically available design documents and lowcost, readily available parts.

Experiential Learning for Undergraduates
Experiential learning (also commonly known as problembased or project-based learning) has been demonstrated to be effective at all levels of the educational continuum (Brodeur, Young, & Blair, 2002;Fevig, Casler, & Straub, 2012;Hall, Waitz, Brodeur, Soderholm, & Nasr, 2002;Mathers, Goktogen, Rankin, & Anderson, 2012;Mountrakis & Triantakonstantis, 2012;Straub, Berk, Nervold, & Whalen, 2013) and across numerous disciplines (Correll, Wing, & Coleman, 2013;Qidwai, 2011;Reynolds & Vince, 2004;Robson, Dalmis, & Trenev, 2012;Saunders-Smits, Roling, Brügemann, Timmer, & Melkert, 2012;Siegel, 2000). Breiter, Cargill, and Fried-Kline (2013), in the context of undergraduate hospitality management education, surveyed industry perception of the value of experiential education and found that industry perceptions of value included student learning of technical and management skills as well as learning related to intangible aspects of the field. In the context of science, technology, engineering, and mathematics (STEM) undergraduate education, work (Simons et al., 2012) at Widener University demonstrated how student participation in an in-the-field experiential learning exercise increased student understanding (as both self-assessed by students and assessed by their field supervisors) of subject material and its real-world application; it also caused the student participants to gain a better understanding of the needs and intricacies of the populations that they were serving. Students also indicated that participation increased their interest in careers in the field and caused them to learn relevant terminology and time management skills. They also indicated that they gained an appreciation of the duties of a professional job in the field and assessed themselves as "better prepared" for workforce entry or to pursue graduate studies. Bauerle and Park (2012) conducted work that demonstrated the value of an experiential exercise in increasing knowledge retention. They found that students who participated in the experiential exercise increased homework scores by 12% and the scores of those who fully participated (including participating in a tree climbing exercise) increased by 19%. The impact of this work was the greatest for students outside of the STEM disciplines participating in the STEM plant science course. Edwards, Jones, Wapstra, and Richardson (2012) demonstrated that experiential learning techniques increased student engagement. In response to declining honors level (an optional 4th year program for undergraduate students common to many institutions outside of the United States) enrollment, they incorporated experiential elements into all 3 years of their base undergraduate program. While demonstrating correlation only anecdotally (and failing to account for Hawthorne effect attributable changes, though the magnitude of the Hawthorne effect is an open research question; Cook, 1967;Jones, 1992;McCarney et al., 2007), the presented data suggests that the experiential inclusions stemmed the significant decline in enrollment experienced in 2002-2006, with 2007 enrollment at nearly three times the 2006 levels and 30% to 40% higher than enrollment in -2005. Dym, Gilkeson, and Phillips (2012 describe the role of experiential elements in the Harvey Mudd College Engineering program. They show a significantly significant (at p < .05) positive difference in the performance of Harvey Mudd College students as compared with students at 30 other engineering schools. Perhaps more significantly, they demonstrate the efficacy of and ability of first-year students to effectively participate in design projects.

OpenOrbiter Program
The OpenOrbiter program is a student-conceptualized, student-led program that aims to provide opportunities for student research and experience related to spacecraft design and engineering, software design and practical experience in many supporting areas. Student team members defined the project's scope, starting from a loosely defined concept of building a CubeSat, as well as specific program objectives, work areas, and timelines. The name, logo, and other project branding elements were also developed by student participants. The following sections provide an overview of the program and its progress, highlight key learning objectives, and discuss undergraduate participation to date.

Overview
The OpenOrbiter initiative aims to create design materials for the Open Prototype for Educational NanoSats (OPEN) concept and to develop a 1-U (10 cm × 10 cm × 11 cm, 1.33 kg) CubeSat-class spacecraft based on these designs. OPEN is poised to have a positive impact on aerospace engineering, mission critical system software development, and other fields through making a complete set of CubeSat designs, fabrication instructions, testing plans, and other materials freely available. The OPEN designs target a materials cost of no more than US$5,000 Straub,Korvald et al., 2013). This places the cost of the spacecraft at a level that could, in many institutions, potentially be supported by teaching or institutional funds (as opposed to requiring extramural funding). This reduced cost level also acts to enable research projects that may not be able to attain the levels of funding required for more expensive approaches. The reduced cost level also decreases the impact of failure, allowing more freedom to take risks and allow, in the case of educational projects, student leadership and decision making.
The OPEN design is different from traditional CubeSats in that it utilizes vertical insertion of the printed circuit boards instead of physical stacking of horizontal boards. Each of the four sides of the spacecraft, shown in Figure 1, is comprised of a board, which is held in place by corner posts with a retaining track. Electronic connectors are included in both the top and bottom plates, which allow electrical stacking of the boards without requiring physical stacking. This configuration also makes it very apparent if a board is not completely or properly seated, as the top plate cannot be locked in place.
The software that will accompany the OPEN design will run on top of a customized Linux kernel. It has been separated into three primary development efforts: operating software, payload software, and ground station software. A verification and validation group assesses the software created by the other groups to ensure flight-readiness. The operating software controls the moment-by-moment operations of the spacecraft, commanding all sensors, actuators, and subsystems. The payload software plans payload objective performance tasks and processes the data collected during these tasks. The ground station software communicates with the onboard operating software to convey controller instructions in the form of new tasks, task cancellations, and task modifications.

Learning Objectives
A number of learning objectives were set at the beginning of the OpenOrbiter program. These objectives were identified based on a combination of the identification of areas where traditional curriculum was lacking and prospective learning benefits that could be conveyed by a small spacecraft program. These objectives fall into several large categories: technical skills, communications skills, teamwork skills, spacecraft design skills, and time/project management skills. The project also sought to increase student excitement about their field of technical participation and about space in general.
The technical skills category is defined as being comprised of all elements of the team's work which are not spacecraft-specific. These categories should have a loose correlation with a subfield of an academic discipline, based on how teams were divided. Some teams' work covered a few related subfields; in a few limited cases, teams were themselves interdisciplinary due to the nature of the work they were performing.
The communications skills category covered both workplace communications and presentation skills. Communications skills were deemed to be an important focus, as they are enumerated as a required component of various discipline-specific accreditation programs. The lack of interdisciplinary communication skills by graduates was also identified as a prospective problem (as employers would be required, in the absence of its correction in academia, to bear the cost of this reduced productivity and training). Learning related to these interdisciplinary communications skills was deemed to be a benefit that could only be produced by a project, such as OpenOrbiter, with significant interdisciplinary participation.
The category of teamwork skills was comprised of the skills required to participate effectively in a large team. Unlike many class projects where students self-select a group of peers with whom (in many cases) they may already be friends with, OpenOrbiter placed students together based on their thematic interest. While, certainly, many students knew one another, the broad promotion of the project campus-wide resulted in many groups being composed of collections of individuals who were not previously well acquainted. This included the pairing of undergraduate and graduate students, individuals from different disciplines and across multiple year levels. For this reason, the project was an exercise in teamwork skills closely resembling the workplace, where one may be required to work with individuals not previously known or liked by them.
Spacecraft design skills were comprised of spacecraft design-specific technical skills. These included skills and the associated knowledge about the spacecraft design process as well as knowledge and abilities related to designing for the harsh and different environment of space. The validation   designs and their implementation for the space environment was also a critical element of this category. Time and project management were identified as important skills that were also lacking in other areas of the traditional curriculum. While students might have an appreciation for managing their own efforts (though for some, this may be the cram-at-the-last-moment mentality), working with others in a large interdependent project requires a significantly more robust skillset. It necessitates an understanding of what areas represent dependencies for others (and which do not) to facilitate decision making and prioritization in a time-resource-constrained environment. Team and group leaders, in addition to managing their own time relative to project, academic and other commitments, also had to learn the skills required to manage the efforts of others.
While all of these areas were deemed to be important (and covered by the program in some way), assessment during this initial period was limited to a subset. Future work will focus on assessing additional learning objectives.

Undergraduate Participation
Undergraduate participation is ubiquitous throughout the program. Undergraduates have participated in every team and have served as team leaders for several teams. One undergraduate served as a group leader (leading other undergraduate and one graduate student team leaders). Undergraduate participation has included individuals who volunteered, one individual who was funded to work on the program through a competitive internal (to the university) undergraduate researcher support program, and individuals who have participated as part of a class project or independent study. There is at least one instance of an individual who started working on the project as an undergraduate continuing to participate as a graduate student; more transitions of this type appear imminent.
Undergraduate participants have expressed several general classes of reasons for participating. Some partici-pate because the project and the chance to launch something into space at the end excite them. Many participate to improve technical skills in a particular area or to learn a new technical skill. Others have indicated that their reason for participating is to gain experience in working on a team project that is much larger than anything they have been exposed to in classes. Still others are participating to satisfy a specific degree or a course objective.
The undergraduates who have participated have expressed general pleasure with the results so far. Anecdotally, several examples of the program being discussed in an interview (and helping the participant secure an internship or position) have been mentioned to the authors. The following sections present a more formal assessment of program performance in undergraduate students.

Results
An assessment survey was distributed to student participants in the OpenOrbiter program in April 2013 during regular team meetings. Both graduate and undergraduate students responded to this survey; however, only the responses of undergraduates are presented and analyzed in this article. These students spanned all four undergraduate class levels (see Table 1) and three disciplines (see Table 2). They ranged in grade point average (GPA) from 3.0 to 4.0 (see Table 3). The group surveyed included both individuals who participated in a team lead role and who participated as team members (see Table 4). The average amount of time spent by participants was also collected; this is presented in Table 5.

Quantitative Results
Improvement in five of the key educational goal areas was assessed. Reported status by undergraduate students prior to and after program participation is presented in Figure 2. As shown in this figure, undergraduates reported improvement in all five areas. The most significant growth was reported in spacecraft design skills; the second most was reported in focus-specific technical skill growth. Figure 3 depicts the average level of improvement, by category. Figure 4 shows the average level of improvement enjoyed by those who showed improvement in a given category. Note that, for the purposes of calculation, three anomalous prior/post score combinations (indicating an effective decline) have been removed. These data points appear to be clerical errors, as the related attribution levels reported were not negative (as one would expect if an actual decline had occurred). Student respondents were then asked to characterize the program's impact on creating the changes described. They were asked to rate, on a 9-point scale (ranging from 1 = strongly disagree to 9 = strongly agree), whether they agreed with statements indicating the project had improved the skill category in question. Technical skill improvement is attributed to this with a 6.9 average response (just below the 7-agree mark); improvement in space interest received a 6.3 response (also near the agree threshold). The response with regard to presentation skills was less positive: a 4.8 response just below the no-preference level. Figure 5 shows the distribution of responses for each category. For technical skills, 84.6% of responses were in the positive (program was impactful) range. For space interest, 76.9% were in the positive (impactful) range. For presentation skills, only 15.4% were in the positive (impactful) range, whereas 53.8% were in the indifferent category and 30.8% were in the negative (non-impactful) range.   The impact of serving as a team lead was also considered. Four team leads were included among the 13 respondents. All four team leads reported spending between 4 and 7.99 hr per week on the project; only one non-lead participant spent this much time (all of the rest spent between 1 and 3.99 hr per week on the project). The average duration of involvement in the project, for team leads, was slightly longer (0.8 year as opposed to 0.7 year), the average GPA was marginally lower. Team leads were either juniors or seniors (while non-lead participants spanned all four undergraduate years).
Team leads, as shown in Figure 6, attributed technical skill gains to project participation to a greater extent (7.3 vs. 6.8 average) than non-lead participants. They attributed increase in space excitement to the project significantly less (5 vs. 7 average) and improvement in presentation skills marginally less. As shown in Figure 7, they significantly outperformed non-lead participants in all five categories. The most significant outperformance was in improvement in spacecraft design (1.0 greater average improvement), followed by presentation comfort (0.78 average improvement) and technical skills (0.75 average improvement). They outperformed in excitement about space by 0.4 and in presentation skills by 0.3. The level of improvement, for those showing improvement, also was greater for team leads in each category (as shown in Figure 7).
In addition to showing greater improvement across the board, team leads had a greater percentage of respondents showing improvement in four of the five categories (technical skills, design skills, presentation skills, and presentation comfort). Non-lead participants reported a greater number of individuals showing improvement in space excitement. This is depicted in Figure 8.

Qualitative Results
In addition to program assessment conducted through categorical and scale-response questions, an open-ended question was also provided to allow respondents to highlight other areas of value to them. Of the 13 undergraduate respondents, 6 included comments in response to this question. The question was phrased as follows: Please share with us: (a) any areas where you believe the project may have provided you with particular benefit and/or (b) any comments on any of the above questions and/or (c) any other areas of benefit that you enjoyed that were not discussed.
The first highlighted leadership skills, presumably as an area of particular benefit or an area not discussed. The second indicated that the largest benefit that they received was involvement in a large project and the opportunity that this provided for them gaining experience working in teams. This was mirrored by another respondent who also indicated a benefit from group work. The fourth indicated that the project was useful in "introducing" the respondent to "groupbased computer science work"; they also benefited from learning about validation and testing activities. The fifth indicated that the project had "opened" his or her "eyes" to   the diversity of computer science applications. This individual also commented on his or her ability to gain experience in a particular technical skill that he or she otherwise would not have had an opportunity to learn. Finally, the sixth respondent indicated that the program had allowed the direct application of material learned in his or her classes and also expanded his or her knowledge in both these related and other areas.

Analysis of Results
The results presented in the preceding section indicate that the majority of the surveyed participants were upper-level undergraduates with good GPAs (3.0 and higher). Most spent less than 4 hr per week on the project, whereas a few spent between 4 and 8 hr per week on the project. Roughly a third of those surveyed were team leads (there is a high level of correlation between the 4-8 category and being a team lead). Respondents were largely computer science majors (approximately 75%), though electrical engineering and the non-STEM entrepreneurship major were also represented. This distribution was not representative of overall participation in the project. As the students self-selected for participation and the project occurred largely at a single institution, the ability to generalize these results to other projects and other institutions is limited. This being said, the results serve to demonstrate initial successes that serve to justify future work and assessment. The results demonstrate improvement across all measured categories (and anecdotally, based on free-response comments, across some categories not specifically assessed). This improvement was particularly centered in the technical skills and spacecraft design categories, with an average improvement of 20% of the scale in these two categories. Strong attribution (nearly 7-agree) also existed for the technical skill category. Space excitement and presentation skills and comfort also showed improvement (between 10% and 15% of the scale range); however, the attribution for the presentation skills category was not strong. The level of benefit enjoyed by team leads was shown to be significantly greater than for non-lead participants: In two categories, the average level of improvement shown for those who improved was double (or greater than double) that of non-lead participants. In other categories, it was also significantly greater.

Conclusions and Future Work
This article has demonstrated the efficacy of using a small spacecraft development program to facilitate undergraduate education, for a limited number of undergraduate student participants in the OpenOrbiter program. The results presented cover the first academic year of OpenOrbiter operations, following a thematically related predecessor program (which operated for approximately 6 months). It has been shown that participation in this program is generally effective and that team leads enjoy greater levels of benefit than non-lead participants. The extrapolation to other small spacecraft programs is limited due to programmatic differences and the small number of individuals sampled in this work. Future work will include the completion of the OPEN designs and their implementation. Assessment activities are planned to continue (and be augmented) during this time and be extended to assess the benefits to students in multiple small spacecraft development programs.