By: Associate Professor Duan Zhang, Research Methods and Information Science, Assistant Professor Erin Anderson, Educational Leadership and Policy Studies, and Assistant Professor Garrett Roberts, Teaching and Learning Sciences

This project was intended to evaluate the efficacy of a digital badging project in Aurora Public Schools (APS). APS is the fifth largest district in the state of Colorado and serves a diverse student population that is 55.3% Hispanic, 19% Black, 15.1% White, and 4.1% Two or More Races. The students in APS come from over 130 different countries and speak over 160 languages (39% English Language Learners). Many of the students in the district qualify for free and reduced lunch (68%).

The district has implemented a number of initiatives to increase the graduation rate of the diverse student body. The 2015-16 four-year graduation rate was 65%, which is below the national average of 82% (http://nces.ed.gov/). The Postsecondary and Workforce Readiness division of APS developed the APS Digital Badge Program to support district-wide goals for (a) every student to have a plan for their future, (b) every student to have a set of skills to implement that plan, and (c) every student to have the credentials that open doors (http://aurorak12.org/about-aps/fast-facts/). To address college and career readiness, the district is emphasizing 21st Century skills. Students can earn badges in these skills or credentials from five major categories: (a) collaboration, (b) critical thinking, (c) information literacy, (d) invention, and (e) self-direction. Once a student earns all of the badges in a category, local businesses provide the students with experiential learning related to the skill. Students are given the opportunity to earn badges in each classroom in all middle and high schools in APS.

The APS Digital Badge Program is an essential component to the APS strategic plan and the work of the Postsecondary and Workforce Readiness department; however, despite the promise this program holds and the interest of media outlets such as Education Week, the district has not been able to measure the success of this innovative credentialing program. The district reached out to faculty in the Morgridge College of Education for support in evaluating the success of the program. The goal of this Public Good Grant-funded project was to assist Aurora Public Schools in understanding the impact of their digital badging initiative on study outcomes, such as student achievement, as measured by Colorado Measures of Academic Success (CMAS) testing in the middle school and GPA, PSAT, and SAT data in the high school; student graduation rates; and student discipline data.

The Badge implementation data from 2015-2018 showed that among the 80,891 APS students who were in Grade 6-12 during these years, 15.9% of them participated in the Badge program through receiving one or more digital badges. In order to evaluate the efficacy of their participation in an equitable way, propensity matching analysis was conducted to match each participating student with five non-participating students (so as to maximize effective sample size) with similar demographic backgrounds. The analysis ended up creating a control group (those students who did not receive badges, n =64,400) and a treatment group (those students who did receive one or more badges, primarily in middle and/ or high school, n = 12,880). Our hypothesis was that those students who earned one or more digital badges would have more positive student outcomes than those who did not earn badges. A series of subsequent statistical analyses were conducted to explore if the two groups differed on a variety of student outcomes such as standardized test scores, attendance rates, and discipline referrals.

Although we were not able to find a statistically significant difference between the two groups on these student outcomes, we were able to make recommendations regarding the future collection of data, the frequency and distribution of distributed badges, and systematic structures that could support the APS Digital Badge Program moving forward. Regarding project implementation, we suggest that the Badge team shall consider more systematic top-down strategies to market this valuable project to more middle and high school principals with potential teacher incentives for participation. In addition, to evaluate Digital Badge efficacy in a more holistic way, we recommend that school engagement and student social-emotional measures be incorporated as critical outcomes together with parent and teacher focus group interviews about student development dynamics.