Documents

The development and use of TEL should be well integrated into the university’s overall vision and educational goals. Within this context, the university should create an open and stimulating environment that allows faculty to pursue and iteratively refine new ideas involving TEL. Universities need to support the use of data for decision making, both at the instructional level and also at the level of program assessment. Current practices surrounding program evaluation and assessment are already in place at many institutions and can be built upon to this end. Part of building this culture also involves bringing ideas about research-based teaching and TEL into the community through collegial conversations, workshops, and other events (Austin, 2011).

Collectively, faculty may recognize the value of research-based teaching and TEL design, but they decide individually whether or not to change a course or develop a new TEL resource. The calculus of these individual decisions naturally favors the status quo over paying a substantial price (mostly in time and opportunity cost) to innovate and/or to apply a scientific approach to education. We need to adjust the incentives for faculty to engage in effective, iteratively improved TEL design. For example, providing financial incentives (e.g., in the form of internal grants) and then highlighting positive examples are methods that have been successful. In the longer-term, we need to foster and promote a culture that pursues, uses, and values rigorous educational research. For faculty who pursue and build a strong portfolio of such research, this must be recognized as part of the evaluation of their teaching performance and academic achievement.

Faculty members are subject matter experts with the disciplinary knowledge that forms the substance of instruction. They are also the ones with direct experience teaching students this substance. Nevertheless, beyond their expertise and experience, there is an extensive literature of research-based principles and results that can greatly enhance teaching and learning, especially TEL. We need to provide faculty with resources and tools for professional enrichment so they can draw on this literature and put it into practice in their own teaching contexts.

In addition, we need to provide faculty with support — in the form of pedagogical/ technological consultants who can distill relevant results from the literature and translate them to the current TEL design, and production specialists who can help build TEL resources that are collaborative designed (e.g., by a team of multiple faculty members, pedagogical consultants and technologists). Finally, to get to an ideal end-state where faculty and graduate students are comfortable engaging in scholarship and research around teaching and learning, we need to provide opportunities for professional development and exploration in this area.

To address the challenges described above and to consistently produce demonstrably effective TEL resources, we need a systematic approach. We label this recommendation deliberate instruction both to emphasize the analysis and planning that is critical to effective instructional design and to capture the key features of “deliberate practice” (Ericsson et al., 1993). The components of deliberate instruction are:

• Cognitive task analysis: deconstructing a complex task into the knowledge and skills that are necessary for good performance (Clark et al., 2006). Cognitive task analysis helps avoid the expert blind spot (Koedinger and Nathan, 2004) and identifies the required knowledge and skills that instruction must address.
• Instructional alignment: ensuring that instructional activities and assessments are well aligned with clearly articulated learning objectives. When these elements are aligned, assessments provide valid measurement of student outcomes and instructional activities support students in performing well on assessments and achieving learning objectives.
• Repeated practice: creating sufficient opportunities for learners to practice the knowledge and skills they need to learn. Because robust learning does not occur in one trial, repeated practice is critical for developing mastery (especially when combined with feedback)
• Targeted feedback: providing learners with targeted and timely feedback on their performance. Such feedback is critical for learners to refine their approach, correct misunderstandings, and deepen their understanding.

When these components are jointly put in place, the result is instruction that supports learners to achieve the desired outcomes.

Applying a deliberate instruction approach and incorporating research results to the design of TEL resources already can lead to significant improvements in student learning outcomes, but why stop there in applying a scientific approach to teaching and learning?

Especially when TEL resources are designed with repeated practice and targeted feedback, the data from students’ interactions with the material provide a rich dataset on student learning. Such data can drive multiple productive feedback loops (see accompanying figure). Universities need to build the infrastructure for collecting, analyzing, and visualizing such data so that instructors and students can get a window onto the learning process and adapt their teaching and learning accordingly. Data collected from students using a TEL resource can also guide ongoing improvements to the resource’s design, thus ratcheting up its effectiveness over time. Moreover, these data and analyses can be fed back into the TEL system to enable various options for personalizing the learning experience.

Enacting this recommendation involves developing new and better tools for:

• Instrumenting TEL resources for easy, compatible, and secure data collection
• Analyzing the data to provide meaningful results to a variety of audiences, and
• Preparing and supporting various participants (e.g., teachers, students, administrators, course designers) to understand and use the results effectively.

Today, the main emphasis of faculty development is on pedagogy and effective classroom delivery and engagement. Moving forward, we need to shift the focus to a building a stronger understanding of how to apply learning sciences, how to use and develop TEL resources, and how to use data to iteratively improve teaching and learning. A team of instructional design, educational technology, and faculty-development specialists can consult with faculty on the application of learning science to instructional and TEL design, help create TEL resources, educate faculty and graduate students about best practices, and support faculty in carrying out all the steps of deliberate instruction.

Research suggests that a key factor in faculty members’ evaluation — and ultimately adoption — of research-based educational innovations is the compatibility with their teaching context (Montfort, Brown, & Pegg, 2012). Unfortunately, in higher education, teaching tends to be a ‘solo sport’ in which individual faculty members know relatively little about their colleagues’ teaching practices or experiences (cf. Herbert Simon). An online repository will offset this deficit by providing faculty members access to TEL resources, strategies, videos of their use in action, and peers’ assessments of these artifacts based on actual experiences. This repository will enable faculty and administrators at a variety of institutions to directly use these materials and contribute back to the collection.

The Simon DataLab offers one model for a collaborative data repository that supports primary and secondary research. The central repository portal serves as an access point and links to other resources that allow data owners and administrators have the ability to determine who else is granted access as well as what privileges. Visitors can see all available data sets and are be able to request access to others. The repository administrators can ensure that data access is granted according to the project IRB.

Data infrastructures will need to generalize so that they are interoperable across existing learning science or education data repository efforts such as:

• DataShop (http://learnlab.org/datashop)
• Stanford Vice Provost for Teaching and Learning (VPTL) (http://vpol.stanford.edu/research)
• TalkBank (MacWhinney, 2007; http://talkbank.org)
• Databrary (http://databrary.org)
• Education data at data.gov (http://data.gov/education)
• Large-scale standardized assessment data at the National Center for Education Statistics, such as National Assessment of Educational Progress (NAEP) data (http://nces.ed.gov)
• Student outcome data used in San Pedro et al., 2013 (http://www.studentclearinghouse.org)

Attracting researchers to perform secondary analysis on data sets will likely draw from the repository’s existing user base as well as publications that cite the repository’s data. This large community of researchers performing secondary analysis would be interested in the data set once it is available. Creating or joining a collaborative research community devoted to improving learning outcomes through empirical research will aid in facilitating the design, improvement, comparison and dissemination of effective learning metrics, feedback mechanisms and best practices necessary to fuel a global learning revolution. As part of these communities, we advocate bringing together the expertise and resources of university, industry and government stakeholders to collect and store hundreds of high quality data sets and accumulate the best analytic methods available for educational research.

Faculty, curriculum developers, instructional designers and learning researchers should be encouraged to contribute so that all can benefit from their experiences and research. Learning scientists should be encouraged to test hypotheses and new analysis methods on hundreds of datasets contributed by others. Creating a data commons is a critical step in the success of the interdisciplinary research community and provides a unifying network for collaboration.

Different kinds of educational data come with a range of sensitivity and risks to student privacy. An example of low-risk data is simple mouse and keyboard interactions with an educational technology (so-called “click stream” data) where no student identifying information, such as name, address or demographics, is included and where there are no links to other datasets. At the other extreme, classroom or student video data that includes student identifying information is of high risk. Corresponding with the risk are different levels of data security and access. High storage security and limited access, for example, to only a small group of researchers with Institutional Review Board approval is appropriate for highly sensitive data. In contrast, for low-risk data more open access is reasonable. Thus, a one-size-fits-all policy is not appropriate. Instead, we recommend defining commonly agreed upon protections for each pairing of level of data sensitivity and sharing sensitivity. Furthermore, perceptions of risk and sensitivity may vary according to global cultural contexts and associated privacy laws will vary from country to country. Accordingly, we recommend that a global, cross-sector task force be commissioned with the goal of specifying practices that relevant legal bodies (university IRBs, legislative bodies crafting laws such as FERPA) can adopt to protect individuals’ privacy.

Specific practices and procedures should take into account well established “best practices” and principles for privacy preservation. For example, according to Fung et al. (2010), a privacy threat occurs when it is possible to link a record owner to: a specific record in a publicly available data a sensitive attribute in the data such as age or address, or the publicly available data itself. To avoid the creation of a privacy threat, Fung further suggests that the publicly available data should not provide additional information beyond what is already publicly known. Practices for collecting and archiving data should be informed by the principles and practices outlined in foundational conventions and codes such as The Asilomar Convention for Learning Research in Higher Education, the 1973 Code of Fair Information Practices, and Belmont Report of 1979.