A group of instructional designers at Ecampus participated in a book club reading “Ungrading” (Kohn & Blum, 2020). We learned many creative ways of designing assessments through participation in this book club. If you happen to be searching for ideas on designing or re-designing assessments in your teaching, we would highly recommend this book!
The idea of “Ungrading” may sound radical to many of us. Yet instructors at all types of educational institutions have tried ungrading in many different courses, ranging from humanity courses, to STEM courses, and from primary education to higher education. Starr Sackstein (author of Chapter 4 “Shifting the Grading Mindset” of the book) encourages educators to consider “ways to adjust small things in the classroom that will lead to important growth for students”. And this suggestion of starting small is coherent with what James Lang proposes in his book “Small Teaching” (Lang, 2016) and Thomas Tobin’s +1 strategy for implementing new teaching and learning strategies (Tobin & Behling, 2018). Sackstein provides a table comparing the grades vocabulary that focuses on judgement or criticism, with the non-grade vocabulary focusing on assessing and opportunity for improvement.
In chapter 5, Arthur Chiaravalli proposed a way for teaching without grades: Descriptive Grading Criteria, such as A for outstanding, B for Good, C for Satisfactory and I for Incomplete. Do you remember elementary school report cards that use E for Excellent, S for Satisfactory, and NI for Need Improvement type of categories? I think that is exactly what descriptive grading criteria represent.
In chapter 7, Christina Katopodis and Cathy Davidson offer a new approach to start a new term/semester by asking students:” What is Success in this class for you? And How can I help you achieve it?” (p. 107) Katopodis and Davidson also remind us the importance of explaining why when you challenge your students to take their own learning seriously and give students opportunities for metacognitive reflections about the learning activities themselves. Katopodis and Davidson also offer a model of contract grading for Twenty-First Century Literacies and a model of collaborative peer evaluation. Students’ grades in the course come from self-and-peer evaluations using detailed evaluation forms.
In chapter 8, Christopher Riesbeck described his critique-driven learning and assessment design of do-review-redo submission process for his intermediate-level programming course. I have used similar approach in my own teaching before and it works very well for any course with manageable number of students. The advantage for this approach is every one of your students can improve their first submissions based on feedback they receive from the instructor. The disadvantage for this approach is the potentially extended time instructors may spend on providing the feedback and reviewing the submissions and re-submissions. The key to this assessment method is making sure that the workload of providing feedback and reviewing revisions is manageable. In chapter 9, Clarissa Sorensen-Unruh provided her experience of using ungrading in her organic chemistry II course, giving students opportunities to practice evaluating their own work.
And that is only snippets of what I took away from a few chapters from this book. Many resources about ungrading outside the book were shared during our book club meetings, such as two-stage exams, group exams and public exams. To answer a common question that ungrading practices may fit humanity courses more easily, Cyndie McCarley shared “Grading for Growth” blog written and maintained by two math instructors Robert Talbert and David Clark. To learn about all the creative assessment design methods introduced in this book, read it yourself either through library ebook or get a hard copy and enjoy reading, designing and experimenting!
References
Kohn, A. and Blum, S. (2020). Ungrading. West Virginia University Press.
Lang, J. (2016). Small Teaching. Jossey-Bass.
Tobin, T.J. and Behling, K.T. (2018). Reach Everyone, Teach Everyone. West Virginia University Press.
As an instructional designer one of my most important tasks is to hold an initial meetingwith an instructor, or subject matter expert (SME). During that first opportunity to collaborate on an online course design project the conversation typically focuses onkey course design elements. Commonly we discuss the broad approach to the course, anticipated assignments, narrated lecture needs, assessment, video production and support, the mechanics of the development process and more. Rarely does art come up in the conversation.
Chasing Images
Now, that may seem obvious why art is not outwardly discussed. However in the first meeting we do address the topic of look and feel for the course, and images for course banners and any special subject matter art that may need to be displayed. So, visual images are addressed as a way of supporting the learning focus for the course. Finding images that are copyright free and support the general course topic are fairly easy to find in the more obvious online image repositories like Unsplash.com, Pixabay.com, and Pexels.com.
For courses with education, agriculture, forestry, medicine, and other more general themes finding usable images is pretty straight forward. If your institution also owns licensing for GettyImages.com, the largest stock image collection in the world, you have really nice access to images. Even so, sometimes we don’t have access to the right images for the right course.
In those cases I have found myself chasing down images or building visual metaphors using multiple images to create one image that communicates the theme, unique message, or topic of a given course. Scrambling to create images that support a course can be a challenge for instructional designers. We can do it, but it is not ideal.
For those courses where stock images are not helpful there is a need to re-imagine the image sourcing process. Key to this is recognizing two important factors that will influence your new thinking; 1)The need to find images that do not require copyright clearance or purchase and 2) the need to have images that reflect the unique theme required for the course. How might this be done?
Asking the Question
Sometimes the answer to finding the right image for a course is not about the image but the artist who made it. Every college community has dozens of artist who produce many pieces of art. Some might just be what you need for your course. If you found the right artist would you be willing to invite them to contribute their artistic creations to your course? Would you ask an artist to be part of your course design?
That question came up while working with a course instructor, Dr. Mark Edwards, on the development of his sociology course addressing welfare and social services in 2019. In addressing this issue we knew finding stock art for this topic would be difficult and perhaps inappropriate. During our discussion the instructor said he had seen some examples online of street art addressing homelessness that he though might fit the course. I asked him if he would be willing to reach out to the artist and ask him if we could get permission to use his art in the course. Dr. Edwards said yes.
To our surprise and delight the artist responded positively by granting written permission to use photographs of his art in the course (see Figure 1. below). The artist participated knowing the art would be used for this particular course and would be behind a password protected site in Canvas. We also agreed to credit his work and link back to his art source page.
Course Artist-in Residence
Asking an artist to share their work in this way may be a challenging ask on both ends of the question. We did not know what the outcome would be. Some artists vigorously and rightly protect their work. Some correctly want compensation. Some artists see their work differently and are willing to share their creative efforts in the right circumstance or to support the right cause. In this case, I felt it was best for the instructor to make the ask as he could best communicate how the art might dovetail with the outcomes of the course. So, in this sociology course example the ask worked.
In the summer of 2021 I collaborated again with Dr. Edwards. His new course was a graduate course on research methods in the public policy program. At our first meeting he shared that he had already been in discussion with a colleague in Equador who was a chef and artist. The instructor thought about using images from a series of bread baking images from the chef /photographer for the course. The course theme was organized around the analogy of baking bread as a way to explain the art and science of social research methods. We would use the images (see Figure 2. below) to spur and reinforce understanding about what students were learning.
With the development of this second course the concept of a course artist-in-residence became intentional. There are a number of benefits and also limits that come along with this approach to image sourcing and use (see Figure 3.).
Traditionally artist-in residence (AIR) programs are a way for artists to find a place and unfettered time to extend their creative efforts. There are hundreds of such art programs throughout the country and many can be found listed at the Artist Communities Alliance. A wide variety of artists are served by these program.
The Course AIR described here is quite different from traditional AIR programs. Where an AIR participant might create a piece of art as part of their AIR experience the Course AIR is intentionally sharing artistic creations within a course space to help foster the objectives of the course. Depending upon the level of integration of the art with the course activities the role of the Course AIR may vary from contributor to collaborator.
Final Thoughts
The role of an instructional designer in this process is to first assist the instructor in vetting art appropriate for the course. Secondly, the instructional designer helps shape the application or use of the art in the course then designs course pages to showcase and compliment the artist-course connection.
The Course AIR concept has been a happy accident that holds interesting potential. Art comes in various form and is experienced through various senses. What music, photographs, video, illustrations, or other types of art might be just right for your next online course?
Would a Course Artis-in-Residence contribute to achieving the course learning outcomes for a course you are developing? If our experience is any type of guide you will never know unless you ask the question and invite art into the online course design.
Recognition
A special thanks to Dr. Mark Edwards for his work on the courses, willingness to make the ask, and support for this post. Thank you also to my colleague Chris Lindberg for his contributions to this post.
At a recent faculty professional development workshop series, I became aware of faculty’s concerns about addressing the learning needs of students from different cultural and linguistic backgrounds. Students with multilingual and multicultural identities are oftentimes perceived as deficient (Almon, 2014; Flores & Rosa, 2015) and might feel they hold an outsider status (Merryfield, 2000). In my personal experience navigating multiple identities that intersect culture and language, and in my work supporting faculty in their learning design and instructional decisions, I began examining ways in which blended and online learning spaces can offer more welcoming opportunities for students. One of these ways is using a cultural lens and mindset towards inclusive learning design.
Culturally Responsive Approaches
There have been several culturally responsive approaches to teaching and learning. By and large these approaches advocate for the recognition of students’ cultural backgrounds as critical to their learning success (Gay, 2013; Ladson-Bilings, 1994). In fact, a culturally responsive pedagogy (CRP) aims to promote the integration of students’ culture to support their learning experiences. In blended and online learning, this pedagogy can create an environment that “acknowledges, celebrates, and builds upon the cultural capital that learners and teachers bring to the online classroom” (Woodley, Hernandez, Parra, & Nagash, 2017, p.1).
For students whose first language is not English, we first should focus on their strengths and not their deficiencies. These students bring their cultural backgrounds, values, experiences, and language diversity with them to the online learning environment. It is important to recognize that culture is central to teaching and learning; therefore, advancing online and blended learning design should be grounded on dimensions for cultural sensitivity where students’ diverse identities, cultures, languages, and backgrounds are seen through an asset-based lens. This means, recognizing the value in the cultural backgrounds, experiences, and languages of students; and embracing these students’ traits as assets. This asset-base approach can be the first step in developing a mindset for designing and teaching in ways that promote social, academic, and emotional learning for these and ALL students.
Culturally responsive approaches have been mapped out to the three principles of Universal Design for Learning to offer (1) multiple means of engagement, (2) multiple means of representation, and (3) multiple means of action and expression (UDL, n.d.). UDL and CRP can help instructors amplify the opportunities for students from different cultural backgrounds to demonstrate their knowledge when given strategies that incorporate multiple perspectives, experiences, connections to the real world, and choices (Bass & Lawrence-Riddell, 2020; Kieran & Anderson, 2018)
The connection of UDL and CRP offers consideration to inform instructional design choices. Yet, these considerations appear to be adds-on to the design of the learning experience. How can we expand the UDL and CRP connection to embrace a mindset to move towards an inclusive learning design where the cultural and linguistic traits of students are seen from an asset-based perspective? A few dimensions from research and praxis would get us started to help achieve this goal.
Dimensions for Learning Design
The following dimensions for learning design, that expand the connections between UDL and CRP, should be considered whenever possible in the design of blended or online learning experiences. Following are the six dimensions.
Instructor’s reflection
Visual design
Linguistic domain
Content
Interaction
Technology
Identity and Experience
Instructor identity and experience relates to the practice reflection –inner analysis to reveal assumptions about teaching and learning (Jaramillo Cherrez & Jin, 2020). Through these reflections, instructors can identify ways to humanize the learning experience. Instructional designers (IDs) can help instructors engage in a (self) dialogue to explore how the instructor’s identity informs or impacts their teaching and instructional decisions, how they respond to students’ cultural differences and embrace them as strengths, and how instructors could also learn from students.
Design
The visual design of the course and learning materials can have a profound impact on students’ learning experience (Hedberg & Brown, 2002). For students whose first language is other than English in particular, it is important to be aware that these students may come from different cultures and social groups, and thus, visual representations may have positive or negative consequences to their success in the course. Visuals should be carefully chosen because the variety of images, colors, and symbols may affect the message students receive in the class. When using images from pop culture, it is helpful to add context to give more clarity to the instructional purpose. The visual design also relates to the readability of the content and how it is presented and structured. Asking a colleague or friend to read the instructions and descriptions of assignments can help clarify expectations and requirements for students. Bear in mind that what is clear to one is not always clear to others, especially when using complex sentences and terminology of a discipline.
Linguistic Variability
Many students might have a first language different from English. Also, keep in mind that different cultures may have different ways of writing, usually influenced by rhetorical and social contexts (Almuhailib, 2019). There may also be linguistic and cognitive differences in the way that students interpret the information given to them. For some cultures, direct descriptions are fine, whereas for others the context is important before addressing a specific perspective. Some cultures may characterize themselves for being more individualistic and others more holistic, and students, including those whose first language is other than English, can find themselves moving along that continuum. In designing culturally responsive blended or online courses, language matters because of the transactional distance characteristic of asynchronous spaces. Many students may already be pressured to demonstrate “good”, “academic”, “professional” English. One way to be aware of linguistic diversity is to be more explicit with instructions. For example, indicate clearly the use of naming conventions, abbreviations, acronyms, and descriptors in activities and assignments.
Content
The fourth consideration is content. The main suggestion is to try to diversify the curriculum with resources from around the world (e.g., content from scholars from diverse cultures and linguistic backgrounds). Allow students to see themselves represented in the materials. Create activities and assignments that help students explore the concepts in connection to their own backgrounds and communities(e.g., linguistic, cultural) and experiences, and that allow students to move from low to high cognitive tasks (e.g., staged projects). Yet, diversifying the curriculum goes far from bringing into the course content perspectives that are commonly ignored. It involves explicit acknowledgement of the value of the different perspectives and modes of knowledge.
Interaction
Interaction also can benefit from a culturally responsive mindset in that instructors can vary the modes of interaction by using audio/video communication(e.g., assignment feedback, DB, announcements). It is also important to guide and scaffold group activities with resources such as guidelines, group contracts, teamwork guidelines, group rapport activities, conflict resolution resources). Particularly for teamwork, instructors can build group activities early in the course to promote collaborative learning. For online discussions, instructors could allow students to select the tools that they feel more comfortable with using, bearing in mind that many students from different cultural backgrounds might not be familiar or have experience participating in discussion activities. Another suggestion is to promote student-led discussions to help students move from the individual task to the group task. This will allow to vary the cognitive demands that can foster meaningful knowledge construction and organization while also addressing different audiences, styles of writing and analysis, and communication modes.
Technology
It is important to recognize that technology is not neutral. Clearly, it is important to select tools and evaluate them for intended and unintended consequences for students, such as the cost, the technical support, the pedagogical affordances, and the availability in other geographic areas. It will be helpful to consider the different levels of technology skills that students may have and plan on developing guidelines and technical resources (e.g., links to providers, manuals, accessibility and privacy policies) that can help students. Additionally, in considering digital tools it is important to review whether the affordances the tools offer are available to all students, in the different browsers and devices (e.g., tablets, smart phones, browsers). In using digital tools, careful attention should be given to the kinds of data that the tools require students and instructors to share. It is important to read carefully the terms of use, data privacy, and the information that is being collected as a way to understand how the users can trust the tools and their procedures for sharing or not with others the data collected.
These dimensions underscore the need to approach learning design with a mindset that not only acknowledges student multilingual and multicultural identities, but also catalyzes these identities to help students be valued and successful. I consider these dimensions in my instructional design work, and I would like to invite you to consider them next time you design an online or blended learning experience.
Sources
Almon, C. (2015). College persistence and engagement in light of a mature English language learner (ELL) student’s voice. Community College Journal of Research and Practice, 39(5), 461-472.
Almuhailib, B. (2019). Analyzing Cross-Cultural Writing Differences using Contrastive Rhetoric: A Critical Review. Advances in Language and Literary Studies, 10(2), 102-106.
Bass, G., & Lawrence-Riddell, M. (2020). Culturally Responsive Teaching and UDL. Faculty Focus. https://www.facultyfocus.com/articles/equality-inclusion-and-diversity/culturally-responsive-teaching-and-udl/
Dougherty, E. (2012). Assignments matter: Making the connections that help students meet standards. Alexandria, VA: ASCD.
Kieran, L., & Anderson, C. (2019). Connecting universal design for learning with culturally responsive teaching. Education and Urban Society, 51(9), 1202-1216.
Gay, G. (2013). Teaching to and through cultural diversity. Curriculum Inquiry, 43(1), 48-70.
Hedberg, J. G., & Brown, I. (2002). Understanding cross-cultural meaning through visual media. Educational Media International, 39(1), 23-30.
Ladson-Billings, G. (1994). What we can learn from multicultural education research. Educational Leadership, 51(8), 22-26.
Merryfield, M. M. (2000). Why aren’t teachers being prepared to teach for diversity, equity, and global interconnectedness? A study of lived experiences in the making of multicultural and global educators. Teaching and Teacher Education, 16(4), 429-443.
Flores, N., & Rosa, J. (2015). Undoing appropriateness: Raciolinguistic ideologies and language diversity in education. Harvard Educational Review, 85(2), 149-171.
Memory plays the central role in learning – it is “the mechanism by which our teaching literally changes students’ minds and brains” (Miller, 2014, p. 88). Thus, understanding how memory works is important for both instructional designers and instructors. According to modern theories, memory involves three major processes: encoding (transforming information into memory representations), storage (the maintenance of these representations for a long time), and retrieval (the process of accessing the stored representations when we need them for some goal or task) (Miller, 2014). Let’s briefly review these processes and see how they may inform our course design and instruction.
Encoding – What Is the Role of Attention and Working Memory?
How does encoding happen? We receive information from our senses (visual, auditory, etc.), and then we perform a preconscious analysis to check whether it is important to survival and if it is related to our current goals. If it is, this information is retained and will be further processed and turned into mental representations. Thus, attention is the major process through which information enters our consciousness (MacKay, 1987). Attention is limited, and it is to some extent under voluntary control, but it can be easily disrupted by strong stimuli. Attention is crucial for memory, and without attention we cannot remember much (Miller, 2014).
How attention is directed depends on the way the content is presented and on the nature of the content itself (Richey et al., 2011). If the content is intrinsically motivating for the student, it will catch their attention more readily. But beyond that, the manner we design our instructional materials can influence how learners focus their attention to select and process the information, and in turn on what and how much gets stored in their memory. For example, we can ensure that students are guided to the most relevant content first by making that content more visually salient. Or we can tell an engaging story to focus their attention to the concepts that come next.
Working memory is a concept introduced in the 1970s by Alan Baddeley. This model describes immediate memory as a system of subcomponents, each of them processing specialized information such as sounds and visual-spatial information. This system also performs operations on this information and are managed by a mechanism called the central executive. The central executive combines the information from the various subcomponents, draws on information stored in long-term memory, and integrates new information with the old one (Baddeley, 1986).
Some researchers consider attention and working memory to be the same thing; while not everyone agrees, it is clear that they are highly interconnected and overlapping processes (Cowan, 2011; Engle, 2002). Attention is the process that decides what information stays in working memory and keeps it available for the current task. It is also involved in coordinating the working memory components and allocating resources based on needs and goals (Miller, 2014).
The capacity of each of the working memory components is limited. However, these components are mostly independent: visual information will interfere with other visual information, but not much with another type such as verbal information (Baddeley, 1986). Therefore, the most effective instructional materials will include a combination of media, such as images and text (or better yet, audio narration), rather than just images or just text. Graphic by Cheese360 at English Wikipedia is licensed under CC BY-SA 3.0
Storage – How Fast Do We Forget?
In the late 1800s, Hermann Ebbinghaus conducted his famous series of experiments on the shape of forgetting. The result was the forgetting curve (also called the retention curve), which is a function showing that the majority of forgetting takes place soon after learning, after which less information will be lost (Ebbinghaus, 1885). A recent review of studies on the retention curve concluded that the rate of forgetting may increase up to seven days, and slows down afterwards (Fisher & Radvansky, 2018). This interval is useful to consider when planning instruction. A well-designed course will include sufficient opportunities for practice and retrieval during this time, so as to minimize the forgetting that naturally occurs.
Graphic from MIT OpenCourseWare is licensed under CC BY-NC-SA 4.0
Retrieval – How Do We Get It Out of Our Heads and Use It?
While long-term memory is considered unlimited, retrieval (or recall) can be challenging. Its success depends on a few factors. To retrieve memory representations, we use cues—information that serves as a starting point. Since a memory can include different sensory aspects, information with rich sensory associations is usually remembered more easily (Miller, 2014). Visual and spatial cues are particularly powerful: memory athletes perform some mind-blowing feats by using a special technique called “the memory palace”—imagining a familiar building or town and placing all content inside it in visual form (to learn more about this technique, check out this TED talk by science writer Joshua Foer).
Recall is also influenced by how the information was first processed: deep processing (focusing on meaning) will yield superior retrieval performance compared to shallow processing (focusing on superficial features like some key words or the layout of the information). However, equally important is a match between the type of processing that happens during encoding and the one that happens during retrieval (Miller, 2014). For instance, if the final exam contains multiple-choice questions, learners will perform better if they also practiced with multiple-choice questions when they learn the content. Finally, emotions have been shown to boost memory (Kensinger, 2009), and even negative emotions (such as fear or anger) can have a strong effect on recall (Porter & Peace, 2007).
Conclusion – Implications for Instruction
What can we do to maximize our students’ memory potential? Based on these memory characteristics, here are a few strategies that can help:
Make use of graphic design and multimedia learning principles to create attention-grabbing, well-organized instructional materials that include a combination of media.
Include plenty of retrieval practice activities, such as polling during lectures, quizzes, or flashcards. The website Retrieval Practice is a fantastic resource for quick tips, detailed guides, and research. Top things to keep in mind:
Boost retrieval practice through spacing (spreading sessions over time) and interleaving (mixing up related topics during a practice session).
Make sure you plan some sessions for the critical seven-day period after introducing the material.
Consider teaching students the memory palace technique for content that requires heavy memorization.
Support every type of content visually where possible.
Encourage deep processing of the material, for example through reflections, problem-solving, or creative activities.
Ensure that students have opportunities to engage with the material during learning in the same way as they will during the exam.
Try to stimulate emotions in relation to the content. While negative affect can help (for example, recounting a sad story to illustrate a concept), it is probably best to focus on positive emotions through exciting news, inspiring anecdotes, and even more “extrinsic” factors such as humor, uplifting music, or attractive visual design.
Using these strategies will help you create learning experiences where students encode, store, and retrieve information efficiently, allowing them to use it effectively in their lives, studies, and work. Do you have any related experience or tips? If so, share in a comment!
References
Baddeley, A. D. (1986). Working memory. Oxford University Press.
Cowan, N. (2011). The focus of attention as observed in visual working memory tasks: Making sense of competing claims. Neuropsychologia, 49(6), 1401–1406. https://doi.org/10.1016/j.neuropsychologia.2011.01.035
Ebbinghaus, H. (1885). Memory: A contribution to experimental psychology.
Engle, R. W. (2002). Working memory capacity as executive attention. Current Directions in Psychological Science, 11(1), 19–23. https://doi.org/10.1111/1467-8721.00160
Fisher, J. S., & Radvansky, G. A. (2018). Patterns of forgetting. Journal of Memory and Language, 102, 130–141. https://doi.org/10.1016/j.jml.2018.05.008
Kensinger, E. A. (2009). How emotion affects older adults’ memories for event details. Memory, 17(2), 208–219. https://doi.org/10.1080/09658210802221425
MacKay, D. G. (1987). The organization of perception and action: A theory for language and other cognitive skills. Springer New York. http://dx.doi.org/10.1007/978-1-4612-4754-8
Miller, M. D. (2014). Minds online: Teaching effectively with technology. Harvard University Press.
Porter, S., & Peace, K. A. (2007). The scars of memory. Psychological Science, 18(5), 435–441. https://doi.org/10.1111/j.1467-9280.2007.01918.x
Richey, R., Klein, J. D., & Tracey, M. W. (2011). The instructional design knowledge base: Theory, research, and practice. Routledge.
This is the paradox of failure in games. It can be stated like this:
We generally avoid failure.
We experience failure when playing games.
We seek out games, although we will experience something that we normally avoid. (Juul, p. 2)
As a continuation from my last blog post considering grades and Self-Determination Theory, I wanted to take a brief side-quest into considering what it means to experience failure. Jesper Juul’s The Art of Failure: An Essay on the Pain of Playing Video Games will provide the main outline and material for this post, while I add what lessons we might learn about feedback and course design in online settings.
Dealing with Failure
Juul outlines how games communicate through feedback using the theory of Learned Helplessness. Specifically, he focuses on Weiner’s attribution theory, which has three dimensions:
Internal vs. External Failure
Internal: The failure is the fault of the player. “I don’t have the skills to defeat this enemy right now.”
External: The failure is the fault of the game. “The camera moved in a way that I couldn’t see or control and resulted in a game over.”
Stable vs. Unstable Failure
Stable: The failure will be consistent. No recognition of experience gained or improvement. “No matter what I do, I can’t get past this challenge.”
Unstable: The failure is temporary. There is a possibility for future success. “I can improve and try again.”
Global vs. Specific Failure
Global: There is a general inability preventing success. “I am not good at playing video games.”
Specific: Poor performance does not reflect on our general abilities or intelligence. “I’m not good at flight simulators, but that doesn’t mean I’m bad at all video games.”
In general, a combination of Internal+Stable+Global failure feedback would contribute most strongly toward a player adopting a learned helplessness mindset. There is a potential parallel here with course design: when a student does not do well on an assessment, what kind of feedback are they receiving? In particular, are they receiving signals that there is no opportunity for improvement (stable failure) and that it shows a general inability at the given task (global failure)? Designing assessments so that setbacks are unstable (offer multiple attempts and a way for students to observe their own improvement over time) and communicating specific skills to improve (make sure feedback pinpoints how a student could improve) would help students bounce back from a “game over” scenario. But what about internal vs. external failure? For Juul, “this marks another return of the paradox of failure: it is only through feeling responsible for failure (which we dislike) that we can feel responsible for escaping failure (which we like)” (p. 54). This importance of internal failure aligns with what we know about metacognition (Berthoff, “Dialectical notebooks and the audit of meaning”) and the numerous benefits of reflection in learning.
Succeeding from Failure
Now that we have an idea on how we deal with failure, let’s consider how we can turn that failure into success! “Games then promise players the possibility of success through three different kinds of fairness or three different paths: skill, chance, and labor” (Juul, p. 74):
Skill: Learning through failure, emphasis on improvement with each attempt. (This is also very motivating by being competence-supportive!)
Chance: We try again to see if we get lucky.
Labor: Incremental progress on small tasks accumulates more abilities and items that persist through time and multiple play sessions. Emphasis here is on incremental growth over time through repetition. (Animal Crossing is a great example.) (This path is also supported by Dweck’s growth mindset.)
Many games reward players for all three of these paths to success. In an online course, allowing flexibility in assignment strategies can help students explore different routes to success. For example, a final project could allow for numerous format types, like a paper, podcast, video tutorial, interactive poster, etc. that students choose strategically based on their own skills and interests. Recognizing improvement will help students with their skills and helping students establish a routine of smaller, simpler tasks that build over an entire course can help them succeed through labor. Chance is an interesting thing to think about in terms of courses, but I like to think of this as it relates to content. Maybe a student “gets lucky” by having a discussion topic align with their final project topic, for example. For the student in that example, that discussion would come easier to them by chance. Diversifying content and assignment types can help different individuals and groups of students feel like they have “lucky” moments in a course.
Reflecting on Failure
Finally, how do games give us the opportunity to reflect on our successes and failures during gameplay? Juul outlines three types of goals that “make failure personal in a different way and integrates a game into our life in its own way” (pp. 86–87):
Completable Goal: Often the result of a linear path and has a definite end.
These can be game- or player-created. (i.e., Game-Driven: Defeat the ghost haunting the castle. Player-Driven: I want to defeat the ghost without using magic.)
Transient Goal: Specific, one-time game sessions with no defined end, but played in rounds. (e.g., winning or losing a single round of Mario Kart.)
Improvement Goal: Completing a personal best score, where a new high score sets a new goal.
For Juul, each of these goal-types have different “existential implications: while working toward a completable goal, we are permanently inscribed with a deficiency, and reaching the goal removes that deficiency, perhaps also removing the desire to play again. On the other hand, we can never make up for failure against a transient goal (since a lost match will always be lost), whereas an improvement goal is a continued process of personal progress” (pp. 86–87). When thinking about your courses, what kinds of goals do you design for? Many courses have single-attempt assignments (transient goal), but what if those were designed to be improvement goals, where students worked toward improving on their previous work in a more iterative way that replaced old scores with new and improved scores (improvement goal)? Are there opportunities for students to create their own challenging completable goals?
I hope this post shines a light on some different ways of thinking about assessment design, feedback types, and making opportunities for students to “fail safely” based on how these designs are achieved in gaming. To sum everything up, “skill, labor, and chance make us feel deficient in different ways when we fail. Transient, improvement, and completable goals distribute our flaws, our failures, and successes in different ways across our lifetimes” (Juul, p. 90).
Audio is a term we all understand on some level. Commonly we think of audio as the transmission, reception, or reproduction of sound. Audible sound is what we hear. So, when we consider audio as an element of course design we are thinking about how sound can be used to communicate or support learning in an online course environment. Integrating audio in course delivery can be quite powerful if done well.
Perhaps the most common method of using sound in an asynchronous online course is via narrated lecture. The narrated lecture is typically a voice over a slide presentation or screencast. It can be an essential tool of instruction if designed effectively.
Another way the voice of an instructor can be incorporated into a course is via audio feedback on student assignments. These are short audio recordings that have been found to help build a sense of instructor presence in an online course.
Highly focused use of audio is also utilized in subjects where audio is, in essence, the topic at hand. Here we are considering language, music, and media arts courses as examples.
Other valued voices are often brought into the online learning experience via guest interviews in either audio or video format. Not to be left out, the voices of students are increasingly present in online course via tools such as VoiceThread and university provided video portals such as Kaltura Media Space. And with tools such as Zoom recording audio interviews and voice overs is easier than ever. With the availability of media platforms such as YouTube, Vimeo, Amazon Prime, Audible, and Apple Music and Podcasts the ability to bring external audio resources into the online experience seems almost limitless.
Audio is also used as a supporting resource for text-based content in online courses. In this case screen readers may typically provide the audio support. When using audio as a primary learning resource it is necessary to provide text-based transcripts as accessibility options. A fuller description of making audio accessible can be found at the W3C website.
The examples listed above are common ways we integrate audio elements into online learning. Are there other perhaps different ways we might consider? In the next few paragraphs we will explore a few ideas of how we might use audio in specialized ways in course design.
Specialized Audio Use
Orientation & Review Audio
Audio is a great tool to use when smaller segments of media should be used to orient students to a part of an online course. Think about short, more ephemeral, voice messages that can be easily produced and updated from term to term as the course changes. These audio segments need not be highly produced but should be of good quality. This type of audio segment reconnects students with the instructor via voice.
Listen to the orientation audio sample below that was used in an online course: RE 270 – Outdoor Recreation Resources, Behavior and Values | Module III Orientation by Dr. Craig Rademacher; Northern Michigan University c. 2012. (00:02:56). [download orientation audio transcript from Temi.com]
Similar short segments of audio may be used to review sections of content. The review audio may be produced by instructors or students. These audio reviews may be used in preparation for an exam, major project, or collaborations within the course. The goal of such reviews may vary but certainly one goal would be to re-focus the listener to the task at hand while providing timely tips or learning objectives.
One of the things you may have noticed about the audio clip above is the integration of music to the orientation message. Purposeful music selections can support the emotional feel of a course or module being introduced. Music can also serve as an audio cue, or audio branding, for a course. So, selecting audio stingers, or music introductions, can highlight that a particular message or topic is coming or reinforce an emotional tone if carefully planned.
The primary benefit of using audio for orientation and review is that audio is less production intensive making it a quick way to provide feedback. Audio is also fairly easy to edit with a free cross-platform audio tool such as Audacity.
Narrative
The oral traditions of learning go back centuries. Prior to print, learning was interwoven in spoken traditions, legends, and cultural stories. Today story remains a potent vehicle for learning. As you might imagine audio is a great vehicle for story.
An example of this is an Oregon State University political science course titled Governing after the Zombie Apocalypse which was designed and taught by Dr. Rorie Solberg. The story that underpins this course is that a natural disaster has caused the breakdown of the U.S. government. In response citizens must create a new government including a bill of rights and constitution. Students become the citizens creating that new government taking into account marginal populations such as the surviving zombies which are called “blues”due to their virus-caused color.
Audio is used creatively in the course by periodically inserting radio broadcasts about odd happenings around the country. Although not the heart of the effort of the course, these audio presentations, really imagined radio newscasts, provide situational tenor and decision points as students go about creating a new government. Listen to a segment of a mock radio broadcast below.
Mock radio broadcast excerpt: Story by Dr. Rorie Solberg. Produced by Oregon State University Ecampus. Voice acting credit: Warren Blyth (00:02:04)
This segment highlights how audio can be used to shape and carry a narrative through an online course. You might imagine how different narrative audio presentations may support history, literature, or science courses.
Soundscapes & Nature Sounds
Experiencing authentic places or environments is believed to be a valuable form of learning. This idea is a driving force behind field-based learning and experiential learning. Audio soundscapes provide access to authentic acoustic environments that can support online learning about the context of an environment. The environment may be urban, rural, or perhaps in a wilderness-like setting. Soundscapes may also be used to create sense of cognitive and emotional world building that can be used in instruction. Soundscapes typically feature a molar perspective of the acoustic environment.
Other, more specific sounds of nature are potentially positive resources for online instruction. Below is a sound sample of the call of a common raven. Listen to the raven’s call.
Audio recorded in the Beaver Basin Wilderness at Pictured Rocks National Lakeshore in Michigan in 2010 | Craig Rademacher. (00:00:26)
Now imagine how this simple recording may be utilized in a course. It might be used to help identify ravens from crows in an ornithology course. Perhaps this audio clip could be used to help communicate a sense of isolation for listeners studying wilderness values. Or, perhaps it could be used to introduce the poem by Edgar Allen Poe titled The Raven. In each case the audio would be used intentionally to deepen the context or experience of the learner.
Fidelity Matters
Sound quality is an important factor when selecting and using audio in an online course. Audio files need to be of sufficient quality to clearly indicate what you are expecting students to hear. Poor audio, or a confusing sound recording, is experienced as distracting and student will likely tune out. Recording and editing audio does require some knowledge and practice. And there are many places where you can learn to produce audio. LinkedIn Learning offers courses in audio production, podcasting, and even how to select a microphone. So, if you are interested give it a try.
If you are not inclined to produce your own audio content there are resources available where you can find high quality audio for use in courses. Some of these resources are royalty free. Others may require licensing of audio for use.
Final Thoughts
We have reviewed how audio is commonly used in online courses and how we might think about new ways of integrating audio. As you explore the links to resources below start to think about your next course design. How can you augment the text and video you normally use with audio? How might you leverage voice, narrative, or soundscapes to connect online students to the context, authenticity, and humanity of learning? You might want to experiment with audio at first. Start small. If it works, then you will have truly found a sound idea for online course design.
Podcast Resources
There are several ways to find audio podcasts to review for inclusion in a course. Apple Podcasts is a dominant resource in this area. Apple streams over 750,000 podcast shows with over 20 million episodes. Google Play Music is another good resource. Podcast feeds can also be found simply by browsing for podcasts online.
Audiobooks
Audiobooks are found in many online book seller sites such as Amazon (Audible.com). Additionally some more specific sites such as audiobooks.com also provide resources.
New online instructors often express concern about the loss of immediate student feedback they get by teaching in person. These educators count on in-class interaction to help shape their lesson plans in real-time. Student questions, lack of interaction, or even blank looks, help them understand what concepts are difficult for their learners. Others just feel more comfortable with the two-way nature of in-classroom communication.
But teaching in an online environment doesn’t have to be mutually exclusive from gauging student interest and comprehension.
Mud Cards
I was first introduced to the concept of “Mud Cards” or “Muddiest Points” through an open course MIT offered in Active Learning in College-Level Science and Engineering Courses. The instructor described handing out index cards to each student at the end of class asking students to write down an answer to one or more of a few prompts (MIT OpenCourseWare, 2015).
In an online course, this could easily take the form of a weekly survey that looked something like this:
What concept from this week did you find confusing?
Is there anything you found particularly compelling?
What would you like to know more about?
Potential Benefits
The answers received have multiple potential benefits. First of all, instructors will get to look for trends in a particular class.
Are learners missing something central to a course learning outcome?
Is there a concept they need additional resources to master prior to an upcoming exam?
What excites them the most?
Getting this information weekly can provide information that is normally gathered during in-class interactions. It may even be more informative, as participation is likely to be higher (or can be incentivized through participation points). This feedback can be used to add content, perhaps through an announcement at the beginning of the next unit, addressing any common problems students reported. It can also help improve the content or activities for the next iteration of the online course.
The second benefit of an activity like this one is that it is an easy way to introduce active learning to your online course. Active learning, with origins in Constructivism, includes the idea that students build knowledge through “doing things and thinking about what they are doing.”
Rather than passively watching narrated slide-based lectures or videos, or completing assigned readings, they are asked to think about what is being taught to them. Each student, by reflecting on questions like the examples above, takes some responsibility for their own mastery of the content.
3-2-1 (a similar tool)
I recently attended the keynote at the Oregon state Ecampus Virtual Faculty Forum by Tracey Tokuhama-Espinosa (Tokuhama-Espinosa, 2020). At the beginning of her presentation, she told all of us we were going to be asked to email her our “3-2-1.” A 3-2-1, she defined as:
Three things that are new to me
Two things so interesting I will continue to research or share with someone else
One thing I will change about my practices based on the information shared today
Even though I was very familiar with the underlying pedagogical practice she was leveraging, I paid significantly more attention than I would have otherwise to an online presentation. I wanted to come up with something helpful to say. To be honest, suffering from COVID related ZOOM fatigue, it also made sense to ensure the hour of my time resulted in something actionable.
A Word of Caution
The use of a tool like the Mud Cards or 3-2-1 will be successful only if used consistently and students see the results of their efforts. If not introduced early and repeated regularly, students won’t develop the habit of consuming content through the lens of reflecting on their own learning. Similarly, students who never see a response to their input, through a summary or additional explanations, will get the message that their feedback is not important and lose the incentive to continue to provide it.
Conclusion
Introducing a reflection activity like those suggested is a simple, quick way to incorporate active learning into a course while simultaneously filling a void instructors sometimes miss through being able to ask questions of their students in a classroom.
Canvas allows for building anonymous graded or ungraded surveys in which a weekly activity like this would be easy to link to in a list of tasks for a unit of study. It is a low development effort on the part of the instructor, and participation from students shouldn’t take more than 5 minutes.
I will link below to some of the resources mentioned that discuss the use and benefits of Mud Cards and active learning in instruction. If you try it out in an online course, I would love to hear how it works for you.
Hall, S. R., et al. “Adoption of Active Learning in a Lecture-Based Engineering Class.” 32nd Annual Frontiers in Education, vol. 1, IEEE, 2002, pp. T2A-9-T2A-15. DOI.org (Crossref), doi:10.1109/FIE.2002.1157921.
Flexibility is an inclusive practice. Structure is an inclusive practice. Both of these statements are true–yet, many people might wonder how to reconcile these seemingly opposite approaches in their course designs. How does one build a course that is both flexible enough to accommodate the diverse needs of their students, yet structured in a way that is clear and unambiguous? In a practical sense, what do these words really mean?
First, let’s define these terms and consider why each of these approaches are critical to student success. What do we mean by flexibility and structure and why are they both important features of course design and facilitation?
Flexibility
Flexibility is getting a lot of press right now, due to our global pandemic. We are all encouraged to be flexible and understanding of one another and to recognize that most of us, especially right now, are dealing with increased responsibilities. As a student myself, I recall how much relief it gave me to read in a note from my professor that this term is “all about flexibility” along with detail around what this means in the context of our course.
For those of us familiar with online learning, accommodating students with full-time jobs and child or eldercare responsibilities, for example, is not new. However, even for our online students, these responsibilities are compounded by school closures and other distancing measures. Everyone needs additional flexibility, understanding, and support right now. Even you, reader! Let’s be explicit and honest about this in our communications with students and each other.
In the context of our online or remotely taught courses, how do we communicate this to students? Here are a few ideas and suggestions to get you started:
Flexible policies: Saying you will be flexible is not enough. Build flexibility into your policies. For example, if students are required to do field observations for a report or lab, are the guidelines for these observations too restrictive? Might students with mobility challenges or high-risk health considerations be unable to spend extended periods of time outdoors? What alternatives can you provide to these students?
Student choice: Providing your students options will increase their autonomy and engagement. Choice is especially important now because it will allow students to make decisions based, not only on their personal and professional interests, but also based on their individual circumstances, which may have drastically changed in recent months.
Communication: Keeping the lines of communication open is essential. Frequent communication builds feelings of connection so that student needs are more likely to be articulated.
Structure
Building structure into your course means removing ambiguity and avoiding assumptions about your students. Structure does not mean being inflexible. You can be explicit and unambiguous without being rigid.
Two helpful tools for adding structure to your course are rubrics and models, or examples. Rubrics will help you to communicate with your students and will allow you to identify your expectations along with how each criterion will be evaluated. Model assignments will help students to interpret your expectations.
When you don’t have enough structure built into your course, when your expectations are ambiguous, your underrepresented students are disproportionately impacted. This level of ambiguity often results from assumptions about your students’ prior experiences. Assuming they know how to use an LMS or that they have reliable WiFi at home, for example, puts students who don’t have these resources at a disadvantage.
When you don’t have enough structure built into your course, your students will be forced to make assumptions, correctly or incorrectly, about your expectations. Some students may ask questions, but others will do their work and hope for the best. This results in a clearly unequal playing field, exacerbating existing inequalities.
Balance
Given that both flexibility and structure are needed in course design and teaching, whether online, remote, on-ground, or hybrid, how does one balance these competing elements?
Too much structure, and your students will lose agency and motivation. Too much flexibility, and your students may feel ungrounded and directionless.
Here are some tips for finding balance:
Give choice, but include clear parameters for evaluating student work.
Provide multiple lower stakes assessments and stage your course projects, so that students have multiple opportunities to get feedback, correct misconceptions, and earn course points.
Welcome student questions and concerns and share your feedback with the whole class. If one student is asking a question, many others are thinking about asking it and would benefit from the same communication.
Don’t wait for students to request alternatives: odds are high that only your most privileged students will feel comfortable asking for accommodations such as more time or additional feedback. If one student requests an accommodation, others who need similar considerations, may not be asking for them. Why not proactively offer these options to all students?
As a final thought, both structure and flexibility are essential ingredients in the recipe for exemplary teaching. When you find the perfect blend of these elements, all your learners will benefit!
References
Parker, F., Novak, J, & Bartell, T. (2017). To engage students, give them meaningful choices in the classroom. Phi Delta Kappan 99 (2), 37-41.
Sathy, V. & Hogan, K.A. (2019). Want to reach all of your students? Here’s how to make your teaching more inclusive: Advice guide. Chronicle of Higher Education. Retrieved from https://www.chronicle.com/interactives/20190719_inclusive_teaching
Do you ever get the sense that students posting in their online discussions haven’t really engaged with the reading materials for that week? One way to encourage active engagement with course readings is to have students annotate directly in the article or textbook chapter that they are assigned. While it is common to see students annotating in their paper copies of their textbooks or readings, these aren’t easily shared with their peers or instructor. Of course, students could snap a photo of their handwritten annotations and upload that as a reading assignment task, though that does require additional steps on the part of both the student and instructor, and there is no interaction with others in the course during that process. However, it is possible to have students annotate their readings completely online, directly in any article on the web or in their ebook textbook. With this process, the annotations can also be seen by others in the course, if desired, so that students can discuss the reading all together or in small groups as they are reading an article or book chapter online. The benefit to this type of annotation online includes components of active learning, increased student interaction, and accountability for students in engaging with the course materials.
Active Learning
The shift to active learning is a bit like going from watching a soccer game on TV to playing a soccer game. Likewise, reading passively and reading to learn are two different activities. One way to get students actively reading to learn is to ask them to make connections from the course materials to their own lives or society, for example, which they then make into annotations in their readings. Annotation tasks require students to take actions and articulate these connections, all without the pressure of a formal assessment. Furthermore, many students arrive at college not knowing how to annotate, so teaching basic annotation practices helps students become more active and effective learners (Wesley, 2012).
Interaction
“Individuals are likely to learn more when they learn with others than when they learn alone” (Weimer, 2012). Discussion board activities are often where interaction with others in an online course takes place. However, rather than having students refer to a particular reading passage in their discussion board activity, they can simply highlight a passage and type their comments about it right there in the article, no discussion board assignment needed. Others in the course can also read participants’ annotations and reply. With some creative assignment design in Canvas, this can also be set up for small groups. Students may find this type of annotation discussion more authentic and efficient than using a discussion board tool to discuss a reading.
Accountability
A popular way to ensure that students have done the reading is to give them a quiz. However, this is a solitary activity and is higher-stakes than asking students to make targeted annotations throughout a reading. It may make more sense to guide them through a reading with specific annotation tasks. Being explicit about what pieces of the reading students should focus on can help them understand what they need to retain from the reading assignment.
Possible Activities
Student-student interaction: Replace a discussion board activity with a collaborative annotation activity where students can annotate the article as they read. Then they can go back later in the week and reply to each other.
Activate prior knowledge: Ask students to include one annotation related to what they already know about this topic.
Evaluate sources: Find a pop-science article in your discipline that includes weak support for arguments or claims, for example. Ask students to identify the sources of support in the arguments and challenge the validity of the support. Perhaps they could even be tasked with adding links to reliable sources of support for your discipline in their annotation comments.
Nuts and Bolts
Two popular annotation tools are Hypothesis and Perusall. I would encourage you to test these out or ask your instructional designer about your needs and whether an annotation tool would be a good fit for your course learning outcomes.
Open Pedagogy Part 1 – What is the value of going ‘open’?
By Ashlee M. C. Foster, Instructional Design Specialist Oregon State University Ecampus
Are you committed to broadening access to education and knowledge, acknowledging and mitigating barriers, fostering social justice, and designing authentic and renewable learning experiences that contribute to the greater good? Do you employ pedagogical approaches that focus on student agency, collaboration, community, and connection to the public and world at large? If so, you may be an open educator at heart!
This is a three-part blog which will introduce the potential value of open pedagogy (part 1), critically examine considerations and strategies for implementation (part 2), and explore current practitioner examples and design approaches (part 3), which I hope will help you envision open assessments for your courses.
You may be thinking those two little words encapsulate a great deal, and you would be right! I have learned that this is a complex question with various evolving answers among practitioners. Recent literature indicates that there is a shift occurring from Open Educational Resources (OER) centered pedagogy to pedagogy that is focused on the potential impact, collaboration, connection, democratization of education, and the critical inquiry of systems and technology. Both leaders in the field, Robin DeRosa and Rajiv Jhangiani define open pedagogy as, “access-oriented commitment to learner-driven education AND as a process of designing architectures and using tools for learning that enable students to shape the public knowledge commons of which they are a part.” It may help to contextualize this pedagogy by examining your perceived value of the approaches, consider what excites you most, and identify how you personally connect with the pedagogy. Let’s begin by exploring this together!
What values underpin open pedagogy?
I had an invaluable opportunity to attend the Digital Pedagogy Lab Critical Open Pedagogy track, facilitated by Rajiv Jhangiani. Throughout the intense week, our cohort engaged in meaningful discussions centered on what is it that makes someone an educator, open pedagogical approaches, public scholarship, educational technology, the democratization of education, and how open pedagogy can foster social justice. Rajiv asked participants to review his 5Rs for Open Pedagogy and then write a personal interpretation of the values. Specifically, he asked, “What brings you (or others) to this work?” In the spirit of openness, I have shared my initial perception of the values which continue to evolve as I learn more about the field.
Recent literature surveyed educators and asked them to describe how going open impacts their pedagogical approaches. Educators indicated that the open approaches prompted them to find innovative ways for students to obtain and share knowledge, use of new methods and platforms, diversify learning materials to include multi-perspectives, actively teach open literacies, move to a participatory model of teaching and learning from one that was top-down, and to engage in critical inquiry around entrenched knowledge structures.
Additionally, educators shared their perceived value for creating learning assessments that:
go beyond a single course (renewable),
are broadly relevant (inclusive),
allow for student choice when demonstrating learning (agency),
connect to the real world and the learner’s personal interests (relevancy),
amplify multi-perspectives from broad global voices (liberate),
empower students with the knowledge and skills to participate openly (freedom), and for educators and learners to collaborate (participate)!
What are students saying?
These are valuable insights from practicing educators, but what are students saying about open approaches in their classes? In a recent study, 173 students were asked to compare the educational value of open pedagogy to traditional approaches, to identify the types of learning outcomes associated with this approach, and if they preferred open pedagogical approaches to traditional. Out of 169 respondents, 53% of students preferred open pedagogical approaches to traditional classroom teaching practices. Students shared that the open approaches led to increased knowledge of the material, synthesis of information, consideration for the relevance of information, how to bring information together in a meaningful way for diverse audiences, application to real-world issues which they personally connect with, and they found the approaches to be more engaging. However, 20% of students preferred traditional pedagogy. This highlights that the integration of varied approaches may be optimal. I have learned that open pedagogy is not necessarily a silver bullet that can remedy all barriers and challenges associated with closed systems. Rather, it seems to be a tool that can be leveraged to foster social justice, engagement, participation, collaboration, co-construction of knowledge, the democratization of education, and to increase global access to education.
With all that said, let us circle back around to the question posed in the Critical Open Pedagogy workshop, what brings you to this work? I encourage you to reflect on this question. You may even find it helpful to write out your interpretation of the values of open pedagogy and share those with the community. If you feel comfortable to do so, please feel free to share in the comments of this blog. Do you find yourself inspired by this pedagogical approach? If so, I invite you to revisit this blog for Open Pedagogy Part 2 – Critical Considerations for Implementation and explore the resources below.