Evaluation Plan and Report

Evaluation Plan

In order to best establish our evaluation plan, we asked ourselves what we would like to know and measure upon completion of the learning design. The chart that we’ve included shows the questions that we came up with as a team for us to better understand the effectiveness and impactfulness of our learning design.

Milestone-7-9-Development-and-Evaluation-Google-Docs (4).png

Questionnaire/Survey

While we are planning to conduct all evaluation methods that are included above, we wanted to specifically hone in on the Questionnaire/Survey because it was included for multiple questions we had about the learning design and it is an easy way to collect feedback. The Questionnaire/Survey will be given to learners during the last week of the course, at the end of the week 4 session. The survey will be housed on Google Forms and will include 6-8 questions. Scroll through images of the survey here or access the full survey at this link: https://forms.gle/J1NSgn4T1pVcBAJ

Evaluation Report

Purpose

This evaluation was written for our stakeholders who are interested in implementing a course addressing critical digital media literacy skills for older adults. The instruction that is being evaluated centers around an educational program targeting digital media literacy skills for adults aged 60+. The program consists of four face-to-face sessions distributed over four weeks with each session lasting 60-90 minutes. Each class focuses on a different critical aspect of digital media literacy. The breakdown of topics is listed below:

Introduction to critical media literacy and why it matters
Building a community of learners
Identifying misinformation, scams, conspiracy theories, and phishing emails
Curating credible sources for the future

The program also includes an optional social media component for learners to participate in outside of the face-to-face sessions. Learners are encouraged to join a group created and moderated by the course instructor on a specific social media platform where they can post content to their peers and practice actively participating in a digital space.

This evaluation set out to answer key questions to determine the effectiveness of this learning design. The questions are as follows:

Did the instructor find the course easy to run?
Do learners feel that the skills they learned in the course are applicable to their real world experience?
Were learners able to apply what they learned to their lives? Are learners learning what the designers want them to learn?
How well does the design serve learners with varying levels of background knowledge?

Design Process Model

During our design process, we followed a sequence that consists of 5 phases: analysis, ideation, development, implementation, and evaluation. These phases were loosely inspired by the ADDIE model (Analysis, Design, Development, Implementation, and Evaluation). While we were working on each phase, our team always connected back to our learning goals and objectives and adjusted and refined them at each phase. Connecting with our learning goals and objectives thus, became a central part of our design process. Based on our experiences, therefore, we came up with a model in which 5 phases are connected to each other and at the same time, each phase is also connected to our learning goal and objectives. Each phase of the design process is like the phase of our earth at different positions and is revolving around the only sun which functions as a metaphor for our learning goals and objectives.

Analysis Phase

When we started our design process model, we first began with our analysis in order to better understand the available information for our subject matter and the audience for our learning design. We broke down the analysis into 3 sections: needs analysis, task analysis, and learner analysis. We conducted a survey and interviews among older adults over 60, completed research and literature reviews to understand the content available on our subject matter and spoke with subject matter experts to identify our learners’ needs. The Analysis phase is typically where learning designs begin, however, in our learning model, we see it as both the beginning and an iterative process for a learning design. At times throughout our design process we returned to our Analysis phase to either gather information or research new information as needed.

Reflections

Brittany

This project was really impactful for me. As an individual who has been working in learning development and design for some years now, this class and project offered some structure to my lived experience and prior knowledge. The design process is truly a vast process that includes much more than I ever realized prior to this class. It was a great experience to work on a team with my teammates who all brought their unique strengths and areas of expertise. I’m really excited about the potential and future of this project.

Kate

This project really helped me appreciate just how complex the learning design process is. Because this course is part of the DMDL program, I assumed that we would be developing a digital learning design when we first started exploring this topic. However, our design shifted to in-person learning as we continued working through the different milestones. This change really illustrates how varied designs can be. I also grew to really enjoy the collaboration that this project required. I learned to trust my teammates, and we worked together to maximize our individual strengths. While our design still has a long way to go before being ready for a true rollout, I am really proud of the work that we produced together.

Angel

This project helped me to narrate what is the learning design process correctly. And also, this topic is the creative topic I focus on and feel attracted to. When I first started on our design, I thought teamwork was the main content for these steps. From online design to creating how to design, these processes helped me correctly use life experience and academic knowledge mixed to support our project well. All in all, I love my teammates who taught and helped me a lot with my work; I am the only full-time student in this group, so I should make more focus on my assignment on our work and tasks. Also, during our project, I hope these contents could help more older people know and understand media literacy, thus reducing unnecessary scams on their websites. Thank you guys all on my team!

Kyle

Honestly, this project is indeed a challenge for me as it is called. I don’t think I will be able to handle it without working with amazing teammates. Throughout this semester, I can confidently say that I have learned so much from working on this project. I understand what processes a design is going to take and how to communicate with every member in this project including stakeholders. I think this project has been so beneficial to me not only because I am a teacher myself that I will design lessons, but also because I have a passion for creating better learning experiences and solutions for all students in the future.

References

Kurt, S. (2018, December 16). ADDIE Model: Instructional Design. Educational Technology. Retrieved

May 4, 2022, from https://educationaltechnology.net/the-addie-model-instructional-design/