Last week I wrote a post about three types of assessment common in the Ontario school system: Assessment as Learning (AaL), Assessment for Learning (AfL), and Assessment of Learning (AoL). That post described all three and explained their importance. If you haven’t read it yet, I suggest you start there.
Over three weeks, I will break down each of the different types of assessment, as they pertain to instructional design and learning experience design. I will start with Assessment of Learning (AoL). Even though this type of assessment comes at the very end of learning, it’s the simplest to understand, and the most common form of assessment.
Before I delve into AoL as it is used in instructional design and learning experience design, let’s review where the term came from as it relates to teaching in the primary and secondary school system.
Description of AoL from the school system
AoL is a formal assessment of what a learner has learned by the conclusion of the chapter or course. This is often in the form of a test or exam, although in some cases it could also be some sort of project or presentation. In the school system, AoL is also known as summative assessment or evaluation (not the same as evaluation in the instructional design world).
AoL in instructional design
When it comes to the instructional design environment, AoL may look a little different. Quizzes, tests, and exams are common but projects and presentations are rare outside of the school system. The only typical form of authentic assessment outside of the school system is the rarely-used style of workplace experience simulations.
Let’s look at some considerations for AoL in instructional design.
1. Assessment must be based on the learning outcomes
The single most important aspect of creating good assessment, particularly AoL, is creating assessments that are strongly based on the learning outcomes. At the start of the course design process, you should have written some learning outcomes for the course. These outcomes, the instruction, and the assessment should be closely aligned. In fact, if a learner ever had the desire, for each test question they should be able to point to the associated learning outcome and point to the relevant learning content.
Writing effective learning outcomes is a blog topic of its own, but I’d be remis not to include one valuable piece of information here: choose your action verbs wisely! If you stated that a learner will be able to know, understand, appreciate, or learn something, you’ll never be able to assess this.
Instead, your learning outcomes should begin with a measurable action verb. Whether you choose to pick a verb from Bloom’s Taxonomy, Marzano’s Taxonomy, or elsewhere, it should be a verb that can actually be assessed.
Often when creating eLearning, instructional designers feel a bit boxed in. They often feel that they can only use verbs that assess lower order thinking skills. This may be true when using a standard multiple choice test, but if you have the opportunity to include scenarios, you can assess higher order thinking skills as well. I’ll get to scenario-based assessment shortly, but for the time being, I don’t think I can overstate how valuable scenarios can be for AoL. They allow you to assess critical thinking and problem solving. They also are one of the best ways to assess learners on relatively open-ended questions.
If you’d like more information on constructing learning outcomes, I’d recommend this post from Mohawk College.
2. Writing multiple choice tests
Although they’re not my favourite, I’d like to mention multiple choice tests here, because they are so prevalent. Patti Shank and Trina Rimmer each have great articles on multiple choice tests, but here are some brief notes to creating effective multiple choice tests.
- Ensure that the stem of the question is clear and brief. Unless the information is absolutely essential, overly long stems can confuse learners.
- Ensure that the answer options are also brief and direct. Keep them all approximately the same length and use similar language and style. The answer shouldn’t be apparent from the way the options are worded.
- Avoid double negatives. Learners who know the answer may get it wrong simply because they are confused by the wording.
- Try not to connect questions. If a learner answers one question incorrectly, it should not affect their ability to answer any other question.
- For online tests in particular, consider randomizing the questions and shuffling the answer options to reduce cheating attempts. However, one exception is when the answer options to a question are numbers – in this situation, have the numbers go in order from smallest to largest (or vice versa).
3. Creating scenarios
You may be tempted to skip this section if you create compliance training – I urge you not to. Many instructional designers complain that compliance training is only about giving rules and yelling at learners. With that attitude, it makes sense that they only feel that a simple multiple choice test of lower order thinking skills is appropriate for the AoL.
However, I strongly disagree with these thoughts. I make compliance training for a living. Every single course I create is about rules, policies, and regulations. However, I think that by focusing on why those rules, policies, and regulations exist, you can get to the goal of the instruction. With that goal in mind, it’s easy to build effective training scenarios that assess how learners will use the rule, policy, or regulation in the real world. And that is what will actually make a different in the life of the learner.
For example, if the training requirements were to teach learners to do XYZ, don’t simply ask the learners ‘What is XYZ?’ or ‘What are the steps to XYZ?’. This really doesn’t assess whether they understand it, or will ever use it. Instead, ask yourself (or your SME), ‘Why do they want the learners to do XYZ?’. Based on this answer, you can build a scenario.
While I was working on my Master’s of Instructional Design and Technology, I dutifully learned all the theories, principles, and best practices that our instructors told us to learn. But it wasn’t until I came across Cathy Moore’s scenario-based learning design process called Action Mapping that I became truly excited by the idea of designing learning.
Action Mapping teaches instructional designers how to build a course (or assessment piece) through scenarios. Here is a brief overview. The first step is to identify what the problem is, and how you’ll know it’s been solved. Then, assuming the problem can be solved through training, the process first focuses on what they need to be able to do to solve the problem. The information taught to the learners flows from there (not the other way around). Finally, scenario-based activities are developed that focus on what the learner needs to be able to do, and they allow the learner to ‘pull’ information, as needed, to solve the scenario problems. [for more information on pull learning, check out this article by Tom Kuhlmann]
Using Action Mapping, or a similar strategy, can help you design AoL that assesses higher order thinking skills and gives you information that more accurately represents whether the learners understood the course content. Even though Action Mapping is based on how learners attain the content in the first place, the same principles and strategies can be used for AoL.
4. Gamification
I have included gamification as a heading here to let you know that I haven’t forgotten about it, and am not purposefully ignoring it. I will delve into gamification in the Assessment for Learning (AfL) post.
This is mainly because, in my opinion, gamification is an excellent way to learn content, but at the end of the day AoL should focus on how well the learner knows the content without the distractions of points, scores, timers, and leaderboards.
5. Tracking assessments
My final thoughts related to the instructional design of AoL is that before writing your assessment, you should consider how the scores will be tracked. Are you working within an LMS? For an eLearning module will the scores be collected via the SCORM or xAPI course, or will the questions be LMS-based? Make sure you know the answers to these questions before beginning the design of the assessment, because these considerations could have a major impact what you can and can’t do when designing the AoL.
AoL in learning experience design
Learning experience design takes instructional design one step further by considering the learner experience. Since the learner experience doesn’t stop at the end of the course, it continues on with the AoL, let’s look at a few ways that can we ensure the learner is having a positive experience on the AoL.
1. Navigation
If the AoL is not within the eLearning module (or even if it is, but it is significantly different than the rest of the module), you may want to provide the learners with some navigation help before they begin the assessment. You could include a short video, images, or simply text describing how to navigate the assessment.
Unfortunately, I’ve learned this the hard way. While creating my second-ever course during my Master’s degree, I developed a final assessment within the Moodle LMS. Since the assessment environment was significantly different than the course environment, many learners struggled to understand how to navigate through the assessment. This was the number one complaint I received. Since then, I’ve tried my best to ensure that I don’t add to the stress of my learners by familiarizing them with the assessment environment.
2. Clarity and visual design
Clarity of an AoL primarily comes down to clear instructions and appropriate vocabulary (that is consistent with the vocabulary used within the course content). Before releasing an AoL to your learners, have another person read through the instructions and check that they understand them perfectly.
Additionally, it helps to include comments at the start of the assessment related to the required passing grade, the number of questions in the assessment, and an example rubric if rubrics will be used.
Visual design is mostly of concern for AoL completed within an eLearning module. All of the visual design considerations that exist for the content part of the eLearning module continue to matter on the AoL. Distracting or confusing designs can significantly impact a learner’s ability to complete an AoL.
3. Accessibility
Review your learner analysis when preparing your AoL. Although you may consider variety in questions to be important, adding images, audio, or video could be troublesome if you have learners with visual or auditory impairments.
Additionally, any learners who will complete the AoL on a mobile device and will be in a public place may not be able to unmute the course, so they may not be able to answer questions that rely on audio or video.
Continuing to consider the mobile learners, try out your AoL on a mobile device prior to releasing it to the learners. You may find that the radio buttons or checkboxes used to answer questions are too small for your fingers to select.
4. Reduce stressors
Many learner’s experience test anxiety to some level. Don’t make things worse by adding in a timer or trick questions.
Feedback for AoL
As with gamification, I have included feedback as a heading here to let you know that I haven’t forgotten about it, and am not purposefully ignoring it. However, I won’t make it a focus of this post because many learners don’t look beyond their final test score.
Ideally, all learners would carefully read the feedback and review any concepts they answered incorrectly even though they won’t be re-assessed, but this rarely happens in real life. This means that the feedback given while the learners are learning (AaL) and (AfL), which has the possibility of altering learner behaviour, needs to be filled with meaningful comments. So be sure to visit the Assessment for Learning post to read more about this topic.
Conclusion
When I first decided to write this post, I never expected it to be this long! But with learning outcomes, scenarios, navigation, accessibility, and more, there’s lots to consider when creating AoL.
I’ve only skimmed the surface on some topics, but it should be enough to help you refine your online searching. In the future I might explore some of these topics in more depth. Hopefully, you’ve read at least one thing here that you’ve never considered before.
You’ll find the same kinds of information (on slightly different topics) in the upcoming posts on Assessment for Learning (AfL) and Assessment as Learning (AaL). Stay tuned for them.