Backwards Design: How it Ensures Validity

Accurately measuring our students’ learning requires us to know what we expect students to be able to do and to be sure that the assessment is measuring exactly that.  This is important because we want to ensure our assessments are valid and reliable.  Our teaching will be more focused as well.  To accomplish this, we must use Backward Design when planning lessons, courses, or programs.  According to Lesson 1 of Rio Salado’s “Assessing Learners Online” course (2011), the steps of Backward Design are: 

Step 1: Choose the content

Step 2: Learning Outcomes

Step 3: Performance Objectives

Step 4: Assessment

Step 5: Activities/Materials

Backwards design forces us to first look at what we want the students to learn before we start planning activities or assessment.  We determine this through a series of steps.  First we decide what content to teach (i.e. is it a math lesson covering the order of operations or is it a science lesson to teach the scientific method).  We should consult content standards, grade-level expectations/curriculum, and even students’ interests. 

Then, we must write learning outcomes.  These are general statements of what students should be able to do at the end of a lesson, course, or program.  They are similar to goals and they are broader statements of capability.  From those outcomes, we must narrow our focus and write specific performance objectives which include an actual, observable verb that describes what students will actually be able to do to demonstrate their learning.  This is a very key part of the process.  According to Oosterhof, Conrad, & Ely (2008), the verbs chosen must be observable and they must be able to demonstrate that the particular type of knowledge we are teaching has been learned.  The types of knowledge include declarative knowledge, procedural knowledge and problem solving knowledge.  It is important to determine if a lesson teaches declarative knowledge, procedural knowledge, or problem solving knowledge because the objective must be written using a verb that is capable of showing that students have acquired that type of knowledge.  For example, if we want math students to know that operations within parenthesis should always be computed first (declarative knowledge), we must write an objective that asks students to state that knowledge.  However, if we want students to apply the rule (procedural knowledge), then we must be sure our objective asks students to do something that shows us they know how to apply the rule such as solve a problem using order of operations rules.  Otherwise, we may be asking students to demonstrate something other than what our objectives require of them-which, of course, means our assessment results are invalid.

Next, we can design assessments.  Using the solid performance objectives we have written, we can create assessments that will accurately measure what they are supposed to measure.  In other words, they will be valid.  Finally, we can plan activities, lessons, and select materials to be used.  Since, at this stage, we know what we are looking for in students who have learned what we want them to, we can begin to plan learning activities and lessons and choosing materials.  Our lessons will be better designed due to this process.

This method forces teachers to determine what they must observe in students who have met the objectives before they begin to teach instead of teaching first and then trying to come up with an assessment.  In other words, teachers know what to look for to determine who has learned the material.  It ensures that learning outcomes and performance objectives are based on standards and grade-level curriculum, as well as substantiating the validity of the assessment(s).

References

Oosterhof, A., Conrad, R., Ely, D.  (2008).  Assessing Learners Online.  Upper Saddle River, NJ:  Pearson Education, Inc.

Rio Salado (2011).  ELN 122-Evaluation Online Learning- Lesson 1:  Measuring Knowledge Online.  Retrieved from Rio Salado website:  https://www.riolearn.org/content/ELN/ELN122/ELN122_INTER_0000_v1/lessons/lesson01.shtml.

Performance Assessments in Online Education

Performance Assessments are used in traditional, face-to-face settings as well as online settings.  They are a valuable tool in either setting, but have specific applications, advantages, and limitations in each.  I will define and discuss them and their advantages and limitations in online education.

According to Oosterhof, Conrad, & Ely (2008), performance assessments involve students with performing a task, which involves several steps and requires specific skills, usually in order to create a product.  They are used in skills based and educationally based classes as well as on the job training and evaluation.  They are not a multiple choice, true/false, fill-in-the blank type of test.  However, a performance assessment can be a written document if the course objectives require it.  For example, in order to determine if a student has internalized and can use/apply grammar rules, a teacher might observe actual writings from the student.  A performance assessment must be designed to measure an explicit objective or goal from the course.  In other words, the task or product of the performance objective must themselves be course objectives or goals.

Performance assessments can be graded by looking at the process the student goes through to complete the assessment or they can be graded by examining a final product.  Often, the process is simulated-especially when it is too expensive or dangerous to have the student perform the task in reality.  For example, simulations are often used in a military training facility to prepare soldiers for a mission/battle.  Sometimes, however, the performance assessment does involve having the student actually perform a task and/or create a product.  The instructor will observe and judge either the process of creating the product or the product itself, but not usually both.  For example, in a cake baking class, a student’s final might be to bake a cake and the instructor would sample the cake and judge it (the final product) based on the qualities of a good cake in order to determine if the student has met the objective of the class (baking a cake).  Of course this is a very simple example, from a skills based class instead of an educational class.  However, performance assessments can be used in every type of educational class as well.

The instructor/observer has two choices about how to structure the assessment.   He/she can either prompt the student to do specific steps/tasks in the process or they can simply observe without prompting.  The specific situation being observed will usually dictate this decision.  For example, if a company is hiring someone to work in a customer call center, they may want to observe the person helping a customer on the phone without prompting them to find out if the person has the right skills necessary for the job.  However, this will make it difficult to determine the range of skills the person has because not every situation can be covered in one to a few calls.  According to Oosterhof, Conrad, & Ely (2008), prompting students tends to bring out maximum performance whereas not prompting them will produce a student’s typical performance.  Because of this, when a supervisor or instructor wants to observe personality traits, work habits, and a willingness to follow prescribed procedures, he/she should not prompt the worker/student.  However, prompting is needed if he/she wants to see how well a student can explain a concept orally, write a paper, or play a musical instrument.

Since performance assessments involve students actually doing something, they are able to determine well a student’s skill at activities like playing a musical instrument, sports performance, creating and conducting an experiment in a science lab, or creating works of art.  Written tests would not be able to accomplish this.  However, due to space and time separation in asynchronous, online settings, performance assessments can’t be used online to measure these types of skills.  Therefore, online education is limited to what types of classes/skills can be taught online as well as what types of skills can be assessed online. 

Due to the space and time separation with asynchronous, online courses, observing the process versus the product is also very limited or impossible.  In other words, online instructors are not able to create performance assessments that directly observe and measure the steps/tasks a student will complete because schedules of online instructors and students do not often coincide with each other, hence one of the reasons for taking classes online.  This does not rule out using performance assessments in online settings to grade the product, which is how it is usually employed in online settings.  In this way, the instructor can indirectly observe the process by looking at the product.  However, it does present a problem for using performance assessments as formative assessments in online settings since formative assessments rely on observing the process that one goes through to create a product.  Since performance assessments often help us to see and understand a student’s reasoning and understanding, (either directly through observing the process or indirectly through observing the product) they can provide insights about students that written tests cannot.  Having online students participate in creating an actual product (such as an education student creating an actual test) will help the instructor better determine what the student can do with the course knowledge than a written test (about rules and procedures for creating a test) could determine.

In online settings, performance assessments can be used to teach complex skills if they are used effectively because students who know they will have to complete a performance assessment will learn the material differently than if they were studying for a written test.  Also, the teacher will be more likely to teach students differently in order to ensure their success with the performance assessment.  In other words, they will not only memorize information, they will try to understand it so they can use and apply it.  This should be the goal of both face-to face and online settings because our goals should always be to help student use higher levels of thinking.   Since many online classes are designed for working professionals, their reasons for taking the classes require them to not only know something (declarative knowledge), but also be able to use and apply the knowledge (procedural knowledge).  They also tend to need to be able to overcome difficulties and solve problems in their work environments.  Because of this and the fact that performance assessments can measure problem solving skills, whereas written tests cannot, they are a very natural, well-suited, and valuable tool in an online educational setting.

Another reason performance assessment is well suited for online learning is because they require less security than written tests require.  Often, the scoring plan or an actual model will be given to the students before they are required to complete the performance assessment, whereas the answer key for a written test would never be given to the students.  This is especially helpful in an online environment since security issues are often more difficult to overcome than in face-to-face settings.

References

Oosterhof, A., Conrad, R., Ely, D.  (2008).  Assessing Learners Online.  Upper Saddle River, NJ:  Pearson Education, Inc.

Balance between Online and Offline Administering and Scoring of Assessments

Both constructed response tests and fixed response tests are valuable tools for eLearning.  A constructed response test asks students to provide an answer.  An example of this type of test is a fill-in-the-blank.  Fixed response tests are tests such as multiple-choice, true-false tests, or even matching.  These two types of tests each have different qualities which make them beneficial in eLearning for specific purposes.  However, according to Oosterhof, Conrad, Ely (2008), due to technical and practical issues, there must be a balance between administering these types of tests online and having the computer score them or administering them in writing and having the teacher grade them.

Fixed response tests sample the content more adequately than constructed response tests, such as essay tests, because more questions can be included on the fixed response test than on an essay test due to the fact that it takes longer to complete an essay test than a multiple-choice or true-false test.  Because they sample more of the content, they can be used as summative assessments.  However, they can work just as well as a formative assessment since instructors can often program the online assessment to give feedback to students when a wrong answer is chosen.  Fixed response tests measure procedural knowledge such as concepts and rules more readily than constructed response tests.  They structure the problem to be addressed more effectively than constructed response test because constructed response items often can have several correct responses based on the learner’s interpretation of what the test item is asking.  The fixed response test eliminates this by providing only four responses to choose from with only one correct answer. 

Constructed response tests are not subject to guessing like fixed response tests.  Also, more time is required to build a fixed response test than a constructed response test.  They have three characteristics which make them best used in for formative tests:  they are best designed for use with key words of a unit or lesson, they tend to measure lower levels of thinking such as recall of information, and they tend to have a higher scoring error.  They can have a higher scoring error (if scored online and not by the instructor) because the answer supplied by the student may be counted as incorrect even if it is correct due to the computer not having a complete list of all possible answers.  For example, if a student answers a question in the plural form of the answer, but the computer answer key only has the singular version of the answer, it will be counted incorrect even though it is right.  Fixed response items are more easily scored by a computer than constructed response items because they contain only one correct answer.  However, essay tests do a better job of directly measuring the behaviors required in the instructional objective.  Therefore, they may work well as a summative.  Since they do not sample a broad area of the content domain being studied, and since they allow learners to explain their logic, they may serve a formative purpose better however.  Due to the time constraints with essay items, both in taking and scoring the exam, they are usually used more sparingly.

Due to several reasons, sometimes it is better to have these tests administered and graded online by the computer and sometimes it’s better to have them drawn up and graded by the instructor.  Making an online test can be time consumptive and difficult depending on the capabilities of the software or learning management system (LMS) being used.  Therefore, if it saves time, it may be best to hand administer and score the test.  As mentioned above, some assessments don’t lend themselves well to computer grading just yet (such as essay and completion items).  However, with online administration, feedback can be given immediately.  This is an important factor for effective feedback.  Instructors can diagnose problems (problems with test validity and formative assessments of achievement) while a test is being administered online.  All of which is necessary for effective instruction and student growth and learning.  Another important reason to administer and score online has to do with the nature of online courses.  Since students in open entry/open exit online courses may have different start dates, it makes sense to allow students access to tests automatically based on that start date.  Students taking the test at different times would make it difficult for an instructor to grade several different assessments at the same time, therefore, online testing with computer generated scores eases the burden of the instructor.  Of course, we can’t forget to discuss security issues with both paper and pencil and online assessments.  Depending on the purpose and importance of a test, the security required will vary.   Most formative assessments will require less security than summative assessments.  Distributing tests to students may allow them to copy and distribute the test to other students.  Therefore, it may be beneficial to have it administered online where the software or LMS being used can prevent a student from copying and pasting the test.  Another method would be to have the student take the assessment at a secure facility with passwords and supervision.

Again, deciding on how to administer and score assessments in an online setting depends on technical and practical issues which will vary by the school and the technology being employed.  Generally speaking, we need to use technology to support and improve the teaching/learning experience and not to add extra burdens or impediments.

References

Oosterhof, A., Conrad, R., Ely, D.  (2008).  Assessing Learners Online.  Upper Saddle River, NJ:  Pearson Education, Inc.

Differences and Uses of Constructed Response vs Selected Response Tests in eLearning

A constructed response test asks students to provide an answer.  An example of this type of test is a fill-in-the-blank.  Fixed response tests are tests such as multiple-choice, true-false tests, or even matching.  They each have different qualities and appropriate uses for eLearning.  Some differences have to do with the advantages and disadvantages of each type of test. 

Some of the advantages for fixed response tests include that they sample the content more adequately than constructed response tests such as essay tests because more questions can be included on the fixed response test than on an essay test.  This is due to the fact that it takes longer to complete an essay test than a multiple-choice or true-false test.  They measure procedural knowledge such as concepts and rules more readily than constructed response tests.  They can ask learners to classify responses as examples or non-examples and apply a rule by asking them to find the response that correctly applies the rule.  They structure the problem to be addressed more effectively than constructed response test because constructed response items often can have several correct responses based on the learner’s interpretation of what the test item is asking.  The fixed response test eliminates this by providing only four responses to choose from with only one correct answer.  Fixed response items are more easily scored by a computer than constructed response items because they contain only one correct answer.  Constructed response items may have correct answers that were unanticipated by the instructor/designer and therefore not included in the computer’s answer key.  This means that a student may answer correctly, but the computer scores it as incorrect because the answer was not included in the answer key.  Computer generated feedback can be provided for responses to individual items.

Some of the disadvantages of fixed response items are that they are subject to guessing, whereas constructed response tests are not.  Also, more time is required to build a fixed response test than a constructed response test.

Since fixed response tests are able to sample more of the content than constructed response tests, they can be used as summative tests.  Also, they can be more easily scored automatically and therefore work very well in an online environment.  However, they can work just as well as a formative assessment since instructors can often program the online assessment to give feedback to students when a wrong answer is chosen.  Since constructed response items such as fill in the blank tests are best designed for use with key words of a unit or lesson and they tend to measure lower levels of thinking such as recall of information, they would be best employed as a formative assessment.  Also, since they do seem to be associated with a higher scoring error, this is the best place for them.  However, essay tests do a better job of directly measuring the behaviors required in the instructional objective.  Therefore, they may work well as a summative.  Since they do not sample a broad area of the content domain being studied, and since they allow learners to explain their logic, they may serve a formative purpose better.  Due to the time constraints with essay items, both in taking and scoring the exam, they are usually used more sparingly.

References

Oosterhof, A., Conrad, R., Ely, D.  (2008).  Assessing Learners Online.  Upper Saddle River, NJ:  Pearson Education, Inc.