Dissertation Project

ABSTRACT

The purpose of the study was to assess animated demonstrations to consider the worked example and variability effects (Paas and van Merriënboer, 1994; Sweller & Chandler, 1991), and determine if demonstration users would exhibit a delayed performance decrement, described as Palmiter’s animation deficit (Lipps, Trafton & Gray, 1998; Palmiter & Elkerton, 1991). The study used Techsmith Morae to measure and record performance time, accuracy and relative condition efficiency (RCE) (Paas and van Merriënboer, 1994; Techsmith, 2004). It addition, it developed a new metric called performance efficiency (PE), which is an objective measure that is similar to relative condition efficiency (Paas and van Merriënboer, 1994). Results revealed the animated demonstration groups (demo+practice and demo2+practice groups) assembled the week one problem scenario in significantly less time than the practice group. This is positive evidence for the worked example effect (Sweller & Chandler, 1991) but now given animated demonstrations. In addition, subjects from the demonstration groups were found to be significantly more efficient (given performance efficiency), than those from the practice group. Finally group performance did not differ a week later, providing no evidence of Palmiter’s animation deficit.

INTRODUCTION

“A worked example is a step-by-step demonstration of how to perform a task or how to solve a problem” (Clark, Nguyen, Sweller, 2006, p. 190). This study considered animated demonstrations from both a human performance technology (HPT) and cognitive load perspective, to measure performance time, accuracy  & relative condition efficiency (RCE) (Paas and van Merriënboer, 1993; Paas, Tuovinen, Tabbers, & Van Gerven, 2003). The study also developed a new metric called performance efficiency (PE).

Lipps, Trafton and Gray (1998) described an effect known as "Palmiter’s animation deficit." Lipps et al. describe this as a short term performance gain by learners using animated demonstrations (during early skill acquisition), but a significant loss in long term retention. Palmiter describe the phenomenon this way: “the demonstration groups became significantly slower between the training and delay test session...accuracy between sessions decreased significantly for the demonstration groups and increased significantly for the text-only group” (Palmiter & Elkerton, 1991, p. 260). However this animation deficit may be in conflict with the well-known "worked-example effect" (Sweller & Chandler, 1991) given the claims of Lewis (2005), who proposed animated demonstrations act like worked examples. That is those learners who study these animated worked examples would be expected to have improved learner performance over their problem-solving peers. Therefore this study contrasts several instructional conditions to compare learner performance.

METHODS

Learners (n=122) were expected to learn how to select, move, rotate, and hide layers – the Mr. Potato head problem, within an Adobe Photoshop Elements document (Adobe Systems, 2002). All learners watched a brief overview, that introduced the subject matter, next a JavaScript randomly assigned learners to instructional conditions.

Research Questions

The following research questions are based on the possible outcomes of the instructional conditions:

Question 1: Is there a significant difference among the instructional conditions, relative to performance time?

Question 2: Is there a significant difference among the instructional conditions, relative to accuracy?

Question 3: Is there a significant difference among the instructional conditions, relative to “relative condition  efficiency?”

Question 4: Is there a significant difference among the instructional conditions, relative to “performance efficiency?”

Given the methodologies of cognitive load research, it was necessary to measure several dependent variables: perceived mental effort, performance time and accuracy. These dependent variables may be addressed on their own, or in combination via constructs like relative condition efficiency or performance efficiency. Each of these variables are measured with a screen recording and analysis software (Techsmith Morae).

Acquisition phase

All learners were asked to open a folder on the desktop of their computers and to double-click on the “start” icon, a short cut which led them to a web-based survey (survey 1). Once learners answered all questions and submitted the survey, all learners were forwarded to the introductory overview, next a JavaScript randomly assigned each learner to an instructional condition (shown below). These instructional conditions taught learners how to use layers within Adobe Photoshop Elements 2.0:

demo - Learners watched an animated demonstration that assembled the Mr. Potato head document with Photoshop Elements but did not practice.
demo + practice - Learners viewed the Mr. Potato head demonstration and assembled the Mr. Potato head problem.
demo 2 + practice - Learners watched a different collage- based animated demonstration that taught the same skills, but they assembled the Mr. Potato head problem.
practice – Learners watched the overview and then assembled the Mr. Potato head problem.

Overview
Condition 1 Demonstration (only)	Condition 2 Demonstration + Practice	Condition 3 Demonstration 2 + Practice	Condition 4 Practice (only)
no practice

Initial practice for conditions 2,3, and 4

After viewing the instructional materials some groups of learners (conditions 2,3, and 4) must reassemble a disassembled "Mr. Potato head" document w/ Adobe Photoshop Elements 2.0.

Disassembled Mr. Potato head	Reassembled Mr. Potato head

After taking part in an instructional condition, all learners concluded the activities by completing a post-treatment survey.

Retention phase (Week Two) - Delayed Practice with a different problem scenario

One week later, all learners assembled a different problem scenario (the picnic problem). This problem requires learners to use their newly learned skills in a new context. This new scenario is similar in many respects to the Mr. Potato head document, but is somewhat more complicated. In this new problem scenario, learners must reassemble a disassembled scene with Adobe Photoshop Elements 2.0

The Picnic Problem	A Reassembled Picnic Scene

Survey 2

In addition to gathering more direct, objective measurements of cognitive load, Survey 2 is intended to gather introspective data to grasp a more subjective measure of cognitive load. Paas (1992) pioneer this type of cognitive load measurement.

Survey 2 also asks the learner if they have used Photoshop over the past week. This is done to see if long term retention has been altered by interaction with the software over the intervening week.

RESULTS

A MANOVA of performance time and accuracy was used to compare group performance. During week one, it was hypothesized that the demonstration learners would out-perform those in the practice condition. During an analysis of the assumptions of the MANOVA, several multivariate outliers were detect and removed. In addition  transformations were employed. Once the MANOVA was conducted it was found that there was a significant difference between groups, Wilks’ Λ=0.68, F (2, 68) = 6.83, p <0.0001, η²=0.32. Post hoc comparisons with Scheffé’s test (p<0.025) revealed that the demonstration groups (demo+practice and demo2+practice groups) assembled the problem, in significantly less time than the practice group, which is positive evidence for the worked example effect (Sweller & Chandler, 1991) given animated demonstration. During week two, a similar MANOVA revealed no differences between groups.

While this study considered the learner performance from a human computer interaction (HCI) perspective, it also considered learners from a cognitive load perspective, by measuring relative condition efficiency (Paas and van Merriënboer, 1993). In addition, it developed a new measure called performance efficiency. During week one, the demonstration conditions were found to be significantly different F (2, 68) = 3.69, p=0.03, given relative condition efficiency. This is positive evidence of the variability effect. However in post hoc comparisons these instructional conditions were not found to differ. Performance efficiency was found to be significantly different, during week one, F (2, 68) = 12.95, p<0.0001, and post hoc comparisons with Scheffé’s test (p<0.05) revealed the demonstration learners were significantly more efficient, than the practice learners. During week two, groups did not differ, suggesting once learners had practiced procedures they would all perform equally well.

DISCUSSION

Those who studied animated demonstrations assembled the week one problem, in significantly less time than those who learned through problem solving. This result is consistent with the worked-example effect (Sweller & Chandler, 1991; Tuovinen and Sweller, 1999). In addition, Performance efficiency was found to be significantly different, since week one demonstration learners were significantly more efficient, than practice learners. Since week two performances did not differ, the findings of the current study do not support Palmiter’s animation deficit given animated demonstrations, further confirmation of the results by Lipps, Trafton, and Gray (1998).
In addition to finding evidence of the worked example effect, this study demonstrated the utility of a new metric called “performance efficiency,” and used this measure to compare several instructional conditions, to objectively analyzed the relative efficiency of learner performance.

Well-known companies, like Bank of America, Amazon.com and Microsoft, are all using animated demonstrations (demos) as a way to teach clients how to use their products and services. The results of this study show this practice improves learner performance and this method of instruction is an efficient means of teaching procedural skills. Therefore, it is the recommendation of this study that developers continue to use this effective and now evidence-based strategy, to provide learners with an efficient instructional strategy for accomplishing procedure-based learning.

REFERENCES

Adobe Systems. (2002). Adobe Photoshop Elements 2.0 [Computer program] Mountain View, CA

Cooper, G., & Sweller, J. (1987). Effects of schema acquisition and rule automation on mathematical problem-solving transfer. Journal of Educational Psychology. 79(4), 347-362.

Clark, R.C., Nguyen, F., and Sweller, J. (2006a). Efficiency in learning: evidence-based guidelines to manage cognitive load. San Francisco: Pfeiffer.

 Lewis, D. (2005). Demobank: a method of presenting just-in-time online learning. In the Proceedings of the Association for Educational Communications and Technology (AECT) Annual International Convention (vol 2, p. 371-375) October 2005, Orlando, FL.

Lipps, A. Trafton, J. and Gray, W. (1998). Animation as documentation: A Replication with Reinterpretation. In the Proceedings for the 45th annual conference of the Society for Technical Communication Arlington, VA. retrieved June 12, 2003 from http://www.stc.org/proceedings/ConfProceed/1998/PDFs/00006.PDF

Paas, F., Tuovinen, J. E., Tabbers, H. K., & Van Gerven, P. W. M. (2003). Cognitive load measurement as a means to advance cognitive load theory. Educational Psychologist, 38 (1), 63–71

Paas, F. G. W. C., and van Merrienboer, J. J. G. (1993). The efficiency of instructional conditions: An approach to combine mental-effort and performance measures. Human Factors, 35(4), 737-743.

Palmiter, S. & Elkerton, J. (1991). An evaluation of animated demonstrations of learning computer-based tasks. In the Proceedings of the SIGCHI conference on Human factors in computing systems: Reaching through technology. New Orleans, Louisiana, United States. p.257 - 263

Sweller, J., & Cooper, G. A. (1985). The use of worked examples as a substitute for problem solving in learning algebra. Cognition and Instruction, 2(1), 59-89.

Techsmith (2004).TechSmith Morae1.0.1[Computer program] Okemos MI