Closing the “ADDIE” Loop

Contrary to the belief that the interests of instructional design and learning analytics work at cross purposes, I present the case for analysis-analytics collaboration for the benefit of the learner. Hear me out, ye learned jury of courseware sponsors and learners, before you passeth judgment …

Opening Argument

Your Honors! In the case of the Instructional Designers versus Learning Analysts, I call upon the much-touted and oft-misinterpreted ADDIE model (acronym for “Analysis-Design-Development-Implementation-Evaluation” model) to bear final witness before my closing arguments on the necessity for Instructional Design (ID) and Learning Analytics (LA) to synergize in order to complete the learning cycle.

In these arguments, I aim to:

• Restate the ADDIE model based on Returns On Learning Investment.
• Describe the link between Learning Objectives and Business Results.
• Explain the feedback loop from Training Evaluation to Needs Analysis.
• Outline the steps for complementing Instructional Design and Learning Analytics.

ADDIE Model & ROLI

Even as instructional designers are aware of the ADDIE model, few are able to “design” and “develop” that enables “evaluation” by the learning analyst. Caught in the middle is the training manager sweating over “implementation”. Eventually, when the analyst waltzes in and “evaluates” the courseware performance, the feedback is quite … err … anti-climactic!

Now, why would any sane-minded instructional designer seek feedbacks that show them in not-so-rich light, you may ponder. Let me counter: why would they not, if the feedback could improve their future performance. Ergo, this means the contents of the analytics report must be predominantly forward-looking, NOT just a critique. The onus … ahem … is upon the analyst—or is it? Well, if you are one of those believers in reactive performance, rather than the proactive, then yes. Else, the courseware designer is better off checking with their courseware analyst PRIOR TO design.

Let’s take a look at the typical outlook of these courseware performance feedbacks—Learning Analytics Reports (LARs). These reports analyze the benefits accrued to either the learner or sponsor (or both). Whatever may be the perspective, LARs are essentially a documentation of the Returns on Learning Investment (ROLI)—how much positive/ negative returns accrued to the learner/ sponsor from the investments on courseware development, delivery, and evaluation. Here, may I quickly add that learning intangibles (such as, time spent in learning, cost of not training, etc.) are also counted in these investments and returns.

Wow! Given such expectations, how would the beleaguered designer analyze, design, and develop the courseware? Read on …

LOs & Business Results

Any instructional designer worth their salt would tell you (quite ceremoniously) that each and every courseware has a set of Learning Objectives (LOs), inviolable as they are. The designer would quickly add that each LO maps exactly to a chunk of subject matter—affectionately called the Learning Object (convenient, isn’t it?). These learning objects, in turn, must map to test items—practice quizzes and assessment activities, the designer would close. To drive home the point, the designer would emphatically draw an invisible line (in the air) connecting the aforementioned three thingies, and baptize the trinity as the “learning triangle”.

Enter the intrepid learning analyst, and the question, “OK, but what’s the ultimate consequence of this fancy Bermuda triangle?” is let loose not unlike a fox amid the placid chickens. Need I say that a feather or two is ruffled in the coop?

Modern instructional designers, thanks to analytics, are increasingly able to answer the analysts’ question. They assiduously make the connection between LOs, subject matter, test items, AND business results. Efficient designers, in fact, use the accrual of stakeholder benefits—“business results”—as a touchstone to validate the inclusion of each learning objective, object, and test item in the courseware. Why take the course if nobody likes it or learns from it; why learn if nobody uses it on the job or adds to the bottom-line, right?

From Evaluation to Analysis

Even as the learning analyst compiles the LAR of a training program, much thought goes into documenting both the observations-analyses (on past performance), as well as inferences-recommendations (on future performance). This 360-degree view of both the past and the future provides the instructional designer a holistic perspective on which to restart their “training needs analysis” phase.

The LAR feedback could be for one or both of these key stakeholders: courseware learner and training sponsor. The former is interested in whether the courseware was “likeable” and “learnable”, whilst the latter wants to know if the training was “implementable” on the job and “contributing” to the bottom-line. Those of you who know the Kirkpatrick’s model would readily recognize these stakeholder benefits as the “Famous Four” levels of the LA pyramid.

So, if you are sitting down to design a courseware, you may want to first analyze which of the above benefits are sought by your courseware stakeholder. If the LAR is already available, check whether the benefits have accrued; if not, ask why. Surely, you wouldn’t want to bark up the wrong tree, post design!

Design-Analytics Synergy

This cleverly drawn (I think) illustration depicts the two keys to successfully unlocking the learning benefits from a training program by the instructional designer and the learning analyst. For such a synergistic partnership to exist between these two competencies, each partner needs to follow four simple steps.

Let me start with the designer:

1. At the very start of design, ROLI measurement needs to be planned into the training program. For this, the designer may want to either consult with the analyst or training manager on how the program’s effectiveness is expected to be measured. The designer could seek answers to the following questions: “What are you going to measure—learner satisfaction, content learning, job implementation, or business contribution?” “Which design considerations would impinge on these measurements?” “What are the benchmarks for successful results?” and “How would the program’s effectiveness be measured?”
2. Next, the learning objectives need to precisely align with the business expectations unveiled from the above step. The designer could validate the necessity of each LO with this single, simple question: “If I delete this LO, will the business results get affected?” Should the answer be a “Yes”, move on to the next LO.
3. Then, the courseware components should be designed keeping in mind the development and delivery costs. For example, large page margins and fonts increase the number of pages to be printed by the instructor, not-so-educative animations and interactivities (Ah! the ubiquitous “bells-and-whistles”) skyrocket production costs and download times, complex equipment requirements constrain workshop facilitators, etc. Please be kind to downstream users.
4. Finally, when the LAR is received, the document must be seen as a process improvement proposal, not a faultfinding exercise. The observations-analyses sections should be objectively studied for historical insights, and the inferences-recommendations sections should be carefully analyzed for future design enhancements. Remember, these are “suggestions”, not “statutes”. Fair enough?

Now, the steps for the analyst:

1. Prior to commencing research, the instructional design of the training program needs to be thoroughly studied in order to deduce the needs of the learner (and the mind of the designer, perhaps). The analyst could look for answers to these questions: “What is the instructional goal of this courseware?” “Who are the target learners and how do they take this courseware?” “What are the different courseware components and why are they included?” and “Is the training program compact or unwieldy?”
2. Then, the connection between instructional design and business outcomes needs to be made. A simple question would suffice: “Who benefited how and what was the cost?”
3. Next, during data collection, both the tangible and intangible costs and benefits need to be defined and measured. For this, extensive liaising with both the number-crunchers (such as finance and accounts staff) and the people-interfacers (i.e., sales, HR, quality teams) is warranted … a tall order, I hope not!
4. Lastly, when compiling the LAR, the phraseology of the observations-analyses sections should be as quantitative and objective as possible, and the inferences-recommendations sections as optimistic and forward-looking as plausible. Focusing on the instructional design of the training program would lend significant scientific credence to the report. Hmmm, analyze this!

Closing Argument

I close by reiterating the premise that the ADDIE model is not a “waterflow” but a “lifecycle”—the quintessential component being the “feedback” from analytics to “analysis”. I exhort all you instructional designers and learning analysts to convert to the evolving paradigm that your competencies complement, not supplement, each other. Failing to appreciate this synergy could result in you two key stakeholders canceling out each other’s efforts to deliver learning value.

Your Honors—Learners and Sponsors—the defense rests!