Closing the “ADDIE” Loop

Contrary to the belief that the interests of instructional design and learning analytics work at cross purposes, I present the case for analysis-analytics collaboration for the benefit of the learner. Hear me out, ye learned jury of courseware sponsors and learners, before you passeth judgment …

Opening Argument

Your Honors! In the case of the Instructional Designers versus Learning Analysts, I call upon the much-touted and oft-misinterpreted ADDIE model (acronym for “Analysis-Design-Development-Implementation-Evaluation” model) to bear final witness before my closing arguments on the necessity for Instructional Design (ID) and Learning Analytics (LA) to synergize in order to complete the learning cycle.

In these arguments, I aim to:

• Restate the ADDIE model based on Returns On Learning Investment.
• Describe the link between Learning Objectives and Business Results.
• Explain the feedback loop from Training Evaluation to Needs Analysis.
• Outline the steps for complementing Instructional Design and Learning Analytics.

ADDIE Model & ROLI

Even as instructional designers are aware of the ADDIE model, few are able to “design” and “develop” that enables “evaluation” by the learning analyst. Caught in the middle is the training manager sweating over “implementation”. Eventually, when the analyst waltzes in and “evaluates” the courseware performance, the feedback is quite … err … anti-climactic!

Now, why would any sane-minded instructional designer seek feedbacks that show them in not-so-rich light, you may ponder. Let me counter: why would they not, if the feedback could improve their future performance. Ergo, this means the contents of the analytics report must be predominantly forward-looking, NOT just a critique. The onus … ahem … is upon the analyst—or is it? Well, if you are one of those believers in reactive performance, rather than the proactive, then yes. Else, the courseware designer is better off checking with their courseware analyst PRIOR TO design.

Let’s take a look at the typical outlook of these courseware performance feedbacks—Learning Analytics Reports (LARs). These reports analyze the benefits accrued to either the learner or sponsor (or both). Whatever may be the perspective, LARs are essentially a documentation of the Returns on Learning Investment (ROLI)—how much positive/ negative returns accrued to the learner/ sponsor from the investments on courseware development, delivery, and evaluation. Here, may I quickly add that learning intangibles (such as, time spent in learning, cost of not training, etc.) are also counted in these investments and returns.

Wow! Given such expectations, how would the beleaguered designer analyze, design, and develop the courseware? Read on …

LOs & Business Results

Any instructional designer worth their salt would tell you (quite ceremoniously) that each and every courseware has a set of Learning Objectives (LOs), inviolable as they are. The designer would quickly add that each LO maps exactly to a chunk of subject matter—affectionately called the Learning Object (convenient, isn’t it?). These learning objects, in turn, must map to test items—practice quizzes and assessment activities, the designer would close. To drive home the point, the designer would emphatically draw an invisible line (in the air) connecting the aforementioned three thingies, and baptize the trinity as the “learning triangle”.

Enter the intrepid learning analyst, and the question, “OK, but what’s the ultimate consequence of this fancy Bermuda triangle?” is let loose not unlike a fox amid the placid chickens. Need I say that a feather or two is ruffled in the coop?

Modern instructional designers, thanks to analytics, are increasingly able to answer the analysts’ question. They assiduously make the connection between LOs, subject matter, test items, AND business results. Efficient designers, in fact, use the accrual of stakeholder benefits—“business results”—as a touchstone to validate the inclusion of each learning objective, object, and test item in the courseware. Why take the course if nobody likes it or learns from it; why learn if nobody uses it on the job or adds to the bottom-line, right?

From Evaluation to Analysis

Even as the learning analyst compiles the LAR of a training program, much thought goes into documenting both the observations-analyses (on past performance), as well as inferences-recommendations (on future performance). This 360-degree view of both the past and the future provides the instructional designer a holistic perspective on which to restart their “training needs analysis” phase.

The LAR feedback could be for one or both of these key stakeholders: courseware learner and training sponsor. The former is interested in whether the courseware was “likeable” and “learnable”, whilst the latter wants to know if the training was “implementable” on the job and “contributing” to the bottom-line. Those of you who know the Kirkpatrick’s model would readily recognize these stakeholder benefits as the “Famous Four” levels of the LA pyramid.

So, if you are sitting down to design a courseware, you may want to first analyze which of the above benefits are sought by your courseware stakeholder. If the LAR is already available, check whether the benefits have accrued; if not, ask why. Surely, you wouldn’t want to bark up the wrong tree, post design!

Design-Analytics Synergy

This cleverly drawn (I think) illustration depicts the two keys to successfully unlocking the learning benefits from a training program by the instructional designer and the learning analyst. For such a synergistic partnership to exist between these two competencies, each partner needs to follow four simple steps.

Let me start with the designer:

1. At the very start of design, ROLI measurement needs to be planned into the training program. For this, the designer may want to either consult with the analyst or training manager on how the program’s effectiveness is expected to be measured. The designer could seek answers to the following questions: “What are you going to measure—learner satisfaction, content learning, job implementation, or business contribution?” “Which design considerations would impinge on these measurements?” “What are the benchmarks for successful results?” and “How would the program’s effectiveness be measured?”
2. Next, the learning objectives need to precisely align with the business expectations unveiled from the above step. The designer could validate the necessity of each LO with this single, simple question: “If I delete this LO, will the business results get affected?” Should the answer be a “Yes”, move on to the next LO.
3. Then, the courseware components should be designed keeping in mind the development and delivery costs. For example, large page margins and fonts increase the number of pages to be printed by the instructor, not-so-educative animations and interactivities (Ah! the ubiquitous “bells-and-whistles”) skyrocket production costs and download times, complex equipment requirements constrain workshop facilitators, etc. Please be kind to downstream users.
4. Finally, when the LAR is received, the document must be seen as a process improvement proposal, not a faultfinding exercise. The observations-analyses sections should be objectively studied for historical insights, and the inferences-recommendations sections should be carefully analyzed for future design enhancements. Remember, these are “suggestions”, not “statutes”. Fair enough?

Now, the steps for the analyst:

1. Prior to commencing research, the instructional design of the training program needs to be thoroughly studied in order to deduce the needs of the learner (and the mind of the designer, perhaps). The analyst could look for answers to these questions: “What is the instructional goal of this courseware?” “Who are the target learners and how do they take this courseware?” “What are the different courseware components and why are they included?” and “Is the training program compact or unwieldy?”
2. Then, the connection between instructional design and business outcomes needs to be made. A simple question would suffice: “Who benefited how and what was the cost?”
3. Next, during data collection, both the tangible and intangible costs and benefits need to be defined and measured. For this, extensive liaising with both the number-crunchers (such as finance and accounts staff) and the people-interfacers (i.e., sales, HR, quality teams) is warranted … a tall order, I hope not!
4. Lastly, when compiling the LAR, the phraseology of the observations-analyses sections should be as quantitative and objective as possible, and the inferences-recommendations sections as optimistic and forward-looking as plausible. Focusing on the instructional design of the training program would lend significant scientific credence to the report. Hmmm, analyze this!

Closing Argument

I close by reiterating the premise that the ADDIE model is not a “waterflow” but a “lifecycle”—the quintessential component being the “feedback” from analytics to “analysis”. I exhort all you instructional designers and learning analysts to convert to the evolving paradigm that your competencies complement, not supplement, each other. Failing to appreciate this synergy could result in you two key stakeholders canceling out each other’s efforts to deliver learning value.

Your Honors—Learners and Sponsors—the defense rests!

Learning for Downtime

In the manufacturing industry, production downtime during annual shutdowns can cost millions. This is an excellent opportunity for L&D managers to contribute to the business bottom-line. Let’s say there’s a large petroleum refinery that requires a 15-day shutdown for its yearly maintenance operations. As the operators are predominantly trained and skilled (through repetition) on routine operations, the shutdown period is difficult for everyone, not to mention revenue loss. During shutdown, there are many planned and unplanned activities to be performed in a stipulated time. Everyone has to work harder to finish things faster. Operators perform several procedures at a time due to the additional work load. Though they may be trained during induction on all these procedures, they are generally not trained to take care of several procedures simultaneously. Consequently, work slows down—hence the 15-day shutdown. To compensate, production managers drive operators to work harder during downtime to restart the plant early.

However, learning consultants advise that exceptional skills be given more focus than routine skills, so there’s less wastage. During the initial orientation / training, when plant operators are trained on both routine and maintenance operations, the latter should be given more emphasis. Later, a couple of weeks before the scheduled shutdown, the L&D manager could make the operators take simulations or walkthrough of shutdown, maintenance, and restart procedures. This way, when the actual shutdown occurs, operators perform the same with equivalent speed as routine operations. Even a 20-25% reduction in downtime adds significantly to plant throughput and profitability.

To illustrate, large refinery companies like Reliance process about 0.66 million barrels of crude oil per day. Given that crude refining margins on refinery products is $11 per barrel, even if the L&D manager can save a day of turnaround, their direct contribution to EBITDA = $7.25 million or Rs.30 crores. The investment in such turnaround training would just be a fraction of the returns. In addition, the intangible benefits accrued could include:

• Reduction of stress in supervisor-operator interactions
• Increase in operator confidence
• Reduction of safety hazards

Learning Analytics Demystified

What you can’t measure, you can’t manage. Yes, we all know that, but seldom are able to translate this “knowledge” into “action”—especially when it comes to measuring the effectiveness of our training programs. Now, why is this so difficult, I wonder, when stalwarts such as Dr. Donald Kirkpatrick and Dr. Jack Phillips have already shown us the step-by-step way to measuring learning outcomes—three decades ago.

Hmmm … let me see! To start with, if you were the trainer, wouldn’t you be interested in knowing if your learners LIKED the training program? Kirkpatrick would say yes, and then defined this measurement as Level 1. Next, he wondered whether it would be relevant to measure if the learners LEARNED from the training program. That’s Kirkpatrick Level 2 for you. Then again, if you happened to be the line manager who set aside precious production time to send the learners to learn, you may be quite keen to demonstrate that your subordinates USED those learnings from the training program on their jobs. Eureka! That’s Level 3, said Kirkpatrick. Clearly, Kirkpatrick was left with some unanswered questions for the top brass (who happens to be looking over the shoulders of the line manager). Enter Level 4—Business Results—which answers the million-dollar question of the top boss, “Did all these training programs impact the corporate bottom-line?” Kirkpatrick was pleased with these four levels, and left at that.

Clearly, that wasn’t enough for Dr. Jack Phillips, Kirkpatrick’s student. With increased focus on corporate governance and scrutiny of, for, and by the Board of Directors, Phillips went for the jugular—Return on Investment!!! Topping the Learning Analytics (LA) pyramid, Dr. Phillips averred that enumerating the achievements of training programs at this level is the highest possible way to justify future investments. The Board smiled, the line manager squirmed, … and the trainer went pale.

OK, thanks for this fantastic pyramid, Drs. Kirkpatrick and Phillips, but how do I implement this on my training programs—you may ask. Worry not, and follow this 1-2-3 approach:

1. PREPARE: Plan for LA even before creating the training programs. Else, you will NOT be able to measure beyond Level 1. Get your training programs “designed” by professional instructional designers—not subject matter experts. Align the employee’s performance areas to the company’s result areas.
2. SELECT: Identify which courses you want to measure at what level. Use the one-half rule—measure all courses at level 1, 50% of courses at level 2, 25% of courses at level 3, 12.5% of courses at level 4, and 6.25% of courses at level 5. This is because the higher you climb the LA pyramid, the more expensive it gets; you may want to invest on your most strategic projects for ROI measurement.
3. IMPLEMENT: Measure at level 1 and 2 using in-house resources. For level 1, just collate the post-training feedback forms. For level 2, study the post-assessment results of your training program. At levels 3+, get an external consultant—this could save you from accusations of being “unscientific”, “biased”, and “political”, when the truth is found. In India, few companies, such as the Mumbai-based Aptech, offer LA consultancy.

Last, but not the least, if and when you catch the LA-tiger by its tail, please, please, please, remember the MOST important premise: “Learning Analytics is NOT a fault-finding exercise, but a learning business process improvement operation.”

From R&D to L&D (5)

Delivering for Business Results

After getting the right content in place, Bob has to arrange to deliver the training to his 2,000+ workforce spread across 12 different countries. This workforce spoke 5 different languages, like, Finnish, Chinese, Romanian, Brazilian and Russian. Training courses were hosted on the LMS of the organization. Bob formed a new team called “Learning Task Force” (LTF), comprising Trainers, Mentors and LMS administrators.

Coaching took the form of Instructor-Led Training (classroom based), Web-Based Training, and Teleconferences, as well as posting of exercises and questions to a discussion database. Internal customers were quick to see the value-add, and qualitative feedback was positive. Many employees started to lean on the training staff as an internal help desk. The online courses could be taken anytime anywhere in 5 different languages with multiple users logging into the same course. They were modular and self-paced. Industry experts and renowned professors were roped-in as online mentors with whom the employee could “chat” to discuss topics.  This made the online courses very popular amongst the employees.

The initial challenge faced by Bob was getting employees attend the classroom and blended training programs. Learners tended to procrastinate and needed a deadline (the end of the session). To create some early success, Bob provided incentives for learners to participate and complete the program, on enrollment some trinkets/cufflinks were given away. Then the employees were allowed to take a trial class on elearning to make sure the online format would suit their needs. Once they were ready, participants were asked to commit to a learning agreement. Part of the agreement was that learners had to complete the program by a specified date, including filling out an evaluation form, in order to gain access to the remaining classes. Finally, all employees needed to have their manager’s approval—and support—to take these courses during work hours. Combined, these efforts guaranteed some rather high completion rates!

The LMS gave the LTF a complete control on tracking the progress of the users and their evaluation. Employees saw value in this format of training and nearly 60% completed the courses assigned to them.

Happy employees make happy customers; this was learnt the hard way at Acme. Bob has now filled the skill gap that existed in his employees, but it’s of paramount importance to measure what is the success of this initiative.  As rightly said, “what can’t be measure can’t be managed” … Find out in my next post how Bob implements the measurement tools to training …

“Oh so ezee, LA for me!”

A certain Training & Development manager of a leading financial institution wanted to “measure the effectiveness” of her training programs, but did not know how. As she was a potential customer, and as I am the “technical expert”, our Sales team insisted that I “educate” this young (and purportedly, inexperienced!) T&D manager on the basics of learning analytics (LA).

On the appointed day, as I gently walked her through Kirkpatrick’s theory and Phillips’ methodology, I was quite amused by her periodic quips of “Ah, that’s easy!”, “Oh, that’s simplistic enough …”, and “Hmm! is that all?” I knew where this was going, but my poor Sales colleague was stumped when she finally declared, “Learning analytics is so easy that even we can do it in-house!!!”

The learned lady was evidently pleased with her grasp of the theory, blissfully unaware of the fact that it’s not the ideation but the implementation that makes or breaks measurement programs (and consequently L&D careers) in organizations. Respecting Knowles’ postulates of adult learning (experience, including mistakes, form the basis for learning), I wished her the very best.

From R&D to L&D (4)

Developing Quality Content

Now that the board is convinced about going the eLearning way, Bob has the Herculean task of developing quality content “in-house”. This is a crucial step as it is the stepping stone to realizing the eLearning vision of the organization. Right content is the key to right output!

Bob sets up a team of Subject Matter Experts (SMEs) from across the functions, such as production, sales, quality, logistics, marketing and customer services. This he christens as the “Content Engineering Team” (CET), which brings to the table decades of experience in their respective functions and industry.

The first task at hand for Bob is to list the various classroom and eLearning courses that need to be developed for this new training initiative. He gets his team to study the requirements against the currently run courses and the market need. The study reveals that the curriculum needs major revamping in terms of new topics to be introduced. Dozens of duplicated, underused classes were culled from the roster, and predominantly classroom-based courses were transformed into blended solutions of Web-delivered courses and traditional training approaches.

The CET is in action to contribute and gather course material for the training programs. Bob co-ordinates with various departments to get the content out. The major challenge faced by Bob was getting the content developed on time. The CET could not give 100% to developing content as their primary task would set different priorities.

Bob moved around like a headless chicken to get the content out. But he did not lose heart. He got the team work extra after-office hours and during weekends to get the right content out. The content went through a series of validations and treatments before it finally shaped up.

With the content in place, the next step was to put it in a presentable form of learning. Screen captures and photographs were clicked with handicams and mobile cameras. The courses were thus developed in ILT and WBT formats.

Bob is now half way through in realizing the vision of the organization.  How will he proceed further with implementation? Would he experience the same success as earlier in delivery of the elearning training? Find this out in my next post…

Best practices are for the incompetent?

At the recently concluded TOI HR Forum in Mumbai, K Ramkumar, HR Head of ICICI Bank, raised an imperious eyebrow or two from amongst the audience when he declared, “Best practices are for the incompetent!”

The seasoned orator that he is, Ramkumar did warn the audience beforehand that he was about to make a “provocative statement”. The statement surely is provocative enough for me. What about you? Do share your thoughts on Ramkumar’s quotable quotes either through our latest poll or by posting a comment to this post.

From R&D to L&D (3)

Managing Resistance to Change

A major change from traditional training programs to eLearning and new genre of learning proposed by Bob generated a lot of resistance and reservations among the management. The management strongly felt that this new-fangled e-learning would be a nonstarter for the organization. A section of the Board didn’t want employees to take responsibility for managing their own development because of “time-tested, conservative” views. Some argued on costs, while others on implementation strategies. They showed apprehensions towards this new format of L&D.

Bob’s evangelism on eLearning convinced the Board that the strategy would involve a blended form of classroom and online training, thereby creating a hybrid learning environment in the organization. He brought out examples and research studies that corroborated his claim that the eLearning initiative is not a mere training program but a successful, feasible and financially viable business strategy that could change the organizational structure for the better. He proposed a phased transition into eLearning over a 1-year period. He detailed out the benefits that eLearning would bring to the organization like, optimizing the cost, bringing in training consistency, reaching a wider audience and, importantly, not taking people off their work to undergo training.

Bob also campaigned for the acquisition of a Learning Management System (LMS), which would bring in comparable benefits of classroom interactivity, by enabling the learner to interact with his/her tutor by mail/chat. To tackle the issue of making the employees buy-in this new mode of training, Bob suggested circulating internal monthly newsletters to highlight usage increments, feature different course each month and promote learning groups.

Bob seems to have convinced the Board to go the eLearning way, but what would he do next? How would he implement this vision for the organization? Find out in the next post…

From R&D to L&D (2)

Aligning L&D Goals to Corporate Goals

While the Directors were frantically thinking of ways to improve the bottom line, the HR Head, Kim Foster had been thinking over a possible way out to improve business performance. She could see Business development through PEOPLE DEVELOPMENT.
The Board after a detailed discussion — decided on a paradigm shift from focus on R&D to L&D.
Bob White the L&D manager was entrusted the responsibility strategizing an L&D program that would align with corporate goals. Aligning L&D Goals to Corporate Goals was something that Bob never thought of. He was completely at sea, he always understood that L&D was to provide information, skills and change attitude but never linked it to corporate goals.
Kim asked Bob to come with an L&D plan for the organization that would position the employees and their services as a differentiated factor in the marketplace. After pondering over it for days, Bob finally came up with an L&D strategy to defend his company from the shackles of competition and dipping market share.
This strategy involved a major revamping of the present Training and Development program. E-learning was introduced as a new format of learning, which was self paced and engaging.

How would the board react to this new eLearning concept of learning proposed by Bob and how Bob champions this find out in the next post…

HR – Key to Quality Manpower

Today, paucity of talented resources is a key issue the corporate world is struggling to overcome. Companies recruit employees just to add up the numbers. If you ask any organisation, they will say they are short of “N” hundred/ thousand resources. The widely used corporate parameter to measure erosion in resources is: percentage attrition – again just another number. So, can this scenario change? Can organisations afford to just keep on giving higher and higher salaries to retain talented resources or poach talented resource from competition? Again, recruiting from another company means shifting the fundamental problem.

The solution to this problem is: hire and groom resources. In the modern corporation, HR’s responsibility goes beyond adding numbers to the resource pool. Comprehensive, enterprisewide learning solutions, which are effecient in the short and effective in the long term, is the way forward …

Lean manufacturing shows us the way!