E-Learning Concepts and Techniques

Chapter 9 – E-Learning Evaluation

Bonus Podcast

An interview with professor, Dr. Mary Nicholson, of the Evaluating E-Learning course at Bloomsburg University can be found at:

To Chapter Menu

9.1 - Introduction to Evaluating E-Learning

Amy Roche

An essential aspect of an e-learning course or curriculum is the evaluation of e-learning. Of course you are probably asking why? Well, there are numerous reasons and benefits for doing so. First and foremost is that evaluation is part of any instructional design model. For example, the highly popular ADDIE model stands for analysis, design, development, implementation, and EVALUATION. Without completing this portion of the instructional design model the e-learning course is incomplete. Evaluation is a key aspect of any instructional design model due to the fact that the course cannot be tailored, redesigned, and improved upon unless this is done. Evaluation consists of numerous attributes; however it basically means assessing the effectiveness and possible improvement of a course/curriculum.

Now you might be thinking... Now I know that evaluation is part of an instructional design model, but why is it included in these models? Well the reasons are numerous and most reasons directly relate to the improvement of course/curriculums. This can include making sure the training is liked by the learners, ensuring that the learners gained information in the process, assure that learners are accountable for the information they obtained in the training, assess learning outcomes, and find and fix quality issues in the training as well as learn how to make training courses and curriculums better in upcoming projects. Each of these reasons can improve the training if fixed. For example, if after the evaluation the training is altered to make sure the training is liked by the learners it can lead to an increase participation in the training, an increase in learner retention, ensuring that it accommodates different learning styles, etc.

In addition to increasing the quality of training another critical reason for evaluation is to assess the value of the training. This is essential because training is a part of business. If it is not deemed valuable within the organization, then the amount spent on training is often reduced. Reasons for evaluation in this particular area include adding value to the organization, justifying the investment in training, assessing the effect the training has on profitability, the impact the training has on employee's work habits, effectiveness and efficiency of the training, assessing the effect of customer satisfaction from the training, etc.

Now that you know the answer to why evaluate e-learning you might be asking who, what, when, and where to evaluate. Well, this depends upon how you evaluate e-learning. There are numerous ways to do this and in this chapter we will discuss some ways including Kirkpatrick's Levels of Evaluation, ROI, Metrics and Learning Analytics, Balanced Scorecards, and Evaluation Plans. When going through these ways to evaluate e-learning remember the reasoning and justification for evaluating e-learning and how it can pertain in real world situations.

To Chapter Menu

9.2 – Kirkpatrick's Four Levels of Evaluating Learning

Amanda Ireland and Andrea Mummert with Mike Moran

9.2 Overview

In education and learning new methods of thinking are constantly generated by people who think outside the box. This is why it may seem unusual that a model for assessing training effectiveness that was introduced nearly fifty years ago is still highly regarded and considered to be one of the best.

In 1954, a doctoral candidate at the University of Wisconsin in Madison, Donald L. Kirkpatrick completed his Ph.D. dissertation entitled Evaluating a Human Relations Training Program for Supervisors. Based on that dissertation, he wrote a series of articles for the American Society of Training Directors (now ASTD) beginning in 1959. The articles were entitled: Evaluating Reaction, Evaluating Learning, Evaluating Behavior, and Evaluating Results. According to Kirkpatrick, At that time training professionals were struggling with the word 'evaluation.' There was no common language and no easy way to communicate what 'evaluation' meant and how to accomplish it. Trainers began to use his four levels and passed them along to other professionals. The four levels together afford the trainer a meaningful evaluation of the training by looking at it from various angles.

To Chapter Menu

Level One: REACTION

What is reaction in training evaluation? Simply put, it reports if participants liked or disliked the training. This would resemble a customer satisfaction questionnaire in a retail outlet. At the First Level of evaluation, the goal is to find out the reaction of the trainees to the instructor, course and learning environment. This can be useful for demonstrating that the opinions of those taking part in the training matter. A Level One evaluation is also a vehicle to provide feedback and allows for the quantification of the information received about the trainee's reactions.

The intent of gathering this information is not to measure what the trainee has learned, but whether the delivery method was effective and appreciated. Non-training items may have a deep impact on the training session and need to be considered. These items include, but are not limited to environmental and other conditions surrounding the learner at the time of training. Level One questions might include the following:

In gathering the data for this first step, it is important to do so soon after the training is completed. It is most presented as a form to be filled out by the learner. The following are some methods used to collect the data for Level One:

Benefits of gathering Level One information are far-reaching. For example, the trainer or instructional designer may be misled into believing there is a shortcoming in the material presented, when it may have simply been an environmental issue. The data can be gathered immediately and most trainees participate readily because the information gathered is non-threatening and shows concern for their feelings. The information, in addition to ease of gathering, is not difficult to analyze. Finally, when a current group is relating a positive experience, other potential trainees are more at ease with a decision to learn.

To Chapter Menu

Level Two: LEARNING

As stated in the above section, Kirkpatrick's Level One Evaluation assesses the reaction of the trainees. Kirkpatrick's Second Level stretches beyond reaction and assesses the learning, also known as knowledge, skills and attitude (KSA) of the learner. More specifically, Level Two data can describe the extent to which participant attitudes changed and if relevant knowledge and skills were increased by the training. Level Two data is valuable for answering the basic question Did the participants learn anything?

The measurement methods of Level Two tend to require more time, effort, and care than Level One. Some methods used to evaluate Level Two are mentioned in the following list:

The pretests are often used to determine the knowledge of the content before the training. The post tests are used to measure the amount of knowledge/understanding of the content after the training. The pretest and posttest are developed before the content is complete. This will ensure that the content meets the learning objectives. The score of the pretest and post test are summarized so that the trainers can monitor if the training has made an impact on learning. Interviews and observations can be useful, but it should be considered that this data could be subjective and may reflect other factors that do not apply to this level of evaluation.

To Chapter Menu

Level Three: PERFORMANCE

Level Three of Kirkpatrick's Evaluation Model incorporates Level One and Two and extends it one step further. Level Three measures the direct correlation between KSA and the behaviors of the learner in their daily job. Level Three Behavior can also be the most validating assessment for the training program's effectiveness.

Level Three evaluations normally take place three to six months after the training has occurred. By waiting three to six months the learners are given an opportunity to implement the new skills/knowledge learned in the training. It is nearly impossible to pinpoint when the transfer of knowledge/behaviors takes place, therefore the trainers/mangers have important decisions to make when considering evaluation. When making this decision the trainers/managers must keep in mind the following factors:

The evaluation methods for Level Three Behavior are as follows:

The observations are performed by the supervisors/managers to observe that the knowledge, behavior, and skills are being applied to their daily work. The interviews are a useful resource but can be a time consuming way to gather the information especially if it is a large organization/company. A survey can acquire sufficient information, as long as the questions are asked appropriately. The most recent addition to methods of gathering Level Three information is coaching. Coaching or Performance Coaching employs a change agent who has the responsibility of demanding and driving behavior and performance changes. The demand and drive of behavior is done in a supportive yet challenging way.

To Chapter Menu

Level Four: RESULTS

Kirkpatrick's fourth and final level of evaluation involves results – the impact that can be derived from training. Level Four Evaluations can produce data that can, in addition to the other three levels, give concrete evidence as to overall value of the training program. Results of a Level Four Evaluation can be specifically useful when reporting and achieving the buy-in of higher level management. The data can also be used to suggest or justify further training efforts and investments.

Level Four Evaluations can produce hard data on such factors that relate to cost, quality, and morale. Data from Level Four is often collected from management and reporting systems that are already in place within an organization. Examples of specific tangible measures that can result from Level Four Evaluations are listed below along with measures that would be considered intangible.

Tangible results
Intangible results

Although the data may be seemingly easy to gather, Level Four Evaluations are the least likely to be conducted within an organization. There are several factors that attribute to this finding:

Kirkpatrick, however, clearly designated that by completing each of the Four Levels of Evaluation would give evaluators a well-rounded indicator as to the value of a training program.

To Chapter Menu

9.2 Summary

Kirkpatrick's Four Levels of Evaluation have consistently proven since their creation, that each level has particular benefits and unique challenges. As the Level of Evaluation increases, the complexity and difficulty of data and data collection also increases. Keep in mind however, that while the higher levels may require more cost, time, and complexity, they also result in the most valuable measurements that a training program could benefit from. Despite time and new evaluation innovations, Kirkpatrick's idea still remains one of the most widely used models of evaluation today.

To Chapter Menu

9.2 Resources

To Chapter Menu

9.3 – Learning Analytics

Kristin Longenecker with Vincent Basile and Pete Mitchell

9.3 Introduction

We function in a results-oriented business environment. Organizations are under constant pressure to demonstrate that their investments of time and money in projects and processes produce measurable benefits. Training expenditures are still seen by many companies as a cost, not an investment. In many cases, these costs are considered to be money lost. Many feel that organizations are only able to appreciate the value of their training expenditures when they can calculate the money saved, or perhaps even earned, by investing in training solutions. Learning analytics supports this by using business analysis practices to study the impact of learning on learners and the organization.

To Chapter Menu

Why Learning Analytics?

Training expenditures can be a huge investment for any organization. When properly utilized, learning analytics can help management demonstrate fiscal responsibility while providing justification for training budgets. It gives an organization the information needed to make more effective training decisions. The information derived from learning analytics can also help guide decisions regarding the type, format and content of training programs. Learners can also be held accountable for their attendance and level of participation.

To Chapter Menu

What to measure?

The critical areas of learning analytics measurement are efficiency, effectiveness, and compliance. Training decisions most likely fall into one of these three areas, efficiency, effectiveness and compliance.

Efficiency measures are concerned with the dollar amount spent on learning in the organization. These measurements include return on learning investment, learning dollars spent per employee, hour, or per course.

Effectiveness measures are concerned with the outcomes of the learning. These measures include course completion rates, certification rates, and measurable improvements due to specific training programs such as increased sales or reduced number of accidents.

Compliance issues are becoming increasingly important for all types of organizations. Compliance measurements include certification rates, compliance percentages either across the organization or in individual areas, and can even track the organizations risk of falling out of compliance and indicate the areas that need to be improved.

To Chapter Menu

Planning for Learning Analytics

An old proverb states, failing to plan is planning to fail. In any endeavor, planning allows us to determine where we wish to go and how we're going to get there. When implementing a learning analytics program, planning is of critical importance.

Start by referring to your organization's goals and strategic vision. These overall goals are used to determine an organization's operational strategies. Training efforts should, in turn, be linked to these goals and strategies. Your learning analytics program should be designed to provide key measures that show the connection between training efforts and meeting those goals.

It is essential to obtain executive support for the learning analytics program. One way to achieve this is to educate your organization's executives on Kirkpatrick's levels of evaluation. Despite much well-publicized information to the contrary, many executives still believe that training evaluation is limited to smiley-face forms (Kirkpatrick Level 1) filled out at the end of a lecture. They have very little understanding of the usefulness of information that is provided at the Learning (Level 2), Performance (Level 3), Business Impact (Level 4) and ROI levels. Present learning analytics as one more tool that shows a management commitment to fiscal responsibility. Showing a direct link between learning analysis measures and operational strategies and goals will further underscore the value of the program.

Many new programs receive administrative approval, however, only to die on the vine due to a lack of ongoing budget support. Once executive support has been obtained, funding for learning analytics should be included in the budget process. This should include start-up costs and ongoing evaluation costs. The percentage of a training budget that should be directed to the learning analytics program relates directly to the scope of the program. Are you interested in evaluating the benefits from a single course, a multi-course training program, or all training efforts for your organization? Generally, the wider the scope of the analytics program, the lower the level of evaluation performed. Partly due to their higher cost and increased difficulty of application, higher level evaluations tend to be focused on smaller training areas.

In many cases, flowcharts and other types of graphic aides can provide a visual guide to the process as you work to determine your information goals and the best way to achieve them. Treat your learning analytics as any other business project. Consider all aspects and develop a project timeline before committing resources.

To Chapter Menu

Implementing Your System

Managing change is a challenge that organizations face every day. Once you have made the decision to implement a learning analytics program, it's important that management interest and support remains positive and visible. Take steps to identify the stakeholders in the program. Their input is essential in both the planning and implementation phases. Your stakeholders may include managers, supervisors and staff members. They may come from operations departments, your training department, clients and contractors. All will have a different point of view that should be considered as you develop and implement your program.

Recognize that some stakeholders may view the gathering and analysis of this data as a threat. Some individuals may become quite vocal about their concerns. Others may hold their concerns back, while maintaining a passive resistance to the change. Take steps to minimize these concerns by bringing stakeholders into the implementation process as soon as possible. Maintain a continued emphasis on the projected benefits of the program. As always, frequent and open communication is an essential and effective tool to use when winning support.

In order to determine the type of analysis that will be performed, it's necessary to identify the questions that you wish to ask and the type and sources of the data you will need. In most cases, data from the lower Kirkpatrick levels is easier and less costly to collect and analyze. If you do use smiley-face forms, the information that you obtain from them is still useful. In addition, you may also wish to gather data from other existing sources. If your organization uses a learning management system (LMS), for example, you may already have a considerable amount of data available on course completion, test scores, etc. By starting with the information that you have, you support familiarity with the program and make the transition much easier. As your organization's comfort with the learning analysis process grows, you can begin to address information that relates to higher levels of learning evaluation.

To Chapter Menu

Custom-Built or Off-the-Rack? Making the Decision

If you are implementing a vendor-supplied system, key members of your technical staff will need to work closely together with their representatives. If you are implementing an in-house developed system, you'll need to assemble a team consisting of individuals from management, plus staff from your training and IT departments. In either case, close work by team members is needed to determine your most effective implementation plan.

Most organizations spend a significant part of their time emphasizing the factors that make them unique. It's only natural, therefore, to assume that you will need a unique solution to meet your learning analytics needs. In some cases, this may be the best approach. Before making this decision, however, several factors should be considered.

Does your company have sufficient internal expertise in both the development of the required technology and the learning analytics process? Do you have the ability and resources necessary to keep up to date with new developments in the learning analytics field? Companies that meet these criteria generally have a core business focus that involves software system development and application.

How do the costs associated with initial development and implementation of the program compare with the cost of a solution provided by an outside vendor? Further, how do ongoing costs of system maintenance and upgrades compare?

Does development of an internal solution give your company a competitive advantage? This may be related to improved managerial and operational efficiencies, or to an ability to market your proprietary system as another factor that differentiates you from your competition.

Do you have a need to share information with organizations with which you have a direct link or with other organizations with whom you have partnered? In some cases, this is facilitated by the agreement to use a common learning analytics platform.

To Chapter Menu

Data Storage

The manner that learning analytics data is stored within an organization is crucial. There are several factors to consider. First, the structure of the training program should be examined. If the courses in your training program are organized hierarchically, data from those courses should be similarly organized. By accurately reflecting the course and program structure in your data storage plan, you can measure the effectiveness of a small piece of the program (such as one course), a series of related courses, or of the entire program.

The information that is gathered should be directed into one centralized location. This pulls all of the pieces together and helps make evaluation of the information easier for the organization or outside vendors. When storing large volumes of data, it can also be helpful to come up with a plan of maintenance. Even the best designed analytics engines can slow down over time if a plan to maintain this data is not established early in the process. Hourly or daily backups of the database should be conducted to prevent any loss of information.

To Chapter Menu

Data Processing

A key factor when processing stored data is how it will be formatted. Standardizing files to a specific format can be advantageous in terms of exporting and analyzing information. The intervention of an Analytics Tool can expedite this process greatly. There are numerous on-line analytic processing (OLAP) tools designed to do this very task. When selecting one of these tools be sure to consider price, functionality, and the ability to filter criteria such as instructor, course, and program. For example, XML is a very powerful tool that can help structure and organize information. A standard system of XML tags can be developed to help export data to a given analytics tool for processing. This system can then easily be modified as the needs of the organization evolve.

To Chapter Menu

Data Reporting

When reporting the findings of a given request, the presentation of the information is vital. In most cases, the use of appropriate charts and graphs can represent the data more effectively than text-only reports. They give us the ability to quickly interpret figures, so that what is being represented is more easily understand. This, in turn, makes evaluating performance much more effective.

Reporting software may also require maintenance. When this is the case, it is very important that the need for this is accurately communicated to its users. Downtime due to maintenance should take place during the most convenient times for the report users. This way the reporting process can be carried out most efficiently in terms of time and available resources.

To Chapter Menu

The Learning Dashboard

A dashboard is the name given to web-based tools that are used for reporting information in a concise and easily accessed manner. In this case, your dashboard is the part of your data reporting system that provides a quick summary of the information gained through your learning analytics process.

When selecting software, the presence of a dashboard is a very important feature. The purpose of this interface is to display measurements of key factors that help you quickly evaluate your training program. The more clearly this information is displayed, the more quickly and easily your training can be evaluated.

Most software comes with a beginning template to help design your dashboard. Be sure to use graphs with time axis and gauges. This allows the measurements of key factors to be shown over time. Keep in mind that the goal of your learning dashboard is to allow you to assess your training program at a glance and you'll be right on track.

To Chapter Menu

9.3 Conclusion

In an article on the Knowledge Advisors website, Jeffrey Berk referred to learning analytics as, the process by which learning professionals analyze critical indicators within their business to not only continuously improve but to demonstrate value to stakeholders and make better decisions to optimize learning investments.

In other business areas such as finance, inventory control, manufacturing and sales, analytics are tools that have long been used for information gathering and decision-making. The increasing interest in learning analytics brings the power of these tools to the training field.

To Chapter Menu

9.3 References

To Chapter Menu

9.4 – Balanced Scorecards

Amy Roche

9.4 Introduction

A Balanced Scorecard is a framework for using the organization's strategic business objectives and applying them as a set of performance indicators that measure the success of the organization. The success is indicated from four perspectives including Financial, Customer, Internal Business Processes, and Learning and Growth. The Balanced Scorecard relates these four different areas with each other and creates a dynamic relationship.

To Chapter Menu

History of the Balanced Scorecard

The Balanced Scorecard was created by Robert S. Kaplan and David P. Norton in 1992. It originally was created to measure private industry financial and non-financial performance and was created because Kaplan and Norton wanted a solution to the weaknesses of previous management systems. Kaplan and Norton wanted a clear way to measure finances in a balanced format. This was done by having the four perspectives be in balance with one another in terms of finance. In order to create the Balanced Scorecard, Kaplan and Norton used the Hoshin Planning as a basis. The Hoshin Planning is an organization-wide strategic planning system that is used throughout Japan. The Balanced Scorecard eventually adapted to become a Performance Management system for both private and public organizations. In addition the emphasis switched from financial and non-financial performance to business strategies.

To Chapter Menu

Reasons to use the Balanced Scorecard

Balanced Scorecards are used for numerous reasons. First and foremost it defines the organization's business strategy, facilitates organizational chance, and measures performance. In addition, the Balanced Scorecard assesses the organization on all levels and across the entire organization. By doing so the Balanced Scorecard strengthens the unity between the different perspectives. The Balanced Scorecard helps drive performance on all the levels that, in turn, improve bottom line results. This is done by reducing cost and improving productivity. In addition, the Balanced Scorecard aligns the strategic activities to the strategic plan. This allows the real deployment and implementation of the business strategy and the organization can get feedback on their strategy and alter it accordingly.

In addition, the Balanced Scorecard collects data that can be used in numerous ways. One way is to assess performance of different geographical locations that can lead to the identification of gaps and allows them to be fixed. Data collected allows the identification of best practices within an organization and expanding these best practices throughout the entire organization. The data collected is concrete and provides a rational basis for making decisions on budgets and control of processes. Along the same lines, there is accountability and incentives based upon real data and not subjective numbers. The data also allows for a basis for deciding which improvements of the organization need to be made first and how to implement resource allocation processes.

The Balanced Scorecard allows for the measurement of the organization in other ways. This includes ensuring that the correct measures are included in the business strategy, identifies key measures that are missing, tracks the key measures of the business strategy, and allows for benchmarking performance against outside competitors. Other uses of the Balanced Scorecard include the encouragement of good management and an increased understanding of the organization's business strategy and the organization itself.

To Chapter Menu

The Four Perspectives

As previously stated the four perspectives include Financial, Customer, Internal Business Processes, and Learning and Growth. The key behind the Business Scorecard is to keep these four perspectives in balance with one another. To fully understand the Balanced Scorecard it is essential to learn more about these areas.

The Financial perspective focuses upon financial objectives and their representation of long-range profit goals. In government or educational settings, finances are measured in goals of efficiency and maximizing the return on investment. In other organizational settings, finances are measured on bottom dollar results. Within any business funding and profits will always be a priority; however if this is too much of a priority it can be off balance with the other perspectives and actually hinder the organization instead of helping it. Therefore it is essential to balance the financials with the rest of the perspectives. To go along with this risk assessment and cost-benefits should be used and included within this perspective.

The Customer perspective focuses upon the ability of the organization to provide quality goods and services to the customer and to have customer satisfaction. It is essential in any organization to keep customers, internal and external, satisfied. If the customers are not satisfied they will find another organization to fulfill their needs. Poor performance from this perspective is a good indicator of the future decline of the business in the financial perspective. When creating metrics for this perspective, customers should be analyzed by type of customer and types of processes should be analyzed by the process type that provides the goods or services to specific customer types.

The Internal Business Processes perspective focuses upon improving upon business processes. Metrics are used to identify and assess how well the organization is running and whether or not the end result of these processes are adequate. The end results of these processes include the products, services, and customer satisfaction within the Customer perspective. The processes that are identified are within three different categories including strategic management processes, mission-oriented processes, and support processes. Strategic management processes focus upon the bottom line results, while mission-oriented processes are special government office functions which encounter unique problems and are not easy to measure. Support processes are more repetitive and are easy to break down, measure, and benchmark using metrics. The Internal Business Processes perspective includes the numerous processes that are used to achieve the other Balanced Scorecard perspectives.

The Learning and Growth perspective focuses upon the ability of employees, resources, and organizational alignment to manage the business and adapt to change. This perspective includes employee training and the corporate culture of the organization as it relates to individual and organizational improvement. The success of any organization is based upon their ability to keep up with rapid changes of technology. This depends highly upon the employees learning about new technology and applying it on their job. In this day and age there is a problem of not being able to find new technical employees and there is a decline in keeping older employees up to date with training. It is essential to reverse this trend by changing corporate culture to incorporate training and self-improvement within their organization. By doing so, there will be a balance upon the perspectives and the success of the organization will be increased.

Kaplan and Norton also focus upon that learning is more than training. According to them, learning includes the availability of mentor and tutors, collaboration of employees, communication of employees, and the use of technological tools. In addition, metrics within this perspective can guide the organization to focus training where it is needed the most. For more information about the four perspectives , how they pertain to one another, and an excellent graphical representation of the balanced scorecard go to http://www.balancedscorecard.org/basics/bsc1.html.

To Chapter Menu

How to Implement a Balanced Scorecard

The hardest part about the Balanced Scorecard is how to implement it within an organization. There is no standard step by step approach; however, the phases of implementing a Balanced Scorecard fall within three broad categories. These categories include mobilization, design and rollout, and sustainable execution (Kaplan & Norton, 2006). The time span for these phases are best estimated as three to six months for mobilization, six months for design and rollout, and twelve to twenty-four months for sustainable execution (Kaplan & Norton, 2006). While this might seem time consuming, it is essential to take the time within each step to ensure the step is completed effectively. This is done because everyone within the organization must be on board and it is essential to make sure that the Balanced Scorecard is balanced.

The mobilization process is the first step to implementing a Balanced Scorecard. The main purpose of this step is to prepare everyone in the organization about the major changes that will happen in the near future. There is always a reluctance to change, so it is essential easing everyone into the change and to get everyone on board as soon as possible so that the process will go smoothly. The first change that should that be done is to get commitment to the Balanced Scorecard across all levels of the organization, especially those in leadership positions. Another step within the mobilization process is to identify ideas and key points that should be focused upon in the vision and strategy of the organization. This includes doing research and assessing the needs of each of the four perspectives. Suggestions to conduct this include choosing an audience within each of the four perspectives, identifying their requirements, determining potential gaps in performance for the audience, and set audience priorities. After this is completed, it is critical to narrow down the priorities and the end result in the vision and strategy of the organization. The reasoning for this is the organization can only focus upon a certain number of priorities at a time due to resource limitations.

Now that the priorities of the organization are narrowed down the next step is to create a clear, identifiable vision and strategy. The vision should be completed in a timely matter due to the completion of previous steps which inherently created the focus of the vision. In order to produce the strategy, the organization must determine how the organization's vision pertains to the internal processes of the organization and how to change the internal processes in order to fulfill the organization's vision. By accomplishing this step, the organization is translating their vision and strategy in order to start the Balanced Scorecard. Once this is done the mobilization phase of the Balanced Scorecard is complete and the focus can now be on the design and implementation of the Balanced Scorecard.

Design and implementation is the second step to implementing the Balanced Scorecard. This step focuses upon how the vision and strategy affects the four perspectives and how to align the objectives and results of the perspectives to the vision and strategy. The first step to accomplish this is to identify that the areas that link the vision and strategy to the results. This includes areas within the four perspectives and can include specific areas such as financial performance, operations, innovation, and employee performance. Once this is done it is essential to create strategic objectives that support the business vision and strategy along with keeping in mind the areas that link the vision and strategy to results.

Once the strategic objectives are determined, the organization must come up with a way to measure the success of the objectives. In order to do this, the organization must select and design metrics. This includes determining what should be measured, defining what success is, and ensuring that there are enough resources to implement to do everything which has been determined thus far. In addition, it is important to create short-term and long-term goals. By having goals within each of these categories, the success of the Balanced Scorecard can be found easily. Before the metrics are finalized, it is essential to make sure that the metrics are accepted throughout the entire organization. If everyone is not on board with the metrics, then most likely the Balanced Scorecard will not be a success. Once the metrics are determined and accepted, the implementation plan for the Balanced Scorecard must be determined.

In order to have a successful implementation of the Balanced Scorecard, there must be a plan of action. Although this varies from organization to organization, there are some good guidelines/suggestions to follow. One suggestion is to use organization Intranets and technology capabilities to ensure communication throughout the rollout of the Balanced Scorecard. In addition, it is important to aid in aligning the entire organization to the vision and strategy. This can be done in numerous ways including offering training in improvement areas, creating a reward system for improvements, and ensuring that the strategy and vision of the company is everyone's job by doing such things as having a tie-in to employee performance. Once the plan of action is complete, the rollout of the Balanced Scorecard can be done. After the rollout is completed, the improvement of the processes of the organization will start due to aligning the organization to a clear vision and strategy. These improvements will continue to improve during the next phase of the Balanced Scorecard and will hopefully continue in the future of the organization.

The final phase of implementing a Balanced Scorecard is sustainable execution. This phase focuses upon making the Balanced Scorecard part of the everyday procedures of the organization. There is no "set" way to do this; however it just takes time. In addition, sustainable execution is ensuring the value of the Balanced Scorecard and assessing the success of the goals objectives. This success is determined by the metrics that were made in the design and rollout phase. It is essential to check the results of the plan that was made before in order to continually improve upon the Balanced Scorecard. To assess the success of the Balanced Scorecard the organization must collect and analyze performance data and compare the results to the metrics. Once this is done, weaknesses can be found and can be fixed accordingly.

To Chapter Menu

9.4 Conclusion

By having the Balanced Scorecard being used on all levels of an organization as well as providing the results of the Balanced Scorecard to employees there will be a continual process of aligning performance with the organization vision and strategy. This will ensure that the company will continually improve upon goals. In addition, the Balanced Scorecard will allow the organization to learn what works best both externally and internally in the organization. By learning what works best, the organization will become highly adaptable and will become more competitive than without using the Balanced Scorecard.

To Chapter Menu

9.4 References

To Chapter Menu

9.5 Evaluation Models

Michael Bond and David Cerreta

Evaluation of e-learning, learning and training programs has become a widely researched area. This is due to the fact that the performance of learners and education itself is vital to enabling learners to reach their full potential. In understanding how our learners "learn" will enable us to make better decisions when trying to educate them or implement trainings in general.

In evaluating learning or instruction we are searching for many things:

Keep in mind that these are only a few of the thousands of questions one could ask about the effectiveness and efficiency of a training module or educational course. One could ask a vast array of questions depending on what it is that you are trying to evaluate. For now, we'll keep it simple and just try and get a basic idea of what evaluation and evaluation models are.

Next, we're going to compare, contrast and critique 3 different evaluation models. In order to effectively examine an evaluation model we must first explain what an evaluation program is and find the purpose that it serves. Are all evaluation models the same? Are they all evaluating the same things? Can different models be used in different circumstances or contexts?

These are some of the questions we should ask ourselves when examining the following evaluation models.

For now we can safely say that an evaluation model is systematic approach that will guide us in measuring the efficiency and effectiveness of a training, a course or an educational program.

The three models we'll be reviewing in this particular article are the following:

Comparisons/Contrasts of Each Model

Kirkpatrick's 4 Levels of Evaluation

Kirkpatrick has defined four levels of evaluation (Winfrey, 1999).

Kirkpatrick Level 1
Reactions/Learner Satisfaction - Ask questions like, Did they like it? or Was the material relevant to their work? - This type of evaluation is often called a smile sheet.
Kirkpatrick Level 2
Learning - Goes beyond satisfaction, attempts to assess the extent that learners have advanced in skills, knowledge and attitude
Kirkpatrick Level 3
Transfer – At this level trainers attempt to determine if the skills, knowledge, or attitude has been effectively transferred into everyday life activities as required, measuring at this level is difficult because it is almost impossible to predict when the behavior change has taken place so the time to evaluate must be carefully calculated and chosen.
Kirkpatrick Level 4
Results - this level measures the success of the program in terms that managers and executives can understand, such as, increased production, improved quality, decreased costs, reduced frequency of accidents, increased sales, and even higher profits or Return On Investment (ROI).
Stufflebeam's CIPP Evaluation Model
Context, Input, Process, and Product Evaluation
Daniel Stufflebeam's CIPP Model (2002) is based on the premise that evaluations should assess and report an entity's merit, worth, and significance as well as, the lesson learned.

The CIPP model's main theme is that evaluation's most important purpose is not to prove, but to improve.

Flashlight Triad Model

According to Educational Technologies at Virginia Tech, The Flashlight Triad Model covers 5 distinctive steps ("Evaluation Models").

1. Overview and Confronting the Blob
Brainstorming brings to light all the elements that feed into whatever it is you are evaluating.
2. From Blob to Issue
To narrow down the Blob this step focuses on choosing which elements you think are crucial to evaluate (usually only 1 or 2).
3. From Issue to Triad
Take the elements you chose and create Triads which consist of the type of technology that is employed (T), the activity the technology enables (A), and outcomes expected from the activity (O).
4. From Triad to Data
From the Triads, now questions are created to gather data about the Triads; these questions usually fall into one of 5 categories:
  • Technology
  • Interaction of Technology and Activity
  • Activity
  • Interaction of Activity and Outcomes
  • Outcomes
Note: there is a pre existing Flashlight Network Question Database of over 500 question types that can be used other than these (Evaluation Models).
5. From Data to Next Steps
After analyzing the data collecting and answering the questions created modifications are made.

9.5 Conclusion

In our conclusion we propose the best approach to evaluation would be taken using the Kirkpatrick model. This is due to the fact of the longevity of the system and the many times it has been effectively used when approaching evaluations.

Kirkpatrick's four-level evaluation model has been acknowledged as the standard in the training field because of its simplicity and its ability to help people think about evaluating criteria. It has provided a highly successful framework that has clearly met an organizational need, and it has become well known in HRD departments around the country. Through this model, you not only get to know the real meaning and the purpose of the evaluation, but it guides you in the exact ways to effectively conduct evaluations.

Kirkpatrick's evaluation model is has been in existence since the 1950's and is continued to be accepted today. In addition, technology and creativity have only added to maximizing its benefits for the modern corporation.

Kirkpatrick's goal was to clarify what evaluation meant. The model clearly defined evaluation as meaning "measuring changes in behavior that occur as a result of training programs." Originally, the model itself is composed of four Levels of training evaluation. A fifth level, ROI has been added since then. The fifth level, "What is the return of learning investment?" was the brainchild of Dr. Jack J. Phillips, Ph.D., author, consultant and Knowledge Advisors advisory board member and strategic partner (Winfrey, 1999).

Return on investment has been a critical issue for trainers and top executives in recent years and is a topic frequently listed on meeting agendas. This technique probably should receive more emphasis from educators than it has in the past.

For more information on Kirkpatrick's theory on learning evaluation theory , please visit Businessballs.com at http://www.businessballs.com/kirkpatricklearningevaluationmodel.htm (Kirkpatrick's learning and training evaluation theory).

To Chapter Menu

9.5 References

To Chapter Menu

Creative Commons License 
This work is licensed under a Creative Commons License .

Contact Information

  • Pamela Berman
  • 2221 McCormick Center
  • Bloomsburg PA 17815-1301
  • USA
  • pberman@bloomu.edu

Last updated October 13, 2006

Graphics by David Cerreta