Formative Assessment Models — Easy As Riding A Bicycle


boy-on-bikeWhen I got my first “two-wheel” bike, it was a pretty scary deal. I didn’t know it at the time, but it was filled with “formative assessment.”

My sturdy tricycle didn’t require any balance, not a lot in the way of steering skills, no real braking capability–and the “summative performance assessment” was if I had fun and didn’t fall off. But this new bike measured me every step of the way.

I like to use that example for teachers when I talk about the difference in the two main types of measurement in meeting learning goals.

One way is for us to just keep teaching, and teaching, and teaching–and at the end of the module–give a test. Some students will do well, some not so well, and some fail. (What we used to call the Bell Curve.) That’s a Summative Model, because it sums up what happened. (No, not because “only some students learn.”)

Let’s Try A Formative Assessment Model–And Master CTE Standards For All Students!

In this new model, students are a big part of the measurement. And measurement goes on all throughout the learning process–not just at the end. Learning objectives are posted and each student knows what they must be able to do, and can check off each competency. (Think of those merit badges that Boy Scouts earn.) Here’s a quick 3 minute video that gives you some ideas.

Common Core Formative Assessment Tools

With this model, all students can reach competency with all standards. But it may be messy–since some may be working on different parts of the curriculum at different times, some may be helping others, and some may take a little longer to complete all of the “badges.” Our traditional model assumed that everyone would learn everything in the same amount of time. And when that didn’t happen, we just gave them an “F” and moved on. That doesn’t work anymore.

Here’s a great chart featuring eight ways to include formative assessment in your existing learning:

formative assessment models

Online Tools For Formative Assessment

There are also a large number of online tools available for formative assessment–check out some of these from the NWEA blog:

  • Kahoot - A game-based classroom response system, where teachers can create quizzes using Internet content.
  • Backchannel Chat - Similar to TodaysMeet, this site offers a teacher-moderated version of Twitter. An extension of the in-the-moment conversation might be to capture the chat, create a tag cloud and see what surfaces as a focus of the conversation.
  • GoSoapBox - Free for less than 30 students, this all student response system works with the BYOD model, so no charge for a clicker. One of the most intriguing features for me is the Confusion Meter
  • Lino - A virtual corkboard of sticky-notes so students can provide questions or comments on their learning. These can be used like exit tickets or during the course of a lesson.
  • Poll Everywhere - Teachers can create a feedback poll or ask questions. Students respond in various ways and teachers see the results in real-time. As Steven indicates, with open-ended questions you can capture data and spin up tag clouds to aggregate response. You should note that Poll Everywhere has a limit to the number of users. Mentimeter (which we’ve listed below) does not which makes it a little more functional.
  • Socrative - Engaging exercises and games that engage students using smartphones, laptops and tablets.
  • AnswerGarden - A tool for online brainstorming or polling, educators can use this real time tool to see student feedback on questions.
  • Ask3 - This app for the iPad allows students and teachers to collaborate on lessons both in and outside of the classroom. Questions can be posted to specific classrooms set up in the app, and students can add their thoughts, answers, and thinking to the whiteboard.
  • AudioNote - A combination of a voice recorder and notepad that captures both audio and notes for student collaboration.
  • BubbleSheet - An app that allows students to complete assignments and common assessments using an iPhone or iPad Quizzes up to 10 questions are free.
  • Coggle - A mind mapping tool designed to understand student thinking.
  • Conceptboard - This software facilitates team collaboration in a visual format – similar to mind mapping, but using visual and textual inputs. Compatible on tablets and PCs, Conceptboard can work from multiple devices.
  • Five Card Flickr - Designed to foster visual thinking, this tool uses the tag feature from photos in Flickr.
  • ForAllRubrics - This software is free for all teachers and allows you to import, create and score rubrics on your iPad, tablet or smartphone. You can collect data offline with no internet access, compute scores automatically and print or save the rubrics as a PDF or spreadsheet.
  • Formative Feedback for Learning - An iPad app that is designed to foster and encourage communication between students and teachers. Through a conference setting it uses icons to prompt discussions. Dylan Wiliam, our resident formative assessment expert, says, “Formative Feedback for Learning looks very useful. I can see myself recommending it to others.”
  • Google Forms - A Google Drive app that allows you to create documents that students can collaborate on in real time using smartphones, tablets and laptops.
  • iBrainstorm - An iPad app that allows students to collaborate on projects using a stylus or their finger on screen.
  • I>Clicker - A device that helps facilitate all student response to polls, questions and other teacher-led discussions.
  • iLEAP Pick a Student - Helps the teacher pick a student from the class, and uses turn-based selection so every student is selected before a student is picked again. Supports multiple classes and has a number of selection options.
  • InfuseLearning - A platform by which teachers can engage all students on any device, getting valuable formative feedback along the way.
  • iThoughts - This mind mapping app for Apple’s iFamily® is a great visual tool to help you brainstorm ideas, plan projects and themes, and set goals. As students discuss ideas and possible answers to discussions, educators can visually see the path that their thinking takes, helping to understand how students are learning.
  • Mentimeter - Allows you to use mobile phones or tablets to vote on any question a teacher asks, increasing student engagement.
  • Padlet - Provides an essentially blank canvas for students to create and design collaborative projects. Great for brainstorming.
  • Pick Me! - An easy to use app for the iPod, iPad and iPhone that facilitates random student selection. Can be organized by class for convenience.
  • PollDaddy - Quick and easy way to create online polls, quizzes and questions. Students can use smartphones, tablets, and computers to provide their answers and information can be culled for reports.
  • RabbleBrowser - An iPad app that allows a leader to facilitate a collaborative browsing experience.
  • QuickVoice Recorder - Another free voice recording app for the iPhone or iPad that allows you to record classes, discussions or other project audio files. You can sync your recordings to your computer easily for use in presentations.
  • Random Name/Word Picker - This tool allows the teacher to input a class list and facilitates random name picking. You can also add a list of keywords and use the tool to have the class prompt a student to guess the word by providing definitions.
  • SMART Response VE (for SMARTboards) – A cloud-based software that enables students to respond to planned and spontaneous questions and take quizzes using any of their favorite Internet-enabled devices, from anywhere.
  • Tagxedo - A tag cloud generator that allows you to examine student consensus and facilitate dialogue.
  • ThinkBinder - A collaboration tool that allows students to ask questions and discuss topics in a group, share, create and work together on almost any project.
  • TodaysMeet - This online collaboration tool allows educators to create a “room” in which students can share ideas, answers and thoughts to lectures and lessons. Educators can view student responses in real time for evidence of learning.
  • VoiceThread - Allows you to create and share conversations on documents, diagrams, videos, pictures or almost anything. This facilitates collaborative student discussion and work.
  • Vocaroo - A free service that allows users to create audio recordings without the need for software. You can easily embed the recording into slide shows, presentations, or websites. Great for collaborative group work and presentations.
  • Wordle - Generates tag clouds from any entered text to help aggregate responses and facilitate discussion.
  • XMind - A mind mapping software for use on computers and laptops.

(Note:  This post was first published on the blog at AES Education, where it became hilariously popular.)

Assessment Without Fear



smartkidI was six years old.  My first grade teacher Mrs. Larsen was handing back our papers, and I knew my educational career was finished.  I’d tried as hard as I could to keep my letters within the wide-lined paper — both capitals and lower-case — but by the end of the page things had gotten pretty messy.

Kenny, one desk ahead of me, had a paper in his hand with a slew of red marks on it.  Alice, the class show-off (just across from me) had a “worked hard” sticker on her paper — she’d already been called out for her perfect work.  And now it was my turn.

Yup.  Life was over.  I was destined for failure.  I’d gotten a “-2″ on the test.

A New Way Of Thinking About Assessment


Earlier this month, I had the honor of presenting a workshop for a room full of Career and Technical Educators at the ACTE show in Las Vegas.  My topic was “Assessment FOR Learning: How Measuring Success DURING Learning Turns Testing Upside Down”.  Rather than wait until the end of the learning, teachers use assessment every day (without those red markers and embarrassment) to engage students and work together with them towards success.

To model that behavior, I didn’t give a speech.  I worked together with all that experience in the room to build answers to questions about how we could improve, after describing the idea.  I had a great helper — Anne Kuser from the home office scribed all the information up on big yellow sheets, and I’ve added it to my PPT deck on Slideshare.

Assessment for learning v2 from Dick Carlson

[slideshare id=29291852&doc=assessmentforlearningv2-131217103143-phpapp02]

I’d love to hear what you think!

Yes, It’s True! Learning Makes Your Brain Hurt


I read this research on the Internet, so it has to be true.  Learning hurts your brain. I first began to suspect this was true in 4th Grade Spanish class with Mrs. Gonzalez, where I was called “Ricardo” for reasons that remained a mystery to me all year long.  (Hey — I’m just visiting your country for an hour a day.  I’m not planning on renouncing my citizenship and living here as an ex-pat like Hemingway, lady.) But I digress.

The researchers exposed J20 and wildtype mice to new cages to increase neuron activity. Surprisingly, after 2 hours in the novel environments, the number of gH2A.X -positive neurons spiked in the brains of both healthy and diseased animals, primarily in areas critical for memory formation and learning—suggesting that the brain activity itself was triggering DNA damage.

Interestingly, the damage was resolved in wildtype mice within 24 hours back in their home cages, but the damage persisted in J20 mice. Furthermore, the damage was higher in J20 mice, which had up to three times as many gH2A.X-positive neurons—and the differences could be detected as early as 1 month, before the J20 mice began exhibiting cognitive symptoms. The results suggest that perhaps the high levels of amyloid in the brains of these mice was preventing DNA repair. SOURCE

So — Should You Be Worried?

A slightly “less scientific” version of the findings published on Ars Technica suggests, reassuringly, that this “shouldn’t keep you up at night.”  And that it may even provide some useful insights into the cure of Alzheimer’s. But if you’ve already substantially damaged your neurons by moving to a new cage, you may not even be aware of what you’ve lost.  Check around your desk for degrees that you may have earned, patents in your name, and photos of you with famous people.  You should also ask close family members if you’re a big deal in any way at least once per day.

As educators, I think most of us are already aware of this situation.  I can say that any time I’ve tried to instill learning in any meaningful amount, it makes my brain hurt in a direct relationship to how hard I’m teaching.  That’s why, as I get older, I teach less and less and listen more and more.


(First published on


How Can I Force Students To Spend More Time Learning?


A client recently relayed a comment to me from a classroom teacher.  The teacher was disappointed in our e-learning unit because some students would just quickly “click through” the content, rather than spending several minutes on each screen.  The instructor wanted to know if we could modify the product to force the students to spend a couple of minutes looking at each screen, before they were allowed to click “next” and proceed.  My client asked how I would respond to the question.

Here’s what I said — I’m curious, how would YOU respond?

Would Forcing Longer Time On A Screen Help?

I usually start this answer (if I think the teacher can take a little gentle ribbing) by asking if, before they hand out the textbooks, they glue down the corners of all the pages.  When they say “no” I let this lead into having the group list the uses of a textbook – not just a front-to-back read, but for reference, to use for review before testing, to quote in projects, to refresh memory during open-book tests, and so forth.

From there, I start talking about how e-learning content can be used in many different ways by a learner.  Some learners will start at the beginning, and go from front to back.  Some will dip their toes in at different places.  Some will scan for subjects of interest.  Some will use the “search” function to find a specific subject.  (I stop to point out that, as educators, we don’t get a vote in this.  It’s how learners build knowledge in the world today.)  So we need to understand that, and build it into our understanding of how we help them facilitate their learning.

What Has Changed In Learning, Anyway?

Back when I was in the 3rd grade, Mrs. Solem had a huge amount of control over what I learned and exactly how that happened.  Today, learners expect to be able to make a lot of decisions about what and how they’re going to learn.  (They make GOOD and BAD decisions, of course.  That’s to be expected.)  Our job as teachers and facilitators is to help guide them as they do this, to show them how to make good choices, and (ultimately) measure the outcome of their choices and achievement.  The learners own their success or failure.

Technically, it’s easy to set a minimum screen time.  Practically, we’ve found that learners just go browse the web or read email or paint their nails or do something else.  I’ve seen no solid research that there’s a corresponding increase in learning.  Much like gluing the pages down in a textbook to force students to spend more time on each one, you’re just going to frustrate the participant.

The larger question that needs to be asked here is why does the learner skip through the content, not learn, and end up with “some quiz and test scores so low”?

Given Good Content And Teaching, Why Don’t They Learn?

If we assume the content is good quality, and the teaching is good quality – why do students skip through it and fail the tests?  There are enough answers there to fill a thesis dissertation, but I’ll focus here on just one.  Motivation.

If we’re trying to teach a student how to write an MLA footnote, and the student sees no reason that they need to learn to do so – I could make them stay on that page for an hour and they probably won’t learn the content.  (Unless I tell them there’s a test they have to pass to graduate.) Google the difference between “intrinsic” and “extrinsic” motivation for extra credit.

But if I tell the student the next screen will show them how to double their score at “Angry Birds” they will stay on that screen until they’ve squeezed every drop of knowledge out of it.  Because they want that information for themselves and see the value.

So as a teacher, one of the most useful things you could possibly do is to show your students that what you’re teaching is connected to their real lives, and actually means something to them.  And that’s a huge, huge job.

So — what do you think?  Do you glue your virtual pages down?


Get Your Head Out Of The Cl-ASS-room — eLearning Is Different


I spend lots of time developing learning that lives online.  e-learning, web content, videos, podcasts, support information, application forms for tractor assembly jobs — so I fancy myself as a bit of an expert in how this type of content is consumed.

(Many other people in my discipline would use less flattering terms to describe me — including references to rodents, orifices and familial relationships.  But even if true — I design a bunch of online learning.)

As part of that, I joust regularly with people who teach and train in something called “The Real World”.  I have a hazy memory of this place — it involves chalk boards, rows of desks, and children in freshly-pressed jumpers smiling up at me as I whack their knuckles with a wooden ruler.  I rarely teach there anymore — nobody wants to pay my embarrassingly high prices, and I keep trying to click on individual students and block them.

baggageAs they move into the world of online teaching, most “real world” practitioners attempt to bring all their baggage with them.  And as the airlines have found, the more baggage you allow the more difficult it is to get the damn thing off the ground.  So I’m here today to show you a few of the cherished icons of classroom instruction you’re going to have to leave at the gate if you want to succeed in the online world.

Everyone Does Not Stay Together
In your classroom, you can exert a good deal of control that all students advance at a controlled pace — by assigning readings, presentations, and in-class activities.  You’ll find that online students will lag behind and race ahead.  Some will have questions about section 14 on the third day of class.  Oopsie.

So you’d best be prepared on Day 1 to teach the whole thing, or you’ll take the wind out of the sails of the students who are really engaged.  And you’ll have to be willing to support someone who’s going back to the beginning for a refresher during the last week.

Everyone Expects Personalized Support
Blame it on Tony Hsieh of Zappos — a large number of your students will now expect to interact with you via email, chat, FaceBook, Twitter, LinkedIn, Google+, Snoozle, Schmaltz, Fizzle, Abdalo, Whackadoodle — ok, I was just making up those last few.  But you get the picture.  If you’re not careful about designing how you set up your assignments, your workload will go up exponentially.

Be sure that you set your assignments up so that students interact with each other, rather than always depending on you.  Be sure that you become an Online Facilitator, and let go of the idea that all learning comes from you.  Be sure that you spend some time learning new skills for this new role that you’ve taken on.

Everyone Will Not View Every Screen
Do you have a DVR?  If so, I bet you don’t watch commercials.  Or the boring parts.  And your e-learning students are going to do that, as well.  In your classroom, they sat their in their seats and pretended to listen during the boring parts.  Online, they’re just going to skip past the things they don’t want to pay attention to.  (I always find it amusing when teachers complain about learners ignoring boring online content.  I ask them what they, personally, do during the sermon in church.)

So you’d better make sure the e-learning you’re using is interesting, engaging, and makes your learners want to pay attention.  Or find a way to introduce it so they will.

Quit Measuring Stuff You Don’t Track
There’s no reason to add in all those little “Check Your Understanding” and “Quick Quiz” screens throughout the e-learning, unless you’re going to collect the data and use if for something.  Only three possible outcomes:

  1. Student actually knows the answer. Wow.
  2. Student doesn’t know the answer.  Unless you force them back through the content, you just make them feel dumb.  Wow.
  3. Student skips past the test — which the majority will do.  Wow.

If you feel you must do this, just have a question and the correct answer on the next screen, like a flashcard.  That way you’re reinforcing a positive.  There’s good data to support that.  And you can re-use that content for test prep at the end of the course.

Let The Inmates Build The Prison
As I’ve mentioned above, you need to start thinking more like a “facilitator” than a “teacher”.  You’re guiding this group of learners through the curriculum, and no longer the main source of knowledge.  Let them learn from each other, from resources you provide (and that they find and vet through you), bring in live humans via Skype or Webcasts, have them do original research and share — be creative in how each new class discovers information.

Each course will look different, and that’s ok.  Each group of learners will approach the problems in a different manner, and the shared knowledge that they create will be unique.  That’s one of the amazing parts of online learning — those “Poindexters” that sit in the front row will fade into the background, and you’ll meet a whole new group of people you never heard from before.


You May Be Certifiable


(Addendum added Nov 6, 2011)

I keep getting myself in trouble. Let me be the first to admit it. And this time, it’s because I’m proud of the work I did, and I hate to see other people cheapen the meaning of it. In this case, we’re talking about the word “Certified”.

When you drive your car into a garage, you see signs for an ASE Mechanic.  If you want to climb into the cockpit of a jet airplane, you’ve got to talk to a guy (or gal) who’s holding a CFI Rating. And, if you want to get into the server room at work, you’re gonna need a little something called an MCSE (Microsoft Certified Systems Engineer) Certificate(Full disclosure, I spent a good part of my career writing big complicated courses to teach people to get that last one — which hundreds of thousands, if not millions of people — now hold around the world.)

My point is, when you bill someone as being “Certified” in something, you’re saying to the public at large that you’re in some way reassuring them that this person can actually perform the tasks expected of them in a competent manner.

The ASE Mechanic can diagnose the problem with your car, order the parts, install them and then test your car and return it to you without giving you a dangerous vehicle.

The CFI Flight Instructor can teach your pilot how to fly an airplane in such a manner that you will be safe riding with them, and that if a problem arises they will professionally and quickly resolve it and land the airplane without you ending up a small grease spot on the pavement.

The Microsoft Certified Systems Engineer will be able to install, configure and diagnose your computer servers and systems so that your data is safe, your users are happy, and that that guy from Nigeria doesn’t get into your bank account.

Now, of course, there ARE organizations that provide “certification” of crystal ball gazers, balloon animal makers, and those who make sculptures out of dryer lint.  Anyone can “certify” someone in a skill — coaching, finger-snapping, apple carving — and that’s their right.  While I think it’s amusing, I don’t really have a big problem with them.

But a year or two ago, a Large Learning Organization (Let’s call it the “Amazing Society for Trivial Development” decided that the world needed them to “provide a way for workplace learning and performance professionals to prove their value to employers and to be confident about their knowledge of the field.”

Great!  Sounds just like my definition above — reassuring, performance, competent…

Well, hold on there, Hoss.  The bar’s not all that high.  Here’s what you need:

1.  Three years in a “related” field.

2. “Pass” a multiple choice test (which apparently has not been validated)

3.  Submit a “sample” of your work (PPT deck?  Drawing of you in a classroom?)

Oh — one other little thing.  A check for $1,000 dollars. For a machine-scored test and to have a “blue-ribbon panel” look at your course plan?

Just for comparison, a Microsoft Certification test (for a single cert) right now costs around $200.  And having it will immediately boost your income about 25% or more in most markets.  Because the people who pass this test have to pretty much take a week of classes and then study really, really hard before they can pass.

If you got all five certs, you’d double your income.  Those tests would be about $1000 out of your pocket.  Probably a good deal.

So let’s get back to the Large Learning Organization.  Do they have any documentation that their certification maps directly to more income? Well, they do claim that the Fortune 500 “prefer their candidates” but they don’t give any hard data. If there were actual stats, I bet they’d quote them.

But come on — who in their right mind would say these people are competent at what we do? E-learning?  Instructor led?  Webcasts?  Curriculum Design?  Needs analysis?  Assessment?  Social Media?  Coaching?  Psychometrics? Data Analysis?

But (from their own data) the LLO has now administered their test to 3,605 people and charged users about $1000 each.  I’m not a math whiz, but that comes out to Three Million, Six Hundred and Five Thousand Dollars! Now they did have mimeograph costs for the multiple-choice tests, and they had to print up some nice blue ribbons for the panel, but other than that — pure profit. And people say Bernie Madoff was smart!

(Full Disclosure:  They will give you a $200 discount on the test if you join the org or are a member.  But that money just goes in another pocket, so I’m not discounting it.)

This causes an itch under my saddle because they’re supposed to be a Professional Learning Organization, and they should know better.  If it was a bunch of Balloon Animal Professionals, I’d cut them some slack and say “Well, what do they know about training and certification?”  But these people CLAIM to be the best and brightest.  In fact, I bet some of them even HAVE this certification. (Interesting side note would be to see if EVERYONE paid full price, or if the insiders got it discounted or for free.  I know what I’d be betting — how about you?)

So there you have it.  Call me cranky, call me snarky, or call me somebody that’s proud of my profession.  Proud of my skills.  Proud of the people that I work with every day who really provide Training and Development at a high level of quality.

And embarrassed to be associated with a money-machine that grinds out pretty certificates.










Addendum:  Several people have written me personally, asking why I hate people who have certifications from the Large Learning Organization, or think that those people don’t have skills in learning.  PERISH THE THOUGHT!  I’d bet that a large number of people who coughed up the $10000 fee for this overpriced and poorly-designed piece of irrelevant documentation are likely highly skilled learning professionals.  I know several of them personally, and would recommend them highly, depending on what it is you wanted them to do.

The point is that this Four Letter Acronym really has nothing to do with whether they are skilled or not — it merely proves they can write a check and answer some multiple-guess questions.  It’s not a certification for a potential client that they can actually provide the services that the client needs, or that they have the skills that map to the project at hand.

Were I at the beginning of my career, with no other evidence of my skill (like advanced degrees or client referrals) I’d probably hold my nose and write the check.  But I’d have to take a shower every time I put the acronym after my name.


My Friend Wasn’t Killed By Social Media


I lost a friend yesterday.  Trey Pennington of Greenville — father, husband, grandfather, speaker, teller of stories and launcher of Social Media Clubs — could no longer fight the darkness and took his own life.  Twitter and Facebook echoed with the stunned and shocked responses from his friends, acquaintances and those who had been helped by this kind and gentle man.

Less than 24 hours later, I’m already seeing the first jabs coming online, talking about how this loss may have been caused by The Interwebs.  “Online Relationships just aren’t real.”  “We need to connect with actual people.”  “This Social Media stuff is all junk.”

What complete bunk.

If you know Trey, you know his demons were in the real world.  People who deserted him, institutions that turned their back on him after years.  I’m not giving details because he wouldn’t want me to.

But the online community — we loved him unconditionally.  He had hundreds of thousands of people who knew him as a kind, helpful, thoughtful soul.  He spoke around the world — most recently, here in Columbia a couple of weeks ago at my request.  He did it at no charge.  (A man who could command thousands of dollars for a single keynote appearance.)

We chatted, and put off dinner for another day because he was rushing home to work on a presentation the next day.  We knew we had time, because he was feeling great.  He’d lost weight, gotten tanned, and had all his dark demons at bay.

Then he returned to the real world. And it killed him.

I don’t doubt there are problems in the online world.  But I’m not going to let people sully the legacy of my friend with this kind of shit. He made social media a better place for so many people, and we’re all worse off now that he’s no longer with us.

I hope there’s really good wireless up there on that cloud, buddy.


“Instruction Does Not Cause Learning…”


I work with lots of folks who create and deliver some kind of instruction — training, documentation, presentations and such. I’m planning to have this quote from Etienne Wenger tattooed on my forehead in large letters, just so they can think about it.

“Instruction does not cause learning; it creates a context in which learning takes place, as do other contexts. Learning and teaching are not inherently linked. Much learning takes place without teaching, and indeed much teaching takes place without learning.”


Serving Up Some Sacred Cows Of Learning

Georgia being nosey!

Yesterday I ran across an article in The Atlantic showing “The School Of The Future” that lives in a local YMCA. While they don’t seem to have a Quidditch field or a landing pad for flying cars, it sounded like a nice idea.  Working with the community, sharing the pool, innovating in education — what’s not to love?

Then I came to the author’s contention that “Eighty percent of charter schools don’t produce better results than traditional public education. And sadly, some results are much worse.” While not labeled as “op-ed”, I’m guessing Kathleen Kennedy Townsend does have a particular axe she’s trying to sharpen, here.

Since I have a nodding familiarity with education, and more exposure to self-designated experts than is healthy for most adults, I wondered where this had come from.  So I started doing some searching.  And searching.  Best I could do (‘cuz, of course, they don’t reveal SOURCES in the Atlantic) was some questionable research from a LARGE MAINSTREAM EDUCATIONAL INSTITUTION that had numbers that looked a little bit like that, if you read really quickly and ignored the details.  And the source.  And didn’t ask any questions.

(I don’t have anything against experts.  Really.  Some of my friends are experts.  I’d even let my daughter marry one, if she was the last woman on earth and had a trust fund.) But in this case, I think it might be useful to stop for a moment and talk about the process of measuring results.  Something that we do a lot in my line of business.

If a client hires me to teach punch press operators to run their machines faster, one option is for me to just show them how to turn the speed dial up to “11″ and step back.  Job Done!  The machines are now running faster, I’ve accomplished the goal, and I want my check.  Ka-Ching!

(There will, of course, be a corresponding loss of fingers, an increase in damaged materials, lots of lawsuits, and explosions when the red-hot machines finally blow up — but I achieved my goal with 100% success!)

Still confused?  Let’s make it even simpler.  Let’s imagine that you’ve got a Dairy Barn. And you want “better results”.

Our goal is to get more milk from your herd.  Each morning, we’ll measure how many gallons of milk you get from group “A” and how much from group “B” and see which is “BEST”.  (Best, in this case, meaning more milk.)

At the end of the first week, Group “A” wins.  With weak, runny milk because the foreman fed them food with lots of fluid.  We couldn’t sell it because the fat content was so low, but there was more of it.

New goal — highest fat content.

At the end of week two, group “B” is the winner.  The foreman ordered in extremely expensive food, which raised the fat content for the second group nice and high.  But the cost per cow was triple what we usually pay, so we ended up selling the milk at a loss.

Week three — we aim for highest profits.  And the foreman butchers all the cows and sells them at market, showing a huge profit.  Oopsie.

So be careful when you hear people in Education talking about measuring results, or who gets “better results”.  Without asking a lot of very specific questions, and without a lot of experience at designing and delivering actual education to actual people, you may get slaughtered.