Evaluation – Isn’t there an off the peg solution?

(Learning Log for OLDS MOOC Week 7)

I’ve just spent a couple of hours looking through all the resources on Cloudworks about evaluating learning design and found it all quite frustrating.

The first thing that frustrated me was the number of times I clicked into Clouds to discover they were empty.  I realise that they have been set up in the expectation that people will fill them with things, but it has resulted in a huge number of hyperlinks and wasted effort on my part.  When you’re busy, it’s just irksome.

The other thing I found frustrating was that there is a lot of information about how to evaluate, about the theory behind it and about how to construct an evaluation method, but what I want is a tried and tested evaluation method to use – not to make. It is all very interesting if evaluation is the thing you’re interested in, but not so much if you’re only interested in getting a job done.  Sure, I could design my own evaluation method but I think someone far more knowledgeable than I must have done so already. There should be something that I can pick up and use.  I don’t mind providing feedback and helping to improve it, but I doubt that my reinvention of the wheel will turn any more smoothly than previous versions.

Maybe, every situation, every course needs its own evaluation method and tools but I doubt it.  I can only assume that where I am thinking of online courses as the product of learning design, others are thinking of smaller units which may not need the whole bag of tools.

Anyway, this gives the impression I didn’t find anything I could use which isn’t true.  I explored the Cloudscape called Evaluation Toolbox, looked in all the clouds and then discovered that the Cloudscape was in a Cloudstream which itself contained a cloud called FevaTools -Formative Evaluation Tools for Online Course Design. (I’m slowly getting the hang of this Cloudworks though am not sure I’d want to explain it to someone else).

In this cloud was a link to a website called fevertools  at San Diego State University.  “Welcome to fevatools! This is your one-stop portal to find out everything you want to know about formative evaluation tools to help you gauge the success of your courses, especially hybrid and fully online courses.”

Sounds too good to be true and a little ‘un-academic’ but I may explore this further since it supplies tools to evaluate different aspects of a course and also different tools for each aspect.  I have an online Psychology module I need to look at this week and that can be my guinea pig. But that’s enough for today and the evaluation can be done on Thursday as tomorrow needs to be spent thinking and talking about electronic submission, marking and feedback.

The Convergence Session Today.

I attended the convergence session today – Google Hangout streamed to youtube with questions from non-hangout people (myself included) via twitter.  A few things interested me –

1. when running an online course outside a VLE and on multiple systems it is difficult (impossible) to collect data on user participation and hence it is difficult to evaluate the course design properly and difficult to identify ‘at risk’ students and help them.

2. When you run a course with different tutors each week they do not experience the course fatigue that students do who are on the course every week.

3. Someone said on twitter that they planned to do this OLDSMOOC course again. I hadn’t even considered that but it got me thinking that I might just do so (if they let me!).  Now I understand the theory I am more ready to put it into practice and if I did it again I might try to assemble a team at my university to work on a real project. (Or would this fail because they would be on their first attempt?). Anyway, something to think about.

4. Davinia Hernández-Leo from Universitat Pompeu Fabra spoke about  LdShake which is a social platform that allows teachers to share resources. I will be interested to see whether this gets used outside the geographical area in which these teachers work.  if I had time I’d explore LdShake, though in reality I know I won’t ever have time 😦


  1. Bob, I share some of your frustration with the challenge of finding things (such as evaluation tools) in this MOOC. Different participants are using so many different tools to shape and share their knowledge, and as a learning facilitator in the MOOC, I can’t seem to get a handle on where it is all happening. The Cloudscape tool probably has a lot of potential, but I certainly have not mastered it. I don’t use Twitter so I know I am missing a lot of interaction there. It is 10:30 PM on the last night of Week 7 and I just stumbled on your thoughtful reflection above. With respect to your posting, I don’t think there is an off-the-shelf evaluation plan that will fit every online course development initiative. But you and your colleagues could use some of the tools in this MOOC to set up a flexible protocol that could be applied whenever you are designing, producing, and implementing new online courses. This might begin with a content review by subject matter experts, an instructional design review using a set of dimensions of effective online learning (see for instance this Australian tool: http://members.optushome.com.au/~lsiragusa/pages/overview.htm ), an alignment review by a skilled designer like yourself, a heuristic evaluation, and even some formal usability testing. Setting up the protocol would take some time and I appreciate how little time you have, but once a protocol is established, efficiencies would be realized. I am not familiar with the “fevatools” site you mentioned above, but I will be very surprised if it provides the one stop shop you are seeking. I’ll take a look at it. Meanwhile, thank you for your frank and useful feedback. – Tom Reeves

  2. I’ve shared my frustrations with Cloudworks from the start … and had some experience a year ago so entered the cloud with a sense of dread. I stuck at it and found some odd ways in. What mattered was the contact with people I got to know – as they gave up it became inevitable that I would do so too, not least because I had more pressing matters. H809 a postgraduate module, partially produced by the same team as it comes from the Open University stable, but a very different beast. More like getting on a bus with four to five stops a week. A weekend for an assignment every five weeks and a longer sojourn to produce a short dissertation at the end. Four tutors groups each with less than sixteen people in each. I liken my cloudworks experience to Fresher’s Fair … every day of the week. Every time I came in I wondered around getting interested in what other people were doing, sometimes landing their by mistake. So a Fresher’s Fair with some 12 entry doors on several floors with the people behind each stall mostly changing too. Your brain gets tired of the overload, the lack of landscape and in this sense ‘Cloudscape’ is the right term, for the wrong reasons. I took some pictures of a Constable painting ‘Study of Clouds’ in the Ashmolean Museum when I was in Oxford on Friday. I’ll post it in my blog. What was I doing in Oxford. Hankering after ‘the real thing’ – a chance to meet and talk with some people in the flesh, this at an talk on Virtual Worlds in Japane at the Centre of Social and Cultural Anthropology hosted by the Oxford Internet Institute. After a while, all this online stuff has you eager to meet likeminds in person. Which I guess why conferences will flourish, not die out, and then from pure e-learning vs. traditional learning the blended form might become the long term, preferred pattern. The best of all worlds, with landscapes, water courses, digital oceans and shifting clouds.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

Postcards for Petrolheads

Classic cars caught unawares in their natural environment.

Ros Walker

...learning, education, technology and stuff

HistorianRuby: An Historian's Miscellany

Early modern historian. Loves gender, women's, social & royal histories. Ventures elswhere when interest is piqued. Blog may cover above themes or something a little more random. Find me on Twitter @ruthrblair


Historian and geographer, writer and researcher

Laurence Marks

away from the office..


Random cars we like

The Ali Lowe Commentary

The view from the shed...

The Lure of Speed

Vintage Motoring Blog

Rob Appleyard

My thoughts on learning, technology and stuff that makes me tick

%d bloggers like this: