Activity 21 – Project Evaluation

What have we learned from the evaluators comments on our project?

·      We received a comment that a diagram showing the learners progress would help make the text clearer – so maybe the lesson here is that in a prototype it helps the evaluators if they can see the pedagogical progression of what you are trying to achieve.

·      I think this links to my own point below about the link to pedagogical intent not being clear in the project I evaluated – an evaluator will have the perspective of ‘what learning is this trying to achieve, and how effectively is it achieving it – so a diagrammatic representation of intended learning progress could be very helpful.

·      Such a diagram could also help the teacher during the course, to ensure that the learning is moving in the right direction.

·      Good feedback to adjust language in learning materials to suit the target learners.

·      Good feedback about the need to flag, as the course progresses, how the course could be adapted for delivery to immigrants/refugees.

What have I learned from evaluating other projects?

·      It can be very difficult to understand exactly what people want you to evaluate – I think the heuristics need to be very focused and clear.

·      The format of the feedback form is quite important – I had to keep paging up and down to refer to the more detailed descriptions of some of the heuristics beneath the table – this was quite clunky and quite a disincentive to give the evaluation a full focus.

What would I change in our project, in light of these new insights?

·      The link to pedagogical intent was not clear in the project I evaluated – so when designing my own heuristics in the future I will try and make this link more consciously.

·      I would try and make the heuristics very focused and clear.

·      I would try and make the feedback form extremely easy to use, with no requirement to page up and down the screen all the time.

Provide a diagram of intended learning progress in future?

What are the advantages and limitations of this evaluation process?

·      Advantages

o   You get fresh perspectives on your work.

o   This could save you wasted costs from developing a module too early.

·      Limitations

o   The experience is, by definition, not a complete one – so there risks being a distortion of the actual experience of the user.

·      I think it is very difficult to evaluate a long list of heuristics in a short space of time. So I think it would be better to assign one or two heuristics to each evaluator and get them to focus on only those.

·      I think it is also better to have specific parts of the prototype that you want people to test, to see if they work – rather than just tell people to go through the whole thing systematically.

What does the heuristic evaluation process neglect?

·      It neglects fresh insights / comments that may be unrelated to the heuristics. The designers write their own heuristics, so risk being limited by their own outlook on what is effective.

It neglects a dialogue between the evaluator and the project team - the comments are made in one direction, without discussion and iteration.

·      I think it may tend to encourage negative feedback (albeit constructive) and not positive feedback. The template does allow for positive feedback (e.g. giving ‘excellent’ ratings), but as I was completing the evaluation I found I tended to feel I needed to flag things needing changing because they weren’t working, rather than also flagging good aspects that could be expanded or reinforced.