e-Assessment software

http://www.assessmentfocus.com/adaptive-testing.php


http://www.assessmentfocus.com/adaptive-testing.php  

Week 23 Activity 9

Table 3: Comparing ‘Ann Other’ Bales’s analysis by hand and by OpenMentor

Bales’s categories	Total number of comments categorised by hand	Total number of comments categorised by OpenMentor
Positive reactions A1 Shows solidarity A2 Shows tension release A3 Shows agreement
Attempted answers B1 Gives suggestion B2 Gives opinion B3 Gives information	12	7
Questions C1 Asks for information C2 Asks for opinion C3 Asks for suggestion	1	0
Negative reactions D1 Shows disagreement D2 Shows tension D3 Shows antagonism	6	1

Both Open Mentor and myself have no Category A categorisations.

OM only seems to pick up some of the comments.

Each category, B, C, and D, shows differences between OM and myself – quite material.

I think OM is fine as a quick piece of broad feedback to give a tutor a sense of the type and number of comments being made, but its use as a tool presupposes that the tutor already has good feedback skills – so I wouldn’t use OM as a tool for teaching tutors how to give better feedback. OM’s categorisations are not necessarily the same as a tutor’s would be, and a tutor’s view of the right balance of A,B,C and D comments for a specific student could be very different from OM’s.

Activity 21 – Project Evaluation

What have we learned from the evaluators comments on our project?

·      We received a comment that a diagram showing the learners progress would help make the text clearer – so maybe the lesson here is that in a prototype it helps the evaluators if they can see the pedagogical progression of what you are trying to achieve.

·      I think this links to my own point below about the link to pedagogical intent not being clear in the project I evaluated – an evaluator will have the perspective of ‘what learning is this trying to achieve, and how effectively is it achieving it – so a diagrammatic representation of intended learning progress could be very helpful.

·      Such a diagram could also help the teacher during the course, to ensure that the learning is moving in the right direction.

·      Good feedback to adjust language in learning materials to suit the target learners.

·      Good feedback about the need to flag, as the course progresses, how the course could be adapted for delivery to immigrants/refugees.

What have I learned from evaluating other projects?

·      It can be very difficult to understand exactly what people want you to evaluate – I think the heuristics need to be very focused and clear.

·      The format of the feedback form is quite important – I had to keep paging up and down to refer to the more detailed descriptions of some of the heuristics beneath the table – this was quite clunky and quite a disincentive to give the evaluation a full focus.

What would I change in our project, in light of these new insights?

·      The link to pedagogical intent was not clear in the project I evaluated – so when designing my own heuristics in the future I will try and make this link more consciously.

·      I would try and make the heuristics very focused and clear.

·      I would try and make the feedback form extremely easy to use, with no requirement to page up and down the screen all the time.

Provide a diagram of intended learning progress in future?

What are the advantages and limitations of this evaluation process?

·      Advantages

o   You get fresh perspectives on your work.

o   This could save you wasted costs from developing a module too early.

·      Limitations

o   The experience is, by definition, not a complete one – so there risks being a distortion of the actual experience of the user.

·      I think it is very difficult to evaluate a long list of heuristics in a short space of time. So I think it would be better to assign one or two heuristics to each evaluator and get them to focus on only those.

·      I think it is also better to have specific parts of the prototype that you want people to test, to see if they work – rather than just tell people to go through the whole thing systematically.

What does the heuristic evaluation process neglect?

·      It neglects fresh insights / comments that may be unrelated to the heuristics. The designers write their own heuristics, so risk being limited by their own outlook on what is effective.

It neglects a dialogue between the evaluator and the project team - the comments are made in one direction, without discussion and iteration.

·      I think it may tend to encourage negative feedback (albeit constructive) and not positive feedback. The template does allow for positive feedback (e.g. giving ‘excellent’ ratings), but as I was completing the evaluation I found I tended to feel I needed to flag things needing changing because they weren’t working, rather than also flagging good aspects that could be expanded or reinforced. 

Activity 19 – reflecting on the experience of developing the prototype

·      Funny how, again, we arrived at a division of work (2 weeks of the prototype each) without needing to formally allocate work – everyone just got on with it. The features table was a useful tool that enabled people to hold up there hands to volunteer for the bits they wanted to do. We had no clashes.

·      I guess it would have been interesting to see what would happen if 2 people had wanted to do the same bit.

·      I think in practice we are all quite relaxed and ‘non-territorial’ – the work is sufficiently pressured and the resources sufficiently constrained that I think we all intuitively recognise the value of letting people pitch in wherever and whenever they can.

What was my role in this phase?

·      I wrote the prototypes for Week 2 and Week 7.

·      I joined in the team calls we had to discuss points of confusion.

Was the process clear and efficient?

·      It was reasonably clear.

·      Efficient? I think it was sufficiently efficient for this stage of the process. We didn’t duplicate work. All 4 team members contributed their energies.

·      Ideally the prototype would have been opened up for review earlier – we opened it up on Friday 14^th June, one day after the official completion date.

·      I wonder if we couldn’t have reduced the workload by only prototyping parts of the storyboard in detail – we have ended up prototyping the whole course really.

Were my expectations met?

·      I think so. Perhaps we could have focused more on 'what works' for different learning objectives, and this would make us prototype in more depth in certain areas. E.g. if we want people to know how to use some of the digital tools, have we really prototyped something that lets us see if the learning works? Are there sufficient opportunities for learners to experiment and practise?

The advantages and limitations that I perceive in constructing a prototype

Advantages

·      Focuses the mind.

·      Forces you to see the flow of design.

·      Makes it easier to see the antecedents and precedents required for all steps of the process.

Disadvantages

·      Risks forcing a linear view to be taken when the design should still be very fluid and iterative.

·      I think we’ve found it difficult in practice to know which bits of the prototype to prioritise.

Team call 14 June 2013

Good to have this call with Asanka and David. I also had a catch-up call with John yesterday as I couldn’t make the team call earlier that day.

We also all had a call together on Tuesday 11^th – so 4 calls in the space of 8 days.

Interesting what this says about the value of synchronous discussions.

Points from the call

·      The prototype  is looking pretty good.

·      Lots of disconnects in the prototype –e.g. Asanka has introduced a ‘forum’ tool in week 6, and feels she should now work backwards to earlier weeks to have forums available there as well. Very commendable that she wants to achieve consistency, although in the interests of time I think we were all relaxed if she doesn’t have the time to do these changes today before we open the prototype for evaluation.

·      Asanka will open the prototype at midday today.

·      Asanka also thought that the instructions to our reviewers were too general (IE just asking them to work through the whole prototype systematically) – and we would be better to give more detailed instructions, perhaps asking evaluators to focus on specific activities.

·      We discussed some confusion around the TMA template – are we supposed to be writing about our experience in an H817 context? I said I think we are supposed to imagine we are a design team working in the context we have imagined (an NGO course for teachers of immigrants/refugees) – so a lot of our context will be real (using the digital storyboard, needing to have team calls across different timezones etc) and a lot will be imaginary (the NGO, the trainee teachers, the refugees etc). Similarly, the ‘protagonists’ will include ourselves, the trainee teachers, the personas we imagined etc.

Learning point

·      I threw together a draft set of heuristic protocols and instructions two nights ago.

·      I was quite apologetic to the group about this, as I felt this was a very rough, early draft – more for my benefit to help me get my thoughts in order rather than something useful for the team to collaborate on.

·      I even felt it might be easier for the team to start with a fresh template.

·      In practice, the team has taken my draft and quickly enhanced and added to it – it is fine now.

·      The learning point for me is to realise that, in this collaborative work, perhaps it can sometimes work fine to post something in quite a rough form for people to refine and develop.

Learning Journal Week 19

Features table – I find the prioritisations very difficult to review.

I’m struggling to properly refer back to the storyboard.

Or, indeed, to the document we did on synthesising all our thoughts.

Haven’t we just skipped all that and gone for our prototype?

Also, haven’t we prototyped everything?  The prioritisation of elements hasn’t really worked. It has felt quicker and simpler just to write out the whole thing.

Are we missing something?

The ‘work it out on your own!’ rubric of activity 18 feels a tad unhelpful.

Cohort-wide call 9 June 2013

What an enjoyable call this was.

It was my first time to connect with many of the other people in the tutor group – Nuala, Nicola, Paige.

We should define the user of our product quite narrowly – the narrower the better.

The focus of the TMA is primarily about our learning on the block, not about the excellence of the website itself.

The purpose of a design narrative is to capture the design process so that in the future you don’t repeat the same mistakes.

‘Prototyping’ is most effective if you are testing whether something specific ‘works’ – maybe this helps address my confusion from last week about which bits to prototype.

It would be good to have a look at how design narratives are used in architecture, if there is time.

Week 18 Learning Journal

I guess the main focus this week was extracting features from the storyboard (activity 16)  and selecting features to include in the prototype (activity 17).

We discussed this a lot during our team call on 7 June 2013.

Points discussed on the call:-

• On our call we weren't really sure what a 'feature' means in this context, and why this stage of extracting and selecting features was really necessary.
•  David offered to complete the features table, leaving space for us all to fill in our prioritisations.
• [Having now gone through some prototyping, I can maybe understand a bit better the need for an extracting and selecting stage - when you have a clearer sense in your mind of the learning outcomes you are trying to achieve, then maybe you can sense which parts of your storyboard need particular focus].
• For the overall reflection on this process, one of the things I'd like to give more thought to is whether we could have done this extraction and selection stage more effectively.
• In practice, I think we've ended up prototyping pretty well everything!

Team dynamics

• Good and always improving as rapport levels get higher. John has separately commented on the stage of the call when he sought clarification on whether the course was focused on the teachers as the learners, or whether it was intended that the immigrants/refugees should be the learners. He wrote in his post that he felt gently corrected by the rest of the team in a professional and objective way.
• I find that everyone has suggestions and comments that add value - for example, on this call, Asanka could foresee that if we all prioritised the feature table using the same prioritisation columns, then disagreements in the prioritisation would be hard to track and understand. She therefore suggested adding columns so that we can individually give our prioritisation.
• Asanka also pushed for us to make the design more learner-centred wherever possible. The example we discussed was whether we present learners with examples of great digital stories, or whether we ask the learners to go and look for great examples.
• This led on to a discussion on whether or not we could write a prototype that differed from the storyboard - the team had split views on this, and I was asked to clarify with Paige, which I did on the cohort-wide call on Sunday 9 June.

John offered to do a note of the call - Paige had suggested we recorded the call - we all thought this was a good idea but couldn't record in skype, so took extensive notes instead.

Wish we'd done this on earlier calls as this would make it easier to reflect on the whole process.