Activity 21 – Project Evaluation

What have we learned from the evaluators comments on our project?

·      We received a comment that a diagram showing the learners progress would help make the text clearer – so maybe the lesson here is that in a prototype it helps the evaluators if they can see the pedagogical progression of what you are trying to achieve.

·      I think this links to my own point below about the link to pedagogical intent not being clear in the project I evaluated – an evaluator will have the perspective of ‘what learning is this trying to achieve, and how effectively is it achieving it – so a diagrammatic representation of intended learning progress could be very helpful.

·      Such a diagram could also help the teacher during the course, to ensure that the learning is moving in the right direction.

·      Good feedback to adjust language in learning materials to suit the target learners.

·      Good feedback about the need to flag, as the course progresses, how the course could be adapted for delivery to immigrants/refugees.

What have I learned from evaluating other projects?

·      It can be very difficult to understand exactly what people want you to evaluate – I think the heuristics need to be very focused and clear.

·      The format of the feedback form is quite important – I had to keep paging up and down to refer to the more detailed descriptions of some of the heuristics beneath the table – this was quite clunky and quite a disincentive to give the evaluation a full focus.

What would I change in our project, in light of these new insights?

·      The link to pedagogical intent was not clear in the project I evaluated – so when designing my own heuristics in the future I will try and make this link more consciously.

·      I would try and make the heuristics very focused and clear.

·      I would try and make the feedback form extremely easy to use, with no requirement to page up and down the screen all the time.

Provide a diagram of intended learning progress in future?

What are the advantages and limitations of this evaluation process?

·      Advantages

o   You get fresh perspectives on your work.

o   This could save you wasted costs from developing a module too early.

·      Limitations

o   The experience is, by definition, not a complete one – so there risks being a distortion of the actual experience of the user.

·      I think it is very difficult to evaluate a long list of heuristics in a short space of time. So I think it would be better to assign one or two heuristics to each evaluator and get them to focus on only those.

·      I think it is also better to have specific parts of the prototype that you want people to test, to see if they work – rather than just tell people to go through the whole thing systematically.

What does the heuristic evaluation process neglect?

·      It neglects fresh insights / comments that may be unrelated to the heuristics. The designers write their own heuristics, so risk being limited by their own outlook on what is effective.

It neglects a dialogue between the evaluator and the project team - the comments are made in one direction, without discussion and iteration.

·      I think it may tend to encourage negative feedback (albeit constructive) and not positive feedback. The template does allow for positive feedback (e.g. giving ‘excellent’ ratings), but as I was completing the evaluation I found I tended to feel I needed to flag things needing changing because they weren’t working, rather than also flagging good aspects that could be expanded or reinforced. 

Activity 19 – reflecting on the experience of developing the prototype

·      Funny how, again, we arrived at a division of work (2 weeks of the prototype each) without needing to formally allocate work – everyone just got on with it. The features table was a useful tool that enabled people to hold up there hands to volunteer for the bits they wanted to do. We had no clashes.

·      I guess it would have been interesting to see what would happen if 2 people had wanted to do the same bit.

·      I think in practice we are all quite relaxed and ‘non-territorial’ – the work is sufficiently pressured and the resources sufficiently constrained that I think we all intuitively recognise the value of letting people pitch in wherever and whenever they can.

What was my role in this phase?

·      I wrote the prototypes for Week 2 and Week 7.

·      I joined in the team calls we had to discuss points of confusion.

Was the process clear and efficient?

·      It was reasonably clear.

·      Efficient? I think it was sufficiently efficient for this stage of the process. We didn’t duplicate work. All 4 team members contributed their energies.

·      Ideally the prototype would have been opened up for review earlier – we opened it up on Friday 14^th June, one day after the official completion date.

·      I wonder if we couldn’t have reduced the workload by only prototyping parts of the storyboard in detail – we have ended up prototyping the whole course really.

Were my expectations met?

·      I think so. Perhaps we could have focused more on 'what works' for different learning objectives, and this would make us prototype in more depth in certain areas. E.g. if we want people to know how to use some of the digital tools, have we really prototyped something that lets us see if the learning works? Are there sufficient opportunities for learners to experiment and practise?

The advantages and limitations that I perceive in constructing a prototype

Advantages

·      Focuses the mind.

·      Forces you to see the flow of design.

·      Makes it easier to see the antecedents and precedents required for all steps of the process.

Disadvantages

·      Risks forcing a linear view to be taken when the design should still be very fluid and iterative.

·      I think we’ve found it difficult in practice to know which bits of the prototype to prioritise.

Team call 14 June 2013

Good to have this call with Asanka and David. I also had a catch-up call with John yesterday as I couldn’t make the team call earlier that day.

We also all had a call together on Tuesday 11^th – so 4 calls in the space of 8 days.

Interesting what this says about the value of synchronous discussions.

Points from the call

·      The prototype  is looking pretty good.

·      Lots of disconnects in the prototype –e.g. Asanka has introduced a ‘forum’ tool in week 6, and feels she should now work backwards to earlier weeks to have forums available there as well. Very commendable that she wants to achieve consistency, although in the interests of time I think we were all relaxed if she doesn’t have the time to do these changes today before we open the prototype for evaluation.

·      Asanka will open the prototype at midday today.

·      Asanka also thought that the instructions to our reviewers were too general (IE just asking them to work through the whole prototype systematically) – and we would be better to give more detailed instructions, perhaps asking evaluators to focus on specific activities.

·      We discussed some confusion around the TMA template – are we supposed to be writing about our experience in an H817 context? I said I think we are supposed to imagine we are a design team working in the context we have imagined (an NGO course for teachers of immigrants/refugees) – so a lot of our context will be real (using the digital storyboard, needing to have team calls across different timezones etc) and a lot will be imaginary (the NGO, the trainee teachers, the refugees etc). Similarly, the ‘protagonists’ will include ourselves, the trainee teachers, the personas we imagined etc.

Learning point

·      I threw together a draft set of heuristic protocols and instructions two nights ago.

·      I was quite apologetic to the group about this, as I felt this was a very rough, early draft – more for my benefit to help me get my thoughts in order rather than something useful for the team to collaborate on.

·      I even felt it might be easier for the team to start with a fresh template.

·      In practice, the team has taken my draft and quickly enhanced and added to it – it is fine now.

·      The learning point for me is to realise that, in this collaborative work, perhaps it can sometimes work fine to post something in quite a rough form for people to refine and develop.

Learning Journal Week 19

Features table – I find the prioritisations very difficult to review.

I’m struggling to properly refer back to the storyboard.

Or, indeed, to the document we did on synthesising all our thoughts.

Haven’t we just skipped all that and gone for our prototype?

Also, haven’t we prototyped everything?  The prioritisation of elements hasn’t really worked. It has felt quicker and simpler just to write out the whole thing.

Are we missing something?

The ‘work it out on your own!’ rubric of activity 18 feels a tad unhelpful.

Cohort-wide call 9 June 2013

What an enjoyable call this was.

It was my first time to connect with many of the other people in the tutor group – Nuala, Nicola, Paige.

We should define the user of our product quite narrowly – the narrower the better.

The focus of the TMA is primarily about our learning on the block, not about the excellence of the website itself.

The purpose of a design narrative is to capture the design process so that in the future you don’t repeat the same mistakes.

‘Prototyping’ is most effective if you are testing whether something specific ‘works’ – maybe this helps address my confusion from last week about which bits to prototype.

It would be good to have a look at how design narratives are used in architecture, if there is time.

Week 18 Learning Journal

I guess the main focus this week was extracting features from the storyboard (activity 16)  and selecting features to include in the prototype (activity 17).

We discussed this a lot during our team call on 7 June 2013.

Points discussed on the call:-

• On our call we weren't really sure what a 'feature' means in this context, and why this stage of extracting and selecting features was really necessary.
•  David offered to complete the features table, leaving space for us all to fill in our prioritisations.
• [Having now gone through some prototyping, I can maybe understand a bit better the need for an extracting and selecting stage - when you have a clearer sense in your mind of the learning outcomes you are trying to achieve, then maybe you can sense which parts of your storyboard need particular focus].
• For the overall reflection on this process, one of the things I'd like to give more thought to is whether we could have done this extraction and selection stage more effectively.
• In practice, I think we've ended up prototyping pretty well everything!

Team dynamics

• Good and always improving as rapport levels get higher. John has separately commented on the stage of the call when he sought clarification on whether the course was focused on the teachers as the learners, or whether it was intended that the immigrants/refugees should be the learners. He wrote in his post that he felt gently corrected by the rest of the team in a professional and objective way.
• I find that everyone has suggestions and comments that add value - for example, on this call, Asanka could foresee that if we all prioritised the feature table using the same prioritisation columns, then disagreements in the prioritisation would be hard to track and understand. She therefore suggested adding columns so that we can individually give our prioritisation.
• Asanka also pushed for us to make the design more learner-centred wherever possible. The example we discussed was whether we present learners with examples of great digital stories, or whether we ask the learners to go and look for great examples.
• This led on to a discussion on whether or not we could write a prototype that differed from the storyboard - the team had split views on this, and I was asked to clarify with Paige, which I did on the cohort-wide call on Sunday 9 June.

John offered to do a note of the call - Paige had suggested we recorded the call - we all thought this was a good idea but couldn't record in skype, so took extensive notes instead.

Wish we'd done this on earlier calls as this would make it easier to reflect on the whole process.

Week 17 Learning Journal - Reflecting on the storyboard process (activity 15)

Reflection on this activity

·      Good fun. The storyboard tool is good to use, feels like a table you can shift things around on in a free way.

·      I liked having to structure my thoughts in this way. It helped me, in particular, to focus initially on ‘learning outcomes’.

The role I played

·      I felt I played my part.

·      I drafted my own storyboard.

·      I reviewed David’s, John’s and my storyboards and tried to synthesize the combined learning points – I think this synthesis is helpful (at least to me!)

·      As part of this I tried to make the learning outcomes quite detailed and specific. This then made it easier to think ‘how would each of these be assessed?’

·      I updated our draft storyboard partly with the increased detail but felt I should check with my team whether they found this useful, I’m aware that I sometimes want to go to a greater level of detail than may be required at certain stages.

What I have learned

·      Again, the value of collaborative processes – e.g. John’s ideas of having learners.

Questions and concerns

·      Of course the ideal storyboard, for me, would be a physical board that I could stick things on and move around – something that I could stand back from and get an intuitive feel for the shape and balance of the whole thing. I like the storyboard tool we are using, but I find it hard to ‘see the whole picture’ on my screen.

·      I’m not sure we’ve incorporated enough of our work to date in arriving at the storyboard – the design principles etc. Some are there – e.g. the phasing and scaffolding. But it has felt as is we’ve pressed on to something new, and the previous work is lodged in the back of our minds rather than directly connected.

Activity 12 - Reflection on the Search Process

I used the OU library and a general Google search on ‘Digital storytelling’, and then on ‘digital storytelling immigrants’ and ‘digital storytelling refugees’.

The OU library search came up with quite a rich list of articles. I followed the recommended instructions and reviewed a lot of abstracts – I guess I realized that it was possible to narrow down choices to look at things which were quite similar to our project – a case study on DS and immigrants for example.

I also chose some that were not directly about immigrants/refugees but which seemed likely to have similar issues – so one of Tanzanian schoolchildren using digital storytelling to come to terms with their views on HIV and AIDS.

I really enjoyed looking at the website produced by this Tanzanian project, with the actual video uploads of the mini-dramas enacted by the schoolchildren. They all wore African tribal masks, which gave me the idea for the ‘avatar’ design principle.

I have to say that seeing these children’s videos was a very nice counterbalance to reading the academic papers – it really grounded the theory in some tangible reality.

Later in the fortnight, at the stage of thinking about Storyboards, I did another google search, this time on ‘digital storytelling tools’ and ‘digital storytelling courses’ – this search yielded a whole lot of very practical blog posts and guides on actual technologies to use and the structure of some actual courses being run. These are not academically reviewed materials, but nevertheless still very useful as we pull our thoughts together.

Learning reflections week 16

Learning Reflections Weeks 16 

Module 3 Design

·      I like this DesignStudio format – it is a nice idea to try and replicate a studio online.

·      I find the configuration too clunky however – it takes a long time to find other teams’ projects and then drill down to specific pages and make comments. So we are not replicating the same experience as wandering around from board to board in a studio and having chats.

·      So I like the principle of what we are trying to replicate very much, but I am frustrated by the practice. I think I would have benefitted from some initial hand-holding. And I think the goal of having dialogues with other teams (which Yishay made clear in his call) could maybe be more emphasized in future iterations of this module.

·      I’m painfully aware that this may just be my own personal learning style – I don’t get the point of things by reading. I understand far more by doing, I also appreciate listening, and I very much like being shown some ideas in practice by someone else – this just results in much faster learning for me. I wonder if people in our cohort with different learning styles have got to grips with the OpenDesignStudio much earlier?

·      These thoughts have fed in to our own learning design, which I feel should be directive initially, so that learners can understand the tools they are using and the basics of storytelling. Then they can experiment and construct on their own or in teams.

·      I am in the fourth week of the six-week module 3 and only now starting to appreciate the importance of cultivating feedback dialogues with other teams. This is too late, I should have realized this earlier.

Team dynamics

·      Team dynamics have been less pronounced this week because half the team have had other things to deal with. Both Asanka and John have had to cope with a lot of personal disruption in the first part of the module.

·      Writing this makes me realize that maybe I haven’t done enough to reach out to them and encourage them to remain engaged despite the disruptions. I know that if it were me having to deal with all these other things, I would get massively demotivated about my OU studies, and I would find it very hard to re-engage with them.

·      We were able to update John on the Friday call for Week 16. Neither he nor Asanka could make the Week 17 Friday call, and I couldn’t make an update call for John on the Saturday the day after our routine call.

·      That said, John has been able to start to get involved again in Week 17, and I found his design pattern on ‘The Immigrant and Refugee Voice’ very thought-stimulating.

·      So despite the disruptions, I feel that as a team we still have progressed fine.

Call with Yishay

·      I couldn’t join the live call as it was at 3am my time, so I listened to the recording (it was nice, however, to be able to see that everyone was asked to do a poll on their preferred times, and the most popular time was chosen).

·      I must do more work on online identities, it is such an interesting area. Having seen Yishay’s OU profile and read the module 3 course notes, I had formed quite a distinct view of him and the thinking behind the module. When I listened to the call, my view was completely transformed.

·      It makes me think that in course design, we need to over-communicate the context and spirit that we are trying to foster.