Oh my goodness, what a right old mess the UK has gotten into over this years school exams. Cancelled exams, statistical models, algorithms to ensure that the dreaded “grade inflation” didn’t happen all conspired to make what can only be described as an omnishambles.
Last week, the Scottish government did a swift U-turn on their results which has put pressure on the rest of the UK to do the same. As I write this a news alert has just popped up on my phone saying the PM has confidence in Education Secretary Gavin Williamson and Ofqual. Back in “normal” times that language was a signifier of a resignation or a sacking, however these days it may well mean that the PM does have confidence in his minister, and the agency despite the mixed messaging from them both over the weekend.
Perhaps one positive thing to come out of this mess is the start of a public debate about statistical modelling, the development and use of algorithms and the implicit and explicit bias that they almost always promote.
However, this is a very messy business and there has been a huge amount of human complicity and error here too. In was pretty obvious in March that these exams would not go ahead.
Students themselves have (quite rightly) been very vocal, and visible in their anger, dismay and outrage at the overriding ‘logic” of the bigger pattern and the curve taking precedence over them as individuals. w.
The blame games have already started, with opposition parties seeing huge political capital to be made. Calls for public inquiries , discussions about what to do next year are all I fear detracting from what is the fundamental issue – our over reliance on exams.
If we had more continuous assessment and less reliance on final exams, if/ when another pandemic strikes or covid-19 has another spike, we wouldn’t have to worry about exam results or models to moderate grade inflation. Students work could be judged on their merits, there would be confidence in the marking through a shared learning outcomes (which if I am not mistaken do already exist). A more holistic view of students as people, with ideas, with agency, with the ability to express. share and reflect on their views would emerge.
We could allow students to exploit digital technologies to develop their portfolios, to share their work more openly, to develop more cross curricular activity, to develop agency and critical thinking skills. Much of this does happen in schools but still, the only thing that really counts are those final exams. That incredibly stressful, unfair and to be honest quite archaic way of testing memory not knowledge and understanding.
It’s said by many commentators that our current PM is a “crammer”. Had jolly japes at Eton, crammed for exams and through his loquacious use of slightly arcane language (see what I did there!) got the grades and the interview patter to get into Oxford and sustain his career in politics and journalism. The final result is what matters – Brexit, the last UK election, the ‘war’ on covid. . . . unfortunately we all have to suffer the chaos of the this period of uncertainty as we rumble from disaster to disaster.
We could change the way we assess our children as they leave school. Teachers already have the skills, knowledge, understanding and technology to do it, we just need to rethink time, space and place for on going assessment. It would be cheaper and more effective imho to spend money on that than on a public inquiry into what has and is still happening with this years results.
I have quote above my desk from a post I saw on social media early on in lockdown, it says “in the rush to return to normal, use this time to consider which parts of normal are worth rushing back to” (attribution Dave Hollis). I find it so sad that we seem to be rushing headlong back into exams instead of seriously contemplating the alternatives. Is this not is the perfect time to change that old “normal” to a far more equitable “new normal” for assessment?
Bob Harris wrote an excellent article last week about the new DfE strategy educational technology. This post is not about that per se, I can’t add any more to Bob’s excellent critique, rather it’s about what has been stuck in my head since reading the post, the focus on attainment not learning.
Learning is not included in the report, much to the surprise of many. Bob reports this explanation from Deborah McCann, head of ed-tech at the DfE who “ … astonished many attendees when she admitted that the term “learning” was deliberately excluded from the strategy. She said: “We have focused on attainment. There’s a view that ‘learning’ is a bit of a weak term really and there is a lot more that we are talking about – attainment and outcomes. That’s why you don’t see it in the strategy. … learning is the process, obviously, but what we want to see is attainment.”
Increasingly I am seeing attainment as a key strategic goal. I’ve seen a number of how to develop an effective digital strategy etc papers from ed tech/publishers and attainment, retention and outcomes are prominent but there’s little about learning. Apart from maybe a bit about personalised learning being enhanced through data and AI. You too can have a totally unique, homogenised personalised learning experience . . .
We can see this focus on attainment amplified outside education, particularly in run up the European Parliament elections and the focus on the attainment of Brexit.
When Therese May won the last Tory party leadership contest she famously said “Brexit means Brexit “. Over the last 3 years it has become increasingly clear that no-one has any idea what Brexit actually means (and has forgotten it’s a made up tabloid word). However the attainment of it, has become all consuming.
As far as I can see, there has been no attempt to learn about the process of leaving the European Union by the hard line right Brexiteers, or to engage the electorate in a meaningful discussion about just what that would entail. The promises of saving money that would go back into the NHS were backtracked on as soon as the referendum was over.
Early this week I heard a Brexit party candidate being interviewed the the radio. He was claiming that Brexit was the only way to improve the NHS, education and all the things people really care about. When challenged by the interviewer on what couldn’t been done through existing parliamentary and government process on these issues, he paused for a bit then said something along the lines of “well I haven’t research any of that but I know it will all be easier once we are out of the EU”. At that point, dear reader, I didn’t crash the car, but I may have shouted a few expletives at the radio.
The attainment of Brexit was his overriding focus, the details of how that could be done, what would happen next – not really that important. The lack of learning and process around the understanding of what Brexit is, and this all consuming focus attainment of Brexit has created serious consequences.
We now have a zombie government, Nigel Farage back on the campaign trail, Boris Johnston setting himself up to lead another Tory Brexit charge. In the meantime our current national problems such as housing, education, the NHS never mind the global environmental crisis are, to my mind, being ignored as the attainment of Brexit overrules them all.
Perhaps if our current government, and all leave political parties had taken a bit more time to really learn about the process of exiting the second largest trading block in the world, 40 years of trade and related treaties, human rights legislation etc, etc, and then share that in a meaningful way with the electorate, we would actually know what Brexit means. Then we could go through a meaningful learning process to decide if that really is what we need. In the meantime I’ll take learning over attainment any day.
Here is an advert you might remember that kind of sums it up for me.
I confess to a more than a bit of this sentiment, and not just in relation to OER, “Much of the OER movement has a bad attitude about platforms.” I am always wary when the focus is on developing platforms and not developing the people who will use these platforms.
I was once in a meeting where I put forward the “people and process not platforms and products” case. I was told that what was being discussed was platform “in the Californian sense of platform”. . . I’m sure a classic WTF look must have passed over my face, but it was explained that this meant people as well as technology. Geography aside, three years later this sense of platform doesn’t seem to be that wide spread or acknowledged. Maybe I need to go to California. But I digress.
Not long before the Wiley post I was reading the Pearson White Paper on learning design. It caused me a bit of unease too. Part of me was delighted to see learning design being recognised by, whatever might happen to them, a significant player in the education technology provider field. Using learning design to help product design is a bit of a no brainer. Technology should be driven by educational need or as Pearson put it :
“Products and systems that effectively leverage learning design can deliver superior learning outcomes.”
One example in the paper referred to work they had done in social science classes
“we quickly recognized that students were easily distracted by conventional textbooks. This told us we needed to eliminate distractions: any extraneous cognitive load that doesn’t promote learning. Fortunately, our learning design work reveals many proven techniques for accomplishing this. REVEL segments all content into manageable pieces and presents it via a consistent structure. It provides strong signaling cues to highlight key material and places all relevant content on screen simultaneously to offer a continuous, uninterrupted experience”
Which kind of related to this point from the Wiley post:
“Our fixation on discovery and assembly also distracts us from other serious platform needs – like platforms for the collaborative development of OER and open assessments (assessments are the lifeblood of this new generation of platforms), where faculty and students can work together to create and update the core materials that support learning in our institutions. Our work in OER will never be truly sustainable until faculty and students jointly own this process, and that can’t happen until a new category of tools emerges that enables and supports this critical work. (Grant money for OER creation won’t last forever.)
And don’t even start trying to explain how the LMS is the answer. Just don’t. “
Well of course Pearson do try to explain that:
“As testing progresses, we can overcome problems that compromise outcomes and build a strong case that our design will support learning. The very same work also helps us tightly define assessments to find out if the product works in real classrooms”
Of course they don’t really touch on the OER aspect (all their learning design stuff has been made available with CC goodness) but I’ll come back to that.
That phrase “if the product works”, I keep coming back to that. So on the one hand I have to be pleased that Pearson are recognising learning design. I have no argument with their core principles . I agree with them all. But I am still left with the niggle around the assumption that the platform will “do” all the learning design for both staff and students. That underlying assumption that if only we had the right platform all would be well, everything could be personalised, through data and analytics and we’d have no retention issues. That niggles me.
I was part of a plenary panel at the HESPA conference last week called “the future of learner analytics” where a number of these issues came up again. The questions asked by this group of educational planners really stimulated a lot of debate. On reflection I was maybe a bit of a broken record. I kept coming back not to platforms but people and more importantly time. We really need to give our staff and students (but particularly our staff) time to engage with learning analytics. Alongside the technical infrastructure for learning analytics we need to asking where’s the CPD planning for analytics? They need to go hand in hand. Cathy Gunn, Jenny McDonald and John Milne’s excellent paper “the missing link for learning from analytics” sums this up perfectly:
“there is a pressing need to add professional development and strategies to engage teachers to growing range of learning analytics initiatives If these areas are not addressed, adoption of the quality systems and tools that are currently available or underdevelopment may remain in the domain of the researchers and data analysis experts”
There seems to be an assumption that personalisation of learning is a “good thing” but is it? Going back to learning design, designing engaging learning activities is probably more worthwhile and ultimately more useful to students and society than trying to create homogenised, personalised chunked up content and assessments. Designing to create more effective engagement with assessment and feedback is, imho, always going to be more effective than trying to design the perfect assessment platform.
In terms of assessment, early last week I was also at a Scotbug (our regional Blackboard user group) meeting, where I was in a group where we had to design an assessment system. This is what we came up with – the flipped assessment – aka student generated assessments.
Not new, but based on pedagogy and technology that is already in use ( NB there’s been a really great discussion around some of this in the ALT list this weekend). I don’t think we need any new platforms for this type of approach to assessment and feedback – but we do need to think about learning design (which encapsulates assessment design) more, and give more time for CPD for staff to engage more with the design process and the technologies they either have to, use or want to use. This of course all relates to digital capability and capacity building.
So whilst we’re thinking about next gen platforms, learning environments, please let’s not forget people. Let’s keep pressing for time for staff CPD to allow the culture shifts to happen around understand the value of OER, of sharing, of taking time to engage with learning design and not just having to tweak modules when there’s a bit of down time.
People are the most important part of any learning environment – next gen, this gen, past gen. But people need time to evolve too, we can’t forget them or try to design out the need for them for successful learning and teaching to take place. Ultimately it’s people that will make the product work.
Oops, I did it again. I’ve now managed to complete another MOOC. Bringing my completion rate of to a grand total of 3 (the non completion number is quite a bit higher but more on that later). And I now have 6 badges from #oldsmooc and a certificate (or “statement of accomplishment”) from Coursera.
But what do they actually mean? How, if ever, will/can I use these newly gained “achievements”?
Success and how it is measured continues to be one of the “known unknowns” for MOOCs. Debate (hype) on success is heightened by the now recognised and recorded high drop out rates. If “only” 3,000 registered users complete a MOOC then it must be failing, mustn’t it? If you don’t get the certificate/badge/whatever then you have failed. Well in one sense that might be true – if you take completion to equate with success. For a movement that is supposed to be revolutionising the (HE) system, the initial metrics some of the big xMOOCs are measuring and being measured by are pretty traditional. Some of the best known success of recent years have been college “drop outs’, so why not embrace that difference and the flexibility that MOOCs offer learners?
Well possibly because doing really new things and introducing new educational metrics is hard and even harder to sell to venture capitalists, who don’t really understand what is “broken” with education. Even for those who supposedly do understand education e.g. governments find any change to educational metrics (and in particular assessments) really hard to implement. In the UK we have recent examples of this with Michael Gove’s proposed changes to GSCEs and in Scotland the introduction of the Curriculum for Excellence has been a pretty fraught affair over the last five years.
At the recent #unitemooc seminar at Newcastle, Suzanne Hardy told us how “empowered” she felt by not submitting a final digital artefact for assessment. I suspect she was not alone. Suzanne is confident enough in her own ability not to need a certificate to validate her experience of participating in the course. Again I suspect she is not alone. From my own experience I have found it incredibly liberating to be able to sign up for courses at no risk (cost) and then equally have no guilt about dropping out. It would mark a significant sea change if there was widespread recognition that not completing a course didn’t automatically equate with failure.
I’ve spoken to a number of people in recent weeks about their experiences of #oldsmooc and #edcmooc and many of them have in their own words “given up”. But as discussion has gone on it is apparent that they have all gained something from even cursory participation either in terms of their own thinking about possible involvement in running a MOOC like course, or about realising that although MOOCs are free there is still the same time commitment required as with a paid course.
Of course I am very fortunate that I work and mix with a pretty well educated bunch of people, who are in the main part really interested in education, and are all well educated with all the recognised achievements of a traditional education. They are also digital literate and confident enough to navigate through the massive online social element of MOOCs, and they probably don’t need any more validation of their educational worth.
But what about everyone else? How do you start to make sense of the badges, certificates you may or may not collect? How can you control the way that you show these to potential employers/Universities as part of any application? Will they mean anything to those not familiar with MOOCs – which is actually the vast majority of the population. I know there are some developments in California in terms of trying to get some MOOCs accredited into the formal education system – but it’s very early stages.
Again based on my own experience, I was quite strategic in terms of the #edcmooc, I wrote a reflective blog post for each week which I was then able to incorporate into my final artefact. But actually the blog posts were of much more value to me than the final submission or indeed the certificate (tho I do like the spacemen). I have seem an upward trend in my readership, and more importantly I have had lots of comments, and ping backs. I’ve been able to combine the experience with my own practice.
Again I’m very fortunate in being able to do this. In so many ways my blog is my portfolio. Which brings me a very convoluted way to my point in this post. All this MOOC-ery has really started me thinking about e-portfolios. I don’t want to use the default Coursera profile page (partly because it does show the course I have taken and “not received a certificate” for) but more importantly it doesn’t allow me to incorporate other non Coursera courses, or my newly acquired badges. I want to control how I present myself. This relates quite a lot to some of the thoughts I’ve had about using Cloudworks and my own educational data. Ultimately I think what I’ve been alluding to there is also the development of a user controlled e-portfolio.
So I’m off to think a bit more about that for the #lak13 MOOC. Then Lorna Campbell is going to start my MOOC de-programming schedule. I hope to be MOOC free by Christmas.
Professor David Boud got this year’s eAssessment Scotland Conference off to a great start with his “new conceptions of feedback and how they might be put into practice” keynote presentation by asking the fundamental question ‘”what is feedback?”
David’s talk centred on what he referred to as the “three generations of feedback”, and was a persuasive call to arms to educators to move from the “single loop ” or “control system” industrial model of feedback to a more open adaptive system where learners play a central and active role.
In this model, the role of feedback changes from being passive to one which helps to develop students allowing them to develop their own judgement, standards and criteria. Capabilities which are key to success outside formal education too. The next stage from this is to create feedback loops which are pedagogically driven and considered from the start of any course design process. Feedback becomes part of the whole learning experience and not just something vaguely related to assessment.
In terms of technology, David did give a familiar warning that we shouldn’t enable digital systems to allow us to do more “bad feedback more efficiently”. There is a growing body of research around developing the types of feedback loops David was referring to. Indeed the current JISC Assessment and Feedback Programme is looking at exactly the issues brought up in the keynote, and is based on the outcomes of previously funded projects such as REAP and PEER. And the presentation from the interACT project I went to immediately after the keynote, gave an excellent overview of how JISC funding is allowing the Centre for Medical Education in Dundee to re-engineering its assessment and feedback systems to “improve self, peer and tutor dialogic feedback”.
During the presentation the team illustrated the changes to their assessment /curriculum design using an assessment time line model developed as part of another JISC funded project, ESCAPE, by Mark Russell and colleagues at the University of Hertfordshire.
Lisa Gray, programme manager for the Assessment and Feedback programme, then gave an overview of the programme including a summary of the baseline synthesis report which gives a really useful summary of the issues the projects (and the rest of the sector ) are facing in terms of changing attitudes, policy and practice in relation to assessment and feedback. These include:
*formal strategy/policy documents lagging behind current development
*educational principles are rarely enshrined in strategy/policylearners are not often actively enaged in developing practice
*assessment and feedback practice doesn’t reflect the reality of working life
*admin staff are often left out of the dialogue
*traditional forms of assessment still dominate
*timeliness of feedback are still an issue.
More information on the programme and JISCs work in the assessment domain is available here.
During the lunch break I was press-ganged/invited to take part in the live edutalk radio show being broadcast during the conference. I was fortunate to be part of a conversation with Colin Maxwell (@camaxwell), lecturer at Carnegie College, where we discussed MOOCs (see Colin’s conference presentation) and feedback. As the discussion progressed we talked about the different levels of feedback in MOOCs. Given the “massive” element of MOOCs how and where does effective feedback and engagement take place? What are the afordances of formal and informal feedback? As I found during my recent experience with the #moocmooc course, social networks (and in particular twitter) can be equally heartening and disheartening.
I’ve also been thinking more about the subsequent twitter analysis Martin has done of the #moocmooc twitter archive. On the one hand, I think these network maps of twitter conversations are fascinating and allow the surfacing of conversations, potential feedback opportunities etc. But, on the other, they only surface the loudest participants – who are probably the most engaged, self directed etc. What about the quiet participants, the lost souls, the ones most likely to drop out? In a massive course, does anyone really care?
Recent reports of plagiarism, and failed attempts at peer assessment in some MOOCs have added to the debate about the effectiveness of MOOCs. But going back to David Boud’s keynote, isn’t this because some courses are taking his feedback mark 1, industrial model, and trying to pass it off as feedback mark 2 without actually explaining and engaging with students from the start of the course, and really thinking through the actual implications of thousands of globally distributed students marking each others work?
All in all it was a very though provoking day, with two other excellent keynotes from Russell Stannard sharing his experiences of using screen capture to provide feedback, and Cristina Costa on her experiences of network feedback and feeding forward. You can catch up on all the presentations and join in the online conference which is running for the rest of this week at the conference website.
The latest webinar from the JISC Assessment and Feedback programme will take place on 23 July (1-2pm) and will feature the SGC4L (Student Generated Content for Learning) project. Showcasing the Peerwise online environment the project team will illustrate to participants how it can be used by students to generate their own original assessment content in the form of multiple choice questions. The team will discuss their recent experiences using the system to support teaching on courses at the University of Edinburgh and the findings of the project. The webinar will include an interactive session offering participants the opportunity get first hand experience of interacting with others via a PeerWise course set up for the session.
For further details and links to register for this free webinar are available by following this link.
Finding common understandings is a perennial issue for those of us working in educational technology and lack of understanding between techies and non techies is something we all struggle with. My telling some of the developers I used to work with the difference between formative and summative assessments became something of an almost daily running joke. Of course it works the other way round too and yesterday I was taken back to the days when I first came into contact with the standards world and its terminology, and in particular ‘bindings’.
I admit that for a while I really didn’t have a scoobie about bindings, what they were, what the did etc. Best practice documentation I could get my head around, and I would generally “get” an information model – but bindings, well that’s serious techie stuff and I will admit to nodding a lot whilst conversations took place around me about these mysterious “bindings”. However I did eventually get my head around them and what their purpose is.
Yesterday I took part in a catch up call with the Traffic project at MMU (part of the current JISC Assessment and Feedback programme). Part of the call involved the team giving an update on the system integrations they are developing, particularly around passing marks between their student record system and their VLE, and the development of bindings between systems came up. After the call I noticed this exchange on twitter between team members Rachel Forsyth and Mark Stubbs.
so @thestubbs has just explained to me that ‘binding’ info in system 1 means telling system 2 to do something special with relevant data
Next week sees the CCA Conference and there is a pre-conference workshop where some of the assessment related standards work being funded by JISC will be shared. The workshop will include introductions to:
• A user-friendly editor called Uniqurate, which produces questions conforming to the Question and Test Interoperability specification, QTIv2.1,
• A way of connecting popular VLEs to assessment delivery applications which display QTIv2.1 questions and tests – this connector itself conforms to the Learning Tools Interoperability specification, LTI,
• A simple renderer, which can deliver basic QTIv2.1 questions and tests,
• An updated version of the comprehensive renderer, which can deliver QTIv2.1 questions and tests and also has the capability to handle mathematical expressions.
There will also be demonstrations of the features of the QTI Support site, to help users to get started with QTI.
The workshop will also provide an opportunity to discuss participants’ assessment needs and to look at the ways these might be addressed using the applications we have available and potential developments which could be part of future projects.
If you are interested in attending the conference, email Sue Milne email@example.com with your details as soon as possible.
Those of you who regularly read this blog, will (hopefully) have noticed lots of mentions and links to the Design Studio. Originally built as a place to share outputs from the JISC Curriculum Design and Delivery Programmes, it is now being extended to include ouputs from a number of other JISC funded programmes.
The Transforming Assessment and Feedback area of the Design Studio now has a series of pages which form a hub for existing and emergent work on assessment and feedback of significant interest. Under a series of themes, you can explore what this community currently know about enhancing assessment and feedback practice with technology, find links to resources and keep up to date with outputs from the Assessment and Feedback and other current JISC programmes.
This is a dynamic set of resources that will be updated as the programme progresses. Follow this link to explore more.