The day after the night before: thinking about data #codesign16

I’m writing this the morning after the night before the US 2016 election. What a day to be writing about data, and in this context learning analytics. The same thing happened to me earlier this year with the Brexit result, I was giving a keynote about learning analytics the day after that night too.

Once again the pollsters – even Nate Silver, the data, got it wrong.  Those pesky people who weren’t on the data gathering radar went out and voted – and not in the way that the data predicted.

I don’t know if Facebook “called it” or not but there are some pretty big questions to be asked about the predictive algorithms being used for elections.  What this all means is not something I’m going to talk about in this post (you might want to read this excellent response to the election result from Lorna Campbell in that respect), but I cannot not mention it today.

You may be aware that Jisc is running its codesign challenge just now and “how can we use data to improve teaching and learning” is on of their 6 challenges.

With both government and students focusing on value for money and a quality student experience, it is essential that universities and colleges are directing their resources appropriately to deliver the best learning experiences to students. Universities will need to draw on the vast array of data and information available not just to demonstrate the quality of their teaching for the TEF, but to examine what makes the most difference to students’ learning, employability and overall satisfaction with their experience, in order to continuously improve their offer. Fuller text about the challenge is at: https://www.jisc.ac.uk/rd/get-involved/how-can-we-use-data-to-improve-teaching-and-learning

Jisc have fielded a number of questions:

• How are you currently using data to improve learning, teaching and student outcomes?
• What would be the key questions about learning and teaching that you would like to see explored through data-driven approaches?
• What data is needed in FE to better understand which aspects of course design and teaching lead to higher success rates for their learners?
• How can a data-driven approach lead to improved quality and greater understanding of higher education, without risking losing its richness and diversity?
• Should the UK HE offer be more strongly shaped by higher-level skills needs in the national and international economies?

Like many colleagues in the sector, I am trying to get a handle on our institutional analytics capabilities. I don’t really know what we want to do yet, but just have this feeling that there are indeed some “actionable insights” that we could get from a more systematic exploration and visualisation of our data that would help us in some way to address all of the questions above. However, before I can do that I have to overcome my biggest problem and that is actually getting at the data. I’ve blogged about this before and shared our experiences at this year’s ALT conference.  It is still “a big issue.”  As is consent, and increasingly, data processing agreements.

This is one area where I think Jisc could (and through the current effective learning analytics programme are beginning to)  make a significant contribution to sectoral knowledge and understanding.

I worry about the lack of understanding and subsequent misunderstanding about secondary use of data; about multiple, varied DPA agreements between institutions and vendors, that may inadvertently forget about student consent, about the still unanswered question of who actually owns student data – is it institutions or the individual students?   I know none of this is new, but it is still challenging.

I’m still very wary of predictive analytics, and the dangerous metrics and measurements that are being discussed as what can still only be seen as proxies for “student engagement”  I couldn’t actually answer this question, it’s not a binary choice or even reality for me just now.

However I do think have more insight into our data can help us in terms of our understanding.  Understanding engagement is a huge task. Do we even have a clear understanding of what we mean by engagement? In TEF terms is this engagement and ergo success (passing) in assessments? Does this relate only to digital interactions, what about our offline engagement? How do we measure critical reflection, perhaps 5-10-30 years after leaving formal education?

One thing’s for sure today of all days we have seen how predictive data analysis has failed to measure human behaviour and engagement or lack of it.

That said I’m looking forward to the rest of the Jisc discussion as it unfolds. If you have any views then, please join the discussion using #codesign16 and this Friday (11th November) there is a tweet chat at 12.30 -1.30.