School Data Pseudoscience

I intend to write a short series of posts regarding the effective use of data in schools. This is out of keeping with the rest of my blog, but a matter of personal interest to me. Before I became a teacher my career involved the reporting and analysis of data for large banks (salary, age, qualifications, performance ratings, that kind of thing) so I have a good understanding of data interpretation. I also have a statistics degree from UCL, but here I’m not presenting any statistical analyses. I’m just offering my views in the hope of hearing the opinions of others.

Performance Benchmarks - Alps
I'm going to start with a brief look at Alps target grades. My initial thoughts on the subject are based on small amounts of empirical evidence - I have not yet taken the time to do any formal research or analysis and I would be incredibly grateful if anyone could point me in the direction of anything that has already been written on the subject.

Background
Like most organisations, schools use data to benchmark performance. FFT data and RAISEonline data are widely used in the UK. It is understandable that schools want to know how they're doing compared to others. It is sensible for leadership to evaluate the effectiveness of their strategies.

Until last year my school used Alis data for internal analysis of A level grades. However the target grade that students were aware of - the one that appeared on their reports - was their UCAS predicted grade which had been determined by their teachers.

Last year our new headteacher invested in Alps data. Alps reports are very well presented in a 'we know what we're talking about and we have good software' kind of way. I'm not suggesting the people at Alps don't know what they're talking about, I'm sure their methods are robust, but I want to explore the effectiveness of Alps grades from a teacher's perspective.

Setting students up to fail?
Let's take two of my students as examples - I'll call them Amy and Beth. They both did AS level maths with me last year.

Both Amy and Beth did well at GCSE. They achieved a good number of A* and A grades. According to their Alps targets, they'll both achieve straight As at A level, whatever subjects they take. Now we all know about the step up from GCSE to A Level - As are very hard to achieve at Key Stage 5. To do so, students need to be both very bright and very hard working.

In the first week of Year 12, our students sat an entry test in maths (more on this here). The test covered basic maths skills including arithmetic, fractions and solving equations. The results were worrying - a large number of students scored less than half marks on this very simple test. I intervened with remedial lunchtime sessions in the first few weeks of the school year. This was effective for some pupils - they responded well to the message that their maths wasn't up to scratch. They took the remedial sessions seriously, did the extra work to catch up and ultimately did well in their summer exams.

Amy achieved a mere 16% in her entry test. I immediately raised concerns with her Head of Year. In my opinion maths was not the right choice for Amy. But she stayed in my class. I worried endlessly about her throughout the year. She was neither a natural mathematician nor a hard worker. By the end of the first half term I knew that she would probably fail the course. I always want the best for my students so I kept encouraging and supporting her. I was also anxious because I knew I'd be criticised if she didn't meet her Alps target. I'd be thought of as an underperforming teacher who failed to help her students reach their potential. 

Poor Amy was being set up to fail. Her Alps target was misleading. She was new to my school so I never saw her GCSE grades but perhaps she had done well in arts and humanities. Anyone with any sense knows that if a pupil is good at humanities at GCSE it does not follow that they will be good at A level maths. But Alps targets are not subject specific and this is one of their fundamental flaws.

The numerous interventions I tried with Amy were not effective. She didn't put in her share of the effort. She was full of excuses and didn't try hard enough. The maths didn't click for her. In her reports throughout the year I gave her D grades but even that was aspirational. 

As expected, Amy got a U in AS maths in the end. Her grades in other subjects were low too - she certainly didn't achieve the straight As that Alps had promised her. Dare I say that she just wasn't clever enough?

I believe in the power of aspirational targets as a motivator but targets must be realistic or they become meaningless. Were these pseudoscientific Alps targets of any benefit to Amy? No. Was it all my fault that Amy didn't meet her targets? No. She probably blames me though.




Beth was very similar to Amy at the outset. She struggled in maths from the word go but she worked hard all year and got there in the end. She got a C. I was overjoyed on results day. She may feel like a failure because she was two grades off her Alps target, but to me she should be very proud of her amazing result. I thought she might get a U so I'm chuffed with what she achieved - I worked hard for that C and so did she. She should be happy. What a shame her Alps target begs to differ. To me, Beth is a success. In a school driven by meaningless Alps targets, she is a failure, and so am I.

A measure of teacher performance?
I taught some very bright young mathematicians last year. They achieved As even though their Alps targets were Bs. Should I take all the credit for this outperformance? No. Their targets were wrong. They were always capable of achieving As. I saw that from the start of the year, and I told them so. I helped them reach their real potential, not the limited potential dictated by artificial targets. But in the same way I shouldn't be criticised when girls like Beth get a grade C, I shouldn't take all the glory for strong performance. My students sat the exam, not me.

I know I do a good job with my classes so why am I made to feel like a failure by SLT? Why are teachers constantly told they're letting people down? I really do everything I can for my students. I don't appreciate the criticism. And to determine my pay based on Alps targets? Surely anyone can see that's not going to have the desired effect. Performance-related pay is apparently meant to make me a better teacher. But objectives have to be realistic, sensible and achievable otherwise they are utterly pointless. By holding back my pay - money that I need to support my family - because Amy and Beth didn't achieve their targets, the school is not motivating me at all. They are frustrating me and driving me away. They will lose talented teachers and the situation will worsen.

I have one more example that's worth a mention. It's a small case study but opens the door to wider analysis. Last year we had two Further Maths classes in Year 13. The 10 students in each class had very similar Alps targets, but the teachers felt that one class was weaker than the other. Nevertheless, the classes were taught in exactly the same way. Same teachers, same lessons, same resources, same exams. One class met their targets and the other didn't. So was it the teaching that determined whether or not the students met their targets? Both classes had the exact same teaching experience and yet performed differently. The Alps report for one class told the teachers they should have done better. The Alps report for the other class said the teachers had done a good job. Go figure.

Conclusion
Some people seem to have the view that there is a perfect cause-and-effect correlation between the performance of teachers and the performance of their pupils. Clearly this is not the case - there are numerous other influencing factors. Students' performance is not a clear indicator of my effectiveness as an employee. I appreciate the need for objective measures of performance instead of wholly subjective measures, but these Alps targets are all wrong. The data makes no sense. Evidence suggests that in teaching, performance related pay does not impact teachers' performance. If we're being judged against targets that we don't support, it's clear to see why not.

In my opinion there are far better uses for school money than buying Alps reports. Instead we should just tell all students to aim high and do their best. SLT should trust and support teachers to help students reach their potential. When I did my A levels I wasn't given a target grade. I don't believe they're necessary or even desirable.

I've been critical of Alps targets here but I am very open to hearing counter-arguments and evidence for their effectiveness. Please comment below or email me if you have a view or can point me in the direction of any relevant research, other than that written by the people behind Alps.





***********************
Update 6/9/14

Thank you to everyone who has emailed me their views on this.

I highly recommend the excellent article "Target Setting doesn’t have to be hard work..." by @CharlotteSISRA. I feel better knowing that the fault lies with my SLT who are clearly misusing the data. Alps grades are estimates and should be combined with professional judgement to make predictions. An aspirational element can be added to predictions to set targets. It makes total sense. My school are confusing estimates with targets and therefore missing the vital teacher input that makes them meaningful. Can I have a volunteer to tell my headteacher please?

fft.org.uk

3 comments:

  1. Couldn't agree more. Had a really bright GCSE class in my NQT year who did brilliantly, and a decidedly mediocre and (despite all my efforts) lazy GCSE class last year who failed to meet their targets. I don't think I should be blamed for the weaker class any more than I should be praised for the bright one. A teacher can potentially be a limiting factor, but for the most part we provide a facilitating environment, and those who are determined to learn won't be stopped while those who are determined not to can't be forced.

    Aside from performance related pay making the teaching profession less attractive to current and prospective teachers, to work your socks off all year only to be told you've failed doesn't do much for staff morale.

    ReplyDelete
    Replies
    1. I'm glad I've found someone who agrees with me! I sometimes rant about this at school, but I find that in education it's hard to voice an opinion without falling out of favour with SLT (which could further impact pay and progression! Grrr). I totally agree with what you say about how teachers can be a limiting factor, I know of an example where Alps reports have backed up concerns about an underperforming teacher - but not exposed them in the first place. A good HoD is aware of issues with his/her teaching staff without the need for expensive reports.

      What you say about morale is spot on - the majority of teachers do work their socks off all year and should be celebrated for doing so.

      Delete
  2. More comments here: https://twitter.com/oldandrewuk/status/507794732341149696

    ReplyDelete