Thanks for this post Chris. For someone like me whose interest in education comes from the effective teaching side, I feel you're giving a good argument from the viewpoint of the "other side", so to speak (say, the mainstream education academics), without dismissing the importance of effective teaching, or attempting to demonize it by linking it to authoritarian politics. Mainstream education academics are often fond of saying proponents of effective teaching methods "lack nuance"; you've done a good job showing what said nuance would actually look like. I'll certainly be reading your other posts with interest.
I'm aware of the importance of qualitative educational research. It can answer questions that quantitative research cannot answer, but at the same time quantitative research can also answer questions that qualitative research cannot answer. Regarding Jo Boaler, you say that "Peer review eventually did its job, but ‘eventually’ was after California had built its new curriculum based on her work." Isn't one of the problems that even if there had been no academic malpractice, the type of research Boaler was doing simply did not have the necessary power to justify driving policy? I see this often from mainstream education academics, who claim that the kind of research effective teaching proponents point to should not be viewed as strong enough to drive educational policy (for the very reasons you highlight in your post), but do not seem to think there is any problem if their own (qualitative, with a limited sample size) research is used as a model for policy changes. This is what causes me to think that mainstream education academics are often ideologically-driven: research is good if it supports "good" ideas.
Here in Quebec, the government announced a few years ago that it would launch an Institute of excellence in education, which I understand to be a "what-works project" of the kind you describe. Many education academics took to the media to complain about how this lacks nuance and would favour a specific kind of research (quantitative, large-scale) and a specific goal for school (effectiveness and results) over the kind of research that they prefer doing and their own view of what school is for. I personally think the creation of an Institute of excellence in education, if it happens, will mostly be a good idea. True, it will probably be focused on a particular goal and on a particular type of research, but that should not stop education researchers from working on what they want to work on. Though they'll have to understand that their research also has limitations and should not be used to justify large reforms of the education system of the kind we've seen in the last few decades (and which gave disappointing results).
Thanks Marc. This is a great perspective, and it illustrates exactly why it's time to move past what has at times been a partisan view about what education research is for.
It's really interesting to get the Quebec angle on the 'what works' project. Again, I think a dichotomous view of these projects doesn't help move education research forward. One of the EEF's most valuable contributions, in my view, is their 'teaching and learning toolkit' – their synthesis of existing research. Like the large-scale trials, it has its limitations, but it's had huge influence and impact across schools. Undoubtedly, these projects have improved what I do in the classroom.
'Isn't one of the problems that even if there had been no academic malpractice, the type of research Boaler was doing simply did not have the necessary power to justify driving policy?' To an extent, yes. But the kind of work she was doing is the sort of thing that can feed into the trials the EEF and other 'what works' centres support. Their pilots, which they scale up, have to start somewhere, and you need people willing to try something new and question the status quo to generate the ideas that are worth scaling (with the 'scaling up' caveats I mention in the post).
'This is what causes me to think that mainstream education academics are often ideologically-driven: research is good if it supports "good" ideas.' Totally agree – but it's applicable to 'both sides'. If only there were a way of moving past this! The issue is that each side thinks its own view is 'evidence-based' (albeit with a different view of what constitutes 'evidence'), and therefore can't be ideologically driven. But ideology drives the kind of questions we choose to investigate and the methods we use – and I wouldn't exempt myself from that, although I do try to see the progress each perspective has enabled.
I'll leave it there for now. Great comment. For what it's worth, I think you've caught the issues with education research more accurately than Piper did.
Hi Chris, I recently came across Kelsey Piper through her recent piece about the flaws of educational research - I wasn't aware of her before. Very much enjoyed reading your counter response to her rather pessimistic counter response. The importance of qualitative research for school accountability and continual learning outcomes certainly seems like something that was missing from Piper's account, which you explored thoroughly. Thank you again for your thoughts on this, looking forward to your next piece!
Thanks Sam - glad you enjoyed it and great to have your thoughts. Your series (https://substack.com/@samuelkammin/p-192354518) shows why it's more important than ever to find out what's actually going on inside schools. Looking forward to your next one!
Why does improving quantitative research imply not doing qualitative research?
This seems like you agree with Piper that quantitative research can be improved but then want to complain that she is not mentioning qualitative research.
I notice that you do not go into any research on qualitative research methods or look at how other disciplines tackle it.
Instead of a hey you missed this other topic why not provide something useful that moves the needle on qualitative research.
You seem to want to create an argument where none was needed.
Thanks for reading, Stan, and for the comments. Going in order:
1 & 2. ‘Why does improving quantitative research imply not doing qualitative research?’ It doesn't, obviously. My piece wasn't arguing against improving quant research (I do it myself) - it was flagging that (a) qual was absent from Piper's framing, and (b) several of her proposed fixes are already in use (the EEF being the obvious example), with practical limitations she didn't engage with.
3. ‘I notice that you do not go into any research on qualitative research methods or look at how other disciplines tackle it.’ Interestingly, in the Nature article, across all social science disciplines only about 20% of papers cited seemed to make replicable quant claims at all, meaning (by my reading) that the qual:quant ratio across these is similar. There's scope to do this, just not here.
4. ‘Instead of a hey you missed this other topic why not provide something useful that moves the needle on qualitative research.’ Improving qual methods deserves its own post, and honestly more reading than I've done. Decent starting points: Billig's Learn to Write Badly (pan-social-sci criticism) and Hammersley's ‘Questioning Qualitative Inquiry’. Might come back to it. My point, as I said, wasn't that qual research is better, just that it's necessary.
5. ‘You seem to want to create an argument where none was needed.’ Fundamentally, I disagree that education research is ‘broken’, and to claim that, you need to ask what it's for. I think Piper missed central features of education research - the role of qual, discussion of the purpose of education research, and the existence of large-scale ed research following some of the methods she suggested but didn't seem aware were already being used. To me, that warranted a reply; happy to agree to disagree.
Happy to continue but won't be able to pick it up until later. Thanks again for the comments.
Piper is writing about a problem in education research. That is the lack of the same progress in improving quantitative research seen in other disciplines. I am not sure if you think this is the case or not. You give examples. But the links you give are not directly addressing how well the field has adopted similar practices to other fields. It would take a lot of digging through the links you provided to see if there is even any claim about this there.
Arguing that there is lots of other non-quantitative research seems irrelevant to Piper's point and a distraction. Arguing that she doesn't address the purpose of education also seems irrelevant. Whatever the purpose, surely the quantitative research should address known short-comings and aim to be as good as possible.
You might worry that you come across as attempting to win an argument by changing the ground rather than addressing the other persons key point.
If you look at your recent piece of curiosity where the research data was withheld because the funder sees it as competitive information they don't want to share, I'd of thought you would be fully behind Piper's complaint. Having a standard where to publish, the data and code must be public is in your interest if you want to base your work on the conclusions drawn from it.
'I am not sure if you think this is the case or not.' The Nature article on its own makes the case that quant ed research lags on transparency, and I agree the field should catch up. Education research has real ethical hurdles around data sharing, particularly anonymity (but so do other fields). To access the UK government's pupil tracking database you have to jump through various hoops. Fully transparent? Debatable. But there is at least a mechanism.
So, agreed: we need to share data and think harder about how to work around the obstacles. On code sharing, it should be the default and there’s no good reason not to.
Thanks for reading the curiosity piece - hope this covers that too. I use and value publicly available data like the OECD's, and per my PISA post, I share code where the analysis warrants it. The curiosity data didn't really - the analysis was pretty basic, and it's sadly unlikely to be a policy driver (though there's always hope).
'Arguing that there is lots of other non-quantitative research seems irrelevant to Piper's point.' Strongly disagree here. Piper's framing wasn't 'quant ed research has problems', it was that education research as a discipline is broken. If you treat 'fix the quant problems' as equivalent to 'fix education research', there’s a lot you overlook.
Anyway, I think we've worked this one through about as far as it can usefully go. Appreciate the discussion - I'll let the post and the replies stand on this for now, and look forward to picking it up on the next one.
Thanks for this post Chris. For someone like me whose interest in education comes from the effective teaching side, I feel you're giving a good argument from the viewpoint of the "other side", so to speak (say, the mainstream education academics), without dismissing the importance of effective teaching, or attempting to demonize it by linking it to authoritarian politics. Mainstream education academics are often fond of saying proponents of effective teaching methods "lack nuance"; you've done a good job showing what said nuance would actually look like. I'll certainly be reading your other posts with interest.
I'm aware of the importance of qualitative educational research. It can answer questions that quantitative research cannot answer, but at the same time quantitative research can also answer questions that qualitative research cannot answer. Regarding Jo Boaler, you say that "Peer review eventually did its job, but ‘eventually’ was after California had built its new curriculum based on her work." Isn't one of the problems that even if there had been no academic malpractice, the type of research Boaler was doing simply did not have the necessary power to justify driving policy? I see this often from mainstream education academics, who claim that the kind of research effective teaching proponents point to should not be viewed as strong enough to drive educational policy (for the very reasons you highlight in your post), but do not seem to think there is any problem if their own (qualitative, with a limited sample size) research is used as a model for policy changes. This is what causes me to think that mainstream education academics are often ideologically-driven: research is good if it supports "good" ideas.
Here in Quebec, the government announced a few years ago that it would launch an Institute of excellence in education, which I understand to be a "what-works project" of the kind you describe. Many education academics took to the media to complain about how this lacks nuance and would favour a specific kind of research (quantitative, large-scale) and a specific goal for school (effectiveness and results) over the kind of research that they prefer doing and their own view of what school is for. I personally think the creation of an Institute of excellence in education, if it happens, will mostly be a good idea. True, it will probably be focused on a particular goal and on a particular type of research, but that should not stop education researchers from working on what they want to work on. Though they'll have to understand that their research also has limitations and should not be used to justify large reforms of the education system of the kind we've seen in the last few decades (and which gave disappointing results).
Thanks Marc. This is a great perspective, and it illustrates exactly why it's time to move past what has at times been a partisan view about what education research is for.
It's really interesting to get the Quebec angle on the 'what works' project. Again, I think a dichotomous view of these projects doesn't help move education research forward. One of the EEF's most valuable contributions, in my view, is their 'teaching and learning toolkit' – their synthesis of existing research. Like the large-scale trials, it has its limitations, but it's had huge influence and impact across schools. Undoubtedly, these projects have improved what I do in the classroom.
'Isn't one of the problems that even if there had been no academic malpractice, the type of research Boaler was doing simply did not have the necessary power to justify driving policy?' To an extent, yes. But the kind of work she was doing is the sort of thing that can feed into the trials the EEF and other 'what works' centres support. Their pilots, which they scale up, have to start somewhere, and you need people willing to try something new and question the status quo to generate the ideas that are worth scaling (with the 'scaling up' caveats I mention in the post).
'This is what causes me to think that mainstream education academics are often ideologically-driven: research is good if it supports "good" ideas.' Totally agree – but it's applicable to 'both sides'. If only there were a way of moving past this! The issue is that each side thinks its own view is 'evidence-based' (albeit with a different view of what constitutes 'evidence'), and therefore can't be ideologically driven. But ideology drives the kind of questions we choose to investigate and the methods we use – and I wouldn't exempt myself from that, although I do try to see the progress each perspective has enabled.
I'll leave it there for now. Great comment. For what it's worth, I think you've caught the issues with education research more accurately than Piper did.
Hi Chris, I recently came across Kelsey Piper through her recent piece about the flaws of educational research - I wasn't aware of her before. Very much enjoyed reading your counter response to her rather pessimistic counter response. The importance of qualitative research for school accountability and continual learning outcomes certainly seems like something that was missing from Piper's account, which you explored thoroughly. Thank you again for your thoughts on this, looking forward to your next piece!
Thanks Sam - glad you enjoyed it and great to have your thoughts. Your series (https://substack.com/@samuelkammin/p-192354518) shows why it's more important than ever to find out what's actually going on inside schools. Looking forward to your next one!
Why does improving quantitative research imply not doing qualitative research?
This seems like you agree with Piper that quantitative research can be improved but then want to complain that she is not mentioning qualitative research.
I notice that you do not go into any research on qualitative research methods or look at how other disciplines tackle it.
Instead of a hey you missed this other topic why not provide something useful that moves the needle on qualitative research.
You seem to want to create an argument where none was needed.
Thanks for reading, Stan, and for the comments. Going in order:
1 & 2. ‘Why does improving quantitative research imply not doing qualitative research?’ It doesn't, obviously. My piece wasn't arguing against improving quant research (I do it myself) - it was flagging that (a) qual was absent from Piper's framing, and (b) several of her proposed fixes are already in use (the EEF being the obvious example), with practical limitations she didn't engage with.
3. ‘I notice that you do not go into any research on qualitative research methods or look at how other disciplines tackle it.’ Interestingly, in the Nature article, across all social science disciplines only about 20% of papers cited seemed to make replicable quant claims at all, meaning (by my reading) that the qual:quant ratio across these is similar. There's scope to do this, just not here.
4. ‘Instead of a hey you missed this other topic why not provide something useful that moves the needle on qualitative research.’ Improving qual methods deserves its own post, and honestly more reading than I've done. Decent starting points: Billig's Learn to Write Badly (pan-social-sci criticism) and Hammersley's ‘Questioning Qualitative Inquiry’. Might come back to it. My point, as I said, wasn't that qual research is better, just that it's necessary.
5. ‘You seem to want to create an argument where none was needed.’ Fundamentally, I disagree that education research is ‘broken’, and to claim that, you need to ask what it's for. I think Piper missed central features of education research - the role of qual, discussion of the purpose of education research, and the existence of large-scale ed research following some of the methods she suggested but didn't seem aware were already being used. To me, that warranted a reply; happy to agree to disagree.
Happy to continue but won't be able to pick it up until later. Thanks again for the comments.
Piper is writing about a problem in education research. That is the lack of the same progress in improving quantitative research seen in other disciplines. I am not sure if you think this is the case or not. You give examples. But the links you give are not directly addressing how well the field has adopted similar practices to other fields. It would take a lot of digging through the links you provided to see if there is even any claim about this there.
Arguing that there is lots of other non-quantitative research seems irrelevant to Piper's point and a distraction. Arguing that she doesn't address the purpose of education also seems irrelevant. Whatever the purpose, surely the quantitative research should address known short-comings and aim to be as good as possible.
You might worry that you come across as attempting to win an argument by changing the ground rather than addressing the other persons key point.
If you look at your recent piece of curiosity where the research data was withheld because the funder sees it as competitive information they don't want to share, I'd of thought you would be fully behind Piper's complaint. Having a standard where to publish, the data and code must be public is in your interest if you want to base your work on the conclusions drawn from it.
Thanks Stan. A few quick clarifications:
'I am not sure if you think this is the case or not.' The Nature article on its own makes the case that quant ed research lags on transparency, and I agree the field should catch up. Education research has real ethical hurdles around data sharing, particularly anonymity (but so do other fields). To access the UK government's pupil tracking database you have to jump through various hoops. Fully transparent? Debatable. But there is at least a mechanism.
So, agreed: we need to share data and think harder about how to work around the obstacles. On code sharing, it should be the default and there’s no good reason not to.
Thanks for reading the curiosity piece - hope this covers that too. I use and value publicly available data like the OECD's, and per my PISA post, I share code where the analysis warrants it. The curiosity data didn't really - the analysis was pretty basic, and it's sadly unlikely to be a policy driver (though there's always hope).
'Arguing that there is lots of other non-quantitative research seems irrelevant to Piper's point.' Strongly disagree here. Piper's framing wasn't 'quant ed research has problems', it was that education research as a discipline is broken. If you treat 'fix the quant problems' as equivalent to 'fix education research', there’s a lot you overlook.
Anyway, I think we've worked this one through about as far as it can usefully go. Appreciate the discussion - I'll let the post and the replies stand on this for now, and look forward to picking it up on the next one.