Tuesday, 17 April 2012

Good news: Work Experience works

Six weeks ago I wrote here and here that:
"the debate on work experience seemed almost entirely divorced from the evidence of whether work experience actually improved the employment opportunities of jobless young people. "
and
"DWP [the Department for Work and Pensions] should attempt to produce a proper counterfactual analysis that would allow us to come to a considered judgement on the programme's success. "
DWP has now done just that, publishing today an impact assessment of the Work Experience programme.  The analysis compares participants on the work experience programme with "comparable" (as determined by sophisticated statistical techniques) non-participants. 



The details are highly technical, but the key findings are:
"estimates suggest that WE had a strong and beneficial impact on the likelihood of a participant receiving benefit compared to the impact if they had not participated - participants became much less likely to be on benefit than non-participants; the central estimate is -6 percentage points by week 13"
and 
"work experience had a strong and beneficial impact on the likelihood of a participant being in employment compared to non-participants - participants were more likely to be in employment than non-participants: the central estimate reached +8 percentage points."
Here is a chart illustrating the first point:




A team here at NIESR, led by David Wilkinson and including me, performed a peer review of the analysis. As the DWP publication says:
"NIESR concluded that the methodology was sound, and that the key findings and conclusions - in particular the finding that work experience had a significant impact reducing benefit receipt and increasing employment for participants, compared to otherwise comparable non-recipients – appeared robust."
This is excellent news, for two reasons. First, it shows that work experience works.  I argued in my original article:
"My view - shared by most of those who study and evaluate such programmes - is that work experience is likely to improve outcomes for young people, albeit not as much as implied by Ministers' claims. " 
This is precisely what the impact assessment says. The impacts are not huge: 16 weeks after starting the programme, 46 percent of participants are off benefit, compared to 40 percent of comparable non-participants; 35 percent are recorded as being in employment, compared to 27 percent of non-participants (the figures for employment are probably somewhat less reliable, for data reasons).  However, the impact is very clearly positive, and given that the programme is relatively cheap, it is almost certainly a worthwhile investment. 


Second, DWP - both those who commissioned this work, and the extremely professional and competent analysts who  performed it -  should be congratulated on performing a detailed and robust impact analysis, subjecting it to outside scrutiny, and publishing the results. Of course, this should happen, where feasible (and it usually is in some form) for all major social programmes; and it is a pity it took a political row to get us here.  Nevertheless, they did the right thing, and they deserve credit. 


What, then, should we conclude from the recent debate about the effectiveness or otherwise of work experience?


First, the question I asked above "whether work experience actually improves the employment opportunities of jobless young people?" has been answered; it does. As our review suggested, and as the impact analysis rightly acknowledges, there is much more analysis that could be be done, and many important issues and questions remain; but in my view it is highly unlikely to overturn this basic point. Critics of the programme should accept this, and move on to, as the Commission on Youth Unemployment suggested, consider how it could be improved. 


Second, these events show that not only do Ministers have a responsibility to ensure that policies are properly evaluated, and that the debate takes place on the basis of the best available evidence, but the government would actually be better off if they did so.  If DWP had never published this initial, highly selective and hence misleading, analysis, then the debate about work experience would have been hugely more positive; a good programme would not have been tarnished in public because the lack of proper evidence meant both sides were able to make misleading claims


Third, the fact that the actual impact of the programme is much smaller than implied by Ministers' claims at the time - but at the same time that it is clearly a good and cost-effective programme - should lead to a greater sense of perspective about the impact of policies.  Most social programmes don't help everyone who participates - either because their outcomes remain poor despite the programme, or because they would have had a good outcome anyway. 


That is clearly true here; the estimate suggests that despite the work experience programme, 54% of participants were still on benefit afterwards; and of the 46% who left benefit, 40% would have anyway.  This implies that about  86% of the observed impact is "deadweight"; this, of course, was precisely the argument that the government used to justify the abolition of the Educational Maintenance Allowance (EMA).  


As I and a number of other economists pointed out at the time, this was a misuse of the evidence in order to construct a spurious argument against EMA; equally, it would be spurious to use the fact that most of those leaving benefit are "deadweight" to criticise the work experience programme.  It is the costs and benefits of programmes that matter. It is to be hoped that when using today's analysis, as I am sure the government will (and indeed should) to justify the programme, that Ministers will recognise that by the same token the arguments they made to justify the decision to abolish EMA were misleading, and that this will not happen again. 


Finally, Ministers should admit it when they get the facts wrong. I noted six weeks ago that the Prime Minister's statement, at Prime Minister's Questions, that of those young people going on to the programme

"around half of them are actually getting work at the end of these schemes."

was simply false (Sarah Teather subsequently made the same mistake).  As far as I know, the Prime Minister has not corrected this error, even though it was extensively analysed by Fullfact Now that a government publication has set out the facts clearly, it is to be hoped that the Prime Minister will comply with the Ministerial Code, which states

"It is of paramount importance that Ministers give accurate and truthful information to Parliament, correcting any inadvertent error at the earliest opportunity."

I think this is important, as a signal not only of respect for the Code, but for the accurate and proper use of evidence in the public policy debate. 



6 comments:

  1. Firstly, thank you for this blog, you really help an economic novice such as myself understand the strengths and weaknesses of key economic policies.
    However, in this instance I disagree with your application of this study. To me,this study provides evidence that participation in the work experience scheme is better than non-participation. However, importantly it does not confirm whether the presence of a work experience scheme improves employment nationwide. My argument being that rather than companies hiring people for cheap on work experience schemes (then paying half full-time), they could have hired people on paid jobs from the start. Companies were filling jobs on the cheap that may have reduced the necessity to pay full-rate for labour.
    I accept that in many cases companies may not have employed anyone, however my prediction is that enough would have to account for the small 6% benefit from the scheme.

    ReplyDelete
  2. Yes, don't we need to know something about general equilibrium effects?

    ReplyDelete
  3. “It is the costs and benefits of programmes that matter”

    Work experience works, but not as well as Work Trials

    Unfortunately the level of publically available analysis is not comparable to the above, but a quick google* suggests that work trials produce 35-50% job outcomes at a cost of £25-50 per person, with BIS suggesting around 20,000k Work trials a year although these are not officially recorded.

    The only difference between this JCP led programme and the top down “work experience” initiative being that Work trials can only last up to 30 days and have to have a possibility of a real job at the end – not just free Christmas labour.

    Your analysis suggests commensurate if not worse outcomes and this FOI (http://thoughcowardsflinch.com/2012/02/29/revealed-how-the-dwp-lied-about-cleggs-youth-contract/) puts the cost to the taxpayer of £372 per WE placement compared to less than £50 of a work trial placement.

    *http://www.bis.gov.uk/files/file46462.pdf ; http://www.fsb.org.uk/News.aspx?loc=119&rec=7652 ;
    http://base-uk.org/sites/base-uk.org/files/[user-raw]/11-06/rrep694.pdf

    More analysis could be done, but it would appear that at 8 times the price and potentially worse outcomes, the best way of improving work experience would be to take the cash and have jobcentre plus arrange more work trials instead.

    If I were an unemployed youth I would rather 4 weeks experience with a real possibility of being taken on rather than 4 weeks without that assurance, even if it meant the taxpayer forking out 8 times less for this opportunity.

    Pending better analysis on Work Trials the current conclusion would be that in order to politically distance themselves from schemes associated with the previous Government; the coalition has opted for poorer value for money alternative to Work trials, with the opportunity cost being greater unemployment for young people.

    ReplyDelete
  4. this sounds a bit like the graduates earn more and therefore more people will earn more if more people are graduates argument.

    basically work experience is primarily a sorting mechanism for an employer, so someone with it, stands out from someone without it, but if everyone has it, and it takes on a uniformity, by becoming compulsory and organised by the dwp, then the process of sorting would change, and it may well become a question of what sort of work experience. therefore if work experience is compulsory for young unemployed people, its value is then reduced.

    statistical analysis should be an addition to, not a replacement for, understanding of how humans think.

    ReplyDelete
  5. First off, thanks for the analysis but I'm seriously concerned by this comment which I think raises a major problem:

    "basically work experience is primarily a sorting mechanism for an employer, so someone with it, stands out from someone without it"

    There are two questions here, 1) does work experience work for the individual who gets it and 2) does it lead to an overall effect on unemployment.

    Because what might well be happening is, work experience people get more jobs but at the expense of everyone else, the total number of jobs doesn't change.

    In which case it would be unclear what society gains from the program.

    To put it more bluntly, if I went around handing out convincing fake degree certificates to random unemployed people, they would probably find it easier to get a job than unemployed people who didn't get one of my fakes but it wouldn't help in the fight against unemployment.

    ReplyDelete
  6. What are we calling being in employment? Does this cover zero-hour contracts, a few hours a week, so-called apprenticeships at £2.60ph, or a full-time job at least at minimum wage? The distinction is important surely.

    ReplyDelete