Civil Service Reform 1

This is the first of a series of notes which provide more detail about, and comment on, the many attempts – some successful, most not – to ‘reform’ the UK Civil Service. This note summarises reform programs through to just before Tony Blair’s government was elected in 1997.

I strongly recommend that you read the Civil Service Reform: Introduction and Efficiency Programs sections of this website before reading these more detailed notes. You might also like to read about Civil Service Reform Syndrome in order to understand why Ministers and senior officials fail, time and time again, to establish a change program with a realistic chance of success. Then read these more detailed notes for information about the following key reports and initiatives, as well as about many more less prominent government initiatives, external reports and analysis.

Key Reports

Links to these reports will be found in the detailed text below:

The 1700s and 1800s

Michael Coolican's No Tradesmen and No Women provides an excellent history of the civil service.  The following is a very brief summary of its early chapters.

The foundations of a detached civil service based on sound finance to run a modern imperial country can be found in the reforms implemented from the mid 1700s which slowly chipped away at the ability of office-holders to sell or pass on (what they regarded as) their property right to a relative. But the phrase ';civil servant' was first used by the East India Company which trained its young recruits at Haileybury.

As the years passed, the activities of the East India Company became increasingly governmental, and intertwined with those of the British Government in London.  Pressure grew to allow open competition for appointment by the Company, and this was important background to the parallel thinking that led to the Northcote-Trevelyan Report - see further below.  Ironically, given modern concerns, the strongest pressure came from a small group intent on ensuring that Oxbridge graduates could in future work for the East India Company.  Haileybury according closed in the late 1850s, to reopen a couple of years later as the 'public school' that we know today. 

Michael Coolican notes that: 

[This development threw away] everything that had been achieved by the more enlightened directors of the East India Company in ensuring potential recruits were taught relevant subjects [together with] fifty years practical experience of delivering that education. [This] volte-face had major implications for the British civil service as well as the Indian, and it is also possible that it played a significant part in the development of the British education system';s long-standing aversion to vocational training.  The government had turned its back on professional administrators;  the way had been paved for the reign of the gifted amateur.

Northcote-Trevelyan

'The great and increasing accumulation of public business, and the consequent pressure on the Government' caused the Treasury's Charles Trevelyan to become interested in civil service reform.  His particular hobby horse was that work should be divided so that low-grade tasks, which he called 'mechanical' should not be carried out by 'intellectuals'.  This eventually led to the separation of recruitment from universities from the recruitment of clerks, thus facilitating the dangerous divorce of policy-making from policy implementation.

In other ways, too, Trevelyan was not so far-seeing, or as interested in the merits of competition, as we nowadays assume.  Michael Coolican (see above) describes the post-report debates in which Trevelyan was often on the 'wrong' side.  Robert Lowe probably deserves more credit than Trevelyan.  There is more detail here.

Be that as it may, the 21st Century UK civil service retains many of the characteristics of the service that was created as a result of the1854 Northcote-Trevelyan report on ‘the Organisation of the Permanent Civil Service’.  

The eventual result of Northcote-Trevelyan was a civil service appointed on merit and through open competition, rather than patronage, with the following core values:

The first Civil Service Commissioner was appointed in 1855.

The Macaulay Report on the selection and training of entrants into ‘the Civil Service of the East India Company’ was also published in 1854.  This report was in some ways a 19th century version of the 1968 Fulton Report on the structure of the UK Civil Service. Like the Northcote Trevelyan Report, the text is mercifully short and to the point, and the authors were delightfully honest when not 100% sure of their recommendations:-

“… we are inclined, though with much distrust of our own judgment, to think that …”.

They were also well aware of how their recommendations could be perverted:

“We propose to include the moral sciences in the … examination … Whether this study shall have more to do with mere words or with things, whether it shall degenerate into a formal and scholastic pedantry, or shall train the mind for the highest purposes of active life, will depend, to great extent, on the way in which the examination is conducted.”

And the training was to be thorough:

“[The new recruit] should study [Indian history], not merely in the works of Orme, of Wilks, and of Mill, but also in the travels of Bernier, in the odes of Sir William Jones, and in the journals of Heber. … He should understand the mode of keeping and checking accounts, the principles of banking, the laws that regulated the exchanges … [etc.].”

The Northcote Trevelyan reforms took a long time to become fully embedded in civil service culture - arguably until the early 1900s.   

The Civil Service that developed through the rest of the 1800s was far from stuffy - or at least no more stuffy than was normal for that period. David Price, in Office of Hope, describes the 1909 creation of Labour Exchanges by politicians Beveridge and Churchill:

Charles Rey, the new General Manager, [was] the antithesis of Beveridge, with no pretence to academic qualifications but 'excitable and most practically energetic. We had for ever to be raiding the Treasury for new staff and and premises and Rey, always spoiling for a fight, was the spearhead of our forays.' ... while [Rey's] team was regarded [by other officials] as engaged in 'a great gamble ... a company of pirates' the young men saw themselves as 'high adventurers giving shape to one of the great social reforms of the quinquennium'.

Another novelty was the extent of external recruitment of managers and staff, breaking through the established practice of selection by academic written examination, which tended to favour young inexperienced people ... successful candidates included businessmen, trade unionists, a regular soldier, and an American gold speculator who claimed to have run a labour exchange in Chicago, with a revolver provided as part of the office equipment.

But there was still some patronage.  Germanophobe senior Foreign Office officials such as Bertie, Hardinge and Nicolson - all key participants in the discussions that led to the First World War - were all well connected at the anti-German and pro-French Court of King Edward VII, and assisted each other's careers.

Haldane

The next major set of reforms came about as a result of the 1918 Haldane Report published at the end of the First World War. Haldane recommended the development of deeper partnerships between Ministers and officials so as to meet the more complicated requirements of busier government as substantial executive ministries emerged from the first world war - though they were still only one-tenth of the size of their modern counterparts.

The report's impact came through the interaction of two closely-linked ideas:

The relationship between civil servants and Ministers thus became one of mutual interdependence, with Ministers providing authority and officials providing expertise. And that is how it is supposed to work today, undisturbed by a number of proposed reforms, summarised below.

The Great Debate

The period following the 1918 Haldane Report, and particularly after the Second World War, was marked by an intensely political debate about the culture and other aspects of the UK civil service.  On the one hand, critics such as Tommy (later Lord) Balogh felt that Haldane's creation of unified civil service was a victory of the permanent bureaucrats which had created an essentially dilettante Mandarinate whose erudition kept dangerous thoughts well away.  On the other side, so to speak, senior officials such as Ernest Gowers and his mates in 'The Loan Collection' were proud and very defensive of what they eventually came to regard as 'the Civil Service Tradition'.   The debate was eventually resolved (probably to no-one's total satisfaction) via the Fulton Inquiry.  Further detail of this debate and the background to Fulton is in a separate part of this website.

Other Developments After 1918

The 1919 Bradbury Report created a new establishment branch of the Treasury (it would now be called 'human resources') to oversee pay and civil service organisation.

Although there has been little if any fundamental reform of the UK civil service since the 1920s, there have been numerous far-reaching efficiency initiatives, summarised here and below, and in later notes in this series.   These have led to real improvements in the way in which the big departments are managed, including much faster internal communication and much greater use of the internet for handling transactions with the public.

Minister/civil service relations reached 'glorious harmony' after World War 2 as they worked together to introduce the Beveridge welfare state, the National health Service etc. etc.    Unlike 1918/19 this meant that there was no post-World War 2 formal examination of the wartime performance of the British state, and so no 'Haldane 2' Report.  Lord (Peter) Hennessey later remarked that this was a missed opportunity to reform Whitehall.  And it has been downhill ever since with politicians increasingly seeing civil servants as 'part of the problem', not 'part of the solution', as evidenced in my later notes on civil service reform.

The 1949 Handbook for the New Civil Servant provides a pretty good portrait of the post-war profession.  I particularly like the fact that a new recruit could in due course expect to be assessed on their 'personality, force of character, judgment, taking responsibility, address, tact and zeal'.  Not a bad recipe!

The 1955 Priestley Royal Commission led to the Government accepting that Civil Service pay should be determined by fair comparison with the earnings of comparable work outside the Service. Click here for more information about Civil Service pay.

The Plowden Committee on Control of Public Expenditure: control of civil service pay, conditions and establishments - reported in 1961.

According to Bristol's Rodney Lowe (emphasis added):

'The Plowden committee on the control of public expenditure has been described as a milestone in the modernization of postwar British government. Certainly it effected major changes in both the Treasury's structure and personnel and, by securing the establishment of the public expenditure survey committee, gave subsequent governments the opportunity to plan public expenditure rationally in relation to prospective resources. Ultimately, however, the committee was a failure. The civil service was re-examined by the Fulton committee within five years and public expenditure soon escalated out of control.

The Plowden committee thus represented a major lost opportunity. The time had been ripe for a fundamental political and administrative adjustment to the needs of the extended postwar state; but the committee failed to build the necessary political, parliamentary or public support for its recommendations. The reason for failure was its restricted nature as an internal enquiry with largely ineffectual ‘outside’ members, which enabled vested Treasury interests increasingly to dictate its deliberations. A more open enquiry would have stimulated and brought the best out of the ‘modernisers’ within the Treasury.'

Fulton

Harold Wilson was Prime Minister from 1964 to 1970 and then again from 1974 to 1976. He was a great moderniser in many ways, and a fan of technology, and in 1966 tasked the Fulton Inquiry with overhauling the structure of the UK Civil Service.  As noted above (in The Great Debate) the background to, and findings of, this inquiry may be found in a separate part of this website.

The Wider Issues Review

Edward Heath's 1970-1974 Conservative Government imposed pay restraint and other measures which led to the first ever national civil service strike in 1973.  This in turn led to the Wider Issues Review whose report Civil Servants and Change was published under the second Wilson Government in 1975. It is an interesting document in all sorts of ways. For a start, it was written in close collaboration with the civil service staff associations acting through the National Whitley Council. There is no overt Ministerial involvement, and not even a Ministerial foreword to the report.

The Whitley Council's own foreword to the report notes that '... the confidence and effectiveness of the Service depend very much on Ministers ... there is a risk of imposing greater burdens and stresses on the Service than it can in practice bear.' The report notes that both pay and 'deeper more complex reasons' had caused the recent industrial action. '... civil servants feel that they have been mucked about a lot in the last five or ten years. So there is an atmosphere of sourness in many parts of the Service, and we have found it at every level.' ... 'Ministers can help ... by recognising their responsibility as employers ... they can avoid discrimination against the public service in the application of their economic and social policies'.

Here are some other interesting extracts from the report:

... the Service has changed in several respects over the years ... Older civil servants joined when recruitment was highly competitive; before the war some schools would inscribe on the honours-board the name of a boy who was accepted into the Civil Service as an executive officer ... But other jobs have become more attractive ... and today the very much larger numbers of executive officer entrants do not regard the Civil Service or themselves as very special ... the majority [of civil servants] do not have traditional white collar attitudes and do not aspire to them.

... the economic rewards must be fair, but they will never be excessive;

More generally, though, some perceptive commentators felt that Edward Heath thought too much like an official and too little like a sceptical politician and, as a result, both allowed the civil service too much power to prevent things being done, and exaggerated what could be achieved by the official machinery. One example was said to be Heath's unfounded belief in the power of the Department of Trade and Industry to enforce the Industrial Relations Act.

Harold Wilson & Tony Benn

Wilson's post-1974 second government was marked by increasingly angry clashes with the left wing of his party, represented in particular by Secretary of State for Industry, Tony Benn. Senior officials at the Industry Department found themselves caught up in these quarrels and occasionally required Mr Benn to sign Ministerial Directions before they would implement his decisions. But he and his Permanent Secretary parted on reasonably good terms - see their exchange of letters here.

Dave Richards and Martin Smith have written a perceptive blog (The Lessons of Tony Benn as a Cabinet Minister: Breaking the Rules and Paying the Price) summarising the troubled history of Mr Benn's relationship with his officials,  By way of background, it is worth remembering that Mr Benn had previously been a very successful Postmaster General and technology minister.  But his attempts to implement the 1974 Labour manifesto promises were not supported by neither the Prime Minister nor his Cabinet colleagues and this led to a breakdown in the normal symbiotic, trusting relationship with his departmental officials.

Thatcher Government Reforms

Mrs Thatcher's Conservative Government came into power in 1979. Hugh Stephenson noted, in his book Mrs Thatcher's First Year, that '[her] arrival ... was the biggest jolt that that Civil Service had experienced in living memory. For a while the whole Whitehall system almost visibly juddered ... It was a culture shock. The elite administrative grade had come to think of itself as the guardian and trustee of national continuity .. The Prime Minister and a small group of sympathetic ministers ... were arguing that its ideas and advice had proved bankrupt, that now was the time for an entirely new approach.'

But there was nevertheless no great change in the service's fundamental culture or characteristics. The Whitehall system proved itself able to absorb the changes at the very top, with the staff in the Prime Minister's office in No. 10 Downing Street continuing to serve Mrs Thatcher as they had served her predecessor. Sir Kenneth Stowe as her Principal Private Secretary managed a smooth take-over, while Sir Robert Armstrong, the new Secretary to the Cabinet, appeared effortlessly to take on the role as her most senior closest official adviser. The role of Head of the Civil Service, which had been designed by Fulton to be held by the head of the Civil Service Department, was held by Sir Ian Bancroft. It was to him that the task of “de-privileging” the Civil Service was given, together with that of controlling Civil Service pay and pensions. But the Prime Minister’s Policy Unit, an organisation established by Harold Wilson, but now headed by Sir John Hoskyns and close to Margaret Thatcher, was looking closely over his shoulder.

The Thatcher Government accordingly concentrated on reforming the economy, and institutions outside government, and on improving the management of government. But even the efficiency drive got off to a rocky start. A brief biography of Sir Ian Bancroft records that:

Bancroft's experience and achievements made him a strong candidate to succeed Sir Douglas Allen as head of the home civil service when Allen retired at the end of 1977. But by then the continuing problems of morale and increasing militancy over pay issues in the civil service were having to be handled in a political environment which had become far more critical of all aspects of public administration. On appointment as head of the civil service Bancroft found that the economic situation and policies of the time gave very little room for manoeuvre.

His problems increased with the change of government in 1979. The new prime minister, Margaret Thatcher, was determined to improve the efficiency of the civil service by bringing in private sector methods and by cutting staff numbers. She was very critical of what she regarded as the negative attitudes of senior civil servants when they were made to contemplate new ideas. Bancroft accepted that the civil service would have to be reduced substantially and that the Civil Service Department would have to play a key role in that reduction, but he felt that the management problems with which his senior colleagues would have to grapple as numbers were reduced and posts eliminated were insufficiently understood in no. 10. His attempt to persuade the prime minister that some of her criticisms of senior civil servants were unfair was seen as an example of the weakness which the prime minister believed permeated government departments generally and the Civil Service Department in particular.

He persuaded the prime minister to give a dinner to permanent secretaries at which he hoped that a better understanding might be achieved, if problems could be discussed informally in a relaxed atmosphere. It proved to be a disaster. The explanation of difficulties merely persuaded the prime minister that very few of the permanent secretaries were ‘one of us’, and she ended the dinner unexpectedly early. From that moment Bancroft's position became increasingly untenable.

The Cabinet Office file for April/May 1980 makes fascinating reading. It covers the preparation for the ill-fated dinner and includes the Permanent Secretaries' subsequent 'thank you' letters as well as preparation for a Parliamentary Statement about cuts in civil service numbers.

You might also like to refer to an annotated version of a slightly odd senior officials' 2013 Tribute to the late Baroness Thatcher.

In his autobiography, Matthew Parris (who worked for Mrs Thatcher) notes that:

[there was] a false contradiction between the recollections of colleagues who said that she wouldn't listen, and those who are sure she did.  Both are right.  Thatcher would take advice on how to achieve what she wanted, but not on what she ought to want. ... she respected know-how ... The only objection [she] would allow was that ... a goal might be simply unachievable.  This, after much bridling, she would hear and if persuaded (often after humiliating the message bearer) accept.

Civil servants should not have found it difficult to accept the principle of the demarcation line described by Mr Parris, but it will have been much harder for her fellow Cabinet Ministers.

There were suggestions that Mrs Thatcher took some tentative steps to politicise the senior civil service.  This is debatable, but it was true that she took a greater interest in senior appointments than had her predecessors.  She wasn't interested in how officials voted.  She was much more interested in whether they were energetic and enthusiastic, although this will have been difficult for the more reserved of the mandarins, and/or those unhappy with her policies.  Tony Blair's attempts at politicisation were more serious - see Part 4 of the next note in this series.

Douglas Wass' 1983 Reith Lecture The Privileged Adviser is another good read, especially bearing in mind that it was almost certainly seen as the Mandarin's riposte to Mrs Thatcher.

The Financial Management Initiative (1982-) sought important improvements in the allocation, management and control of resources.

Next Steps - Executive Agencies

The Efficiency Unit triggered much welcome change throughout government, initially through Rayner Scrutinies and then very noticeably through the 1987 Ibbs Report Improving Management in Government; The Next Steps (published in 1988) which led to much of the executive work of Government being devolved to Executive Agencies.

The authors of the Next Steps Report identified seven points of diagnosis:

  1. 95% of the civil service are delivering services; they generally welcome the management changes to date
  2. senior management is dominated by policy staff with little experience of service delivery
  3. senior civil servants are ruled by ministerial and parliamentary pressures
  4. Ministers are overloaded and inexperienced in management
  5. Departments still focus upon activities and not on results
  6. there are insufficient pressures to improve performance
  7. the Civil Service is too big and diverse to manage as a single entity”.

and recommended that:

Diana Goldsworthy's Setting Up Next Steps and the House of Commons Report The Accountability Debate: Next Steps Agencies include later perceptive analyses of the background to, and creation of, Executive Agencies.

Sir Geoffrey Holland praises the creation of Agencies, but offers a pretty scathing review of privatisation, market testing and contracting out, at pp 45-46 of his 1995 lecture "Alas! Sir Humphrey, I Knew Him Well".

A broadly supportive review of executive agencies (Better government services: Executive agencies in the 21st century) was published in 2002. As ever, however, it found that the civil service did not value 'delivery'. The report noted that 'Delivery experience is rarely found at the heart of departments. Departments’ leadership structures must be built around the skills and experience of delivery as well as policy in order to plan and manage all aspects of achieving outcomes for customers.' And 'The different skills needed for excellence in policy advice and in service delivery are not yet valued equally. Mutual support is essential and both are integral to achieving outcomes effectively. Departments and agencies must work together to bridge the gulf between policy development and implementation and to fill high-level skills gaps in departments and agencies.'

Treasury & Civil Service Committee Report - The Role of the Civil Service

This wide ranging 1993 report examined a number of questions which came up in numerous later studies and reports listed in later notes on this website. 

But probably its most important long-term impact came through its discussion of a Civil Service Act and its call for statutory backing to be given to its proposals for ‘strengthening the essential values and standards of the civil service’ (paras 103-117). Its proposals were for a civil service code and a direct line of appeal to the Civil Service Commissioners in cases of alleged breaches of the Code. In its response, the then Conservative Government was cautious about statutory backing but accepted the proposals for a code and direct line of appeal (The Civil Service:  Taking Forward Continuity and Change). It was, though, another 15 years before statutory backing was forthcoming,

Here are some interesting extracts from the report which discuss other questions to which later Ministerial and other minds were often drawn

Innovation
… the Government believed that reductions in Civil Service numbers would be achieved by “the combination of ever greater pressure on running costs with a greater capacity for front line managers to actually come up with innovative solutions”.

Mr Waldegrave considered the Citizen's Charter, with its focus on establishing clear service standards, measurable outputs and better relationships between services and their users, as central to the Government’s programme of public service reform." It signalled a high level political commitment to quality and value for money in public services and reinforced the position of those seeking to promote innovation in public services.

Entrepreneurship, Risk & Failure
Sir Kenneth Stowe. … emphasised that the administrative character of much of the Civil Service’s work and the requirement for rigid adherence to integrity limited the scope for energy, innovation and an “entrepreneurial” approach.' There is clearly a balance which needs to be struck in this matter. The original Next Steps Report observed that “the culture of the Civil Service puts a premium on a safe pair of hands, not on enterprise. It does not reward the person who says I have saved money. It does not penalise the person who ignores the opportunity to get better value”.

Sir Derek (now Lord) Rayner told the Expenditure Committee “Efficiency in the Civil Service is dependent, as in business, on motivation, and whereas in business one is judged by overall success, in my experience the civil servant tends to be judged by failure”. … [but] …Professor Eric Caines argued that “Nobody is sacked for making mistakes, the deal being that if Ministers are to protect Chief Executives, they for their part must shield Ministers”.  He alleged that a compact of sorts has been struck between Ministers and Chief Executives which ensures that neither of them assumes the ultimate risk”. A Chief Executive was required to accept limited freedom and the need to keep Ministers out of political trouble in return for job security.

Sir John Bourn, the Comptroller and Auditor General, … noted that risks were inherent in a major programme of change, particularly one which involved new responsibilities tor many more people and organisations in the public sector.' … [and]  … a common feature of many of the failings detected by the National Audit Office and highlighted by the Committee of Public Accounts was the involvement of senior staff brought in from outside the public service, unaccustomed to the standards required m the public service and the audit procedures associated with them, and not imbued with its ethos.”

Fixed Term Contracts

Sir Robin Butler believed that fixed-term contracts were likely to discourage frank advice and created a risk of politicisation.' Mr Waldegrave also believed that fixed-term contracts with “a cliff edge" would change the balance between Ministers and civil servants and might discourage robust arguments. The Efficiency Unit Study noted that fixed-term contracts were not common in the private sector, where “employment contracts are designed to tie the individual to the organisation and give a sense of security, rather than to distance them and make them feel that their employment status is always under review’’

Career Management and Succession Planning Study: The Oughton Report

This influential 1993 Efficiency Unit report led to significant improvements in the career management of senior officials whilst challenging some Ministers' prejudices - for instance that it would be good to recruit much more from the private sector: 'The clear evidence of the private sector is that developing staff from within is right in most cases; recruiting from outside necessarily involves higher risk as well as expense'.

It also challenged Permanent Secretary's preference for promoting those officials who had done lots of different jobs rather than those who had built a more focussed career: 'In appraisal more emphasis must be placed on proven achievement, rather than on the range of posts held. Over 60 per cent of the Senior Open Structure have been in their current posts for 2 years or less.'

Comment: Over 20 years on, there is little sign that these two of Oughton's criticisms had been taken seriously. There is still considerable Ministerial pressure to recruit from outside the civil service, whilst the 'four year rule' ensures that most senior officials move far too frequently.

Ministers and Mandarins

This influential 1994 IPPR report, written by William Plowden, was published half way through John Major's steadily weakening premiership. Addressed, in effect, to an incoming Labour government (which arrived in 1997) Plowden reported that 'the civil service as a whole [was] in a state of crisis unprecedented in its history ... the government ... does not regard them as indispensable ... A significant number of senior officials feel that their professional skills are being ignored or abused.'

Plowden noted that the Oughton Report (see above) could be the springboard for many worthwhile changes including recruitment from outside the SCS to counter 'the slow disappearance, without replacement, of the gifted outsiders brought in during the war'. To guard against this damaging the essential apolitical professionalism of the civil service, he also recommended the introduction of a statutory 'code of behaviour' - and the Civil Service Code was indeed promulgated in Continuity and Change - see below - and introduced in 1996.

Key extracts from the report are here.

Continuity and Change

The 1994 Continuity and Change White Paper (i.e. consultation document) and the 1995 Taking Forward Continuity and Change decision document led to

Cut The Red Tape

This was a Trade and Industry Department initiative, mentioned here because it was fairly typical of efforts at the time to cut paperwork and bureaucracy. A summary of the officials' report, and the subsequent department wide instructions, are here.  It was in my view a great shame that ci (copy for information) and ca (copy for action) failed to become established as an improvement on cc copy lists.

Policy Disasters: Explaining the UK’s Record

The theme of Patrick Dunleavy’s 1995 paper was that the UK was a state unusually prone to make large-scale, avoidable policy mistakes which he defines as mistakes made when decision-makers systematically choose to ignore an abundance of critical or warning voices in order to persevere with their chosen policy.  Contributory factors include (emphasis added):

Further Reading

Sir Geoffrey Holland's lecture "Alas! Sir Humphrey, I Knew Him Well" is a good entertaining read and a useful review of many issues that faced government and the civil service in 1995.

The Institute for Government's 2012 report Reforming the Civil Service provides an excellent description and review of the work of the Efficiency Unit in the early 1980s, and of the Next Steps Report.

And do read Note 2 in this series to see what happened after 1997.

Martin Stanley

Spotted something wrong?
Please do drop me an email if you spot anything that is out-of-date, or any other errors, typos or faulty links.