Transformed Labour Force Survey - A Lessons Learnt Review

3 Dec 2024 12:37 PM

The ONS prides itself on delivering independent and relevant statistics and analysis to inform the UK. We work tirelessly, continuously responding to user need and evolving and improving to ensure the highest quality outputs possible for our stakeholders and the public. To enable us to be at our best, we strive for an open, honest and inclusive culture, that identifies problems, and corrects transparently and quickly, providing safe places for colleagues to speak.

In this spirit, in summer 2024 I commissioned an internal exercise into our essential work on the Transformed Labour Force Survey (TLFS) to consider lessons from several years of the project. It was an opportunity to gain clear insight into the main points of learning and give ownership to colleagues involved to speak openly about their experiences so we can move forward collectively.

Today we are publishing the findings with individual names redacted and an accompanying interim action plan which sets out the actions we have already completed, as well as the further work in train to address the issues raised. This exercise has been an important element of our wider approach on TLFS to seek internal and external expertise to inform our approach, bring in new governance and strengthen our stakeholder engagement channels for challenge and support, so we can transition to the TLFS in an orderly and sustainable manner.

I welcome the spotlight and illumination this exercise has brought to a number of key issues on how we best work together to deliver our ambition. I am grateful for the candour with which colleagues have shared their views and experiences to enable us to progress together. I urge colleagues to continue to share their feedback via the many routes available, including through the line management chain as well as our Speak Up framework that provides a clear path to flag and resolve any type of concern. The past few years have not been without their challenges, in particular our significant deliverables during the COVID-19 pandemic, including the Covid Infection Survey and Census 2021 and the financial constraints we continue to manage. Our full focus now is on high quality and sustainable statistics for our users and stakeholders.

I am heartened that ONS teams are so invested and dedicated to make the TLFS project a success. I am proud of the work colleagues have delivered to date and the skills and knowledge they bring to produce statistics with utmost professionalism and impartiality I recognise that there continue to be challenges to work through and I am committed to our plan of action.

I thank colleagues across the organisation for their immense efforts and commitment on developing our labour market statistics, helping us towards our wider mission of statistics for the public good.

Sir Ian Diamond

Interim action plan

Lens 1: Vision

Lesson: Ensure clarity, acceptance, and stability of the vision

Details

• The vision, objectives, and scope of the Transformed Labour Force Survey (TLFS) should be documented and then reviewed and evaluated regularly, with a clear evaluation plan.  
• Easy to find documentation and audit trails should capture reviews and evaluation outcomes, as well as all important changes in direction, objectives, or the overarching vision. 

Actions

• July 2024 undertook a TLFS project review and refresh to evaluate critical work to address design challenges.
• Design approach, including shortened test trials, approved with users and experts and signed off via new governance.
• Prioritising regular communications throughout transformation project; fortnightly internal newsletters, monthly All Hands Calls; regular externals transformation updates to user community.

Lesson: Revisit crisis decisions for long-term sustainability

Details

• Decisions undertaken during crisis management are unlikely to be the best solution for the longer term. 
• Crisis decisions should be reviewed and reassessed when the crisis is under control. 

Actions

• Crisis action during the Covid-19 pandemic, such as the use of alternative data sources to complement the survey methods (for example, PAYE RTI) have been reviewed with our academic experts and action taken.
• Using the new governance and user expert groups to assure any changes to the survey collection, methods and communications.

Lesson: Protect the iterative approach to transformation and continuous improvement 

Details

• Transformation is necessarily iterative.  Plans are subject to revision as knowledge improves and evidence is collected. This approach needs to be protected throughout the project lifespan. In this case, a project dependency (turning off the Labour Force Survey (LFS)) was mandated and prioritised, and the resultant constraints made transformation unviable. 
• Investing in transformation is not a substitute for investing in continuous improvement. Transformation activities draw off resources (such as subject matter expertise) from routine delivery and this affects continuous quality improvement. This should be acknowledged when planning for transformation. 
• The TLFS approach seemed to focus on paced delivery of a finished tool, rather than iterative development of a minimum viable product (MVP). 

Actions

• Due to complexity of project and uncertainty we will be unable to set out potential timetables until Quarter 1 (Jan to Mar) 2025, when the recent tests have been fully assessed.
• Conducted a resource review in August 2024 to identify critical gaps, single points of failure and prioritising addressing of these points and continue to review this as the project evolves.

Lens 2: Design

Lesson: Understand the required quality of outcome from the start   

Details

• Transformation projects that assess or research feasibility should always test, consider, and design for quality of outcome. They should firmly establish the final outcome requirements and success criteria as early as possible. 
• A narrow focus within professions, teams, deliverables, or stages of the data lifecycle leads to siloed thinking about quality. This can result in an incomplete picture of required quality outcomes.  

Actions

• Comprehensive set of quality criteria completed in Autumn 2024 and undergoing further refinement with users via existing user forums.

Lesson: Ensure clarity and a shared understanding of priority user needs from the start

Details

• TLFS inherited a complex set of user requirements which were not always well articulated and understood. There were differing levels of engagement, knowledge, influence and buy-in, which continuously changed over time. This meant that the scope of the project lacked clarity. When coupled with the tension between innovation and retaining consistency, this left the project without a common, shared understanding of the needs it must meet and how those needs should be prioritised. 
• When requirements are identified that are out of scope, there must be a plan to address them. 

Actions

• Working with Aacademics, Stakeholder Advisory Panel, Technical Engagement Group (Bank of England (BoE), HM Treasury (HMT), Office for Budget Responsibility (OBR), Department for Work and Pensions (DWP)) to agree priority issues that remain with survey and will be used to inform the development activity.

Lesson: ONS is consistently overoptimistic in its vision, particularly around non-survey data

Details

• Challenge and assurance functions, and full planning for worst-case scenarios, are essential to prevent the consequences of over-optimism. 
• Financial discounting methods are regularly applied to account for this bias, but they do not mitigate the impact of only designing for best case scenarios. There should be a plan for the most adverse events. 

Actions

• Introduced external critical review from academics, including through the Stakeholder Advisory Panel and the Technical Engagement Group (BoE, HMT, OBR, DWP) to provide challenge and assurance.
• Developing a set of TLFS scenarios to manage user-expectations around transition timelines.

Lesson: Set out a strategic approach to surveys

Details

• The organisation’s strategic approach to surveys swayed, reflected in funding and the language used to articulate ambitions and plans. 
• ONS should commit to surveys for primary data collection, recognising that this does not negate other data sources but proclaims ambition for both.  

Actions

• Corporately agreed strategic investment in Surveys as a whole (plus £18.8m this financial year compared to last year) including fully funding extended parallel run.
• Currently updating the Social Survey Strategy with a target to publish March 2025.

Lens 3: Plan

Lesson: Data gathering tools can become single points of failure 

Details

• LFS (and now TLFS) are the only tools that can meet a wide range of critical user requirements, including statutory ones.  
• Any data source has the potential to become a critical vulnerability if there is not sufficient investment in diversification and contingency planning. 
• Mitigating this risk must be addressed as part of transformation activity.

Actions

• Continuing to look at potential use of administrative data sources to supplement survey data.

Lesson: Manage and control complexity 

Details

• It was too complicated to turn off LFS and switch to TLFS at the same time because not all requirements were met, given the scale of dependency across diverse users on a single survey. 
• Transformation brings discontinuities in data. This requires a strategic approach and buy-in from stakeholders. ONS did not invest enough in research and stakeholders’ engagement (internally or externally) to define a shared approach to potential discontinuities.  

Actions

• Taking action to de-couple decisions between TLFS and LFS, where possible, and considering other ways to simplify the project.
• Introduced external critical review, including through the Stakeholder Advisory Panel and the Technical Engagement Group (BoE, HMT, OBR, DWP) to help define and agree on methodologies and transition plans.

Lesson: Use an evidenced range rather than a single target date when setting milestones 

Details

• Single target dates based on funding availability do not bring positive impact and can lead to unnecessary costs and reputational impact. Projects should use an evidenced range of dates instead.  
• Funding should not be the sole factor in determining timelines and milestone dates. 
• The mandating of a single target date for LFS decommissioning prevented investment in overcoming the most complex technical questions.  

Actions

• Introduced a modular approach to decision-making with data and user feedback forming the evidence-base of decision making.
• Quarter 1 2025 is the design decision point.

Lesson: Set a realistic cost envelope

Details

• While transformation often has dedicated funding and may have dedicated resource, it takes place in the context of the wider organisation and relies on that to succeed. 
• In this case, the substantial budget reductions asked of surveys in 2023 and 2024, and the requirements for maintaining both the business as usual (LFS, which was struggling with response) and continuing the transformation (TLFS) during this period, had a wider impact on the quality of social surveys overall.  

Actions

• Corporately agreed strategic investment in Surveys as a whole (plus £18.8m this financial year versus last year) which enabled us to put more funding towards LFS business as usual alongside TLFS.
• Agreed to data collection priorities being TLFS, LFS and the Labour Costs and Food (LCF) survey.

Lens 4: Transformation leadership

Lesson: Behaviours matter more than process 

Details

• Strong relationships, shared objectives and mutual understanding between project leaders are needed to set up a project for success and to face challenges.  
• Rewarding early escalation of challenges and discussion on risks and providing constructive challenge enables delivery of quality outcomes at pace.  
• Technical concerns need to be escalated to enable constructive challenge and support.
• Sufficient technical leadership should be in place from the outset of transformation projects.

Actions

• Appointed two experienced senior leaders into project executive role to provide joint analytical technical leadership capability.
• Director for Methodology input to support oversight on the conditions for quality and the journey towards stability of the TLFS.
• From Summer 2024, we begun reporting progress to UK Statistics Authority Board on a monthly basis and present information in a way that the Board can understand the risks and issues and provide challenge in a constructive manner.
• Internal communications raising awareness of channels for escalation of known issues and promoting a culture that encourages the surfacing of risks and issues and invites constructive challenge, including the Speak Up framework.

Lens 5: Collaboration

Lesson: Transformation programmes do not start from a blank sheet of paper 

Details

• Lessons learnt from relevant past experience in ONS and other government departments should inform each stage and inform key decision points.  
• Knowledge should be properly recorded and revisited so that key lessons inform future decisions.  

Actions

• Plan to share these lessons learnt and key actions with wider-ONS transformation projects and programmes.
• ONS have recently commissioned Southampton University to do an international review of LFS methods to help us stay abreast of best practice in this area.

Lens 6: Accountability

Lesson: Ensure clear accountability for decisions 

Details

• The project is ambitious and complex enough to warrant being a programme in its own right. Nominating a single Senior Responsible Owner (SRO) accountable for delivery is standard practice and would provide strong oversight, but ONS needs to reflect on how that can work for projects that cut across strategic portfolios. 
• Governance should foster accountability, assurance, and decision-making at the right level.  
• Multiple and fragmented reporting lines to boards with different remits cause inefficiencies, delays, and can lead to inaction.  

Actions

• Agreed new Project Governance (Joint Project Board and Senior Steering Group) with clear focus on decision-making.
• Agreed to manage project in formal phases with next decision point in February 2025.
• Introduced external critical review, including the Stakeholder Advisory Panel and Technical Engagement Group.

Lesson: Transformation needs statistical quality management 

Details

• Transformation activities need to invest more in assessing, defining, articulating, and evaluating quality requirements.  
• Statistical quality requirements should be front and centre from the early stages.  

Actions

• Comprehensive set of quality criteria completed and undergoing further refinement with users.
• Established a Labour Market Stakeholder Advisory Panel to guide the delivery of project.

Lens 7: People

Lesson: Leaders can’t do transformation on their own

Details

• ONS teams are invested and dedicated to make the project a success. However, TLFS has been a difficult and challenging experience for many, which has affected wellbeing and confidence at all levels.  
• Leaders at all levels of the project need the right support to be able to succeed.  

Actions

• Capturing the experience of the teams and their ideas (via regular workshops and relevant feedback forms) on how to make this complex, challenging project more sustainable to deliver without impacting wellbeing.
• Two technical analytical leaders developing proposals for how to improve the project.

Related downloads

• Transformed Labour Force Survey – A Lessons Learnt Review (415.5 kB pdf)