Data Engineer

Job Overview

Salary: £42,002 plus an additional market supplement £2,998 per annum

Contract type: permanent

Band: E / HEO

Closing date: Monday 20th October at midnight

Shape the Future of the Nation’s Memory

At The National Archives, we are more than custodians of the past – we are pioneers in digital archiving, ensuring the UK’s public records are robust, connected, and accessible for generations to come. As a Data Engineer in our small, dynamic Digital Archiving team, you’ll help transform how archival data is managed, structured, and enriched, unlocking its value for public use and discovery.

Why Join Us?
• Your work will directly support the preservation and accessibility of the UK’s documentary heritage, including a new Parliamentary Archives project.
• We nurture talent, value every voice, and support personal and professional development. You’ll join a collaborative, forward-looking team where your ideas are heard and your growth is championed.
• Work with a modern tech stack including Python, SQL, Shell scripting, Talend, XSLT, XML, CSV, JSON, RDF, RDBMS, NoSQL, and graph databases. You’ll help design and build scalable data pipelines, automate data processing, and develop tools for data enrichment and integration
• We invest in your learning, offering opportunities to develop your skills in emerging technologies and data practices that enhance public access to archives.

This Job Would Suit…

If you’ve been working in an entry-level data role and are ready to take the next step, this is an ideal opportunity to grow your skills and responsibilities. We welcome those who are passionate about data, eager to learn, and keen to make a real impact. Data skills are highly valued at The National Archives, and you’ll be part of an active internal data community where specialists from across the organisation share knowledge, collaborate, and solve problems together.

What You’ll Do
• Design and develop scalable, repeatable data pipelines (ETL/ELT) to process, transform, and load archival data.
• Automate data cleansing, validation, standardisation, and enrichment.
• Develop and document schemas, ontologies, and models for archival data.
• Integrate data from diverse sources and formats, supporting better discovery and reuse.
• Collaborate with product teams, archivists, and developers to deliver practical, data-centric solutions.
• Contribute to a culture of openness, quality, and continuous improvement.

Ready to help shape the future of digital archiving?

Apply now and join a team where your work matters – for today, and for generations to come.

We have 2 full time positions available. However, requests for part-time working, flexible working and job share will be considered, taking into account at all times the operational needs of the Department. A combination of onsite and home working is available and applicants should be able to regularly travel to our Kew site for a minimum of 60% of their work time.

Application Process:

• Interview: Interviews will be held on-site during the week commencing 3rd November, and will include a technical test. At interview, you will be assessed on all of the essential criteria listed in the job description below.

• Application: you will be asked to upload your CV and answer the following questions:
1. Please give examples of work you have undertaken to develop your AWS expertise and proficiency of infrastructure as code.
2. Why are you applying for this role?

Your application will be assessed on the following 3 essential criteria:
1. Hands-on experience with AWS (including identity and access management).
2. Proficiency in Infrastructure as Code (Terraform) and scripting (Python, Bash, or PowerShell).
3. Ability to communicate effectively across technical and non-technical teams.

Artificial Intelligence can be a useful tool to support your application, however, all examples and statements provided must be truthful, factually accurate and taken directly from your own experience. Where plagiarism has been identified (presenting the ideas and experiences of others, or generated by artificial intelligence, as your own) applications may be withdrawn and internal candidates may be subject to disciplinary action. Please visit the civil service careers website where you can find further information on the use of AI in the application guidance section.

SC clearance/willingness to obtain SC clearance will be required for this role. This requires candidates to have been resident in the UK for at least the past three years. Please do not apply if you have been resident in the UK for less than three years as your application will be rejected.

Sponsorship:

We are unable to offer sponsorship for this role.

Job Description

Job purpose

Archives are the homes of our collective memory, past and future. The National Archives is the archive of UK government, and one of the world’s leading digital archives. As the scale and complexity of our digital collections continue to grow, we ensure our data is robust, connected, and accessible.

We are looking for a Data Engineer who thrives on solving complex problems, working at scale, and creating the pipelines and models that power access to the nation’s records. You will play a key role in transforming how archival data is managed, structured, and enriched, helping to unlock its value for public use and discovery.

This role is ideal for someone who enjoys designing and building efficient data workflows, developing tools to automate data processing, and collaborating with product teams and specialists to understand data needs and deliver practical solutions. You will bring strong programming skills and a good understanding of data modelling, transformation, and integration across a variety of formats and technologies.

You will work in a supportive, forward-looking team that values transparency, curiosity, and collaboration. You’ll have opportunities to develop your skills in existing and emerging technologies that enhance public access to archives and promote the re-use of data in meaningful ways.

Role and responsibilities

This role maps to the DDaT Data Engineer and Data Analyst roles with elements of the Data Scientist role.

Data processing and pipelines:

• Design and develop scalable, repeatable data pipelines (ETL/ELT) to process, transform and load archival data.

• Create scripts and tools for data cleansing, validation, standardisation, enrichment, and transformation.

• Automate routine data tasks to improve efficiency, accuracy, and consistency.

• Identify and manage connections between datasets, including those external to TNA.

• Research and implement tools for entity extraction and semantic tagging to improve the usability and findability of archival data.

Data modelling and integration:

• Develop and document schemas, ontologies, and other models that represent and structure archival data.

• Connect and integrate data from different internal and external sources, including structured and semi-structured formats (XML, CSV, JSON, RDF).

• Identify patterns and relationships within data to support better discovery and reuse.

Technical advice and collaboration:

• Develop excellent relationships with product teams, advocate for data-centric approaches to software development and help solve data problems.

• Maintain effective communication with external technical partners (e.g. suppliers), representing The National Archives to resolve complex data issues, clarify requirements, and ensure smooth delivery of digital archiving workflows

• Work closely with product teams, archivists, and developers to support data-centric solutions.

• Provide advice and input on data engineering approaches within the team.

• Share knowledge on data standards, validation rules, and transformation practices.

Quality, openness and growth:

• Work in the open and share code and documentation internally (and externally where appropriate).

• Participate in code reviews, technical discussions, and knowledge sharing.

• Contribute to building a data culture focused on reuse, quality, and continuous improvement.

Working conditions

• Normal office environment

• Display Screen Equipment user

Person specification

Essential criteria:

• Data manipulation Strong experience of accurate, reliable, large-scale analysis and manipulation of complex datasets using relevant programming languages and tools (e.g. Python, Shell scripting, SQL, XSLT).

• Data formats Solid understanding of working with different data formats (e.g. XML, CSV, JSON, RDF).

• Database technologies Practical knowledge of database technologies (e.g. RDBMS, noSQL, graph databases, linked data stores).

• Analysis and problem solving Excellent analytical skills, with a structured and proactive approach to solving technical problems. Able to apply a range of techniques to capture, document and communicate requirements, issues, designs and solutions.

• Communication and relationships Strong relationship building and communication skills, with an excellent user focus. Able to work collaboratively in multidisciplinary teams and explain technical issues clearly to non-technical colleagues.

• Prioritisation and organisation Ability to work with high accuracy, attention to detail and organisation, both independently and as a project team member. Able to prioritise competing tasks and deliver high quality work to agreed deadlines.

Desirable criteria:

• Awareness of the value and meaning of archival data and the ethical responsibilities that come with working with public records.

• Understanding of probabilistic, messy, or uncertain data handling.

• Experience in Agile development environments.

• Familiarity with archival principles or metadata standards (e.g. EAD, Dublin Core).

• Experience in semantic web technologies and entity extraction tools.

The Civil Service is committed to attract, retain and invest in talent wherever it is
found. To learn more please see the Civil Service People Plan and the Civil Service
D&I Strategy
.

Benefits 

 

Generous benefits package, including pension, sports and social club facilities, onsite gym, discounted rates at our on-site cafe and opportunities for training and development.  Annual leave entitlement of 22 days per calendar year (rising to 25 after the first year, and incrementally to 30 days after six years) and 10½ days public and privilege holidays per annum.

 

Any move to The National Archives from another employer will mean you can no longer access childcare vouchers. This includes moves between government departments. You may however be eligible for other government schemes, including Tax-Free Childcare. Determine your eligibility at  https://www.childcarechoices.gov.uk/ (opens in new window)

Reasonable adjustments

 

If a person with disabilities is put at a substantial disadvantage compared to a non-disabled person, we have a duty to make reasonable changes to our processes. 

If you need a change to be made so that you can make your application, you should: 

• Contact The National Archives via careers@nationalarchives.gov.uk as soon as possible before the closing date to discuss your needs 

• Complete the ‘Reasonable Adjustments’ section of your application form to tell us what changes or help you might need further on in the recruitment process. For instance, you may need wheelchair access at interview, or if you’re deaf, a Language Service Professional

​

Feedback will only be provided if you attend an interview or assessment.  

 

Security 

 

Successful candidates must pass a disclosure and barring security check. 

People working with government assets must complete basic personnel security standard checks (opens in new window)

 

Nationality requirements 

 

This job is broadly open to the following groups: 

• UK nationals

• nationals of the Republic of Ireland

• nationals of Commonwealth countries who have the right to work in the UK

• nationals of the EU, Switzerland, Norway, Iceland or Liechtenstein and family members of those nationalities with settled or pre-settled status under the European Union Settlement Scheme (EUSS) (opens in a new window)

• nationals of the EU, Switzerland, Norway, Iceland or Liechtenstein and family members of those nationalities who have made a valid application for settled or pre-settled status under the European Union Settlement Scheme (EUSS)

• individuals with limited leave to remain or indefinite leave to remain who were eligible to apply for EUSS on or before 31 December 2020

• Turkish nationals, and certain family members of Turkish nationals, who have accrued the right to work in the Civil Service

Further information on nationality requirements (opens in new window)

 

Working for the Civil Service

 

The Civil Service Code  (opens in new window) sets out the standards of behaviour expected of civil servants. 
 
We recruit by merit on the basis of fair and open competition, as outlined in the Civil Service Commission's 
recruitment principles. (opens in new window) 

 

The Civil Service embraces diversity and promotes equal opportunities. As such, we run a Disability Confident Scheme (DCS) for candidates with disabilities who meet the minimum selection criteria. 

The Civil Service also offers a Redeployment Interview Scheme to civil servants who are at risk of redundancy, and who meet the minimum requirements for the advertised vacancy. 

 

This vacancy is part of the Great Place to Work for Veterans initiative. (opens in new window)

 

Contact point for applicants:

• Name: The National Archives Recruitment Team 

• Email: careers@nationalarchives.gov.uk 

 

Further information 

 

If you feel your application has not been treated in accordance with the Recruitment Principles and you wish to make a complaint, in the first instance, you should contact The National Archives via email: careers@nationalarchives.gov.uk If you are not satisfied with the response you receive from the Department, you can contact the Civil Service Commission at https://civilservicecommission.independent.gov.uk/recruitment/recruitment-complaints/ (opens in new window)