SW Secure Data Environment Senior Data Engineer

Job summary
We are seeking an experienced Senior Data Engineer to provide technical leadership and hands-on expertise within the Southwest Secure Data Environment (SWSDE). SWSDE enables secure use of health and care data for research, innovation, service improvement and transformation. Working with NHS organisations, universities and research partners, it supports better outcomes and data-driven decisions.

The postholder will lead the design, development, testing and implementation of complex data pipelines, models and research-ready datasets. Working with NHS providers, researchers, technical teams and delivery partners, you will turn complex requirements into secure, scalable and reusable engineering solutions.

You will combine strong technical skills with the ability to lead complex engineering activity, solve difficult data problems and develop other engineers. Essential experience includes SQL, Python, data integration, data modelling and modern engineering practices.

NHS or healthcare data experience is valuable, but we also welcome applicants from research, public-sector or other complex, regulated environments who offer strong engineering skills and fresh thinking.

The role is administratively based in Bristol, with applications welcome from across the Southwest. Flexible regional and home working is available, with travel to partner organisations as required.

Main duties of the job
The postholder will:

• Lead the design, development, testing and delivery of complex health and care data pipelines, models and research-ready datasets.
• Translate research, operational and provider needs into technical specifications and secure, scalable, reusable solutions using SQL, Python and modern engineering technologies.
• Work with NHS providers to understand source systems, datasets, coding, data flows and constraints.
• Profile and validate data, resolving quality issues and leading extraction, ingestion, transformation, linkage and curation.
• Automate processes to improve the reliability, repeatability, traceability and efficiency of data delivery.
• Coordinate engineering across projects, datasets and providers; identify risks, dependencies, capacity issues and barriers, acting or escalating as needed.
• Guide and mentor engineers, peer-review code, pipelines, models, transformations and documentation.
• Promote version control, testing, documentation, reusable components and agreed development practices.
• Work with platform, cloud, architecture, cyber security and information governance teams to meet requirements.
• Maintain documentation, metadata, data dictionaries, lineage and quality controls, and improve SWSDE standards, patterns and cloud capability.

About us
We are committed to creating an inclusive organisation that promotes and values diversity. We know from experience that different ideas, perspectives and backgrounds create a stronger, more creative workplace that helps us to deliver the best services. We welcome applications which represent the rich diversity of our community; from people of all ages, disabled people, all genders, people from ethnic minority groups, LGBTQ+ people, people with diverse gender identity and expression, people from all religions and beliefs and other diverse characteristics, and we have processes in place to ensure that all applications are treated fairly and consistently at every stage of the recruitment process.

Job responsibilities
Essential

• Educated to master's level or equivalent senior-level experience within a specialist data, analytics, digital or technical environment.
• Significant experience designing, developing and supporting complex data pipelines, transformations and data integration solutions.
• Strong hands-on experience using SQL, Python and relevant modern data engineering tools and technologies.
• Experience working with large-scale health, research, public sector or similarly complex datasets.
• Strong knowledge of data modelling, data quality assurance, metadata management and data validation techniques.
• Ability to analyse complex datasets, data structures and technical requirements and determine appropriate engineering approaches.
• Experience planning and delivering complex data engineering or integration activity within agreed timescales and resources.
• Ability to manage multiple projects, technical priorities and deliverables simultaneously.
• Experience providing technical leadership, peer review, mentoring and guidance to engineers or technical colleagues.
• Excellent communication skills, with the ability to translate complex requirements and explain technical concepts to technical and non-technical audiences.
• Strong problem-solving skills and the ability to resolve complex data quality, integration and interoperability issues.
• Understanding of information governance, cyber security, data protection and relevant compliance requirements.
• Ability to work collaboratively with data providers, researchers, suppliers and partner organisations across organisational boundaries.
• Commitment to technical quality, inclusive working, knowledge sharing, innovation and continuous improvement.

Desirable

• Experience working with NHS data, health research infrastructure, Secure Data Environments or Trusted Research Environments.
• Knowledge of NHS datasets, coding systems, data standards and healthcare information flows.
• Experience of cloud-based data engineering using Azure, AWS or an equivalent platform.
• Experience of DevOps, automation, infrastructure-as-code or platform engineering approaches.
• Knowledge of interoperability standards or common data models such as OMOP, FHIR or openEHR.
• Knowledge of SATRE, the Data Security and Protection Toolkit, ISO 27001 or other relevant security and assurance frameworks.

Person Specification
Knowledge, Training and Experience
Essential

• Educated to masters level or equivalent experience working at a senior level within a specialist data, analytics, digital or technical environment.
• Extensive specialist knowledge gained through postgraduate study, professional qualification or significant experience in data engineering, data integration, health informatics, analytics, data architecture or digital services.
• Significant experience designing, developing and supporting complex data pipelines, data transformations and data integration solutions.
• Significant experience working with large-scale health and care datasets within NHS, research or public sector environments.
• Experience of designing, developing and maintaining data engineering solutions using SQL, Python and relevant modern data engineering tools and technologies.
• Experience of data modelling, data quality assurance, metadata management and data validation techniques.
• Experience working with complex NHS datasets, data standards, coding systems and healthcare information flows.
• Understanding of information governance, cyber security, data protection and NHS or public sector compliance frameworks.
• Understanding of the NHS, wider health and care system landscape and national digital and data priorities.
• Demonstrates strong digital literacy and effective use of modern digital, data and collaboration tools.

Desirable

• Experience working within NHS, health research, analytics or Secure Data Environment (SDE/TRE) programmes.
• Knowledge of SATRE, DSPT, ISO27001 or other relevant standards and assurance frameworks.
• Experience of cloud-based data engineering using Azure, AWS or equivalent platforms.
• Experience of infrastructure-as code, automation, DevOps or platform engineering approaches.
• Experience of interoperability standards and common data models such as OMOP, FHIR or openEHR.

Communication Skills
Essential

• Excellent communication skills with the ability to explain complex technical and data engineering concepts to technical and non-technical audiences.
• Ability to communicate effectively with NHS providers, researchers, suppliers and partner organisations.
• Strong presentation, facilitation and stakeholder engagement skills.
• Ability to build collaborative relationships across organisational and professional boundaries.
• Ability to translate operational, research or business requirements into practical technical approaches and engineering solutions.

Desirable

• Experience presenting technical findings, data quality assessments or engineering solutions to governance groups and technical forums.

Analytical
Essential

• Ability to analyse highly complex datasets, data structures, mappings and technical requirements and determine appropriate engineering approaches.
• Ability to investigate and resolve complex data quality, integration and interoperability issues where there may be multiple potential solutions.
• Ability to balance technical, operational, information governance and delivery considerations when making recommendations.
• Experience identifying, assessing and resolving technical risks associated with data engineering, onboarding and data processing activities.
• Strong problem-solving skills and ability to work effectively in complex, ambiguous and rapidly changing environments.

Desirable

• Experience analysing platform performance, pipeline performance, data quality metrics or cloud based engineering workloads.
• Experience applying analytical techniques to support data validation, optimisation and continuous improvement.

Planning Skills
Essential

• Experience planning and delivering complex data engineering, integration and onboarding activity within agreed timescales and available resources.
• Ability to manage multiple competing priorities, projects and technical deliverables simultaneously.
• Experience developing implementation plans, technical specifications and engineering work packages.
• Experience coordinating technical activity across multiple contributors, organisations or datasets.

Desirable

• Experience of agile delivery, DevOps or modern technical delivery methodologies.

Management Skills
Essential

• Experience providing technical leadership, mentoring and guidance to engineers, analysts or technical colleagues.
• Experience coordinating technical activity across multiple projects, datasets and engineering priorities.
• Experience undertaking technical peer review, quality assurance and promotion of engineering standards and best practice.
• Experience supporting knowledge transfer, technical development and capability building within engineering teams.
• Experience working collaboratively with suppliers, provider organisations and delivery partners to support engineering delivery.

Desirable

• Experience of line management, supervision or coaching responsibilities.
• Experience contributing to recruitment and technical assessment processes.
• Experience developing engineering capability within multidisciplinary teams.

This post is subject to the Rehabilitation of Offenders Act (Exceptions Order) 1975 and as such it will be necessary for a submission for Disclosure to be made to the Disclosure and Barring Service (formerly known as CRB) to check for any previous criminal convictions.