Lead Data Engineer

Working at CreateFuture CreateFuture is an AI-native consulting partner where people do work that matters and are supported to do it well. We work alongside organisations such as PayPal, adidas, NatWest, FanDuel and Money Saving Expert, building digital products and services that make a difference  while always putting people first.

We’re a team of creators. We write code, shape delivery, build go-to-market strategies, develop AI solutions and create the practices that support our people. We work side by side with our clients, challenging what’s not working and helping them to build the

About the role 

We're looking for an experienced Lead Data Engineer to join our growing Data Practice, working on a Financial Services engagement centred on a large, long-lived SQL Server estate. You'll work as part of a multidisciplinary team helping a major FS client understand and de-risk a database landscape that has grown, unchecked, over 15 years of trading activity.

This is a role for someone who is comfortable going in without a map: a single SQL Server RDS instance holding around 1,100 tables and fifteen years of trade data, with no archiving strategy, no inventory of stored procedures, and tables duplicated across multiple schemas with no reliable way to tell what's actually in use.Part of the engagement's as-is analysis uses Claude to scan the codebase, map integration points, and flag likely dead code — you'll sit at the centre of that process, validating, interrogating, and making sense of what the tooling surfaces.

As a Lead, you'll shape the technical approach to the estate discovery, mentor engineers supporting the analysis, and act as a trusted, credible partner to the client's data and platform teams.

This role is Hybrid - requires 2 days per week on our clients site in Glasgow. 

Key Responsibilities

• Leading the as-is analysis of a ~1,100-table SQL Server RDS estate — establishing what exists, how it's structured, and where the risk sits, in the absence of any existing documentation or stored procedure inventory.
• Working alongside Claude-driven codebase scanning to map integration points, trace data lineage, and identify likely dead code, then applying engineering judgement to validate and prioritise the findings.
• Investigating duplicated tables across schemas to determine which copies are live, which are redundant, and what depends on each — building the tooling and queries needed where none currently exist.
• Designing an approach to 15 years of unarchived trade data — assessing volume, growth, and usage patterns to inform archiving, retention, and future-state options.
• Producing clear, defensible documentation of the database estate — schemas, dependencies, stored procedures, and data flows — that the client can rely on long after the engagement ends.
• Advising on remediation and modernisation options once the estate is understood, from consolidation and archiving through to longer-term platform or cloud migration.
• Leading and mentoring a small team of data engineers on the engagement, setting standards for how findings are verified, tested, and documented.
• Acting as a credible, hands-on technical partner to client stakeholders — including DBAs, platform owners, and risk/compliance contacts — who will be understandably cautious about changes to a live trading data estate.

Skills & Experience

You know your way around legacy SQL Server estates

• You're genuinely strong in SQL Server — schema design, query performance, stored procedures, indexing — and comfortable working in large, poorly documented production estates.
• You've reverse-engineered a database landscape before: worked out what's live vs. dead, resolved duplicated or conflicting schemas, and built confidence in an estate nobody fully understood.
• You understand the particular pressures of financial services data — long retention requirements, audit and regulatory expectations, and the need to prove data integrity for trade and transaction records.
• You're comfortable with ambiguity and incomplete information, and know how to build a reliable picture of a system methodically rather than guessing.

You're comfortable using AI tooling as part of the analysis

• You're open to and effective at using AI-assisted tools (we use Claude) to scan codebases, map integration points, and surface candidate dead code at a scale manual review can't match.
• You know how to sanity-check and validate AI-generated findings against the real system — you treat the output as a strong starting hypothesis, not ground truth.
• You can translate what the tooling surfaces into a clear, prioritised narrative that both engineers and non-technical stakeholders can act on.

You know your tools

• Deep, hands-on SQL Server experience — T-SQL, SSMS, execution plans, and diagnosing performance or integrity issues in large OLTP schemas.
• Comfortable with AWS RDS for SQL Server, including monitoring, performance insights, and operational management.
• Solid scripting ability (Python and/or PowerShell) for building the discovery, cataloguing, and dependency-mapping tooling this engagement needs from scratch.
• Experience with data cataloguing, lineage, or metadata management tooling is a strong plus, even if the client doesn't currently have any in place.
• Familiarity with data archiving and retention strategies for large historical datasets is beneficial.

You thrive in a consultancy setting

• You've led or mentored data engineers and are comfortable owning technical direction on a complex, ambiguous engagement.
• You can navigate the caution and politics of a large FS operator — multiple stakeholders, risk-averse change processes, and legacy systems that can't just be switched off.
• You're happy being onsite in Glasgow two days a week, building trust face-to-face with the client team
• You balance hands-on delivery with client engagement, and know when to push back on scope and when to be pragmatic.

Tech snapshot

• Primary platform Microsoft SQL Server (AWS RDS) — T-SQL, SSMS, stored procedures, execution plans
• Cloud AWS — RDS, S3, and supporting services for tooling and discovery outputs 
• Estate ~1,100 tables, 15 years of trade data, no archiving strategy, no stored procedure inventory, duplicated schemas
• AI-assisted analysis Claude — codebase scanning, integration point mapping, dead code identification
• Languages SQL (T-SQL), Python and/or PowerShell for tooling
• Documentation & lineage - Data cataloguing / lineage tooling (implementation as part of the
  engagement)
• Ways of working Hybrid — client site in Glasgow, 2 days per week

What we’ll offer you:

We trust people to do their best work. That means flexibility over rigid rules, impact over activity, and real investment in your growth both professionally and personally. You’ll be part of a supportive, and friendly culture, surrounded by smart, curious people who care deeply about what they do.

We offer flexible working, including hybrid and remote options. Our office hubs are located in Edinburgh, Leeds, Manchester, London and Bulgaria, with occasional travel to client sites or CreateFuture offices when needed.

We trust you to manage your time balancing collaboration with client time and focused work. What matters is the impact you have, not how busy you look.

Our hiring process

We try to keep our hiring process clear, fair and respectful of your time. We aim to get back to everyone who applies and we will be upfront about where you are in the process.

It usually looks like this:

• Call with our Talent Acquisition Team 
• Role specific capability interview 

Depending on the role, we might also ask you to do a short presentation, a practical or technical task or have a values focused conversation. We will explain what is involved before anything happens.

Inclusion at CreateFuture 

We believe diverse teams build better workplaces and better products. We want CreateFuture to be a place where people feel able to be themselves and do their best work.

If you need any adjustments or support during the application process, just. We will do what we can to help.

We look forward to your application!