Lead Observability & Data Architect / Engineer (SC / DV Cleared)

Full-time

From Big Data/ETL to Telemetry Pipelines: If you have built high-throughput streaming pipelines using Apache Kafka, Airflow, NiFi, or Spark, you already know how to route and transform complex data feeds at scale.
From Enterprise Search to System Observability: If you have engineered core search, indexing, or sharding architectures using Apache Solr, Lucene, or OpenSearch, you possess the exact foundational knowledge needed to scale an enterprise Elasticsearch cluster.
From Cloud Infrastructure to Monitoring Fabrics: If you are a seasoned Cloud or Infrastructure Architect specializing in secure Kubernetes orchestration, Terraform automation, and air-gapped system design, you can easily pivot to defining secure logging architectures.
We are seeking an Enterprise Platform Architect or Senior Data Engineer to design and scale national-critical monitoring fabrics and data hubs. Whether your background is rooted in core Observability platforms (Opensearch/Elastic/Splunk/OpenTelemetry) or high-scale Data Engineering (Kafka, Python, Airflow, OpenSearch/Lucene), your core mission is the same: transforming billions of raw event logs into actionable, secure streaming telemetry.
Should be able to build high-performance data pipelines to ingest, parse, and structure massive event telemetry across hybrid-cloud and air-gapped environments.
Manage large-scale cluster deployments, designing optimized indexing, data life cycle policies, and schema mapping (such as ECS).
Expertise in Elasticsearch/OpenSearch, Splunk, or Solr. Strong proficiency in building visualization dashboards (Kibana, Grafana) and managing telemetry collectors (OpenTelemetry/OTel, Beats, Prometheus).
Experience in designing and scaling data pipelines for high-velocity telemetry. Essential expertise in Apache Kafka, Logstash, or streaming/orchestration frameworks (Airflow, Vector, Debezium).
Hands-on experience architecting scalable, containerized solutions using Kubernetes (K8s), Docker, and infrastructure-as-code (Terraform/Ansible).
Search & Indexing Engine Depth: Prior architecture experience with OpenSearch, Apache Solr, or Lucene core or Elasticsearch.
Strong proficiency in Python, Rust, or Bash for automating log parsing, testing, and deployment workflows.
A strong track record of engineering within highly regulated sectors (eg, MoD, Home Office, BAE, HMRC) with an active UK security clearance (SC/DV).

Vacancy posted 8 hours ago

Similar jobs that could be interesting for youBased on the Lead Observability & Data Architect / Engineer (SC / DV Cleared) in Stratford, Greater London vacancy

AI Platform Engineer
...Join Us as an AI Platform Engineer - Shape the Future of AI at Barclays... ...vulnerabilities, protect sensitive data, and ensure secure software... ...decisions/changes. They may also lead a number of specialists to... ...are expected to demonstrate a clear set of leadership behaviours to...
Suggested
Long-term contract
Permanent
Full-time
Temporary
Hybrid working
Stratford, Greater London
8 hours ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to Lead Observability & Data Architect / Engineer (SC / DV Cleared). Be the first to apply!