Senior Data Engineer Job Description

Senior Data Engineer Job Description Template

Our company is looking for a Senior Data Engineer to join our team.

Coach and mentor both peers and junior members of Engineering in to develop a data-driven culture with supporting practises;
Investigate, test and introduce new ways of working and technologies where appropriate;
Assemble large, complex data sets that meet functional / non-functional business requirements;
Constantly seek new and creative ways to automate workflows and iterate on system improvements;
As part of our Platform crew, collaborate with Platform (DevOps) and Test Engineers on release and test processes in an incremental and agile manner;
Work with data and analytics experts to strive for greater functionality in our data systems;
Build efficient and reliable data pipelines, ingest and transform data sets using SQL, Python & AWS;
Collaborate with cross-functional peers (e.g. Data Architect and Data Scientist – TBH) to development data governance and management processes;
Writing technical specifications for projects as required;
Collaborate with Business Analysts on business insight and new product requirements;
Provide logical and physical data design and database modelling;
Transfer knowledge to DataOps team, ensuring the design and build are documented and hence supportable;
Convert algorithms, models and features into production solutions;
Set the standards for software engineering practices within the team;
There will be a mix of maintenance work and redesigning and optimising old code as well as creating new pipelines.

Experience with persistence and caching technologies (Cassandra, AWS DynamoDB, RDBMS, Redis, Memcached, Hazelcast, etc.);
AWS cloud services: EC2, EMR, RDS, Redshift, Athena, Lambda;
Data pipeline and workflow management tools: Airflow, AWS Glue, Step functions etc;
Hands on experience of building streaming pipelines using Kafka/AWS Kinesis;
Have proven experience working in a cloud environment with GCP being strongly preferred;
Familiarity with Linux/UNIX and tools including Git and SVN;
Proven experience developing large scale data solutions with technology such as Python, Scala, and Spark, preferably for telco, digital and app data;
Familiarity with software engineering principles and product delivery;
Educated to a Bachelor’s degree in Computer Science or related field;
Knowledge of at least one scripting language i.e. Python;
Educated to a bachelor’s degree level in Computer Science or a related field;
Performance tuning queries;
Ability to communicate with diverse stakeholders;
Strong knowledge and aptitude at building and maintaining data warehouses in Redshift/Snowflake;
Post-graduate in Computer Science, Computer Engineering, Physics or a related subject.