Senior Big Data Engineer Job Description Template
Our company is looking for a Senior Big Data Engineer to join our team.
Responsibilities:
- Work in scrum methodology, effectively prioritise and delegate backlogged commitments;
- Engage with liaise with customers over requirements;
- Using customer requirements to create robust technical solutions;
- Taking a lead and utilising the team to solve complex problems;
- Development of the new and existing data pipelines utilizing Amazon AWS/EMR;
- Cooperating effectively with data scientists and developer, mentor more junior data engineers;
- Parse and handle huge data sets;
- Manage requests and stakeholder requirements in a prioritised manner;
- Manage data integration with third party applications;
- Design and implement internal process improvements;
- Deliver large scale algorithms in production;
- Building suitable environments for deployment;
- Build and manage data pipeline architecture.
Requirements:
- Interest in machine learning and AI;
- Knowledge of data management, handling and processing data quality, preparation, cleansing and modelling;
- Hands-on experience with Spark, Airflow, Hadoop, or Databricks;
- 5-7 years of experience as a data engineer or software engineer working with data;
- Excellent knowledge of Python;
- Significant Amazon AWS experience including EMRInterest in machine learning and AI;
- Experience working with privacy and GDP;
- Experience working with AWS (EMR, S3, Athena, Redshift);
- Experience with Scala/Java and Python;
- Previous experience working in an Agile Environment;
- TDD and Automation;
- Bachelor’s degree in computer science, engineering, or equivalent;
- Experience using Spark, PySpark.