Lead Data Engineer

Key Responsibilities

• Define, shape and perfect data strategies in central and local government.
• Help public sector teams understand the value of their data, and make the most of it.
• Establish yourself as a trusted advisor in data driven approaches using public cloud services like AWS, Azure and GCP.
• As employee growth is a huge focus here, we would expect you to contribute to our recruitment efforts and take on line management responsibilities.

Skills, Knowledge & Expertise

• Proficiency in Git (inc. Github Actions) and able to explain the benefits of different branch strategies.
• Strong experience in IaC and able to guide how one could deploy infrastructure into different environments.
• Knowledge of handling and transforming various data types (JSON, CSV, etc) with Apache Spark, Databricks or Hadoop.
• Good understanding of possible architectures involved in modern data system design (Data Warehouse, Data Lakes, Data Meshes)
• Ability to create data pipelines on a cloud environment and integrate error handling within these pipelines.
• You understand how to create reusable libraries to encourage uniformity or approach across multiple data pipelines.
• Able to document and present end-to-end diagrams to explain a data processing system on a cloud environment.
• Some knowledge of how you would present diagrams (C4, UML, etc.)
• Enthusiasm for learning and self-development.
• You have experience of working on agile delivery-lead projects and can apply agile practices such as Scrum, XP, Kanban.
• Can own the cloud infrastructure underpinning data systems through a DevOps approach.
• Design and implement efficient data transformation processes at scale, both in batch and streaming use cases.
• You are a skilled Data Engineer who has delivered data platforms.
• Knowledge of SOLID, DRY and TDD principles and how to practically implement these into a project.
• You can demonstrate a commercial mindset when on projects to grow accounts organically with senior stakeholders.
• You have the experience to improve resilience into a project by checking for software vulnerabilities and implement appropriate testing strategies (unit integration, data quality, etc)
• You are skilled at offering guidance on how one would implement a robust DevOps approach in a data project.
• You can comfortably talk about tools needed such as DataOps in areas such as orchestration, data integration and data analytics.

• Working at a technology consultancy
• Working with Docker and virtual environments as part of the development and CI/CD process.
• Working with senior stakeholders to gather requirements and keep them engaged with
• Experience in working with a team of engineers using a variety of techniques such as pair programming or mob programming.
• Working with data scientists to productionise advanced data deliverables, such as machine learning models
• Working knowledge of statistics
• Working with multidisciplinary digital and technology teams
• Working within the public sector
• Working with data scientists to productionise advanced data deliverables, such as machine learning models

Job Benefits