Data Engineering Technical Lead - Achievement Awards Group

The Data Engineer lead is responsible for the medium-term and day-to-day operations of the data engineering function. This position is responsible for architecting, building and maintaining the data infrastructure that powers data driven initiatives. This role involves managing a team of data and machine learning engineers, this position will lead projects, provide technical expertise and manage the implementation of best practices to ensure the smooth flow of data for analysis and insights.

This position will be accountable for:

• Oversee and manage a team of data and machine learning engineers and provide technical leadership in projects aimed at delivering high-quality, scalable data solutions
• Manage the design and implementation of data driven strategies across the business
• Coordinate the design, build and maintenance of sturdy data architectures
• Participate in the evaluation, selection and implementation of innovative data engineering tools and technologies
• Contribute to the definition of processes needed to achieve operational excellence in all areas of data engineering, including project management

• Manage the design, build and launch of new data models and data pipelines in production
• Implement data quality standards, data governance protocols and security to ensure data reliability and accuracy
• Ensure data integrity by periodically auditing / cleaning data and creating an ecosystem that pre-emptively finds errors and highlights anomalies
• Act as the project manager for data projects, mentor and grow the team to ensure high performance and operational excellence
• Manage the design and build an infrastructure for extraction, transformation and loading of data from a wide range of data sources
• Ensure the build and maintenance of data foundations that include tools, infrastructure and pipelines
• Increase automation and build data solutions at scale to serve the business requirements
• Constantly monitor and improve data processes and infrastructure for performance, scalability and cost-effectiveness
• Act as a bridge between raw data and actionable insights and determine the success or failure of data-driven projects
• Implement and integrate new technologies and methodologies into the data ecosystem
• Contribute to the development of a budget and resource plan to ensure maximum return on investment and the provision of best tools and technologies
• Provide valuable insights that will ensure data driven success of the organisation
• Travel and client engagement: Required to travel and meet with clients to understand their business requirements and ensure data solutions align with their needs
• Manage rigorous documentation of processes, changes and implementations, creating a wealth of knowledge for a clear understanding of the data landscape

Requirements

Knowledge, skills and attributes:

• Forward-thinking, planning and proactive approach
• Excellent data infrastructure and data architecture skills
• Solid operational skills coupled with the ability to provide leadership and oversight to a team and on projects
• Detailed knowledge of building data pipelines, datasets and data architecture
• Knowledge of data security practices, data privacy and security regulations
• Detailed knowledge and proficiency in relational, noSQL databases and data warehouse design, ETL process, modeling and analytics concepts
• Working knowledge of project management methodologies
• Strong problem-solving, analytical, critical thinking and troubleshooting skills
• Working knowledge of the full software development life cycle
• Good knowledge of operating large production clusters, big data technologies on-premise and/or cloud
• Excellent communication and collaboration skills
• Proficiency in languages commonly used in data engineering, e.g., Python, Java, SQL, etc
• Familiarity with cloud platforms and respective big data solutions
• Ability to optimize database performance, queries and ETL processes
• Good decision-making ability
• Informed about best practices in data governance, ethics in data management and industry requirements

Education and training:

• National Senior Certificate (NQF 4)
• Degree in Computer Science, Data Science, Information Technology, Engineering, or a relevant field
• Relevant certifications

Experience:

• At least 6 years’ progressive experience in managing, designing and maintaining large-scale data solutions
• At least 2 years team management experience
• Proven experience managing and guiding project teams
• Experience analysing, architecting / re-architecting data and analytics platforms
• Proven experience designing and implementing data ingestion solutions using different approaches
• Experience building secure data solutions
• Proven experience working with a variety of relational databases, query languages, query authoring and related tools
• Experience with streaming data platforms, e.g., Apache Kafka
• AWS experience
• ETL experience
• Data warehousing
• Data governance
• Python
• Snowflake and Cloud Data Platform Knowledge
• SQL Proficiency
• Big Data Technologies (e.g. Hadoop, Spark)
• Data Modeling
• API Integration
• Data Visualization Tools (e.g. Tableau, Power BI)
• Version Control
• Containerization and Deployment (e.g. Docker, Kubernetes)
• Streaming Data Experience (e.g. Apache Kafka, AWS Kinesis.)