BI Data Engineer - Sanlam

Guided by senior developers you will be responsible for establishing new technology components and reusable solution patterns that can be leveraged by business-facing development teams in their day-to-day solutions. You will constantly be developing and setting new principles, standards, processes, procedures, and guidelines for the wider BI community.

You should be able to communicate technical information to technical teams, as well as be competent in communicating challenges and solutions to project and operational leadership. An understanding of data management solutions and a keen sense of the strategic value of information to an organisation will be of importance.

You will also be responsible for developing data warehousing blueprints, evaluating hardware and software platforms and integrating systems; translating business needs into long-term architecture solutions.

What will you do?

Main Responsibilities:

The performance of the Data Engineer can be described and measured by:

• Define a structured approach to problem solving and delivery against it. 
• Create role-specific design standards, patterns and principles 
• Assist the planning and management of the workload of the team and to ensure delivery
• Load large, complex data sets and make data available for other data engineers
• Identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery, re-designing models for greater scalability
• Working with other data engineers and data modellers, you will design, implement, and manage data vaults, data transformations and the data pipeline
• Identify, design, and implement vault access layers to enable BI products to leverage the data within data vaults
• Monitor and fine-tune data vaults and data transformations on the Cloudera Hadoop stack
• Use modern development and modelling techniques and tools to implement BI and data management solutions, including data quality, metadata and reference data
• Engage with a wide range of technical stakeholders including data scientists, data analysts, business analysts, other data engineers and solutions architects
• Support data stewards to establish and enforce guidelines for data collection, quality improvements, integration, and processes

What will make you successful in this role?

Role Requirements:

Qualifications:

Bachelor’s degree in Computer Science, Statistics, Informatics, Information Systems, Engineering or another quantitative field / National Diploma in an Information Technology related discipline preferred

Work Experience:

The Data Engineer must have at least 3 to 4 years of relevant experience in a similar environment working with the relevant tools and techniques

Technical Knowledge and Experience:

• The Data Engineer is someone with a strong understanding of data, data structures and data sources. Required skills include:
• Application and data engineering background with a solid background in SQL is required 
• Knowledge of database management system (DBMS) physical implementation, including tables, joins and SQL querying.
• Data architecture design and delivery experience preferred
• Experience in Database technologies (e.g. SAP Hana, or similar) or Hadoop components including HDFS, Hive, Spark, Oozie and Impala preferred and highly advantageous.   
• Object-oriented/object functional scripting languages (e.g. Python, Java, Scala or related)
• Knowledge and experience of structured data, such as entities, classes, hierarchies, relationships and metadata.
• Strong Data Engineering background with a specific focus on staging high quality data
• Understanding of data warehousing principles (e.g. Kimball and Vault).
• Experience in agile development
• Ability to comply to and manage data assets under a strict governance framework
• Desirable/ preferred skills include:
• Data warehousing (Kimball and Data Vault patterns are preferred) and dimensional data modelling (e.g. OLAP and MDX experience)
• Experience in developing data pipelines using ETL tools (e.g. SAP Data Services), automation (e.g. Wherescape), scheduling and test automation (e.g. Robot) is desirable
• A solid background in SQL, Information Architecture and ETL procedures is required
• Experience with object-oriented/functional/scripting languages (e.g. Python, Unix Shell scripting, Java, Scala etc.) is preferred but not essential. 
• Data Management technologies (e.g. Informatica Data Quality (IDQ), Informatica Enterprise Data Catalog (EDC), Axon, EBX) 
• Event/Streaming based data pipelines (e.g. Kafka or Nifi) nice to have

Competencies:

• Strong analytical & numerical ability
• Problem-solving skills

• Conceptual thinker
• Good communication / interpersonal skills
• Action / results orientation
• Quality orientation
• Knowledge seeking/learning orientation
• IT Data Analysis 
• Database Knowledge and Monitoring 
• Data Security 
• Risk Management 
• Strong analytical thinking and problem-solving abilities
• Ability to collect, organize, analyze, and disseminate significant amounts of information 
• Strong technical and operational ability 
• Deadline-driven, even in pressurised and fast-paced environments

Personal Attributes

• Organisational savvy - Contributing through others
• Manages complexity - Contributing through others
• Plans and aligns - Contributing through others
• Optimises work processes - Contributing through others