Overview
Data engineer role building and maintaining Azure-based data pipelines, reporting and dashboards.
What you'll do
- Design, review and develop PySpark scripts.
- Test and troubleshoot data pipelines and orchestration.
- Design and develop reports and dashboards in Power BI, including setting up access control with row-level security and DAX queries.
- Establish connections to source data systems such as on-prem databases, IoT devices and APIs.
- Manage collected data in appropriate storage and database solutions such as file systems, SQL servers and big data platforms such as Hadoop or HANA, as project requirements dictate.
- Design and develop data pipelines using PySpark and copy data activities for batch ingestion.
- Perform data integration, e.g. using database table joins or other mechanisms, at the level required by project analysis requirements.
- Deploy pipeline artifacts from one environment to another using Azure DevOps.
What you'll need
- Bachelor's degree in Computer Science or Engineering with 2 years of experience in Azure data engineering, Python, PySpark or big data development.
- Sound knowledge of Azure Synapse Analytics for pipelines, orchestration and setup.
- 1-2 years of experience in visualization design and development with Power BI, with knowledge of row-level security and access control.
- Sound experience in SQL, data warehousing, data marts and data ingestion with PySpark and Python.
- Expertise in developing and maintaining ETL processing pipelines on cloud platforms such as AWS or Azure.
- Team player with good interpersonal, communication and problem-solving skills.
Nice to have
- Azure Synapse or Data Factory experience on cloud ETL pipelines.
- DevOps expertise.
Details
- Location: Singapore.
- Work mode: onsite, no hybrid option.
- Working hours: 8:30am to 6pm, Monday to Friday.
Read the full description and apply on the company’s own careers page.