Senior Data Engineer – AWS/Databricks/Python

Primary Responsibilities:

• Design, develop and maintain data pipelines to extract data from a variety of sources and populate data lake and data warehouse

• Develop the various data transformation rules and data modelling capabilities

• Collaborate with Product Analyst, Data Scientists, and Machine Learning Engineers to identify and transform data to make data understandable.

• Experience in implementing data quality checks, maintain data catalogues, use orchestration, logging, and monitoring tools, data analysis, information architecture concept and implementation, Git for version control and understand various branching strategies

• Use test driven development methodology when building ELT/ETL pipelines

   

Required Experience and Skills:

• min 5 yrs of relevant experience

• Solid experience with AWS services (S3, IAM, Redshift, Sagemaker, Glue, Lambda, Step Functions, CloudWatch)

• Experience with platforms like Databricks, Dataiku

• Proficient in Python / Java, SQL – Redshift preferred, Jenkins, CloudFormation, Terraform, Git, Docker, 2-3 years of Spark – PySpark

• Good communication and SDLC documentation skills.