Duration: 3 months; will likely extend
Location: REMOTE
Team is in charge of improving services for internal customers. Maintenance for data pipelines, adding tests. Qualified TAWs will help with upgrades and implementing new strategies.
Qualifications :
- Bachelor or Master's degree in Computer Science, Engineering, Information Systems or relevant degree is required
- 5+ years of professional work experience designing and implementing data pipelines in on-prem and cloud environments ( S3, EKS )
- Experience with SQL/Relational databases.
- Experience with manipulating structured and unstructured data.
- Experience with distributed data systems such as Hadoop and related technologies (Spark, Trino, etc.).
- Background in both programming languages (Python & Scala ).
- Experience working with databases that power APIs for front-end applications.
- Experience with modern schedulers ( Airflow )
Responsibilities
Support, Design, develop, test, deploy, maintain and improve data pipeline
Designing and developing data processing techniques: automating manual processes, data delivery, data validation, data quality and integrity
Communicate effectively with customers/team members & help with site up challenges.
Must have skills:
- Python
- SQL
- Spark
- Scala
- AWS ecosystem ( S3, EKS )
- Airflow
- Kafka
- Catalog store ( hive or similar )
Nice to have skills:
- Iceberg
To apply for this job please visit www2.jobdiva.com.