What Is This Job?

A Senior Data Engineer is a professional who specializes in preparing big data infrastructure for analytical or operational uses. They are responsible for designing and creating systems that collect, manage, and convert raw data into usable information for data scientists and business analysts to interpret. Their work enables companies to make smarter decisions and optimize their operations.

What Does This Job Do?

A Senior Data Engineer develops and maintains scalable data pipelines and builds out new API integrations to support continuing increases in data volume and complexity. They collaborate with data scientists and business teams to improve data models that feed business intelligence tools, increasing data accessibility and fostering data-driven decision making across the organization.

They play a crucial role in implementing software and methodologies for data correction, reconciliation, and quality checking.

Job Brief

We’re seeking a Senior Data Engineer to enhance our Data Science Team, focusing on implementing and managing data workflows that support machine learning models and large-scale analytics. This role involves designing ETL processes, ensuring data quality, and deploying ML models to production.

The ideal candidate will have a strong computer science background, advanced Python knowledge, and experience with cloud services, SQL/NoSQL databases, and Docker/Kubernetes.

You’ll work closely with our data science and product teams to drive insights and innovations.

Responsibilities

• Designing and implementing ETL processes
• Managing data warehousing solutions
• Exposing and deploying machine learning models to production
• Ensuring data quality and consistency across various sources
• Design and implement ETL processes for data transformation and preparation
• Deploy machine learning models to production environments
• Manage data pipelines for analytics and operational use
• Ensure data accuracy and integrity across multiple sources and systems
• Collaborate with data scientists to support NLP algorithms and analytics

Requirements

• 4+ years of experience in data engineering within a production environment
• Advanced knowledge of Python and Linux shell scripting
• Experience with SQL/NoSQL databases (e.g., Redshift, Postgres, MongoDB)
• Proficiency in building stream processing systems using Kafka
• Familiarity with Docker, Kubernetes, and cloud services (AWS, GCP)
• Bonus: Experience with the ELK stack and machine learning knowledge