Data Platform DevOps

Job Summary
We are seeking an experienced Data Platform DevOps to implementation, and maintenance of our data infrastructure and pipelines. The ideal candidate will have a strong background in data engineering and DevOps principles, with a passion for automation, quality, and governance. You will act as a bridge between technical and business teams, ensuring our data platform is not only efficient and scalable but also reliable and compliant. This role is crucial for enabling data-driven decisions and accelerating the development life-cycle for all data initiatives.

Key Responsibilities

• Design & Implement Data Platforms: Design, develop, and maintain robust, scalable data pipelines and ETL processes, with a focus on automation and operational excellence.
• Ensure Data Quality and Governance: Implement automated data validation, quality checks, and monitoring systems to ensure data accuracy, consistency, and reliability.
• Manage CI/CD for Data: Own and optimize the CI/CD pipelines for data engineering workflows, including automated testing and deployment of data transformations and schema changes.
• Architect & Implement IaC: Use Infrastructure as Code (IaC) with Terraform to manage data infrastructure across various cloud platforms (Azure, AWS, GCP).
• Performance & Optimization: Proactively monitor and optimize query performance, data storage, and resource utilization to manage costs and enhance efficiency.
• Collaborate with Stakeholders: Manage communication with technical and business teams to understand requirements, assess technical and business impact, and deliver effective data solutions.
• Strategic Design: Possess the ability to see the big picture in architectural design, conduct thorough risk assessments, and plan for future scalability and growth.

Requirements

• Experience: 1-3 years of experience in data engineering, data warehousing, and ETL processes, with a significant portion of that time focused on DataOps or a similar operational role.
• Platform Expertise: Strong experience with data platforms such as Databricks and exposure to multiple cloud environments (Azure, AWS, or GCP).
• Data Processing: Extensive experience with Apache Spark for large-scale data processing.
• Orchestration: Experience working with data orchestration tools like Azure Data Factory (ADF), Apache Airflow, or similar.
• CI/CD & Version Control: knowledge of version control (Git) and experience with CI/CD pipelines (GitLab CI/CD, GitHub Actions).
• IaC: hands-on experience with Terraform.
• Programming: Programming skills in Python and advanced proficiency in SQL.
• Soft Skills: Strong stakeholder management, communication, and collaboration skills. The ability to articulate complex technical concepts to non-technical audiences is a must.
• Problem-Solving: Strong problem-solving skills with an ability to analyze technical challenges and their business impact.

Preferred Qualifications

• Data Modeling: Experience with data modeling tools and methodologies, specifically with dbt (data build tool).
• AI & ML: Experience with AI-related technologies like Retrieval-Augmented Generation
• (RAG) and frameworks such as LangChain.
• Data Observability: Hands-on experience with data quality and observability tools such as Great Expectations, Monte Carlo, or Soda Core.
• Data Governance: Familiarity with data governance principles, compliance requirements, and data catalogs (e.g., Unity Catalog).
• Streaming Technologies: Experience with stream processing technologies like Kafka or Flink.
• Containerization: Familiarity with containerization and orchestration tools (e.g., Docker, Kubernetes).
• Open Source: Contributions to open-source projects or relevant certifications.

Benefits
Health Insurance
- At Lotus's, we care about your health Group insurance from a top insurance company is included in your benefits—OPD, IPD, Emergency OPD

Provident Fund
- Lotus's cares about your long-term plan We offer 3% provident fund.

Year-end bonus
- We include variable and performance bonus for our employees.

Attractive Vacations days
- Enjoy our attractive annual leave. Let's say the minimum is 16 days

No overtime
- We work 5 days a week with. We set our own goals and deadlines.

Free car parking space
- No more stress or extra cost if you drive to work. We offer free parking space for our employees.

Best Culture

• Clear focus.
• Diverse Workplace (Our members are from around the world)
• Non-hierarchical and agile environment
• Growth opportunity and career path