Location: 100% remote

Employee Location: India

Position Title: Lead Data Engineer

Job Type: Full-Time

‍

About the role:

We are seeking an experienced and visionary Head of Data Engineering / Lead Data Engineer to build and lead our data engineering team. The ideal candidate will be responsible for designing, implementing, and optimizing our data infrastructure, ensuring seamless data flow, storage, and processing to support business intelligence, analytics, and AI/ML initiatives. This role requires strong technical expertise, leadership capabilities, and a strategic mindset to drive data-driven decision-making across the organization.

‍

You will be working on:

• Architect, develop, and maintain scalable data pipelines and ETL processes.
• Build and maintain web scraping solutions to scrape industry-wise datasets
• Design and implement robust data warehousing solutions.
• Implement and manage data versioning strategies and tools
• Ensure data quality, integrity, and security across all platforms.
• Collaborate with cross-functional teams, including data scientists, analysts, and software engineers, to optimize data workflows.
• Lead, mentor, and grow a high-performing data engineering team.
• Evaluate and implement modern data technologies, tools, and best practices such as distributed data processing frameworks , specialized data formats and ML metadata management tools.
• Monitor and optimize system performance, ensuring high availability and scalability.
• Define and enforce data governance policies and best practices.
• Work closely with leadership to align data strategy with business objectives.

‍

Who are we looking for:

• Experience: 7+ years in data engineering, with at least 2-3 years in a leadership role.
• Technical Expertise: Strong proficiency in SQL, Python, and data warehousing.
• Experience with web scraping tools and data versioning
• Data Technologies: Hands-on experience with big data technologies (Spark, Hadoop, Kafka, etc.).
• Cloud Platforms: Experience with cloud-based data solutions (AWS, GCP, Azure).
• Databases: Proficiency in working with relational and NoSQL databases (Redshift, Snowflake, Big Query, etc.).
• ETL & Data Pipelines: Expertise in building scalable ETL processes and real-time data streaming solutions.
• Leadership Skills: Proven ability to lead and mentor a team of engineers.
• Problem-Solving: Strong analytical and problem-solving skills with a data-driven mindset.
• Communication: Excellent verbal and written communication skills.

‍

Preferred qualifications:

• Experience in implementing machine learning pipelines.
• Familiarity with containerization and orchestration tools (Docker, Kubernetes, Airflow).
• Knowledge of data privacy and security best practices.
• Experience implementing and optimizing machine learning pipelines, particularly those related to LLM training and fine-tuning.
• Familiarity with MLOps tools and platforms (e.g., MLflow, Kubeflow).
• Experience working with large language models and their data requirements.