Data Engineer

About the Role:

As a Data Engineer with 3-6 years of experience at ResourceDekho, you will be instrumental in designing, developing, and maintaining scalable data pipelines that power critical insights for our clients in the IT Services and IT Consulting domain. Leveraging your expertise in Airflow, Pyspark, AWS, Python, SQL, and Snowflake, this role exists to transform raw data into actionable intelligence, ensuring data quality and accessibility to drive informed decision-making and enhance our service offerings.

Role Overview:

ResourceDekho is seeking a talented and experienced Data Engineer to join our dynamic Engineering team. This full-time, hybrid role based in Jaipur offers a unique opportunity to contribute to the core data infrastructure that underpins our IT Services and IT Consulting solutions. You will play a pivotal role in shaping our data strategy, implementing cutting-edge data solutions, and fostering a data-driven culture across our projects, ensuring high performance and reliability of our data systems.

Key Responsibilities:

• Design, develop, and optimize robust ETL/ELT pipelines using Airflow for orchestration and Pyspark for efficient data processing.
• Build and maintain scalable data solutions on AWS, leveraging various cloud services for data storage, computation, and analytics.
• Write clean, efficient, and well-documented code in Python for data ingestion, transformation, and integration tasks.
• Develop and optimize complex SQL queries for data extraction, manipulation, and reporting within relational and columnar databases.
• Manage and enhance our data warehouse architecture using Snowflake, ensuring data integrity, performance, and accessibility.
• Collaborate with cross-functional teams to understand data requirements and deliver high-quality, reliable data products.
• Implement data governance, security, and quality best practices throughout the data lifecycle.

What You'll Work On:

• Developing and enhancing our next-generation data lake and data warehousing solutions on AWS and Snowflake.
• Implementing automated data pipeline orchestration with Airflow to support various client-specific data analytics initiatives.
• Optimizing large-scale data processing jobs using Pyspark to handle growing data volumes and improve performance.
• Building custom data connectors and APIs using Python to integrate diverse data sources.
• Ensuring data quality and consistency across all data assets through rigorous testing and validation processes.

Required Skills & Experience:

• Education: Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or a related technical field (B.Tech, M.Tech, B.E.).
• Experience: 3-6 years of professional experience as a Data Engineer or in a similar role focused on data pipeline development and management.
• Proven track record of designing, building, and optimizing robust, scalable, and efficient data pipelines using modern data engineering practices.
• Solid understanding of data warehousing principles, dimensional modeling, and ETL/ELT methodologies.
• Experience working in an agile development environment, delivering solutions iteratively.

Technical Skills:

• Python: Advanced proficiency in Python for data manipulation, scripting, API integration, and developing data processing applications.
• PySpark: Strong hands-on experience with Apache Spark and PySpark for large-scale data processing, transformation, and analysis.
• Airflow: Expertise in designing, developing, deploying, and managing complex data workflows (DAGs) using Apache Airflow.
• SQL: Expert-level SQL skills for complex querying, data manipulation, performance tuning, and schema design, particularly with analytical databases.
• Snowflake: Solid experience with Snowflake data warehousing, including data loading, performance optimization, role-based access control, and virtual warehouse management.
• AWS: Practical experience with core AWS services for data engineering, such as S3 (data lake), EC2, Lambda, Glue, and familiarity with services like Redshift or Kinesis.

Professional Skills:

• Problem-Solving: Strong analytical and problem-solving abilities to identify, diagnose, and resolve complex data-related issues efficiently.
• Communication: Excellent verbal and written communication skills to articulate technical concepts clearly to both technical and non-technical stakeholders.
• Collaboration: Ability to work effectively within a small, dynamic team, contributing to a collaborative and supportive work environment.
• Adaptability: Demonstrated ability to quickly learn new technologies and adapt to evolving project requirements in a fast-paced IT services context.

Key Performance Indicators (KPI):

• Achieving target uptime and reliability for critical data pipelines (e.g., >99.5%).
• Ensuring timely and accurate delivery of data for downstream analytics and reporting.
• Optimizing data processing jobs for efficiency and cost-effectiveness.
• Contribution to reducing data-related incidents and improving data quality.

Nice to Have:

• Experience with CI/CD practices for data pipelines and infrastructure as code (IaC).
• Familiarity with data governance, data security, and compliance best practices.
• Relevant AWS certifications (e.g., AWS Certified Data Analytics - Specialty).