Location: San Francisco, CA (On-Site)
Duration: Contract (6 months, with potential extensions)
About the Role:
We are seeking an experienced Databricks Data Engineer to join our dynamic team in San Francisco. This role is ideal for someone passionate about data engineering, hands-on in building scalable solutions, and comfortable working in a fast-paced environment. Prior experience supporting supply chain operations and analytics is highly desirable.
As a Data Engineer, you will be responsible for designing, building, and optimizing data pipelines and workflows in Databricks, supporting critical business initiatives. You will collaborate closely with data scientists, analysts, and business stakeholders to ensure high data quality, availability, and performance.
Key Responsibilities:
- Develop, optimize, and maintain large-scale data pipelines in Databricks and Apache Spark.
- Work closely with business teams to understand requirements and translate them into technical solutions.
- Build robust, scalable ETL processes to ingest, transform, and load data from various sources.
- Design and implement data models to support advanced analytics and reporting.
- Ensure best practices around data governance, quality, and security are followed.
- Monitor, troubleshoot, and enhance existing data workflows.
- Collaborate with the supply chain, operations, and IT teams to support end-to-end data initiatives.
- Document technical designs, processes, and tools for ongoing operations and maintenance.
Qualifications:
- 3-5+ years of hands-on experience as a Data Engineer.
- Strong expertise in Databricks (Delta Lake, Databricks Workflows, Unity Catalog).
- Proficiency in Apache Spark, SQL, and Python.
- Solid understanding of ETL/ELT development, big data processing, and cloud data architectures (AWS, Azure, or GCP).
- Experience working with complex, large-scale datasets.
- Exposure to supply chain data (inventory, logistics, procurement, fulfillment) strongly preferred.
- Familiarity with DevOps practices for data pipelines (CI/CD for data).
- Excellent problem-solving and communication skills.
- Ability to work independently on-site in San Francisco.
Nice to Have:
- Experience with MLflow, Databricks SQL Warehouses.
- Familiarity with Airflow, dbt, or similar orchestration and transformation tools.
- Knowledge of data governance frameworks.
Details:
- Location: Must work in-office in San Francisco, CA
- Compensation: Competitive hourly rate, commensurate with experience.
- No sponsorship available for this role; candidates must be authorized to work in the U.S.
Darwin Recruitment is acting as an Employment Business in relation to this vacancy.
Ryan Corbett