Senior Data Engineer
Tech Stack: Python, Airflow, Scrapy, Azure AI Search, Azure, SQL. Come ambitious and hungry!
About the Role
Join our core engineering team to build the data backbone of our AI-powered accounting platform. You will take full ownership of our data ingestion and processing infrastructure—transforming raw, unstructured web data into high-quality search indexes for our LLMs. We value engineers who treat data pipelines as production products and can navigate the complexity of high-scale web scraping.
Responsibilities
- Architect resilient scraping infrastructure: Build and maintain high-volume, compliant web scrapers using Scrapy to ingest financial and regulatory data from diverse sources.
- Power the AI Context Window: Design pipelines to clean, chunk, and index data specifically for Azure AI Search, ensuring our RAG systems have the most relevant and up-to-date context.
- Orchestrate complex workflows: Design and optimize data pipelines (ETL/ELT) using Apache Airflow, ensuring data quality and timely delivery.
- Anti-Bot Evasion & Proxy Management: Implement sophisticated strategies to handle CAPTCHAs, IP rotations, and headless browsing to ensure 99.9% pipeline uptime.
Requirements
- 5+ years of data engineering experience, with a heavy focus on Python.
- Deep knowledge of web scraping and building scraping pipelines at scale (handling anti-bot countermeasures, dynamic content, and headless browsers).
- Experience configuring and optimizing Azure AI Search indexes (vector search, semantic search, hybrid retrieval).
- Proficiency with Apache Airflow for DAG authoring and scheduling.
- Strong SQL skills and experience modeling data for analytics.
Nice to Have
- Familiarity with running workloads on Kubernetes.
- Experience fine-tuning ranking algorithms or scoring profiles in search indexes.
- Knowledge of LLM integration patterns (RAG).
