Skip to main content
Cyble logo

Data Engineer / Senior Data Engineer

Cyble

Bengaluru, Karnataka, IndiaFull-time$20–65K/yrSeen 2w agoStill listed 3 days ago

See where you stand before you apply. No sign-up to start.

You apply on the posting's application page, with no Olive account needed.

Watch jobs like this.

Don't just apply. Show up ready.

Olive works from this exact posting.

At a glance

Compensation
$20–65K/yr
Location
Bengaluru, Karnataka, India
Schedule
Full-time
Work Authorization
Not specified

Olive focuses on jobs in the United States, including remote roles you can work from there.

Job overview

Cyble is hiring a Data Engineer / Senior Data Engineer to build and operate an Iceberg-based data lake powering its AI agents. The role owns pipelines from raw data landing in S3 through curated lake tables, search indexes, and retrieval-ready datasets. The engineer works with security researchers, data scientists, and product engineers in a small, low-process team.

Skills & qualifications

RequiredNice to have

Skills

Amazon Web Services (AWS), nice to haveData Warehousing, nice to haveETL, nice to haveData Modeling, nice to havePySpark, nice to haveDatabricks, nice to haveAirflow, nice to havePython, requiredSpark, requiredApache Iceberg, requiredAWS EMR, requiredData Lake Design, requiredOpenSearch, requiredElasticsearch, requiredScraped Data Ingestion, requiredSemi-Structured Data, requiredAWS Data Stack, requiredContainerized Deployment, requiredPostgreSQL, requiredMongoDB, requiredRAG Architectures, requiredKafka, requiredAI Coding Tools, requiredTechnical Communication, requiredDomain Ownership, requiredTrino/Presto, nice to haveLLM Context Extraction, nice to haveLLM Serving, nice to haveRetrieval Evaluation, nice to haveOCR, nice to haveDocument Extraction, nice to haveMCP Servers, nice to haveRetrieval APIs, nice to haveGraph Processing, nice to haveWeb Scraping, nice to haveAnti-Bot Handling, nice to haveTerraform, nice to haveCI/CD, nice to haveCybersecurity, nice to have

Qualifications

Bachelor's in Computer Science, Engineering or Equivalent Experience, required3+ Years Data Pipeline Experience, required

Full job description

About the Role Cyble's threat intelligence platform ingests cybercrime data at petabyte scale: stealer logs, breach dumps, dark web forums and marketplaces, Telegram and Discord channels, ransomware leak sites, and domain intelligence. That data feeds our AI agents and analyst tooling, and it is only useful when it is clean, fresh, deduplicated, and retrievable. We're hiring a Data Engineer /Senior Data Engineer to build and run the Iceberg-based data lake that powers Cyble's AI agents. You'll own pipelines end to end, from raw scraped and leaked data landing in S3 to curated lake tables, OpenSearch indexes, and retrieval-ready datasets that agents and products query. You'll work in a small, low-process team where engineers own their domains, ship directly, and use AI coding tools as a normal part of the workflow. What You’ll Do at Cyble:

  • Build and operate the data lake powering Cyble's AI agents: ingestion, Data modeling, deduplication, freshness, lineage, and retrieval-ready outputs.
  • Design and run PySpark pipelines on EMR over Apache Iceberg on S3 (Glue Data Catalog), including schema evolution, partitioning, compaction, snapshot expiry, and small-file control.
  • Ingest scraped, crawled, and leaked data from heterogeneous sources (HTML, JSON, CSV, text dumps, archives, feeds, APIs) and handle the realities: encoding issues, schema drift, malformed records, duplicates, and re-crawled content.
  • Build and tune OpenSearch/Elasticsearch indexing pipelines: mappings, analyzers, shard and index lifecycle strategy, bulk ingestion, and query performance.
  • Build the data side of RAG and agent workloads: chunking, embedding generation, vector and hybrid (keyword + kNN) indexing, and datasets the agents can trust.
  • Run streaming and event-driven pipelines on Kafka; orchestrate batch workloads with Airflow.
  • Deploy and operate data workloads on Kubernetes (EKS) and ECS, including Spark jobs, Trino, workers, and supporting services.
  • Own lake and pipeline reliability: monitoring, alerting, data quality checks, backfills and reprocessing, retention, and storage/compute cost control.
  • Use AI coding assistants and agents (Claude Code, Cursor, Copilot, or similar) daily to build, refactor, test, debug, and document, and review AI-generated code critically before it reaches production.
  • Work with security researchers, data scientists, and product engineers on data models for new intelligence products.

What You’ll Need:

  • Bachelor's degree in computer science, engineering, or equivalent experience.
  • 3+ years building production data pipelines in Python and Spark.
  • Deep Spark knowledge on AWS EMR: shuffle, skew, partitioning, memory tuning, AQE, and cost-aware cluster sizing.
  • Hands-on experience with Apache Iceberg (or Delta/Hudi with willingness to go deep on Iceberg): table design, partition evolution, compaction, snapshot management, time travel.
  • Experience building and operating a data lake or lakehouse that serves downstream AI/ML, search, or agent workloads.
  • Strong OpenSearch/Elasticsearch experience: pipeline integration, index design, and performance tuning at scale.
  • Experience ingesting and normalizing scraped or semi-structured data at scale, including deduplication and schema-drift handling.
  • Working knowledge of the AWS data stack (S3, Glue, EMR, Lambda, IAM) and containerized deployment on EKS and/or ECS.
  • Hands-on with PostgreSQL (indexing, query plans, large-table performance) and MongoDB, with the judgment to choose between relational, document, and search stores.
  • Working understanding of RAG architectures: embeddings, vector stores (OpenSearch kNN, pgvector, or similar), chunking strategies, hybrid retrieval, reranking.
  • Experience with Kafka (or similar) and Airflow in production.
  • Hands-on experience using AI coding tools in real data engineering work, with a track record of catching their mistakes.
  • Ability to explain technical trade-offs clearly to non-technical stakeholders, and comfort owning a domain end to end in a lean team.

Bonus Points If You Have:

  • Trino/Presto on Iceberg, especially on Kubernetes.
  • AI/agent-based context extraction in pipelines: LLMs or NER models (e.g., GLiNER) for entity extraction, classification, and enrichment, and their cost and accuracy at scale.
  • LLM serving and inference (vLLM or similar) and evaluating retrieval or extraction quality (recall, precision/recall on labeled samples).
  • OCR and document extraction (Tesseract, PaddleOCR, Textract, or vision-language models) for images, scans, and PDFs.
  • Building data access layers for agents (MCP servers, retrieval APIs).
  • Graph processing (GraphFrames, Neo4j) for entity relationship mapping.
  • Web scraping frameworks (Scrapy, Playwright) and anti-bot handling, or working downstream of such systems.
  • Infrastructure-as-code (Terraform) and CI/CD for data pipelines.
  • Prior exposure to cybersecurity or threat intelligence data.

If you like working in an inclusive environment, you want to advance your career quickly, and your opinion is valued, look no further than Cyble, Inc. We are young, hungry, and ready to impact the cyber security landscape! Cyble, Inc. takes into consideration an individual’s skillset, experience and location in making final salary determination. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, age, disability, protected Veteran status age, or genetics, or any other characteristic protected by law.

Prepare for this role

You've read the whole posting — now see how you match it.