Data Pipeline Operations Engineer Job at SixMap, Inc., Remote

QzFqWHl4a0o4NDAybm1pZ21DVk1pNzhY
  • SixMap, Inc.
  • Remote

Job Description

We are seeking a detail-oriented and technically skilled Data Pipeline Operations Engineer to manage and execute our weekly scanning process. This critical role ensures the timely flow of customer data through our research, scanning, and UI ingest pipeline. The ideal candidate has a mix of programming, database, and Linux system administration skills to handle the various steps in the scanning workflow.

SixMap is the leading Automated Cyber Defense Platform for continuous threat exposure management (CTEM) across today’s largest, most complex and dynamic enterprise and government environments. With zero network impact and zero agents, SixMap automatically discovers all Internet-facing assets across IPv4 and IPv6 to deliver the most comprehensive external attack surface visibility. The platform identifies vulnerabilities, correlates proprietary and open-source threat intelligence, and provides actionable insights to defend against imminent threats with supervised proactive response capabilities. The SixMap team brings deep intelligence community expertise and best practices to the defense of both U.S. Federal agencies and Fortune 500 corporations.

Responsibilities

  • Manage the weekly scanning process, ensuring customer data progresses through research, scanning, and UI ingest phases according to defined SLAs

  • Prepare input files and kick off processes on the scanning cluster via Airflow

  • Monitor and troubleshoot jobs, adjusting parameters like rate files as needed to optimize runtimes

  • Perform data ingest into production databases using SQL and Python

  • Clear data artifacts and caches in between ingest cycles

  • Execute post-ingest data refresh routines

  • Perform quality checks on ingested data to validate contractual obligations are met

  • Identify process bottlenecks and suggest or implement improvements to the automated tooling to increase speed and reliability

Requirements

  • Required Skills:

    • Strong Linux command line skills
    • Experience with Airflow or similar workflow orchestration tools
    • Python programming proficiency
    • Advanced SQL knowledge for data ingest, refresh, and validation
    • Ability to diagnose and resolve issues with long-running batch processes
    • Excellent attention to detail and problem-solving skills
    • Good communication to coordinate with other teams
    • Flexibility to handle off-hours work when needed to meet SLAs
  • Preferred Additional Skills:

    • Familiarity with network scanning tools and methodologies

    • Experience optimizing database performance

    • Scripting skills to automate routine tasks

    • Understanding of common network protocols and services

    • Knowledge of AWS services like EC2

Benefits

  • Competitive compensation packages; including equity
  • Employer paid medical, dental, vision, disability & life insurance
  • 401(k) plans
  • Flexible Spending Accounts (health & dependents)
  • Unlimited PTO
  • Remote Working Options

Job Tags

Remote job, Flexible hours,

Similar Jobs

Cabela's

Loan Processor Job at Cabela's

 ...POSITION SUMMARY: The White River Financial Services Loan Processor secures the in-direct consumer marine financing and process...  ...signature and send to dealership. Follows up with dealerships for signed loan documents. Reconciles loan document discrepancies.... 

SST Direct

Maintenance Manager Job at SST Direct

 ...Seeking a Maintenance Manager for a direct hire opportunity with our client in Southbridge, MA. This position offers full benefits including PTO, Medical, Dental, Vision, and 401k! Salary is between $100,000 - $130,000 per year (based on experience) Responsibilities... 

Clean Harbors

Driver Class B Trainee - Healthcare (CDL Sponsorship) Job at Clean Harbors

 ...Box Truck Driver role. In this entry-level position, you will be paid to train on the job, learning how to safely transport hazardous and non-...  ...all DOT and HAZMAT regulations. Through our Company Paid CDL Program, we will sponsor you to obtain your Class B CDL (and Hazmat... 

CRH

Local CDL Mixer Driver Job at CRH

 ...to Greater Rochester and the surrounding areas. Compensation and Benefits ~ Class B CDL drivers with experience starting at $29/hr Paid CDL driver training opportunities starting at $23/hr Paid Time Off Medical/Dental/Vision/Prescription Responsibilities... 

Guardian Angel Senior Services

Home Health Aide/Certified Nursing Assistant-40 Hours Guaranteed Job at Guardian Angel Senior Services

 ...looking for compassionate Caregivers, Home Health Aides (HHA), Personal Care Assistants (PCA), and Homemakers to care for seniors and disabled adults in the comfort of their homes. No experience? No problem! We offer free trainings. &##128176; PAY & BENEFITS (Why...