Computer Science SME

Job Description

• Design and implement diverse RL environments for software engineering tasks.
• Create evaluation frameworks and graders that measure learnability and partial credit well.
• Daily collaboration with forward-deployed engineers and product contributors.
• Potentially supervise/mentor crowd contributors and answer technical questions (office hours).
• Design tasks: Create realistic coding and systems problems with clear success criteria and useful failure modes.
• Build harnesses: Develop repositories, sandboxes, tools, fixtures, dependencies, and repeatable runtimes.
• Create graders: Use behavioral, unit, integration, and property-based checks to evaluate correctness.
• Calibrate difficulty: Test reference solutions, baselines, and target models; tune tasks to be difficult but learnable
• Improve quality: Diagnose model versus environment failures and review contributor-built tasks.

Skills and Requirements

Strong Python proficiency (table stakes).

• Competent, practical software engineering skills (expected to roll up sleeves).
• Nice-to-haves / preferred:
• Experience with RL/AI, designing agents, AI evaluations, benchmarking.
• Proficiency in at least one additional language (examples: TypeScript/JavaScript, Rust, Go).
• Experience with testing, CI, containers (common across engineers).
• Expert Python plus at least one additional language; strong debugging in unfamiliar codebases.
• CS fundamentals: Algorithms, data structures, complexity, APIs, and sound software-design judgment.
• Systems depth: Expertise in at least one of distributed systems, databases, operating systems, networking, compilers, programming languages, or security.
• Testing and evaluation: Robust test-suite, benchmark, verifier, and failure-taxonomy design.
• Secure execution: Linux, containers, CI, dependency isolation, and safe execution of untrusted code.
• Reproducibility: Versioning, deterministic runs, telemetry, regression testing, and clear documentation.

We are a company committed to creating diverse and inclusive environments where people can bring their full, authentic selves to work every day. We are an equal employment opportunity/affirmative action employer that believes everyone matters. Qualified candidates will receive consideration for employment without regard to race, color, ethnicity, religion, sex (including pregnancy), sexual orientation, gender identity and expression, marital status, national origin, ancestry, genetic factors, age, disability, protected veteran status, military or uniformed service member status, or any other status or characteristic protected by applicable laws, regulations, and ordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application or the recruiting process, please send a request to [email protected].