John Cropezi’s Path into Developer Tools

• Cropezi became interested in programming as a child after his father showed him a simple QBasic program that printed “Hello John” repeatedly.
• He learned by reading QBasic’s documentation and initially focused on recreating DOS and toy operating systems.
• After studying computer science, he worked at startups, Square, Tumblr, and GitHub.
• At GitHub, he worked on contributor pages, the contribution graph, merge queue, and early Codespaces.
• He later joined Jane Street, where the smaller, more focused user base makes it possible to build deeper integrations around the firm’s languages and workflows.

Trade-offs in Language and Tooling

• Jane Street can tailor tools closely to its common languages, build system, and code-review practices. That depth improves end-to-end usability, but risks making systems inflexible when new languages or workflows emerge.
• Cropezi sees good abstractions and replaceable components as important safeguards. The firm also needs ongoing work to integrate newer uses of Python with systems built primarily for OCaml.
• Comparing Ruby and OCaml, he values Ruby’s expressiveness and speed of development, while OCaml’s static types provide useful guarantees and help communicate assumptions across a large codebase.
• Dynamic languages can still work at scale when performance-critical work is moved to other services and the main bottlenecks—often databases or I/O—are addressed directly. Optional typing systems can help with maintainability.
• Powerful abstractions and type-system features have costs. Cleverness can make testing and maintenance harder, so engineering should favor simplicity while preserving necessary functionality.

Building Jane Street’s AI Platform

• When ChatGPT arrived, Jane Street needed a secure way for employees to use it. Cropezi helped build an internal chat application, JaneGPT, with centrally managed credentials and data controls.
• Adding multiple model providers led to a shared layer that lets users and programs switch models without rewriting integrations. It also supports centralized rate limits and metadata controls.
• The team now works across four main areas:
• LLM application platform: Client libraries, model calls, and typed function-calling interfaces for OCaml and Python.
• User-facing tools: JaneGPT, the aid CLI coding assistant, editor integrations, and interfaces in notebooks and internal languages.
• Agent tools: Access to internal information and systems; the firm has more than a thousand tools built by different teams.
• Evaluations and security: Testing model and agent performance, improving their setups, and limiting what they can access or execute.

Evaluations and Security

• Model evaluation is difficult because outputs are stochastic, results can be affected by small changes, and even LLM graders may struggle to judge work at the frontier of model capability.
• Rubrics, deterministic tests, measurable outcomes, and cost or runtime metrics can make evaluations more useful.
• Models should not simply inherit all the permissions of the human using them. Sandboxing and access controls are needed because agents may behave unexpectedly or try to work around restrictions.
• Jane Street moved from a deny-by-default-in-practice approach to an allowlist sandbox using filesystem and network controls. Monitoring tools help identify dependencies a process needs.
• A strict sandbox proved useful for ordinary software development but did not fit every research workflow. The firm now supplements sandbox decisions with an LLM classifier that can allow, deny, or request human approval.
• Security also requires controls above the operating-system level, such as RPC permissions for particular actions.
• AI user identities distinguish agent activity from human activity, enabling appropriate permissions and attribution of AI-assisted code.
• Agents can misuse shared resources when they do not understand their value or cost, so systems designed for human judgment need agent-specific limits and safeguards.

How Coding Tools Evolved

• Early tools included chat, autocomplete, editor-based questions, and experimental function calling. At first, models were unreliable at using tools, and adoption of agentic workflows was limited.
• A command-line prototype made it easier to iterate on editor integrations. As models improved at tool use, it grew into the aid coding assistant.
• The team also developed centralized orchestration so work can be started from interfaces such as chat or Slack and run on developer machines or build hosts.
• Jane Street’s aid mint workflow lets a user describe a feature in Slack; the system creates a feature workspace, starts a build, and dispatches an agent. The work runs under the user’s AI identity.
• This relatively small change in workflow made agent use much more accessible; roughly 45% of released features reportedly begin through this asynchronous agent flow.
• Preconfigured workspaces and build targets reduce setup work for both agents and humans. Since models repeatedly rediscover how to use tools, reliable setup and concise, well-maintained skills are important.
• The team both builds internal tools and supports external ones such as Claude Code and Codex. Internal tools can integrate deeply with Jane Street’s environment, while external tools provide new capabilities and help the team learn. Maintaining both requires attention to security and sandbox compatibility.

How AI Changes the Work of Programming

• AI can take over routine implementation, shifting engineers’ time toward design, review, and verification.
• This can be freeing for people who previously found coding inaccessible, but it may reduce the satisfying “craft” of writing code and the learning that happens during implementation.
• Reviewing model-written code can be harder than reviewing human-written code: it may compile and work while containing subtle, unfamiliar mistakes.
• Models can also help people understand code. For example, review tools could reorganize diffs into a clearer explanation, surface decisions that were tried and abandoned, or use agent session logs to answer reviewers’ questions.
• AI-generated visualizations and other non-chat interfaces may help explain technical systems, although these approaches remain underexplored.
• The team’s role is increasingly both to create useful developer tools and to provide a platform where others can experiment with tools, review methods, and agent workflows.

Opportunities and Challenges Ahead

• Multi-agent systems raise open questions about how agents should share information, coordinate, and manage concurrent work.
• Giving agents broader control over testing could extend automation beyond unit tests to deployments, fuzzing, property-based testing, and longer-running system tests.
• The development loop may also extend into planning, monitoring, and incident response.
• AI makes rapid prototyping dramatically easier. Cropezi describes a long-imagined editor feature that turns pseudocode into real code, prototyped in about half an hour with an LLM.
• AI may also make refactoring and quality improvements economical when they were previously too costly.
• At the same time, lower implementation costs can produce unnecessary features and code “slop.” Teams still need standards, prioritization, and the ability to reject changes that add needless complexity.
• A useful cultural framing is to treat an AI-assisted submission as a prototype: evaluate whether the change is worthwhile, not just whether the code is present.