SWE-agent

v1.1.0: 10s of thousands of training trajectories

We're very excited to announce our new project SWE-smith, generating 10s of thousands of training trajectories for SWE agents.

Using this training data, our LM SWE-agent-LM-32b achieves open-weights SotA on SWE-bench verified with SWE-agent!

image

Apart from that, v1.1.0 is mostly a fix release with minor improvements, in particular adding compatibility with SWE-bench multilingual/multimodal, and SWE-smith. However, please pay attention to the breaking changes below.

Breaking changes

• Changes to trajectory data format. The messages field is replaced by query by @klieret in https://github.com/SWE-agent/SWE-agent/pull/1107
• Renamed many tool bundles that used "windowed" file viewer (defaults and more) by @klieret in https://github.com/SWE-agent/SWE-agent/pull/1147
• Removed review_on_submit tool bundle (replaced by review_on_submit_m) by @klieret in https://github.com/SWE-agent/SWE-agent/pull/1148
• Change in windowed tools (formerly default): Don't append \n to new file by @klieret in https://github.com/SWE-agent/SWE-agent/pull/1114

Added

New dataset support:

• Feat: Support multilingual evaluation by @kabirgh in #1090
• Feat: SWE-smith & multimodal base support by @klieret in #1092

New utilities:

• Feat: Add quick-stats tool by @klieret in #1125

Enhanced

• Feat: Config/override max_output_tokens by @klieret in #1036
• Enh: [#1042] fix(run_batch): handle JSON parsing errors in trajectory check by @FRAOTIAC in #1043
• Enh: Allow to override tools dirs etc. by @klieret in #1046
• Enh: Allow to override path to swe-bench dataset by @klieret in #1093
• Enh: Allow to disable python-standalone for batch by @klieret in #1115
• Enh: More information on skipped exit status by @klieret in #1117

Fixed

• Fix: Setting max_input_tokens to 0 by @klieret in #999
• Fix: Explicitly set log file encoding by @klieret in #1013
• Fix: Ensure pydantic-settings env prefix set by @klieret in #1018
• Fix: run batch processing with modal by @vsee in #1023
• Fix: Catch exit forfeit by @klieret in #1024
• Fix: Use 'latest' image tag for SWE-Bench images by @klieret in #1029
• Fix: Show tenacity retry reasons by @klieret in #1032
• Fix: Compatibility with textual 2.0 by @klieret in #1033
• Fix: Use default trajectories dir according to ENV by @vsee in

<p align="center">

<a href="https://swe-agent.com/latest/">

<img src="assets/swe-agent-banner.png" alt="swe-agent.com" style="height: 7em" />

</a>

</p>

<p align="center">

<a href="https://swe-agent.com/latest/"><img src="https://img.shields.io/badge/Docs-green?style=for-the-badge&logo=materialformkdocs&logoColor=white" alt="Docs"></a>

<a href="https://swe-bench.slack.com"><img src="https://img.shields.io/badge/Slack-4A154B?style=for-the-badge&logo=slack&logoColor=white" alt="Slack"></a>

<a href="https://arxiv.org/abs/2405.15793"><img src="https://img.shields.io/badge/arxiv-2405.15793-red?style=for-the-badge&logo=arxiv&logoColor=white&labelColor=black" alt="arxiv 2405.15793"></a>

</p>

<p align="center">

<a href="https://github.com/SWE-agent/mini-swe-agent/">

<img src="assets/warning.png" alt="mini-swe-agent.com" style="height: 7em" />

</a>

</p>

[!warning]

Most of our current development effort is on mini-swe-agent,

which has superseded SWE-agent. It matches the performance performance of SWE-agent, while being

much simpler.

See the FAQ for more details about the differences.

Our general recommendation is to use mini-SWE-agent instead of SWE-agent going forward.

SWE-agent enables your language model of choice (e.g. GPT-4o or Claude Sonnet 4) to autonomously use tools to

fix issues in real GitHub repositories,

find cybersecurity vulnerabilities, or

perform any custom task.

• ✅ State of the art on SWE-bench among open-source projects
• ✅ Free-flowing & generalizable: Leaves maximal agency to the LM
• ✅ Configurable & fully documented: Governed by a single yaml file
• ✅ Made for research: Simple & hackable by design

SWE-agent is built and maintained by researchers from Princeton University and Stanford University.

📣 News

• July 24: Mini-SWE-Agent achieves 65% on SWE-bench verified in 100 lines of python!
• May 2: SWE-agent-LM-32b achieves open-weights SOTA on SWE-bench
• Feb 28: SWE-agent 1.0 + Claude 3.7 is SoTA on SWE-Bench full
• Feb 25: SWE-agent 1.0 + Claude 3.7 is SoTA on SWE-bench verified
• Feb 13: Releasing SWE-agent 1.0: SoTA on SWE-bench light & tons of new features
• Dec 7: An interview with the SWE-agent & SWE-bench team

🚀 Get started!

👉 Try SWE-agent in your browser: Open in GitHub Codespaces (more information)

Read our [documentation][docs] to learn more:

• Installation
• Hello world from the command line
• Benchmarking on SWE-bench
• Frequently Asked Questions

[docs]: https://swe-agent.com

SWE-agent for offensive cybersecurity (EnIGMA) <a name="enigma"></a>

<img src="https://github.com/user-attachments/assets/84599168-11a7-4776-8a49-33dbf0758bb2" height="80px"></img>

[SWE-agent: EnIGMA][enigma] is a mode for solving offensive cybersecurity (capture the flag) challenges.

EnIGMA achieves state-of-the-art results on multiple cybersecurity benchmarks (see leaderboard).

Please use SWE-agent 0.7 while we update EnIGMA for 1.0.

[enigma]: https://enigma-agent.com

[SWE-bench]: https://github.com/SWE-bench/SWE-bench

[nyu-ctf]: https://arxiv.org/abs/2406.05590

In addition, you might be interested in our other projects:

<div align="center">

<a href="https://github.com/SWE-agent/mini-SWE-agent"><img src="docs/assets/mini_logo_text_below.svg" alt="Mini-SWE-Agent" height="120px"></a>

&nbsp;&nbsp;

<a href="https://github.com/SWE-agent/SWE-ReX"><img src="docs/assets/swerex_logo_text_below.svg" alt="SWE-ReX" height="120px"></a>

&nbsp;&nbsp;

<a href="https://github.com/SWE-bench/SWE-bench"><img src="docs/assets/swebench_logo_text_below.svg" alt="SWE-bench" height="120px"></a>

&nbsp;&nbsp;

<!-- <a href="https://github.com/SWE-agent/SWE-agent"><img src="docs/assets/sweagent_logo_text_below.svg" alt="SWE-agent" height="120px"></a> -->

<a href="https://github.com/SWE-bench/SWE-smith"><img src="docs/assets/swesmith_logo_text_below.svg" alt="SWE-smith" height="120px"></a>

&nbsp;&nbsp;

<a href="https://github.com/SWE-bench/sb-cli"><img src="docs/assets/sbcli_logo_text_below.svg" alt="sb-cli" height="120px"></a>

</div>

Contributions <a name="contributions"></a>

If you'd like to contribute to the codebase, we welcome issues and pull requests! For larger code changes, we always encourage discussion in issues first.

Citation & contact <a name="citation"></a>

SWE-agent is an academic project started at Princeton University by John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press.

Contact person: John Yang, Carlos E. Jimenez, and Kilian Lieret (Email: johnby@stanford.edu, carlosej@cs.princeton.edu, kl5675@princeton.edu).

If you found this work helpful, please consider citing it using the following:

<details>

<summary> SWE-agent citation</summary>

@inproceedings{yang2024sweagent,
  title={{SWE}-agent: Agent-Computer Interfaces Enable Automated Software Engineering},
  author={John Yang and Carlos E Jimenez and Alexander Wettig and Kilian Lieret and Shunyu Yao and Karthik R Narasimhan and Ofir Press},
  booktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},
  year={2024},
  url={https://arxiv.org/abs/2405.15793}
}

</details>

If you used the summarizer, interactive commands or the offensive cybersecurity capabilities in SWE-agent, please also consider citing:

<details>

<summary>EnIGMA citation</summary>

@misc{abramovich2024enigmaenhancedinteractivegenerative,
      title={EnIGMA: Enhanced Interactive Generative Model Agent for CTF Challenges},
      author={Talor Abramovich and Meet Udeshi and Minghao Shao and Kilian Lieret and Haoran Xi and Kimberly Milner and Sofija Jancheska and John Yang and Carlos E. Jimenez and Farshad Khorrami and Prashanth Krishnamurthy and Brendan Dolan-Gavitt and Muhammad Shafique and Karthik Narasimhan and Ramesh Karri and Ofir Press},
      year={2024},
      eprint={2409.16165},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2409.16165},
}

</details>

🪪 License <a name="license"></a>

MIT. Check LICENSE.

<div align="center">

Pytest

build-docs

codecov

pre-commit.ci status

Markdown links

</div>