PyTorch swaps a broken, unlicensed test-picking model for a smaller open one

What changed in PyTorch's checks

As github.com reported in a PyTorch commit note, the project has moved the language-model part of its test-selection system onto new code. That system is called target determination, or TD. For each pull request, it decides which tests are most worth running so that problems surface quickly without running everything.

The change swaps the model behind one of TD's signals. The old setup used CodeLlama-7B. The new one uses Qwen3-Embedding-0.6B, an embedding model, which turns text and code into lists of numbers so that related items land close together. That lets the system rank test files by how relevant they look to a change.

The note labels the risk as medium. The commit appears on both the trunk and viable/strict branches, so it is one change listed twice.

The old step was doing nothing

The note's explanation for the swap is blunt: the previous LLM heuristic had delivered nothing since mid-2026. The retriever did its work and wrote its results, but the next step, packing them into a zip file, failed because the zip tool was not installed in the container the job ran in.

A setting called continue-on-error kept the job showing green anyway. The upload was skipped, and TD simply logged that the expected results artifact could not be found. Nothing looked broken, yet the LLM contributed no signal.

There was a second problem. The code came from two personal repositories, one of them checked out without pinning a specific version, and it carried no license. The new retriever lives in pytorch/test-infra instead.

Smaller model, better ranking

The note gives a head-to-head test on the same events. For 44 breakages on trunk, the main development branch, the new model put the breaking test file in the top 25% of its ranking in 37 to 41 cases. The old setup managed 17, and random guessing would manage 11.

It is also much lighter. The new model has 0.6B parameters, against 7B for the old one. Its index ships as a 1.2 GB bundle, against a 13 GB checkpoint, and it needs no GPU. The index build uses a 16-vCPU CPU runner, where the old job needed an A10G GPU runner.

The index covers 3,634 units drawn from 1,350 test files, and building it takes about 13 minutes, including the model download. The indexer is set to run every two hours and publishes one bundle to a storage location that already serves files anonymously.

In testing, the retrieval jobs downloaded and verified the bundle in 11-12 s and scored 338 test files in 16-21 s. Each job took about 3 minutes, against 5.5 minutes at the median for the old one.

How much it changes real outcomes

The embedding score is not in charge. TD adds it at the weight its existing LLM code already used, 0.25. The authors replayed that against TD's logged rankings using a harness from pytorch/test-infra#9333 and reported three results:

• About neutral on pull-request failures in jobs where every test runs, such as macOS jobs, where TD is off.
• 2-3 more of 37 reverted trunk breakages caught, with none lost.
• No growth in the estimated test time of the selected set.

So the gain is modest in practice. The note also calls this a step toward feeding the embedding score into a learned scorer, a later stage of a wider plan tracked in pytorch/test-infra#9321.

Safety, rollout and the off switch

The design splits the work by trust. The build job has no credentials and reports a checksum of what it built. A separate upload job holds the only credentials, on a 15-minute session. It checks the file against that checksum before publishing, and it only archives the previous bundle if its metadata shows this workflow wrote it.

If fetching the bundle or running retrieval fails, the job stays green, prints a warning and TD runs without the LLM part. Turning the feature off is one line: set the upload-results condition to false.

Timing matters for the merge. The old indexer still runs daily around 00:25 UTC and can move the new bundle into an archive folder. If that happens, the fetch fails softly and the next two-hourly run publishes it again. Older release branches also download the extra 1.2 GB, about 4 s more per run.

After the merge, the authors plan to check that the next PR's TD job logs a non-empty set of LLM results, and to watch shadow recall and CI time saved on the HUD Target Determinator page.