On this page
← papers

Reading Centaur (ICSE)

reviewed · 8 min read · paper-review · testing · neurosymbolic

venue
ICSE
status
reviewed
tl;dr
An LLM proposes rules for DL API inputs, and an SMT solver turns them into valid inputs.
source
www.cs.cornell.edu/~saikatd/papers/centaur-icse2 ↗

1. What is the takeaway message#

the central bet of the paper is that LLMs have absorbed enough DL API usage from the data it has been trained on, that you can extract it into constraints that, when given to an LLM, forces it to take a form that z3 can reason over, with the help of the bridge that is grammar. Grammar plays an important part as it changes the natural language of the LLm into first order logic formulas to be used by the SMT to generate valid outputs. The argument for efficiency is also made effectively as there is a need to pay the LLM cost once and you run it once per API, which is better than prior tools like Pathfinder that keeps on recomputing for every input

One of the main things, that caught my eye was the number: "near 100% of generated inputs are validated", compared to the much lower 43-64% of prior tools. This implies that Centaur is fundamentally reaching deeper than prior tools.

However, one thing that is worth noting and important is that the neurosymbolic label, makes it sound like the neural and symbolic parts are tightly integrated and mutually informing but in practice, it works in a slightly different manner wherein the LLM proposes candidates, and hands it off entirely to a classical SMT. So, it feels that it's more accurate to say the LLm is a smarter enumerator than a random grammar generator because the symbolic part does the actual reasoning.

2. Problem and gap#

Whenever we are building large scale or AI applications, they get constantly corrupted due to the complexity and bugginess of these DL libraries. As said in the paper, DocTer fails in expressing relational constraints, ACETest fails due to path explosion which is evident from Figure 2 for torch.add, and Pathfinder which also has the same issue given by concolic testing. From all these its evident, why they have achieved a max of 64% validity ratio, which I felt could have been better shown as inadequate by discussing the kind of bugs these found, that Centaur couldn't.

The real gap from what i could understand was the real gap wasn't documentation vs code analysis but rather the inability of all predecessor tools to express the relational constrains between the parameters, also the path explosion which they didn't consider.

3. How Centaur works#

I found the following unique points for ending these shortcomings: Grammar, as given in Fig 5a, FOL over tensor properties like ndim, dtype_ and so on, universal quantifiers, conditionals etc that accept multiple types;

  1. Rule generation that uses LLM to feed grammar with docs and error messages to generate the candidate rules which are then validated by centaur and sends back feed back failures. The error messages collected by running tells the LLm where it goes wrong It generates 93.37 rules/API on average in a minute

  2. Constraint learining, where a constraint is a rule with a specific parameter assignment, with error collection and refinement, done by removing each constraint at a time and seeing ig the ratio drops and discarding it ie, a heuristic and not exhaustive manner. 117 valid inputs per API

  3. Fuzzing is applied to translate constraints to Z3 formulas and also solve All the results for these are the biggest evidence for the efficiency of this system. Two diversity tacts are also used here which includes blocking, which is not adding constraints for previously seen values and also bucketing which partitions value space into ranges and also helps sample uniformly.

These are then serialized to JSON corpus and concrete inputs sampled from them at fuzz time.

One more thing that the paper, highly undersells is the practical strength of offline/online separation, which runs down the cost to $0.30. Like the only thing/phase that happens online is the rule generation. Whereas the learning and abstract generation are offline Along with that, the feedback loop with LLM seeing the failure types aalso helps.

4. Evidence and results#

RQ1 — Constraint quality:

15 PyTorch + 15 TF APIs; authors manually wrote ground truth constraints Recall: 88.6% PyTorch, 100.0% TF; Precision: 98.8% PyTorch, 89.8% TF 4 missed constraints: either pruned because they didn't affect validity ratio on seeds, or grammar couldn't express them (string literals) Individually: 100% recall for 27/30 APIs, 100% precision for 22/30 APIs

RQ2 — Validity ratio:

PyTorch: Centaur 97.87% vs TitanFuzz 43.21%, ACETest 47.62%, Pathfinder 64.16% TF: Centaur 99.24% vs TitanFuzz 21.38%, ACETest 57.85%, Pathfinder 29.85% Generates 100x–1000x more valid inputs than baselines

RQ2 — Coverage:

+203 branches vs TitanFuzz, +150 vs ACETest, +9608 vs Pathfinder (Pytorch averages) Statistical significance vs Pathfinder (p≈0); NOT statistically significant vs ACETest and TitanFuzz on coverage TitanFuzz catches up on TF coverage because it implicitly knows hardware-specific memory format rules (e.g., oneDNN int8 NCHW 16-byte alignment) — not expressible in Centaur's grammar

RQ3 — Bugs:

26 total: 13 PyTorch, 13 TF; 18 confirmed, 4 rejected, 4 pending, 1 fixed Categories: 7 crashes, 15 inconsistent (CPU vs GPU), 3 overflow, 1 NaN Example crash: torch.native_channel_shuffle with groups > second dimension size — FPE, valid-looking input per docs Example differential: tf.nn.local_response_normalization with negative beta — wrong output on CPU, cuDNN error on GPU; PyTorch handles it correctly

Ablation:

All 3 components (DocErr, Feedback, ExRules) individually reduce # APIs successfully processed and coverage/validity when removed Removing all 3 together is worst case

Generalizability:

Gemini 2.0 Flash best overall; GPT-5 and Claude Sonnet 4.5 close; Qwen3-Coder-30B weakest (only 39/88 APIs for PyTorch) All except Qwen3 achieve >96% validity ratio

[I know it seems all too number related but that was my understanding of the question]

5. Critique and limitations#

///It might seem much too negative,but the ground truth is that, for RQ1, was written by the same research team that designed the grammar, who may have some unconsious alignment, and it would need some form of third party validation.///(remove in case its not too good, it seems like it to me, this is unrelated)

The sample size for RQ1, is small, ie 15/512 PyTorch APIs which around a 3% sample which makes it hard to generalize for precision/recall claims to all 512.

Also, the coverage gap doesnt match the valid input gap. While centaur generates 500x more valid inputs than baselines but only covers 150-200 more branches shows that the marginal coverage for each valid input seems to be not worth

The refinement is circular, inputs got during refinement are generated by the current constraint set, removing any one constraint may change the inputs that get generate, so that becomes a point where you are testing on shifting distribution, not a fixed one.

*One more thing, is that seed coverage determines constraint quality. Like say if a valid input never appears in seeds, wrong constraints excluding it survive unchalleneged. This is confirmed with int8 (rule_17 for abs excludes int8 even though abs works on int8) and optional parameters (rule_97 for neg's out shape was correct but pruned because seeds never used out)

*Optimal parameter pruning problem where for torch.neg, rule_97 ("out must have same shape as input") is correct and was generated, but was pruned because seed inputs never used the out parameter, removing it didn't change validity ratio, since the generator wasn't producing out anyway

*Grammar expressiveness limit: for torch.svd, out is a tuple of 3 tensors (U,S,V), grammar has tuple(tensor) type, for this, Rule 6 correctly says "out should be a tuple of 3 tensors" generates an integer instead, every corpus generation attempt fails with TypeError. The expressiveness of the specification layer is ahead of this generation layer

*The 44/47 invariant retention rate for torch.matmul vs 1/31 for torch.neg shows the method works best for APIs with many genuine relational constraints and worst for simple unary APIs — the paper doesn't discuss this variance

Other observations: The int8 false constraint and the optional parameter pruning problem both have the same simple fix, diversifying seeds. Includign one call per dtype and one with each optional parameter seems to look like a best one from my side.

6. What the paper achieves#

Achieved by paper: First neurosymbolic API-level fuzzer for DL libraries Novel grammar for first-order constraints over tensor properties Constraint learning algorithm with validity-ratio-based refinement Evaluation against 3 SoTA baselines on 512 PyTorch + 522 TF APIs 26 bugs found, 18 confirmed

Observations:

The grammar is definetly the most technically original and impressive contribution, union types, qynatifiers, conditionals and relational multi parameter rules, getting this right enables everything else.

The concrete bugs(especially in the case of crash and differential examples) are the most convincing evidence that the approach finds things others miss

THe online, offline seperation is an underrated practical contribution, since it makes that expensive LLM work doesn't happen during fuzzing

7. Future directions#

The paper says that the future can extend grammar to include more constraint types, apply to domains beyond DL libraries where input specifications are sparse, and also handle conditional constraints on optional arguments.

Seed diversity problem is unfixed and fixable, include seeds for every dtype, every optimal parameter, 0-dim tensor which is cheap and has high impact

Differential oracle needs automation: per input expected FP error bounds would reduce manual review burder

Cross API constraint transfer can be applied across similar operations like those of unary operations(abs, neg, sqrt, exp) all share the same out parameter, learning once and propagating would increase pool and reduce LLM cost

Instead of binary keep/discard, score each constraint by magnitude of validity ratio drop, and distinguish load bearing constraints from marginal ones, could also surface which constraints are most likely false

Centaur generates far more valid inputs than baselines but barely gets more coverage, understanding why, like maybe it might be because are uncovered branches gated on specific numeric values? on control flow not reachable by structural constraints?. Understanding these would directly inform what constraint types to add next

If a fuzz input satisfying all constraints still crashes, there's a missing constrait. Using crash input as negative examples to trigger another rule search round

8. Open questions#

Centaur generates 100x more valid inputs but only ~150-200 more braches. What is the actual bottleneck?

The refinement validity ratio is computed on input from the current constraint set, if the measuremenet is confounded by the constraint set itself, how do you know you're measuringg constraint necessity rather than just correlation within the seed sample?

one rejected bug was "expected FP behavior" with a 1.07e+09 difference, what is the actual threshold for real inconsistency vs expected FP variance?

The grammar uses integer dtype codes (0=bool, 7=float32, etc.), if the LLM generates a wrong code the rule parses fine but is wrong; how many surviving constraints contain silent numeric errors?

*Based on observations from my code, could have been wrong due to some issue from my side