Updated on 15 September 2026.
An answer about an unpaid invoice is useful only if the amount, customer and payment status belong together. A source citation makes that relationship easier to inspect, but does not prove it. My aim with Souprise is deliberately limited: query business records locally, copy values from the records and perform calculations in code. Its published development reports record 60 correct lookups and 40 correct aggregations in their respective tests. Those results come from specified scenarios using synthetic data and do not establish correctness for every business query. Lookup evaluation, aggregation evaluation
Separating data retrieval from language generation
Souprise is an open-source project for structured records such as invoices, orders and customer accounts. In its default verified mode, rules map a question to a record and field. The answer copies a value from the selected record without using a language model. A separate component handles recognised aggregation requests with decimal arithmetic. Its supported fields and filters define which questions that route can answer. Project description, calculation code
The intended behaviour is to expose conflicting values and refuse requests for unknown entities or inadequate matches. Whether that happens depends on how the system recognises the question and selects records. If no specific field is recognised, the output can contain the matching record instead. That gives the user something concrete to inspect. It does not establish that the original question has been answered. Verified-mode design and limits
Local processing is a central design choice. Verified mode needs no language model. Optional language modes can use a model already stored locally, while the quick-start path otherwise downloads one first. Custom extensions require their own review of data flows. Running locally does not, by itself, establish that a deployment is secure. Deployment options
What the published tests establish
BENCH-5 reports that all 60 lookup answers matched the expected values on a deduplicated corpus. It also records refusals for 20 unknown entities after a correction to entity recognition. The first implementation failed that test because it returned the closest existing match. The report therefore documents a particular tested state. It also shows why similarity alone is insufficient for these queries. BENCH-5
BENCH-6 reports agreement with an independent reference calculation in 40 aggregation cases. The same report evaluates styled mode on 80 questions. In that mode, a local model phrases values already supplied by the deterministic component, and a numeric check is intended to catch changes. At temperature 0, the reported run produced no flagged mismatch, so fallback to the deterministic answer was never triggered. That run does not establish that the check would catch every incorrect model response. BENCH-6
Additional model training did not demonstrate a benefit in the published comparison. Across 60 questions over 2,000 synthetic records, the untuned model scored 0.733 and the tuned model 0.717. The report does not treat that small difference at this sample size as reliable evidence that training made performance worse. For this task, the experiment did not justify the additional training effort. Fine-tuning comparison
Limits of the numeric check
The implementation places a concrete limit on claims of comprehensive verification. In the source revision examined here, _extract_numbers includes numbers of at least 100 and numbers containing a decimal point. It does not capture positive integers below 100 or negative integers. The subsequent comparison checks whether captured numbers occur in the comparison texts. It does not prove that an amount belongs to the right customer or that the surrounding sentence is correct. Describing this as a check of every number would therefore be inaccurate. Numeric-check implementation
The source records impose another limit. A stale payment status remains stale when copied faithfully from an index. A total also requires a clear account of the records included and any missing values. Citations, update rules and inspectable calculations help reviewers find those problems. They do not replace responsibility for the underlying data.
Conditions for deployment
The development reports include tests of access policies and tenant separation. BENCH-9 reports no detected unauthorised output across 200 queries under one specified policy. BENCH-10 records the same outcome for 200 queries across two separate tenant datasets, each containing 1,000 synthetic records. Those findings apply to the tested scenarios. They do not replace checks of user authentication, operating-system permissions, changed datasets or custom extensions. Access-policy evaluation, tenant evaluation
The reports identify these limits themselves. Access policies are objects inside the running process, while associating an authenticated user with the right policy is a separate responsibility. Separate files under one operating-system account do not provide complete isolation from one another. A local pilot therefore needs a defined data scope and a named operational owner. Passing development tests does not establish readiness for every enterprise environment.
The business case
The potential benefit is faster access to records with a traceable connection between an answer and its source. Whether this saves time depends on the work needed to prepare, update and check the data. A local installation still requires operation, maintenance and subject-matter expertise. A useful pilot compares one bounded activity with the existing process and counts corrections and review effort as well as initial response time.
Souprise provides source code, test reports and an approach that can be inspected. The published results support a focused trial where the task fits the supported data and queries. The deciding question is whether people can reliably check and use the answers in their own process.