NADA-Eval

NADA-Eval is a benchmark and evaluation suite for multilingual programming models.

It measures how well AI models understand, write, modify, explain and repair source code when programming is done in human languages other than English.

Most existing coding benchmarks translate the user's question into another language, but keep the code itself in English. NADA-Eval measures something different: whether a model can work correctly with programming languages whose keywords, standard libraries and common programming terminology have been localized.

The benchmark is built on the NADA registry of canonical programming terminology. This registry provides an objective reference for determining whether localized programming terminology is correct, consistent and up to date.

The benchmark is a collection of programming tasks.

Every model is evaluated using the same methodology so that results can be compared fairly.

Some parts of the benchmark are public and are intended for reproducible research and public leaderboards. Other parts remain private to reduce contamination from models training on benchmark questions.

The benchmark is updated over time as programming languages, libraries, frameworks and the NADA registry evolve.

An evaluation is the process of running a model against the benchmark.

The result is a report describing how the model performs across different programming languages, human languages, task categories and levels of difficulty.

Private evaluations can also focus on particular programming languages, human languages, frameworks or customer requirements while using the same underlying methodology.

The benchmark measures many different aspects of multilingual programming. Examples include:

NADA does not provide training data for localized programming.

Instead, it provides a deterministic registry and localization engine that AI systems can deploy locally or integrate directly into their products.

Rather than memorising programming terminology during training and attempting to reproduce it later, a model can consult the registry whenever authoritative terminology is required.

The benchmark measures whether a model understands when this deterministic infrastructure should be used, whether it uses it correctly, and whether it preserves canonical terminology throughout a conversation.

Modern AI systems increasingly rely on external tools rather than attempting to answer every question from their internal parameters alone.

NADA-Eval measures whether a model correctly uses the NADA registry and localization engine when appropriate. This includes questions such as:

NADA provides a downloadable agent skill — an offline package that lets any AI system consult the registry and localization engine locally, with no prior training on NADA required. Models can therefore be evaluated in two configurations: models that have been specifically fine-tuned for the skill or carry first-party, official system prompts that reference it, and models that are unaware of NADA but whose end users have added it as a custom skill.

When to call the tool is itself part of the benchmark. Existing benchmarks already measure general tool-calling ability — the Berkeley Function-Calling Leaderboard is the best known. NADA-Eval asks the same question of a deterministic terminology engine.

Some scenarios require several tool calls in sequence. A user may paste broken localized code: the model must first reason about the problem, then call the engine to canonicalize the code into English, do its repair work, and call the engine again to render the fixed code back into the target locale. In other cases the language itself is ambiguous — Indonesian and Malay share much of their vocabulary — and the model may need the skill's language detection and confidence scoring to identify the locale before any of this work can begin. The benchmark measures whether the model orchestrates these steps correctly: reasoning first, deterministic tooling at the boundaries, canonical terminology preserved throughout.

Real programming conversations often involve several human languages at the same time.

For example, a user may write in Tibetan, paste Python localized into Spanish, receive an English traceback, and request that the final answer be returned in Tibetan.

The benchmark measures whether the model correctly identifies the intended working language for the user, the intended language of the code, and the appropriate language for explanations.

It also measures whether the model unnecessarily switches languages or attempts to translate programming terminology that should remain canonical.

The benchmark distinguishes between canonical terminology, accepted historical terminology, obsolete terminology and invented terminology.

Models are evaluated on whether they consistently use the canonical terminology defined by the NADA registry, both inside source code and in normal written explanations.

This applies equally to programming keywords, standard library terminology, framework terminology and other programming concepts.

Where possible, answers are scored automatically. Many tasks have objectively correct answers.

Localization tasks are scored by comparing the model's output against the NADA registry of canonical programming terminology. Other tasks compare program behaviour, canonical representations or deterministic transformations.

Some evaluation categories include human review where deterministic scoring is not appropriate. These categories are reported separately from deterministic benchmark scores.

NADA-Eval is intended for organisations that build, deploy or study programming AI systems. This includes:

The public benchmark exists so that models can be compared using a common methodology.

Private evaluations use the same methodology but can include additional languages, programming languages, frameworks, libraries, customer-specific task sets and unpublished benchmark items.

Private evaluations may also include detailed reports describing strengths, weaknesses and opportunities for improvement.

The goal of NADA-Eval is to provide a repeatable and transparent way of measuring multilingual programming ability.

Rather than asking whether a model is generally "good at coding," the benchmark measures whether it can work correctly with localized programming languages, use canonical programming terminology, collaborate with deterministic localization infrastructure, and support programmers working in their own human language.

In development. The scoring engine is being consolidated into the core library, and the first reference result — frontier models scored on localized-code comprehension — is in preparation. The first public season follows.