Skill Routing
Date: 09/29/2026 Version: Router v0.1 Type: exploratory experiment / baseline
Introduction
Artificial intelligence agents can have multiple specialized skills, tools, or functions available to solve different types of tasks. An agent can, for example, perform calculations using a deterministic function, count characters exactly, query an external source, check spelling, or respond directly using a language model.
The availability of these capabilities introduces a problem prior to their execution: routing. Before using a tool, the system must determine what type of request it has received and decide which capability should intervene to resolve it.
In this experiment, we specifically investigate this function:
Given a known set of skills, how can a system learn to route a query to the appropriate capability?
We start from a simplified scenario in which an agent has five possible routes. The problem initially consists of learning a function:
query → route
The work is approached incrementally. First, we seek to isolate routing as an independent capability, build a simple baseline, and measure to what extent it can be learned from labeled examples.
Based on this baseline, it will be possible to later compare different routing mechanisms: semantic representations using embeddings, supervised classifiers, specialized models, dynamic tool selection, or mechanisms based on language models.
In a later stage, we are also interested in investigating whether an externally learned routing capability can be transferred or embedded into a model, so that skill selection becomes part of its learned behavior and does not depend exclusively on instructions included in the prompt.
Router v0.1 constitutes the first baseline of this experimental line. Its goal is not to definitively solve the routing problem, but to establish a measurable benchmark against which to evaluate subsequent architectures.
1. Experimental Scenario
We assume an agent that has a known set of five capabilities:
- CALCULATE — Performing mathematical operations using deterministic tools.
- COUNT_LETTERS — Exact counting of letters, characters, or other properties of text strings.
- SPELLING — Spelling check using a specialized tool.
- WIKIPEDIA — Search and retrieval of factual information from Wikipedia.
- DIRECT — Resolving the query directly using the language model when none of the previous tools are necessary.
The router does not execute any of these capabilities nor does it attempt to answer the user's question. Its only function is:
┌───────────────┐
Query ─────────────→│ ROUTER │
└───────┬───────┘
│
┌───────────┬───────┼──────────┬──────────┐
↓ ↓ ↓ ↓ ↓
CALCULATE COUNT SPELLING WIKIPEDIA DIRECT
This allows for studying routing independently of the subsequent performance of each skill.
2. Motivation
In a tool-based system, there are at least two different problems:
- selecting which capability to use;
- correctly executing that capability.
When both functions are delegated to the same language model, it is difficult to determine the cause of an error. An incorrect response may be because the model selected the wrong tool, did not use any tool when it should have, used the tool correctly but misinterpreted its result, or simply failed to generate the response.
Isolating routing allows one of these decisions to be turned into an independent, observable, and quantifiable problem.
For example:
"How many r's are in railroad?"
↓
COUNT_LETTERS
or:
"Who was Marie Curie?"
↓
WIKIPEDIA
The first experimental objective, therefore, consists of determining to what extent this decision can be learned in a specialized manner.
3. Research Question
The initial question of Experiment 001 is:
Can a specialized component learn to route queries to a known set of skills from labeled examples?
From this question, others subsequently arise:
- What types of queries are easiest or hardest to route?
- What language representation is sufficient to perform routing?
- What information does the router's uncertainty provide?
- When should a router abstain from deciding?
- Can it generalize to unobserved linguistic formulations?
- Can it select new tools from their descriptions?
- Can the externally learned capability be subsequently transferred to a model?
- Does an agent's performance improve when the routing decision is decoupled from other capabilities?
These questions correspond to subsequent experiments. Router v0.1 addresses only the first one.
4. Design
To establish a baseline, a simple architecture was deliberately chosen.
Router v0.1 uses:
Text
↓
TF-IDF
↓
Vector representation
↓
Comparison with class centroids
↓
Cosine similarity
↓
Selected route
It does not use:
- semantic embeddings;
- neural networks;
- Transformers;
- language models;
- fine-tuning;
- text generation.
The final implementation was done in pure Python, without external dependencies.
This simplicity is deliberate. The purpose of v0.1 is to establish a performance floor before introducing more sophisticated mechanisms.
5. Representation using TF-IDF
Each query is converted into a numerical representation using TF-IDF (Term Frequency – Inverse Document Frequency).
TF-IDF assigns higher weight to terms that are discriminative within the corpus and lower weight to those that appear frequently in multiple documents.
Router v0.1 uses:
- unigrams: individual words;
- bigrams: sequences of two words.
For example, a query like:
"calculate the square root of 81"
can contain signals like:
calculate
square
root
square_root
While:
"who was Marie Curie"
contains different patterns, among them:
who
was
who_was
Marie
Curie
The system does not understand these concepts in a semantic sense. It learns statistical associations between linguistic patterns and routes.
6. Classification
For each of the five routes, a centroid is built from the TF-IDF vectors of its training examples.
Given a new query:
- the query is transformed into a TF-IDF vector;
- its cosine similarity with the five centroids is calculated;
- the routes are ranked according to similarity;
- the route with the highest score is selected.
Formally:
query
↓
TF-IDF vector
↓
similarity with CALCULATE
similarity with COUNT_LETTERS
similarity with SPELLING
similarity with WIKIPEDIA
similarity with DIRECT
↓
argmax
↓
route
The values obtained are similarities, not calibrated probabilities.
7. Training Dataset
A dataset was manually built of:
100 labeled queries
distributed uniformly:
| Route | Examples |
|---|---|
| CALCULATE | 20 |
| COUNT_LETTERS | 20 |
| SPELLING | 20 |
| WIKIPEDIA | 20 |
| DIRECT | 20 |
| Total | 100 |
The examples include different formulations within each intent.
Some cases were also incorporated to avoid overly superficial associations. For example:
"How many letters does Marie Curie have?"
→ COUNT_LETTERS
versus:
"How many years did Marie Curie live?"
→ WIKIPEDIA
In this way, the presence of an entity like Marie Curie should not be enough on its own to determine the route.
8. Internal Validation
A stratified five-fold cross-validation was performed.
In each iteration, approximately 80% of the examples were used to build the router and the remaining 20% to evaluate it.
The result was:
Mean accuracy: 87.0% ± 8.1 percentage points.
This result was considered only a preliminary estimate, as the 100 examples belonged to the same dataset used during development.
For this reason, it was decided to subsequently perform a completely external evaluation.
9. Evaluation Dataset
Once Router v0.1 was built, a second independent dataset was generated:
100 new queries
with the same distribution:
| Route | Examples |
|---|---|
| CALCULATE | 20 |
| COUNT_LETTERS | 20 |
| SPELLING | 20 |
| WIKIPEDIA | 20 |
| DIRECT | 20 |
| Total | 100 |
These queries did not participate in training.
Router v0.1 remained unchanged during this evaluation.
10. Result
The router correctly classified:
83 out of 100 queries.
Therefore:
External accuracy: 83%.
The performance by route was:
| Route | Correct | Total | Accuracy |
|---|---|---|---|
| COUNT_LETTERS | 20 | 20 | 100% |
| DIRECT | 19 | 20 | 95% |
| WIKIPEDIA | 17 | 20 | 85% |
| SPELLING | 16 | 20 | 80% |
| CALCULATE | 11 | 20 | 55% |
| Total | 83 | 100 | 83% |
11. First finding: heterogeneous performance
The aggregate result of 83% hides important differences between routes.
COUNT_LETTERS presents perfect discrimination in this evaluation set, while CALCULATE reaches only 55%.
Therefore, the difficulty of routing depends strongly on the type of intent one is trying to identify.
This suggests that treating routing only as a global metric can hide relevant system behaviors.
12. Second finding: lexical limitation in CALCULATE
Nine of the seventeen total errors correspond to queries whose correct route was CALCULATE.
The errors appear especially when the mathematical operation is expressed indirectly through natural language.
For example, formulations conceptually equivalent to:
"find half of..."
"distribute in equal parts..."
"I have N boxes with M units..."
"what percentage do they represent..."
may not contain the same lexical patterns as the training examples.
This is consistent with the characteristics of TF-IDF: the system identifies mainly lexical regularities, but does not possess a deep representation of the query's meaning.
This result establishes a clear hypothesis for future versions: a semantic representation could especially improve categories whose intent can be expressed using very different vocabularies.
13. Third finding: decision margin
Router v0.1 also preserves the similarity obtained by the first and second routes.
We provisionally define:
margin = score(route 1) - score(route 2)
During evaluation, errors were observed with extremely small margins.
This suggests that the margin could function as an approximate signal of the router's uncertainty.
A subsequent architecture could use that information to implement abstention:
Query
↓
Router
↓
margin sufficient?
/ \
yes no
↓ ↓
skill escalation
Instead of requiring the router to always decide, the system could recognize situations in which the classification is ambiguous.
This hypothesis was not evaluated in v0.1.
14. Limitations
Router v0.1 presents several deliberate limitations.
Closed set of skills. The five routes are known beforehand. The experiment does not demonstrate the ability to select new tools or generalize to a dynamic catalog of skills.
Small dataset. The training contains only 100 examples. The objective was not to maximize performance but to build an initial baseline.
Manually built dataset. The queries were generated specifically for the experiment and do not yet represent real user traffic.
Predominantly lexical representation. TF-IDF does not directly represent semantic relationships between expressions.
No multi-skill queries. The experiment initially assumes a single correct route per query. A request like:
"Find when Marie Curie was born and calculate how old she was in 1903"
potentially requires several capabilities and falls outside the scope of v0.1.
No abstention class. The router always selects a route, even when the available evidence is weak.
15. Freezing
Following the external evaluation, Router v0.1 is frozen.
The errors observed in router_test.csv will not be incorporated into the training.
Neither will the following be retrospectively modified:
- the examples;
- the representation;
- the classification algorithm;
- the centroids;
- the decision mechanism.
Any modification will give rise to a new version.
This allows v0.1 to be preserved as a baseline for future comparisons.
16. Artifacts
The experiment produced the following artifacts:
router_train.csv
train_router_pure.py
router_v01_pure.pkl
router_test.csv
test_router.py
router_test_results.csv
- router_train.csv — contains the 100 examples used to build the router.
router_v01_pure.pklcontains the resulting model.- router_test.csv — contains the external evaluation set.
- router_test_results.csv — preserves the individual predictions, expected routes, selected routes, and margins.
These artifacts allow for the experiment to be reproduced and audited.
Upcoming Questions
Router v0.1 establishes that even an extremely simple mechanism can learn a considerable part of routing on a closed set of capabilities: 83% of the external queries were directed correctly.
That result does not yet determine which architecture is appropriate. It serves as a starting point for a broader experimental line.
The following stages can study, separately:
- Semantic representation. Compare TF-IDF against embeddings while keeping the classification problem constant.
- Specialized classifiers. Determine if a model trained specifically for routing outperforms the baseline.
- Abstention. Investigate if the margin between routes allows for the detection of unreliable decisions and their escalation to another mechanism.
- Dynamic routing. Move from classes known during training to a catalog of skills described in natural language.
- Skill composition. Allow a query to produce a sequence of tools instead of a single route.
- Integration with agents. Compare an agent that performs tool selection internally against one that receives external routing.
- Transfer to the model. Investigate if examples produced and evaluated through external routing can be subsequently used to train or tune a model that incorporates that capability.
The general question opened by the experiment is, therefore:
How should an artificial intelligence system learn to decide which capability to use for a given task?
Router v0.1 constitutes the first experimental baseline for studying that question.