cognify.

Applied R&D · AI · Data · Analytics

Skill Availability and Routing in Qwen 3.5 0.8B

← Back to logbook

Skill Availability and Routing in Qwen 3.5 0.8B

Date: 09/29/2026
Model: Qwen 3.5 0.8B
Configuration: local via Ollama · thinking=false
Benchmark: 100 queries · 5 routes
Reference: Router v0.1
Status: initial evaluation completed

Introduction

Cognify's first routing experiment started from a distinction between possessing a competence and recognizing the conditions under which that competence should be used.

A system may have a calculator, a search tool, a spell checker, or a function for exact counting. That these capabilities are available and work correctly does not, however, imply that the system adequately identifies when it should use each of them and when it should not.

We will provisionally call this second capacity availability of a competence: that a competence not only exists in the system's repertoire, but can be recognized as relevant and activated in the face of a situation that requires its use.

From this perspective, two problems can be distinguished:

POSSESSING / EXECUTING A COMPETENCE
                ↓
        knowing how to do something

                ≠

AVAILABILITY OF THE COMPETENCE
                ↓
 recognizing when it should be used

Router v0.1 isolated the second problem experimentally. Given a closed repertoire of five capabilities, the system had to recognize which was relevant given a particular query.

The next question was whether a small language model, Qwen 3.5 0.8B, could perform this selection on its own.

1. Experimental question

The question for this stage was:

Given an explicit set of available capabilities, with what precision does Qwen 3.5 0.8B recognize which one should be activated given a query?

We do not yet seek to evaluate whether Qwen can correctly solve the tasks.

For example, given:

"¿Cuántas veces aparece la m en programación?"

we do not evaluate whether Qwen can correctly count the letters. We evaluate a prior decision:

Which competence should be used?

Therefore, the experiment attempts to isolate competence selection from competence execution.

2. Scenario

The exact five routes used to evaluate Router v0.1 were maintained:

  • CALCULAR — Mathematical operations or problems requiring calculation.
  • CONTEO_LETRAS — Counting letters, characters, or letter occurrences.
  • ORTOGRAFIA — Writing revision and spelling error detection.
  • WIKIPEDIA — Retrieving external factual information about people, events, concepts, or places.
  • DIRECTO — Queries that can be answered directly without using the previous tools.

The problem presented to Qwen was:

query
   ↓
Qwen 3.5 0.8B
   ↓
CALCULAR
CONTEO_LETRAS
ORTOGRAFIA
WIKIPEDIA
DIRECTO

The model had to return only one of those five labels.

3. Experimental condition

Qwen 3.5 0.8B was run locally via Ollama.

The following was used:

thinking=false

This decision proved important.

In an initial test with thinking enabled, Qwen produced an extensive reasoning process before selecting CALCULAR. In that output the model correctly identified that "84 dividido 7" corresponded to a mathematical operation, but repeatedly reconsidered alternatives such as DIRECTO before delivering the final answer. Finally it correctly selected CALCULAR.

That configuration produced approximate latencies of 33–42 seconds per decision, so it was not adequate for the proposed benchmark.

With thinking=false, the latency observed during the 100 queries dropped to a mean of:

0.17 seconds per decision

This allowed evaluating immediate route selection without the extended reasoning process.

4. Dataset

The same external set previously used to evaluate Router v0.1 was used:

100 queries

distributed uniformly:

RouteCases
CALCULAR20
CONTEO_LETRAS20
ORTOGRAFIA20
WIKIPEDIA20
DIRECTO20
Total100

One inference was performed per query.

The prompt explicitly described the five capabilities and requested selecting only the corresponding label.

Results

5. General result

Qwen correctly classified:

46 of 100 queries — 46% accuracy.

Router v0.1's reference result on the same benchmark had been:

83 of 100 queries — 83% accuracy.

Therefore, under the conditions of this experiment:

SystemAccuracy
Router v0.183%
Qwen 3.5 0.8B46%

The observed difference is 37 percentage points in favor of the specialized router.

This result corresponds specifically to this dataset, these five routes, this prompt, and this Qwen configuration. It does not yet allow generalizing the difference to other models or routing problems.

6. Result by competence

The comparison by category shows more interesting behavior than the aggregate result.

RouteRouter v0.1Qwen 3.5 0.8B
CALCULAR55%90%
CONTEO_LETRAS100%20%
ORTOGRAFIA80%30%
WIKIPEDIA85%75%
DIRECTO95%15%

Qwen does not simply show inferior performance across all categories.

In CALCULAR, Qwen widely outperforms the specialized router.

Router v0.1 had shown difficulties with mathematically formulated semantic expressions, such as:

"Hallá la mitad de 946."
"Repartí 360 en 9 partes iguales."

Qwen correctly recognized many of these cases as CALCULAR.

This is consistent with a fundamental difference between both systems: Router v0.1 uses lexical patterns via TF-IDF, while the language model can respond to more diverse linguistic formulations.

7. Unequal availability of competences

In other categories the opposite phenomenon appears.

Qwen only correctly identified:

  • 4 of 20 CONTEO_LETRAS queries;
  • 6 of 20 ORTOGRAFIA queries;
  • 3 of 20 DIRECTO queries.

Some examples are particularly illustrative:

"¿Cuántas veces aparece la m en programación?"
→ CALCULAR

when the expected route was:

CONTEO_LETRAS

Also:

"¿Cuántas n hay en internacionalización?"
→ WIKIPEDIA

and:

"¿Cuál es la diferencia conceptual entre eficiencia y eficacia?"
→ ORTOGRAFIA

These results suggest that the availability of a competence is not homogeneous within the model.

The model can easily recognize certain classes of situations and, at the same time, present difficulties activating other capabilities that are also explicitly part of its available repertoire.

8. Competence and availability

The result allows formulating with greater precision the hypothesis that gave rise to the experiment.

Suppose Qwen is capable of correctly performing a given task when explicitly requested:

Contá cuántas "m" hay en "programación".

That would demonstrate an execution capacity.

But if within an agent it has a specific skill for exact counting and, faced with that same situation, selects CALCULAR, a different problem exists.

The competence may be technically available, but not functionally available in the face of the situation that should activate it.

We can represent the distinction as:

CAPABILITY
Can it do it?

AVAILABILITY
Does it recognize when it should do it?

EXECUTION
Does it do it correctly?

A reliable agent requires resolving all three levels.

This experiment fundamentally evaluates the second.

9. A complementary result

The comparison between Qwen and Router v0.1 further shows that both systems present different strengths.

Router v0.1 is particularly strong in:

  • CONTEO_LETRAS
  • ORTOGRAFIA
  • DIRECTO

Qwen is particularly strong in:

  • CALCULAR

This opens an additional hypothesis.

Specialization and general capability might not be alternative strategies, but complementary ones.

A system could use specialized components when the boundaries between competences are known and stable, and resort to more general semantic capabilities when the formulation of the task demands greater linguistic generalization.

This possibility has not yet been evaluated.

10. Invalid outputs

Qwen produced 16 outputs considered invalid by the experimental protocol.

An output was considered invalid when it could not be normalized exactly to one of the five allowed labels.

Therefore, the observed 46% jointly measures:

  1. correct semantic route selection;
  2. adherence to the output contract.

Before interpreting the 16 outputs as cognitive routing errors, it will be necessary to inspect their content.

There may be cases where Qwen conceptually identified the correct competence but produced a label or formulation incompatible with the parser.

For this reason, the current result should be recorded as:

Operational accuracy of the Qwen-router system: 46%.

Not necessarily as a definitive estimate of its semantic classification capacity.

Provisional result

11. Finding

Under a closed repertoire of five capabilities and using the same external benchmark, an extremely simple specialized router obtained 83% accuracy, while Qwen 3.5 0.8B used directly as a routing mechanism obtained 46% with thinking=false.

However, the per-category result shows that Qwen has a clear advantage over the lexical router in certain semantic formulations, particularly in calculation.

The result supports the usefulness of studying competence selection as a differentiated capacity within AI systems.

It is not enough to ask:

What does a model know how to do?

It is also necessary to ask:

Does it recognize when it should put into play each of the things it knows how to do?