Remove modelRouter and add model_providers concept by philipph-askui · Pull Request #224 · askui/vision-agent

philipph-askui · 2026-01-21T07:20:44Z

[Edited] PR Description

To get a quick idea of the new API, I added an example under examples/model_providers.py that you can run
Note: You will need an anthropic API Key for that

This PR:

Removes ModelRouter, ModelRegistry, and model_store
Introduces a provider-based configuration system (AgentSettings)
Renames VisionAgent → ComputerAgent and AndroidVisionAgent → AndroidAgent
Updates docs and examples

Summary

Replaced the ModelRouter/model_store abstraction with three typed provider slots (vlm_provider, image_qa_provider, detection_provider) configured via AgentSettings
Providers own their endpoint, credentials, and model ID — validated lazily on first API call
get() and locate() are now backed by GetTool/LocateTool, which are also available to the LLM during act() — no separate model injection path
Significantly reduced codebase complexity (~4600 lines removed)
Updated all docs and examples to reflect the new API

Key Changes

AgentSettings: single configuration object with provider slots; defaults to AskUI-hosted providers reading credentials from env vars
Built-in providers: AskUIVlmProvider, AskUIImageQAProvider, AskUIDetectionProvider, AnthropicVlmProvider, AnthropicImageQAProvider, GoogleImageQAProvider, OpenAICompatibleProvider
GetTool / LocateTool: wired into the act loop as ToolWithAgentOS — LLM can call them directly during act()
Deleted: entire src/askui/model_store/ directory
Renamed: VisionAgent → ComputerAgent, AndroidVisionAgent → AndroidAgent
Docs: 03_Using-Models-and-BYOM.md fully rewritten; VisionAgent replaced with ComputerAgent across all docs

Further Changes

removes chat-api related code as it was deprecated
removes UI-TARS related code as it was deprecated

Breaking Changes
- VisionAgent is removed — use ComputerAgent
- AndroidVisionAgent is removed — use AndroidAgent
- act_model, get_model, locate_model constructor parameters are removed — use AgentSettings(vlm_provider=..., image_qa_provider=..., detection_provider=...)
- model_store factory functions are removed
- String-based model selection is removed

…del store BREAKING CHANGE: Removed ModelRouter and ModelRegistry classes. Users must now use direct model injection.

…modelrouter

…raises an Error when executing from cache)

…move() methods

…nd AskUiInferenceLocateApi

programminx-askui

Hello @philipph-askui ,

I started to review the code. General remarks:

Changes are to big to review. -> We need to test this heavly
Creating new files, instead of rename/move files -> we don't know which code was already reviewed, which code is new.

Overall it is going in the right direction.

I've reviewed only view files, so you can start working on it. A deeper review is outstanding.

Co-authored-by: Dominik Klotz <105296959+programminx-askui@users.noreply.github.com>

…eview comments

… users

…ve clear separation between Model, Client, messagesAPI and provider

…ing` group

philipph-askui · 2026-02-16T07:57:21Z

I added a new optional dependency group "tracing", that contains all the OTEL packages.

src/askui/models/openai/messages_api.py

programminx-askui · 2026-02-16T07:32:17Z

src/askui/models/shared/tools.py

+        validate_by_alias=True,
+    )
+
+    base_name: str = Field(alias="name", description="Name of the tool")


Does this renaming has influence of any seralized data? e.g. Caching?

no, should still work as the tool name provided by to the model should not change

programminx-askui · 2026-02-16T15:47:17Z

src/askui/model_providers/askui_vlm_provider.py

-        self._model_id_value = model_id
+        self._askui_settings = askui_settings or AskUiInferenceApiSettings()
+        self._model_id = model_id
+        self._injected_client = client


programminx-askui · 2026-02-16T15:50:08Z

src/askui/models/google/get_model.py

+
+logger = logging.getLogger(__name__)
+
+_MAX_FILE_SIZE_BYTES = 20 * 1024 * 1024


Where is the limit coming from? Can we link to the API docs from google in a comment? Otherwise I wouldn't check the file size.

I think I found it here: https://ai.google.dev/gemini-api/docs/file-input-methods

According to a quick research, the limit was increased to 50MB recently. Hence, I updated the limit in the code.

programminx-askui · 2026-02-16T15:51:25Z

src/askui/models/google/get_model.py

+            if len(data) > _MAX_FILE_SIZE_BYTES:
+                err_msg = (
+                    f"PDF file size exceeds the limit of {_MAX_FILE_SIZE_BYTES} bytes."
+                )
+                raise ValueError(err_msg)


Duplicated code. please wrap this an a function check_file_size()

the error messages are different, hence all we could wrap in a function here is the if comparison

src/askui/models/openai/act_model.py

programminx-askui · 2026-02-16T15:56:37Z

src/askui/models/openai/messages_api.py

+from askui.models.shared.tools import ToolCollection
+
+
+def _to_openai_messages(messages: list[MessageParam]) -> list[dict[str, Any]]:  # noqa: C901


Should I review this?

I would expect here an integration tests. Which coveres all message types.

programminx-askui · 2026-02-16T16:13:22Z

src/askui/models/openai/messages_api.py

+"""OpenAICompatibleMessagesApi —
+MessagesApi implementation for OpenAI-compatible endpoints."""
+
+import json as json_lib
+from typing import Any
+
+from openai import OpenAI
+from typing_extensions import override
+
+from askui.models.shared.agent_message_param import (
+    Base64ImageSourceParam,
+    ContentBlockParam,
+    ImageBlockParam,
+    MessageParam,
+    TextBlockParam,
+    ThinkingConfigParam,
+    ToolChoiceParam,
+    ToolResultBlockParam,


Please delete OpenAI Act integration. -> Let's move this to another issue.

src/askui/model_providers/anthropic_vlm_provider.py

src/askui/model_providers/askui_image_qa_provider.py

examples/model_providers.py

…tic `provider_options`

philipph-askui added 3 commits January 21, 2026 06:57

chore: add plan and claude rules

b99056d

refactor: simplify model selection with direct model injection and mo…

7a23ca9

…del store BREAKING CHANGE: Removed ModelRouter and ModelRegistry classes. Users must now use direct model injection.

resolve merge conflict

421d043

philipph-askui changed the title ~~Chore/modelrouter~~ Remove Modelrouter Jan 21, 2026

philipph-askui and others added 12 commits January 21, 2026 08:23

Merge branch 'main' into chore/modelrouter

188ddd2

feat: remove chat related dependencies and components

5c3ade4

Merge remote-tracking branch 'origin/feat/deprecate-chat' into chore/…

dfcda56

…modelrouter

chore: remove dependency on anthropic types

00d359c

fix(agentOS): set default value of name for Display to None (else it …

339717f

…raises an Error when executing from cache)

chore: fix typecheck errors

1d5929b

fix linting errors

021c90b

fix linting errors

2598a46

feat: add model_store

f1460a9

fix linting errors

d8ce5b2

Merge branch 'main' into chore/modelrouter

d4ea0d8

fix linting errors

a0a75b1

philipph-askui changed the title ~~Remove Modelrouter~~ Remove modelRouter and add mode_store Jan 22, 2026

philipph-askui added 11 commits January 22, 2026 15:16

remove UITARS

93ea9ee

remove ModelComposition

b2c79be

update get model interface

f4c3cb2

remove UITars, modelcomposition, and model definition

6222282

Add locate_settings and locate_model parameters to click() and mouse_…

fca6234

…move() methods

Remove AskUiBaseLocateModel class and create LocateApi (Base class) a…

e3dc24a

…nd AskUiInferenceLocateApi

chore: remove GenericActModel

4ac0df4

move HF models and OpenRouter models to model_store

25a3ad8

chore: fix docs

19c8968

fix: make models serializable for telemetry

6217b11

chore: update docs

8f3a9ea

philipph-askui marked this pull request as ready for review January 26, 2026 06:53

philipph-askui requested a review from mlikasam-askui January 26, 2026 06:54

programminx-askui requested changes Feb 12, 2026

View reviewed changes

philipph-askui and others added 15 commits February 13, 2026 06:47

Update docs/00_Overview.md according to Dominiks remarks

e4fcab8

Co-authored-by: Dominik Klotz <105296959+programminx-askui@users.noreply.github.com>

Update docs/00_Overview.md according to Dominiks remarks

6e04f47

Co-authored-by: Dominik Klotz <105296959+programminx-askui@users.noreply.github.com>

Update docs/00_Overview.md according to Dominiks remarks

87f65c1

Co-authored-by: Dominik Klotz <105296959+programminx-askui@users.noreply.github.com>

Update docs/00_Overview.md according to Dominiks remarks

a5d1e32

Co-authored-by: Dominik Klotz <105296959+programminx-askui@users.noreply.github.com>

rename AIElement_locate_model to ai_element_locate_model to address r…

c59e4e9

…eview comments

address review comments

fd911ca

aded resolution for locate to LocateSettings so that it can be set by…

043353d

… users

run format

f22426f

update docs to address review remarks

08df970

add instructions to not generate examples that set env variables in-code

a72b3d0

refactor: model providers according to our coding standards We now ha…

e8e8c59

…ve clear separation between Model, Client, messagesAPI and provider

Merge remote-tracking branch 'origin/main' into chore/modelrouter

fb43999

fix: typecheck errors

f2cea46

fix: add otel + instrumentors as optional dependencies in a new `trac…

0a15539

…ing` group

fix: add tracing dependency group to all

aa1b839

programminx-askui requested changes Feb 16, 2026

View reviewed changes

programminx-askui reviewed Feb 16, 2026

View reviewed changes

examples/model_providers.py Outdated Show resolved Hide resolved

philipph-askui added 10 commits February 17, 2026 07:38

address review comments of dominik

a8a2649

remove openai integration

210b266

fix: liniting issue

afa8bf9

fix: typos

c23753c

feat: change provider init to lazy init

5e934a2

fix: add google api key to template

7732b15

add inline documentation for settings parameters

01646ba

fix: removes obsolete anthropic beta flags from settings

c4d2f48

fix: replace anthropic-specific betas parameter with provider-agnos…

4b17319

…tic `provider_options`

fix: update docs to explain new provider_options param

b82b61f


		logger = logging.getLogger(__name__)

		_MAX_FILE_SIZE_BYTES = 20 * 1024 * 1024

		from askui.models.shared.tools import ToolCollection


		def _to_openai_messages(messages: list[MessageParam]) -> list[dict[str, Any]]: # noqa: C901

Conversation

philipph-askui commented Jan 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

programminx-askui left a comment

Choose a reason for hiding this comment

Uh oh!

philipph-askui commented Feb 16, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

philipph-askui commented Jan 21, 2026 •

edited

Loading