agents/n8n-bale-publicchat-file-fetch/BALE-METRO-FILE-AGENT-DOCS.md

Bale Metro File Agent — Technical Documentation

What this is: an n8n workflow that takes a natural-language chat request, figures out which file the user wants, and pulls it out of a public Bale bot's conversation history — something the official Bale Bot API cannot do, because a bot token can never see or act inside a chat it doesn't own.

Status: working proof of concept. Hardcoded, single-target, built to demonstrate a mechanism — not a production service. Every hardcoded value and known limitation is called out explicitly in this doc so nothing is overstated.

---

1. The problem this solves

The original brief: "you can build this for your own channel, where your bot is admin — but what about a public bot or channel your bot can't be admin of?"

The Bale Bot API (getUpdates, sendMessage, getFile, etc. — the same shape as Telegram's Bot API) only works within chats the bot account is a member/admin of, and only sees activity from the moment it joined onward. There is no bot-token mechanism to:

• open an arbitrary public bot's chat,
• press its buttons,
• or read its message history.

That's a hard protocol boundary, not a missing feature to configure around. Bots generally cannot message other bots or act as a user inside someone else's conversation.

2. The solution: client auth instead of bot auth

aiobale is an unofficial, community-maintained Python library that implements Bale's actual client protocol — the same websocket/RPC protocol the official Bale app and web client use — rather than the smaller public Bot API. Its Client class logs in as a real phone-number account (SMS verification code, exactly like opening Bale on a new device), not a bot token.

Once authenticated, that account can do anything a human tapping around the app could: open any public bot's chat, press its buttons, read full history, download files — because from Bale's perspective it is a person using the app, not a bot.

This is the core trade-off to be upfront about: it isn't the "optimal" or intended integration pattern (it's a real user session, not a first-class bot integration), but it is the only mechanism that can act inside a chat the operator doesn't control.

3. Two approaches considered

Route 1 — press the button, wait for the reply

Send the exact button-label text (e.g. نقشه خطوط) into the bot's chat and wait for its next message. Simple and reliable, but it can only ever fetch one hardcoded thing — there's no way to decide which button to press from a natural-language request without a separate keyword→button mapping table.

Implemented in fetch_metro_file.py (kept in the repo as the minimal working example — this is what proved the mechanism worked at all, before the "decide what to fetch" layer was added).

Route 2 — search history instead of sending a new command (the one actually used)

Since the client account is authenticated as a real user, it can call client.iter_messages() to read the entire history of that bot chat — every file it has ever sent, across every button ever pressed, each with its original caption. Instead of guessing which button maps to a request, the script searches what's already there for a filename/caption/text match.

This is the meaningfully better answer to "can it decide": it doesn't need a hardcoded button map, and it generalizes to anything the bot has ever sent, not just one fixed target.

Implemented in fetch_metro_smart.py — this is the script the live n8n workflow calls.

Honest limitation: Route 2 can only find files the bot has already sent at least once. If nothing matching the keyword has ever appeared in that chat, the search correctly returns "no match" — it does not fall back to pressing a button to fetch something new. Combining both routes (search first, press a button as fallback) is a natural next step, not yet built.

4. Architecture

n8n (Docker, Linux host)                          Bale servers
┌─────────────────────────────┐
│ When chat message received  │  user types a request in the n8n chat panel
└──────────────┬──────────────┘
               ▼
┌─────────────────────────────┐
│ Parse request (LLM)         │  HTTP POST to BazaarLink's chat-completions
│                              │  endpoint. Forces a single-word JSON reply,
│                              │  and explicitly instructs the model to
│                              │  translate the keyword into Persian, since
│                              │  the bot's captions are Persian regardless
│                              │  of what language the user typed in.
└──────────────┬──────────────┘
               ▼
┌─────────────────────────────┐
│ Extract keyword              │  Code node: strips markdown fences, parses
│                              │  the model's JSON reply into {keyword}.
└──────────────┬──────────────┘
               ▼
┌─────────────────────────────┐
│ Run client-auth fetch       │──────▶  fetch_metro_smart.py (Python, runs
│ (Execute Command node)      │         INSIDE the n8n container)
└──────────────┬──────────────┘              │
               │                              │ aiobale Client, logged in as
               │                              │ a real Bale account (session
               │                              │ reused from an earlier SMS
               │                              │ login — no re-auth per run)
               │                              ▼
               │                     iter_messages() over the metro bot's
               │                     chat history → match filename/caption/
               │                     text against the keyword → download
               │                     any match to /mnt/storage/bale-downloads
               ▼
┌─────────────────────────────┐
│ Parse results                │  Code node: reads the script's stdout
│                              │  ({"files": [...], "matched": bool}) back
│                              │  into a normal n8n item.
└──────────────┬──────────────┘
               ▼
┌─────────────────────────────┐
│ Save to disk                 │  Read/Write File node in "read" mode.
│ (visual only)                │  The file is ALREADY on disk (the Python
│                              │  script wrote it) — this node just reads
│                              │  it back so n8n's UI shows the binary
│                              │  preview + Download button.
└──────────────────────────────┘

Key point for explaining this out loud: n8n itself never touches Bale's protocol or the file bytes in transit. It shells out to a Python process that does all of that, and only reads back a small JSON string (a list of file paths) over stdout. The Save to disk node's job is purely cosmetic — showing the file nicely in n8n's UI — not actually saving anything.

5. Why n8n + Python, not n8n alone

n8n's Code nodes run JavaScript, and there is no JavaScript library for Bale's client protocol. aiobale is Python-only, and its Client requires a persistent async connection with a completed SMS login — not something that fits inside a stateless per-execution node. So the design bridges the two: n8n orchestrates the natural-language layer (LLM parsing, decision routing, presenting output), and a standalone Python script does the one thing only it can do (speak to Bale as a logged-in user).

6. Infrastructure notes (the parts that broke, and why)

These are worth knowing even though they're now resolved — they're the kind of thing an interviewer might ask about directly.

n8n's Execute Command node is disabled by default (v2+)

Newer n8n images ship with NODES_EXCLUDE blocking Execute Command for security reasons. Fixed by setting NODES_EXCLUDE: "[]" in the container's environment.

n8n's v2 image dropped apk

The official n8nio/n8n image is Alpine-based but no longer ships the apk package manager itself, so RUN apk add python3 fails with apk: not found. Worked around by copying the apk binary and its support files in from a plain alpine:latest image in a multi-stage Dockerfile, then installing Python 3 and aiobale-py into a venv on top of the official n8n image.

aiobale history-scanning bug on large verified bots

iter_messages() on the metro bot's chat (596K subscribers) threw a pydantic.ValidationError: certain historical messages carry a dict payload (almost certainly a view/seen-tracking structure) in the exact field aiobale expects to be plain text (TextMessage.value: str). This only affects history scans, not live replies — Route 1 never hit it because it never calls load_history.

Fix applied (top of fetch_metro_smart.py): loosen that one field's type at runtime and force pydantic to rebuild its validation schema:

import aiobale.types.message_content as _mc
_mc.TextMessage.model_fields["value"].annotation = Any
_mc.TextMessage.model_rebuild(force=True)

Downstream code then defensively checks isinstance(msg_text, str) before using it, since the field can now legitimately hold a dict instead of text.

Scope of the fix, stated honestly: this patches the specific field/shape that actually broke, based on a real traceback — it is not a guarantee that every possible message shape in Bale's protocol is now safe. A different historical message with a different unexpected shape could still throw a fresh, different validation error. This is a known category of risk with unofficial, reverse-engineered protocol libraries.

7. File inventory

File	Purpose
fetch_metro_file.py	Route 1 (button press + wait) — the first working proof that client auth could talk to a public bot at all.
fetch_metro_smart.py	Route 2 (history search) — what the live n8n workflow actually calls. Includes the aiobale monkeypatch and result deduplication.
fetch_by_description.py	Earlier variant built for scanning your own channel's history (not the metro bot) — kept for reference, not part of the current demo.
bale-metro-file-agent-clientauth.json	The n8n workflow: chat trigger → LLM keyword parse → Execute Command → parse results → visual "Save to disk".
Dockerfile	Adds Python 3 + aiobale-py on top of the official n8nio/n8n image, working around the missing-apk issue.
docker-compose.yaml	Original compose file with two changes: image: → build: ., and NODES_EXCLUDE: "[]" added.
metro_session.bale	The authenticated session file (JWT + auth metadata) from the one-time phone/SMS login. Lives at /mnt/storage/bale-downloads/ so it survives container rebuilds via the volume mount.

8. Hardcoded values (everything that would need to change for a different target)

This was built deliberately hardcoded end-to-end for demo purposes — no credential management, no multi-target config. To point this at a different bot/channel, these are the values to change:

• TARGET_CHAT_ID (in fetch_metro_smart.py) — the numeric uid from that chat's web.bale.ai/chat?uid=... URL.
• TARGET_CHAT_TYPE — ChatType.BOT for a bot chat, ChatType.CHANNEL for a channel, ChatType.PRIVATE for a DM.
• PHONE_NUMBER — only used as a fallback if the session file is ever missing/expired and a fresh interactive login is triggered.
• The BazaarLink API key in the workflow JSON's Parse request (LLM) node.

9. Reproduction steps (from a clean checkout)

1. Place Dockerfile next to your docker-compose.yaml.
2. In docker-compose.yaml: change image: n8nio/n8n:latest to build: ., and add NODES_EXCLUDE: "[]" under the n8n service's environment:.
3. docker compose up -d --build
4. One-time only, on a machine with a terminal/TTY (the container itself has none by default for interactive input): run the login flow to produce metro_session.bale, then copy that file into /mnt/storage/bale-downloads/ (the path the scripts and the volume mount both expect).
5. Copy fetch_metro_smart.py into /mnt/storage/bale-downloads/.
6. Sanity-check in isolation, before involving n8n at all:

   docker exec -it n8n python3 /mnt/storage/bale-downloads/fetch_metro_smart.py خطوط
   

   Expect {"files": ["...tehran-subway1.jpg"], "matched": true}.
7. Import bale-metro-file-agent-clientauth.json into n8n (Workflows → Import from File).
8. Open the workflow's chat panel and type a request in any language, e.g. "get me the metro map."

10. What this honestly proves vs. doesn't

Proven:

• Client auth (not bot auth) can access a chat/bot the operator doesn't control.
• A natural-language request can be turned into a targeted search over that chat's real history, in a different language than the underlying content.
• The mechanism survives real infrastructure friction (container security defaults, missing OS package manager, an upstream library bug) — all diagnosed and fixed with evidence (tracebacks), not guesswork.

Not proven / explicitly out of scope for this demo:

• Generalization to any bot/channel without manually changing the hardcoded target — no dynamic bot discovery.
• Combining Route 1 and Route 2 (falling back to pressing a button when history search comes up empty).
• Production concerns: no credential vaulting, no per-chat request locking for concurrent use, no retry/backoff, no handling for further undiscovered aiobale parsing edge cases beyond the one hit here.
• The account used for client auth is a real personal Bale account — rate limits, terms-of-service exposure, and account-level risk apply in a way they wouldn't for an official bot integration. Worth stating plainly if asked: this trades "it works against public chats" for "it's not the sanctioned integration path."