# Bale Metro File Agent — Technical Documentation

**What this is:** an n8n workflow that takes a natural-language chat request, figures out which file the user wants, and pulls it out of a *public* Bale bot's conversation history — something the official Bale Bot API cannot do, because a bot token can never see or act inside a chat it doesn't own.

**Status:** working proof of concept. Hardcoded, single-target, built to demonstrate a mechanism — not a production service. Every hardcoded value and known limitation is called out explicitly in this doc so nothing is overstated.

---

## 1. The problem this solves

The original brief: *"you can build this for your own channel, where your bot is admin — but what about a public bot or channel your bot can't be admin of?"*

The Bale **Bot API** (`getUpdates`, `sendMessage`, `getFile`, etc. — the same shape as Telegram's Bot API) only works within chats the bot account is a member/admin of, and only sees activity from the moment it joined onward. There is no bot-token mechanism to:

- open an arbitrary public bot's chat,
- press its buttons,
- or read its message history.

That's a hard protocol boundary, not a missing feature to configure around. Bots generally cannot message other bots or act as a user inside someone else's conversation.

## 2. The solution: client auth instead of bot auth

[`aiobale`](https://github.com/) is an unofficial, community-maintained Python library that implements Bale's actual client protocol — the same websocket/RPC protocol the official Bale app and web client use — rather than the smaller public Bot API. Its `Client` class logs in as a **real phone-number account** (SMS verification code, exactly like opening Bale on a new device), not a bot token.

Once authenticated, that account can do anything a human tapping around the app could: open any public bot's chat, press its buttons, read full history, download files — because from Bale's perspective it *is* a person using the app, not a bot.

This is the core trade-off to be upfront about: **it isn't the "optimal" or intended integration pattern** (it's a real user session, not a first-class bot integration), but it is the only mechanism that can act inside a chat the operator doesn't control.

## 3. Two approaches considered

### Route 1 — press the button, wait for the reply
Send the exact button-label text (e.g. `نقشه خطوط`) into the bot's chat and wait for its next message. Simple and reliable, but it can only ever fetch **one hardcoded thing** — there's no way to decide *which* button to press from a natural-language request without a separate keyword→button mapping table.

Implemented in `fetch_metro_file.py` (kept in the repo as the minimal working example — this is what proved the mechanism worked at all, before the "decide what to fetch" layer was added).

### Route 2 — search history instead of sending a new command *(the one actually used)*
Since the client account is authenticated as a real user, it can call `client.iter_messages()` to read the **entire history** of that bot chat — every file it has ever sent, across every button ever pressed, each with its original caption. Instead of guessing which button maps to a request, the script searches what's already there for a filename/caption/text match.

This is the meaningfully better answer to "can it decide": it doesn't need a hardcoded button map, and it generalizes to anything the bot has ever sent, not just one fixed target.

Implemented in `fetch_metro_smart.py` — this is the script the live n8n workflow calls.

**Honest limitation:** Route 2 can only find files the bot has *already sent at least once*. If nothing matching the keyword has ever appeared in that chat, the search correctly returns "no match" — it does not fall back to pressing a button to fetch something new. Combining both routes (search first, press a button as fallback) is a natural next step, not yet built.

## 4. Architecture

```
n8n (Docker, Linux host)                          Bale servers
┌─────────────────────────────┐
│ When chat message received  │  user types a request in the n8n chat panel
└──────────────┬──────────────┘
               ▼
┌─────────────────────────────┐
│ Parse request (LLM)         │  HTTP POST to BazaarLink's chat-completions
│                              │  endpoint. Forces a single-word JSON reply,
│                              │  and explicitly instructs the model to
│                              │  translate the keyword into Persian, since
│                              │  the bot's captions are Persian regardless
│                              │  of what language the user typed in.
└──────────────┬──────────────┘
               ▼
┌─────────────────────────────┐
│ Extract keyword              │  Code node: strips markdown fences, parses
│                              │  the model's JSON reply into {keyword}.
└──────────────┬──────────────┘
               ▼
┌─────────────────────────────┐
│ Run client-auth fetch       │──────▶  fetch_metro_smart.py (Python, runs
│ (Execute Command node)      │         INSIDE the n8n container)
└──────────────┬──────────────┘              │
               │                              │ aiobale Client, logged in as
               │                              │ a real Bale account (session
               │                              │ reused from an earlier SMS
               │                              │ login — no re-auth per run)
               │                              ▼
               │                     iter_messages() over the metro bot's
               │                     chat history → match filename/caption/
               │                     text against the keyword → download
               │                     any match to /mnt/storage/bale-downloads
               ▼
┌─────────────────────────────┐
│ Parse results                │  Code node: reads the script's stdout
│                              │  ({"files": [...], "matched": bool}) back
│                              │  into a normal n8n item.
└──────────────┬──────────────┘
               ▼
┌─────────────────────────────┐
│ Save to disk                 │  Read/Write File node in "read" mode.
│ (visual only)                │  The file is ALREADY on disk (the Python
│                              │  script wrote it) — this node just reads
│                              │  it back so n8n's UI shows the binary
│                              │  preview + Download button.
└──────────────────────────────┘
```

**Key point for explaining this out loud:** n8n itself never touches Bale's protocol or the file bytes in transit. It shells out to a Python process that does all of that, and only reads back a small JSON string (a list of file paths) over stdout. The **Save to disk** node's job is purely cosmetic — showing the file nicely in n8n's UI — not actually saving anything.

## 5. Why n8n + Python, not n8n alone

n8n's Code nodes run JavaScript, and there is no JavaScript library for Bale's client protocol. `aiobale` is Python-only, and its `Client` requires a persistent async connection with a completed SMS login — not something that fits inside a stateless per-execution node. So the design bridges the two: n8n orchestrates the natural-language layer (LLM parsing, decision routing, presenting output), and a standalone Python script does the one thing only it can do (speak to Bale as a logged-in user).

## 6. Infrastructure notes (the parts that broke, and why)

These are worth knowing even though they're now resolved — they're the kind of thing an interviewer might ask about directly.

### n8n's Execute Command node is disabled by default (v2+)
Newer n8n images ship with `NODES_EXCLUDE` blocking `Execute Command` for security reasons. Fixed by setting `NODES_EXCLUDE: "[]"` in the container's environment.

### n8n's v2 image dropped `apk`
The official `n8nio/n8n` image is Alpine-based but no longer ships the `apk` package manager itself, so `RUN apk add python3` fails with `apk: not found`. Worked around by copying the `apk` binary and its support files in from a plain `alpine:latest` image in a multi-stage `Dockerfile`, then installing Python 3 and `aiobale-py` into a venv on top of the official n8n image.

### `aiobale` history-scanning bug on large verified bots
`iter_messages()` on the metro bot's chat (596K subscribers) threw a `pydantic.ValidationError`: certain historical messages carry a dict payload (almost certainly a view/seen-tracking structure) in the exact field `aiobale` expects to be plain text (`TextMessage.value: str`). This only affects **history scans**, not live replies — Route 1 never hit it because it never calls `load_history`.

**Fix applied** (top of `fetch_metro_smart.py`): loosen that one field's type at runtime and force pydantic to rebuild its validation schema:

```python
import aiobale.types.message_content as _mc
_mc.TextMessage.model_fields["value"].annotation = Any
_mc.TextMessage.model_rebuild(force=True)
```

Downstream code then defensively checks `isinstance(msg_text, str)` before using it, since the field can now legitimately hold a dict instead of text.

**Scope of the fix, stated honestly:** this patches the *specific* field/shape that actually broke, based on a real traceback — it is not a guarantee that every possible message shape in Bale's protocol is now safe. A different historical message with a different unexpected shape could still throw a fresh, different validation error. This is a known category of risk with unofficial, reverse-engineered protocol libraries.

## 7. File inventory

| File | Purpose |
|---|---|
| `fetch_metro_file.py` | Route 1 (button press + wait) — the first working proof that client auth could talk to a public bot at all. |
| `fetch_metro_smart.py` | Route 2 (history search) — what the live n8n workflow actually calls. Includes the `aiobale` monkeypatch and result deduplication. |
| `fetch_by_description.py` | Earlier variant built for scanning *your own* channel's history (not the metro bot) — kept for reference, not part of the current demo. |
| `bale-metro-file-agent-clientauth.json` | The n8n workflow: chat trigger → LLM keyword parse → Execute Command → parse results → visual "Save to disk". |
| `Dockerfile` | Adds Python 3 + `aiobale-py` on top of the official `n8nio/n8n` image, working around the missing-`apk` issue. |
| `docker-compose.yaml` | Original compose file with two changes: `image:` → `build: .`, and `NODES_EXCLUDE: "[]"` added. |
| `metro_session.bale` | The authenticated session file (JWT + auth metadata) from the one-time phone/SMS login. Lives at `/mnt/storage/bale-downloads/` so it survives container rebuilds via the volume mount. |

## 8. Hardcoded values (everything that would need to change for a different target)

This was built deliberately hardcoded end-to-end for demo purposes — no credential management, no multi-target config. To point this at a different bot/channel, these are the values to change:

- `TARGET_CHAT_ID` (in `fetch_metro_smart.py`) — the numeric `uid` from that chat's `web.bale.ai/chat?uid=...` URL.
- `TARGET_CHAT_TYPE` — `ChatType.BOT` for a bot chat, `ChatType.CHANNEL` for a channel, `ChatType.PRIVATE` for a DM.
- `PHONE_NUMBER` — only used as a fallback if the session file is ever missing/expired and a fresh interactive login is triggered.
- The BazaarLink API key in the workflow JSON's **Parse request (LLM)** node.

## 9. Reproduction steps (from a clean checkout)

1. Place `Dockerfile` next to your `docker-compose.yaml`.
2. In `docker-compose.yaml`: change `image: n8nio/n8n:latest` to `build: .`, and add `NODES_EXCLUDE: "[]"` under the `n8n` service's `environment:`.
3. `docker compose up -d --build`
4. One-time only, on a machine with a terminal/TTY (the container itself has none by default for interactive input): run the login flow to produce `metro_session.bale`, then copy that file into `/mnt/storage/bale-downloads/` (the path the scripts and the volume mount both expect).
5. Copy `fetch_metro_smart.py` into `/mnt/storage/bale-downloads/`.
6. Sanity-check in isolation, before involving n8n at all:
   ```bash
   docker exec -it n8n python3 /mnt/storage/bale-downloads/fetch_metro_smart.py خطوط
   ```
   Expect `{"files": ["...tehran-subway1.jpg"], "matched": true}`.
7. Import `bale-metro-file-agent-clientauth.json` into n8n (**Workflows → Import from File**).
8. Open the workflow's chat panel and type a request in any language, e.g. *"get me the metro map."*

## 10. What this honestly proves vs. doesn't

**Proven:**
- Client auth (not bot auth) can access a chat/bot the operator doesn't control.
- A natural-language request can be turned into a targeted search over that chat's real history, in a different language than the underlying content.
- The mechanism survives real infrastructure friction (container security defaults, missing OS package manager, an upstream library bug) — all diagnosed and fixed with evidence (tracebacks), not guesswork.

**Not proven / explicitly out of scope for this demo:**
- Generalization to *any* bot/channel without manually changing the hardcoded target — no dynamic bot discovery.
- Combining Route 1 and Route 2 (falling back to pressing a button when history search comes up empty).
- Production concerns: no credential vaulting, no per-chat request locking for concurrent use, no retry/backoff, no handling for further undiscovered `aiobale` parsing edge cases beyond the one hit here.
- The account used for client auth is a real personal Bale account — rate limits, terms-of-service exposure, and account-level risk apply in a way they wouldn't for an official bot integration. Worth stating plainly if asked: this trades "it works against public chats" for "it's not the sanctioned integration path."