1. Marzban VPN Infrastructure (Spain + USA)

What it is: Two production Marzban panels (Spain + USA) serving VLESS Reality and Shadowsocks via Xray-core. Hardened with iptables, fronted by nginx for public subscriptions.

Why VLESS is on port 443

Tested with Atom mobile data: Shadowsocks :1080 connected, but VLESS :2053, VMess :2080 and Trojan :2081 all timed out. The carrier filters uncommon ports. Port 443 (HTTPS) is never filtered, so VLESS Reality moved there:

# On the USA panel host — backup first, always
cp /var/lib/marzban/xray_config.json /var/lib/marzban/xray_config.json.bak2053

# In the Marzban panel: inbounds -> VLESS TCP REALITY -> port 2053 -> 443
# Then restart the container:
cd /opt/marzban && docker compose restart marzban

# Verify xray is actually listening on 443
ss -tlnp | grep ':443'        # expect: xray
# Existing customers must refresh their subscription to get the new port.

Firewall — panel locked to one IP

The Marzban dashboard (:8000) is only reachable from the Spain server. Everything else is dropped at the kernel level, before Marzban ever sees it:

# Run ON the USA Marzban host
iptables -A INPUT -i lo -j ACCEPT                                   # localhost always
iptables -A INPUT -p tcp --dport 8000 -s  -j ACCEPT      # Spain only
iptables -A INPUT -p tcp --dport 8000 -j DROP                       # everyone else
iptables-save > /etc/iptables/rules.v4                              # persist reboot

nginx — public subscriptions without VPN

Problem: customers buy VPN but needed VPN just to fetch their subscription link (both live on :8000). Fix: nginx on :8443 proxies only /sub/* to Marzban; /, /dashboard, /api all return 404. Note Marzban's :8000 is HTTPS with a self-signed cert, so the proxy disables upstream verification:

# /etc/nginx/sites-available/mm-sub-proxy
server {{
    listen 8443 ssl;
    server_name mm.kxant.eu.cc;
    ssl_certificate /etc/letsencrypt/live/mm.kxant.eu.cc/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/mm.kxant.eu.cc/privkey.pem;

    location /sub/ {{
        proxy_pass https://127.0.0.1:8000;   # Marzban is HTTPS (self-signed)
        proxy_ssl_verify off;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }}
    location / {{ return 404; }}              # dashboard stays Spain-only
}}

# Point new subscription URLs at the open port.
# /opt/marzban/.env:
XRAY_SUBSCRIPTION_URL_PREFIX="https://mm.kxant.eu.cc:8443"

# IMPORTANT: `docker compose restart` does NOT re-read .env.
# Recreate the container:
cd /opt/marzban && docker compose up -d
docker exec marzban-marzban-1 env | grep XRAY_SUBSCRIPTION_URL_PREFIX

Silence the bot-login Telegram spam

Marzban notifies the linked Telegram ID on every admin login — and the bot logs in constantly. Remove the link from the bot admin only; keep it on the human admin:

python3 -c "
import sqlite3
c = sqlite3.connect('/var/lib/marzban/db.sqlite3')
c.execute(\"UPDATE admins SET telegram_id = NULL WHERE username = 'vpnbot'\")
c.commit()
print(list(c.execute('SELECT username, telegram_id FROM admins')))
"
# -> [('kbko', 8500073574), ('vpnbot', None)]
# Bot logins: silent. Your manual logins: still alert. 

Hard lessons: Always cp xray_config.json{,.bak} before inbound edits. docker compose restart ignores .env changes — use up -d. Test ports from the user's real network (Atom), not from the server — server-side checks lie about carrier filtering.

How Spain drives the USA panel — one bot, two panels

Spain is the brain, USA is the worker. The bot on Spain holds two Marzban API clients — one per panel — and every command picks the right one from the server key stored on the user row:

# /root/vpnbot/bot.py — two panels, one brain
mz = Marzban(config.MARZBAN_URL, config.MARZBAN_USERNAME, config.MARZBAN_PASSWORD)
mz_us = Marzban(config.US_MARZBAN_URL, config.US_MARZBAN_USERNAME,
                config.US_MARZBAN_PASSWORD)

def mz_for(server):
    """The Marzban client for a Marzban-backed server key."""
    if server == "us_marzban":
        return mz_us
    return mz

# ...inside any handler:
info = await mz_for(server).get_user(mzu)   # `server` comes from the buy flow

The buy flow stores server on the user/order row, so renew, freeze, delete and referral credit all hit the correct panel automatically. The USA panel firewall only allows Spain's IP — the API travels over the internet, but nobody else can reach it.

Two traffics, fully separated — the link can't slow your VPN

A natural worry: does the Spain↔USA hop add lag to customer connections? No — there are two completely separate traffics:

MANAGEMENT (Spain <-> USA)              DATA (customer <-> server)
─────────────────────────              ─────────────────────────
bot -> USA panel API:                   customer -> USA :443 / :1080
  create / renew / delete user            DIRECT — Spain never in path
  a few KB, only on action
hourly watcher: 1 API login            subscription fetch:
                                           customer -> mm.kxant.eu.cc:8443/sub/…
                                           DIRECT to USA nginx

Honest trade-off: if Spain goes down, existing USA customers keep working (their traffic is direct) — but new buys and renewals pause until Spain is back, because the bot lives there. That's why Spain gets a 15-minute freeze auto-recovery plus an hourly health watcher.

Hard lesson: rename the admin → update every saved login

Renamed the Spain panel admin admin → kbko, verified the new login worked — then every bot API call started returning 401 Unauthorized. The panel was fine; the saved logins weren't: /root/vpnbot/.env and /root/resellerbot/.env still said MARZBAN_USERNAME=admin, and the admin user no longer existed:

# Back up both credential files first
cp /root/vpnbot/.env /root/vpnbot/.env.bak-20261018
cp /root/resellerbot/.env /root/resellerbot/.env.bak-20261018

# Update the saved login in both files
# (password redacted here — put the real new panel password)
sed -i "s/^MARZBAN_USERNAME=.*/MARZBAN_USERNAME=kbko/" \
    /root/vpnbot/.env /root/resellerbot/.env
# + set MARZBAN_PASSWORD in both files, chmod 600

systemctl restart vpnbot resellerbot

# End-to-end proof: login → create → delete a test user
curl -s -X POST http://127.0.0.1:8000/api/admin/token \
  -d "username=kbko&password=…"   # → 200 + access_token

Why the watcher healed itself: the hourly health check never caches credentials — it reads them live from /root/vpnbot/.env over SSH on every run. So the first run after the fix went green and sent the recovery notice on its own. Lesson: one credential change means every saved copy — and a watcher that reads live instead of caching is worth it.

2. VidDrop — Telegram Video Downloader

What it is: Paste any video link → get quality buttons + MP3 option → the file arrives in chat. Powered by yt-dlp, capped at 2 concurrent downloads so the VPS never chokes.

Format listing → inline buttons

import yt_dlp, asyncio

def list_formats(url: str):
    with yt_dlp.YoutubeDL({{"quiet": True}}) as ydl:
        info = ydl.extract_info(url, download=False)
    # pick one stream per height, skip absurd sizes
    seen, out = set(), []
    for f in info.get("formats", []):
        h = f.get("height")
        if h and h not in seen and h <= 1080:
            seen.add(h)
            out.append((h, f["format_id"]))
    return info.get("title", "video"), sorted(out)

async def on_link(update, context):
    title, fmts = await asyncio.to_thread(list_formats, update.message.text)
    kb = [[InlineKeyboardButton(f"🎬 {{h}}p", callback_data=f"dl_{{fid}}_{{h}}")]
          for h, fid in fmts]
    kb.append([InlineKeyboardButton("🎵 MP3", callback_data="dl_mp3")])
    context.user_data["dl_url"] = update.message.text
    await update.message.reply_text(f"{{title}}\nQuality ရွေးပါ:",
        reply_markup=InlineKeyboardMarkup(kb))

Download with a semaphore (max 2 at once)

SEMAPHORE = asyncio.Semaphore(2)

async def on_quality(update, context):
    q = update.callback_query
    await q.answer()
    url = context.user_data["dl_url"]
    async with SEMAPHORE:                       # never more than 2 downloads
        if q.data == "dl_mp3":
            opts = {{"format": "bestaudio/best",
                     "postprocessors": [{{"key": "FFmpegExtractAudio",
                                          "preferredcodec": "mp3"}}]}}
        else:
            _, fid, h = q.data.split("_")
            opts = {{"format": fid, "max_filesize": 50 * 1024 * 1024}}
        opts.update({{"outtmpl": "/tmp/viddrop/%(id)s.%(ext)s", "quiet": True}})
        fname = await asyncio.to_thread(download, url, opts)
        await q.message.reply_document(open(fname, "rb"))
        os.remove(fname)                        # don't fill the disk

def download(url, opts):
    with yt_dlp.YoutubeDL(opts) as ydl:
        info = ydl.extract_info(url, download=True)
        return ydl.prepare_filename(info)

Hard lessons: TikTok blocks datacenter IPs — the fix is the user's own logged-in cookies.txt passed via --cookies; never print cookie contents. 50MB is Telegram's bot file cap — enforce max_filesize before downloading, not after.

3. DubLab — AI Video Dubbing Website

What it is: Upload a short video (≤5 min / 250MB) → pick EN↔MM + male/female voice → get a dubbed MP4. Pipeline: faster-whisper transcribes → Gemini translates → Edge-TTS speaks (Burmese voices work!) → FFmpeg muxes. Plus an SRT translator tab.

Step 1 — transcribe with faster-whisper

from faster_whisper import WhisperModel
model = WhisperModel("small", device="cpu", compute_type="int8")

def transcribe(path: str):
    segments, _ = model.transcribe(path, language="en")
    return [{{"start": s.start, "end": s.end, "text": s.text.strip()}}
            for s in segments if s.text.strip()]
# Dependency pins that actually work together (faster-whisper 1.2.1):
#   av==15.1.0  +  huggingface_hub<1
# av 19.x removed the `metadata_errors` kwarg faster-whisper passes -> TypeError.
# av 14.x has no cp312 wheel -> build fails. 15.1.0 is the sweet spot.

Step 2 — translate with Gemini key rotation

Free Gemini quota is tiny (~10–15 req/day/key ≈ 1–2 videos/day). Rotate through keys; when one hits quota, try the next:

import google.generativeai as genai, itertools

KEYS = open("/root/dublab/.env").read().split("GEMINI_API_KEYS=")[1].split(",")
key_cycle = itertools.cycle(k.strip() for k in KEYS)

def translate(text: str, target: str) -> str:
    last_err = None
    for _ in range(len(KEYS)):                 # try each key once
        genai.configure(api_key=next(key_cycle))
        try:
            m = genai.GenerativeModel("gemini-2.0-flash")
            r = m.generate_content(
                f"Translate to {{target}}, return ONLY the translation: {{text}}")
            return r.text.strip()
        except Exception as e:
            last_err = e                        # quota error -> next key
    raise last_err

Step 3 — dub with Edge-TTS (Burmese voices!)

import edge_tts, asyncio

VOICES = {{
    "mm-male":   "my-MM-ThihaNeural",
    "mm-female": "my-MM-NilarNeural",
    "en-male":   "en-US-ChristopherNeural",
    "en-female": "en-US-AriaNeural",
}}

async def speak(text: str, voice: str, out: str):
    await edge_tts.Communicate(text, voice).save(out)

# Per segment, then stretch/compress each clip to the segment's
# timestamp window with FFmpeg `atempo` so lips roughly match.

Step 4 — mux with FFmpeg

# dubbed.mp3 = concatenated per-segment clips, original kept quiet underneath
subprocess.run([
    "ffmpeg", "-y",
    "-i", "original.mp4",
    "-i", "dubbed.mp3",
    "-filter_complex",
        "[0:a]volume=0.25[orig];[1:a]adelay=0[db];[orig][db]amix=inputs=2[a]",
    "-map", "0:v", "-map", "[a]",
    "-c:v", "copy", "-c:a", "aac",
    "output.mp4"], check=True)

Serving — waitress behind nginx, iOS-safe uploads

# waitress on 127.0.0.1 ONLY (never expose the app server directly)
# /etc/systemd/system/dublab.service:
# ExecStart=/root/dublab/venv/bin/waitress-serve --listen=127.0.0.1:5000 app:app

/* iOS Safari bug: fetch() POST with FormData throws
   "TypeError: The string did not match the expected pattern" (WebKit bug).
   Fix: XMLHttpRequest for uploads, fetch only for JSON polling. */
function uploadFile(file) {
  return new Promise((resolve, reject) => {
    const xhr = new XMLHttpRequest();
    const fd = new FormData();
    fd.append("video", file);
    xhr.open("POST", "/api/dub");
    xhr.onload = () => resolve(JSON.parse(xhr.responseText));
    xhr.onerror = reject;
    xhr.send(fd);
  });
}

Hard lessons: Jobs in process memory die on restart — an old job ID becomes invalid after any service restart. Rate-limit uploads (10/hour per IP) or one user eats the whole Gemini quota. The monitor must never auto-reopen the site against an explicit close order.

4. Movie Scraper Bot (Telethon)

What it is: An in-bot movie scraper — searches Telegram channels for movies and serves results inside the bot. Built on Telethon (user client, not bot API) so it can read channel history.

Client setup — secrets never in code

# /root/moviebot/.tg_env  (chmod 600) — the ONLY place secrets live:
# API_ID=123456
# API_HASH=abcdef123456
# (first run does interactive login, then moviebot.session persists)

from telethon import TelegramClient
import os

def load_env(path):
    d = {{}}
    for line in open(path):
        if "=" in line and not line.startswith("#"):
            k, v = line.strip().split("=", 1)
            d[k] = v
    return d

env = load_env("/root/moviebot/.tg_env")
client = TelegramClient("/root/moviebot/moviebot", env["API_ID"], env["API_HASH"])

async def search_movies(query: str, limit=10):
    results = []
    async for msg in client.iter_messages("@movie_channel", search=query, limit=limit):
        if msg.video or msg.document:
            results.append({{"id": msg.id, "text": msg.text or "", "date": msg.date}})
    return results

Dead-token crash loop — the fix

# If the BotFather token dies, the service crash-loops every few seconds.
# That once helped OOM-freeze the whole Spain server (with musicbot).
# Guard: stop + disable until a fresh token exists.
#   systemctl stop moviebot && systemctl disable moviebot
# Then paste a NEW token from @BotFather into .tg_env and re-enable.

Hard lessons: A dead token doesn't fail quietly — Telethon retries forever, eating CPU/RAM in a tight loop. Combined with zero swap, two such bots froze the entire Spain VPS. Monitor for crash-looping services, not just stopped ones.

5. OSINT Bot

What it is: A Telegram bot for quick lookups: /ip /email /user /auto /inn /all /proxycheck. Built for speed — one message in, one answer out.

Proxy handling — file format + graceful fallback

Proxies die constantly. The bot reads proxy.txt (host:port:user:pass per line) and falls back to a direct connection instead of hard-failing:

import random, requests

def load_proxies(path="/root/info/proxy.txt"):
    proxies = []
    for line in open(path):
        line = line.strip()
        if not line or line.startswith("#"):
            continue
        parts = line.split(":")          # host:port:user:pass
        if len(parts) == 4:
            h, p, u, pw = parts
            proxies.append(f"http://{{u}}:{{pw}}@{{h}}:{{p}}")
    return proxies

PROXIES = load_proxies()

def fetch(url, timeout=15):
    last_err = None
    for px in random.sample(PROXIES, min(3, len(PROXIES))):
        try:
            return requests.get(url, proxies={{"http": px, "https": px}},
                                timeout=timeout).json()
        except Exception as e:
            last_err = e                  # proxy dead -> try next
    # All proxies failed: direct fallback (better than nothing)
    return requests.get(url, timeout=timeout).json()

Command skeleton

async def cmd_ip(update, context):
    target = " ".join(context.args)
    if not target:
        return await update.message.reply_text("Usage: /ip 1.2.3.4")
    try:
        d = await asyncio.to_thread(fetch, f"http://ip-api.com/json/{{target}}")
        await update.message.reply_text(
            f"🌐 {{d.get('query')}}\n"
            f"📍 {{d.get('city')}}, {{d.get('country')}}\n"
            f"🏢 {{d.get('isp')}}\n"
            f"🕒 {{d.get('timezone')}}")
    except Exception as e:
        await update.message.reply_text(f"❌ Lookup failed: {{e}}")
# Same shape for /email /user /inn — one fetcher, one formatter each.

Hard lessons: The original crash was a proxy-format mismatch (host:port:user:pass vs code expecting something else) plus missing imports — always validate the proxy file on startup and log which line failed.

6. Health Watchers & Automation

What it is: The monitoring layer keeping everything alive — hourly VPN health checks, 15-minute service watchers, a freeze auto-recovery, quota watchers, and a daily preventive reboot of all three servers.

The watcher pattern — 2 consecutive failures before alerting

This one rule kills 99% of false alarms. A single failed check means nothing; two in a row means something:

#!/bin/bash
# /root/watchers/vpn-server-health-watch.sh (simplified)
STATE=/root/watchers/state/health.json
URL="https://panel.kbko.qzz.io/"

was_down=$(jq -r .was_down "$STATE")
fails=$(jq -r .fails "$STATE")

if curl -sk --max-time 10 "$URL" | grep -q "Marzban"; then
    fails=0
    if [ "$was_down" = true ]; then
        send_telegram "✅ Panel ပြန်ကောင်းသွားပါပြီ"
    fi
    was_down=false
else
    fails=$((fails + 1))
    if [ "$fails" -ge 2 ] && [ "$was_down" = false ]; then
        # Second consecutive failure -> real alert, with fix buttons
        send_telegram "🚨 Panel down (2 checks)\n\n✅ ပြင်ပါ / ⏭️ ထားလိုက်မယ်"
        was_down=true
    fi
fi
echo "{{\"was_down\": $was_down, \"fails\": $fails}}" > "$STATE"
# Silent when healthy. Re-alert only after 2 MORE consecutive failures.
# Fix-approval: next run fixes ONLY if the ✅ button was tapped.

Spain freeze auto-recovery

Standing order: "freezeဖြစ်တိုင်း မင်းဘာမင်းစစ်ပြီး လိုအပ်တာ တခါထဲ လုပ်လိုက်ပါ". Every 15 min: if both SSH and the web check fail twice in a row, reboot Spain via the IONOS panel (max once per incident), then verify and report in Burmese:

# Pseudo-logic of spain-freeze-auto-recovery (15-min cron):
ssh_ok=$(timeout 10 ssh spain "echo ok" 2>/dev/null)
web_ok=$(curl -sk --max-time 10 https://n8n.kbko.qzz.io/ -o /dev/null -w "%{{http_code}}")

if [ -z "$ssh_ok" ] && [ "$web_ok" != "200" ]; then
    fails=$((fails + 1))
else
    fails=0; rebooted_this_incident=false
fi

if [ "$fails" -ge 2 ] && [ "$rebooted_this_incident" = false ]; then
    ionos_panel_restart "SPAIN_IP"   # Restart action, that server ONLY
    rebooted_this_incident=true
    sleep 180; verify_services           # dublab, vpnbot, nginx, marzban
    send_telegram "🔄 Spain freeze → reboot လုပ်ပြီး, service တွေပြန်ကောင်း ✅"
fi

Root cause found (Oct 2025) — and fixed

# NOT a payment issue (contract clean). The real cause:
#  - moviebot (dead BotFather token) + musicbot (login pending)
#    crash-looped every few seconds
#  - 3.8 GB RAM + ZERO swap -> memory-pressure stalls
#    (panel showed RUNNING, but SSH + web were dead)
#
# Fix:
systemctl stop moviebot musicbot
systemctl disable moviebot musicbot
fallocate -l 2G /swapfile && chmod 600 /swapfile
mkswap /swapfile && swapon /swapfile
echo "/swapfile none swap sw 0 0" >> /etc/fstab   # persist reboot

Daily preventive restart (07:00)

# 0 7 * * * /root/watchers/daily-restart.sh
# Spain:  reboot via SSH, verify dublab/resellerbot/vpnbot/nginx/marzban
# USA:    reboot via SSH, verify marzban container
# Windows: restart via IONOS panel API, verify IP + status On
# Report: one Burmese summary, silent if all green.

Hard lessons: Alerts go only to the admin Telegram ID — never to customers. No proactive watchers unless explicitly requested; the approved set is enough.

🛠️ Universal Patterns

Rules applied to every project, learned the hard way:

1. Systemd everywhere: Restart=always, RestartSec=5, enable --now. A service that can't survive a reboot isn't done.
2. Secrets in .env (chmod 600) — never in code, chat, logs, or screenshots. Bot tokens only in env.
3. Backup before changes: cp file{,.bak$(date +%Y%m%d)}. The 30 seconds it takes has saved hours.
4. Create-and-verify before delete — especially user accounts and server migrations. New first, delete old only after the new one works.
5. Inline buttons AND /cmd routes — every bot command, both, always. No exceptions.
6. Burmese-first messages — Unicode only (Zawgyi looks broken), short, Telegram-friendly.
7. Free-only — every project runs on free tiers. Never suggest paid plans.
8. Never break the live VPN bot — new work gets its own token, directory, service, DB, and capped resources.
9. Ask before state-changing work — read-only checks are free to run; anything destructive needs approval (except the standing freeze-recovery order).
10. Name the object in every closing line — a bare "ပိတ်လိုက်ပြီ" makes him ask what was closed.