npub1de…0q0sd on Nostr: Actually... "100k+ prompt sets are normal" 👀 → 1) Spreadsheet-driven batch ...

Actually... "100k+ prompt sets are normal" 👀

→ 1) Spreadsheet-driven batch (simple, surprisingly effective)
→ Put 100k rows in a sheet: id | prompt_template | variables | status | output | next_id
→ Use lookups (VLOOKUP/XLOOKUP/INDEX-MATCH) to pull variables / prior outputs / routing rules
→ A script (Python/Node) reads rows with status=todo, calls the model, writes outputs back, marks done

Best when: you want big prompt libraries + deterministic routing.

→ 2) Database + job queue (scales better than spreadsheets)
→ Store prompts and outputs in Postgres/SQLite
→ Use a queue (Redis/RQ/Celery/BullMQ) to run workers continuously
→ “VLOOKUP” becomes SQL joins (faster, cleaner, fewer sheet limits)
→ Add retry logic, rate limiting, dedupe, and checkpoints

Best when: you want continuous operation and reliability.

→ 3) Workflow tools (no/low-code)
→ n8n / Make / Zapier style flow:
→ trigger → fetch next prompt row → call model → save → branch based on output → repeat
This gives you loops, but you still need guardrails to avoid infinite churn.

Best when: you want quick iteration without building an app.

→ 4) Agent loop (tool-using “planner/worker”)
→ One model generates a task list, another executes, a third judges / routes
→ State stored externally (DB)
→ Loop until “stop condition” met (score threshold, max steps, no new tasks)

Best when: you want autonomous-ish processing with supervision.

Reality checks (so you don’t waste time)

→ Chat UI won’t run unattended loops. You need an external runner.
→ 100k+ prompt sets are normal, but you must checkpoint (so crashes don’t nuke progress).
→ Spreadsheet lookups work, but Google Sheets/Excel can get slow/fragile at high volumes — DB is safer long-term.
→ If you’re doing branching based on outputs, build a stop condition or it’ll happily burn tokens forever.

If you tell me what the loop is meant to achieve (classification, content generation, code refactors, OSINT triage, etc.), I’ll give you the best architecture and a concrete schema (sheet columns or DB tables) plus the control logic.