Fetch Many TikTok Transcripts in Python: Retries

Overview

To fetch many TikTok transcripts, run up to three requests at once, retry only retryable errors, poll speech-to-text jobs, and write each result as you go so a rerun resumes.

Three workers, the plan limit.
Retries blocks and rate limits; polls jobs.
Writes results as it goes; rerun resumes.
Failed requests cost 0 credits.

The script reads a list of URLs, fetches them with three workers (the limit on every plan), retries blocks and rate limits with backoff or Retry-After, polls a job when the API answers 202, and stops cleanly when credits run out.

Ravi has 400 saved videos from a campaign. A first pass failed on the 212th link. He needs a run that records each result immediately and skips what is done on the next start.

The script

fetch_many.py

"""Fetch many TikTok transcripts: 3 workers, retries, job polling, resume.
Usage: python fetch_many.py urls.txt [--out results.jsonl]
Rerun the same command to resume: URLs already in the output file are skipped."""
import argparse, json, os, sys, time, threading
from concurrent.futures import ThreadPoolExecutor
import requests

API = os.environ.get("TTA_API_BASE", "https://api.example.com")
H = {"Authorization": f"Bearer {os.environ['TTA_API_KEY']}"}
STOP = threading.Event(); LOCK = threading.Lock()

def get(path, params=None):
    return requests.get(API + path, params=params, headers=H, timeout=60)

def one(url, tries=4):
    for n in range(tries):
        if STOP.is_set(): return {"url": url, "status": "skipped"}
        r = get("/api/v1/tiktok/transcript", {"url": url}); d = r.json()
        if r.status_code == 202:                                   # speech-to-text runs as a job
            for _ in range(30):
                time.sleep(0.3); j = get(d["poll_url"]).json()
                if j.get("status") == "completed": return {"url": url, "status": "ok", "source": j["result"]["transcript_source"], "words": len(j["result"]["transcript"].split())}
                if j.get("status") == "failed": return {"url": url, "status": "failed", "code": "job_failed"}
            return {"url": url, "status": "failed", "code": "job_timeout"}
        if d.get("success"): return {"url": url, "status": "ok", "source": d["transcript_source"], "words": len(d["transcript"].split())}
        e = d["error"]
        if e["code"] == "insufficient_credits": STOP.set(); return {"url": url, "status": "failed", "code": e["code"]}
        if e.get("retryable") and n < tries - 1:
            time.sleep(float(r.headers.get("Retry-After", 2 ** n * 0.5))); continue
        return {"url": url, "status": "failed", "code": e["code"]}

ap = argparse.ArgumentParser(); ap.add_argument("urls"); ap.add_argument("--out", default="results.jsonl"); a = ap.parse_args()
done = set()
if os.path.exists(a.out): done = {json.loads(l)["url"] for l in open(a.out) if l.strip()}
todo = [u.strip() for u in open(a.urls) if u.strip() and u.strip() not in done]
print(f"{len(done)} already done, {len(todo)} to fetch")
with ThreadPoolExecutor(3) as ex, open(a.out, "a") as f:
    for res in ex.map(one, todo):
        if res["status"] == "skipped": continue
        with LOCK: f.write(json.dumps(res) + "\n"); f.flush()
        print(res["status"], res.get("code", res.get("source", "")), res["url"][-19:])

Captured runs

Eight fixture URLs: five that work (one needs a retry, one is rate limited once, one is a speech-to-text job) and three that fail for good:

First run (mock)

0 already done, 8 to fetch
ok captions 7000000000000000001
ok captions 7000000000000000002
ok speech_to_text 7000000000000000003
ok captions 7000000000000000006
ok captions 7000000000000000007
failed private_video 7000000000000000004
failed video_unavailable 7000000000000000005
failed no_transcript 7000000000000000008

Rerun with four already in the file (mock)

4 already done, 4 to fetch
ok captions 7000000000000000007
failed private_video 7000000000000000004
failed video_unavailable 7000000000000000005
failed no_transcript 7000000000000000008

A key with no credits stops the run (mock)

0 already done, 2 to fetch
failed insufficient_credits 7000000000000000001
failed insufficient_credits 7000000000000000002

What can go wrong

A video that is private or removed fails once and is recorded; nothing retries it.
Do not raise the worker count above your plan’s parallel limit.
The job polling limit here is short because the mock is fast. Use a longer one for the real service.

Get an API key to run it against the real service. Run against the local mock in tools/mock-api, which follows the assumed contract on these pages. It is not the real API, and the output is captured from that mock.

Frequently asked questions

Why three workers?

Every paid plan allows up to 3 parallel requests, so more would only trigger rate limits.

Does it save the transcripts?

It saves a summary line per URL. Add a line that writes the transcript to disk if you need the text.

What if I run out of credits?

The run stops and records insufficient_credits. Top up or change plan, then rerun to resume.