Chunk a TikTok Transcript for RAG (Python)

Overview

To chunk a TikTok transcript for RAG, fetch the JSON segments and join neighbours until a size limit, keeping the start time of each chunk.

One JSON line per chunk with id, start, end, source and text.
The source field lets you rank caption text above speech-to-text.
One request per video.

The script groups segments into chunks, keeps each chunk’s start and end, and notes whether the text came from captions or speech-to-text. TikTok videos are short, so one video is often one to five chunks.

Lena builds a search tool over a creator’s cooking videos. She wants an answer to show the second a tip starts, and to rank caption text above speech-to-text text. Chunks with times and a source field give her both.

Why chunk by segments?

Segments are already sentence-sized and timed. Joining them until a character limit keeps every chunk a clean run of speech and keeps a real start time. Cutting a flat string at a fixed length loses both.

The script

chunk_rag.py

"""Group TikTok transcript segments into chunks for a RAG index.
Usage: python chunk_rag.py TIKTOK_URL [--max-chars 220]"""
import argparse, json, os, sys
import requests

API = os.environ.get("TTA_API_BASE", "https://api.example.com")

def fetch(url):
    r = requests.get(f"{API}/api/v1/tiktok/transcript", params={"url": url},
                     headers={"Authorization": f"Bearer {os.environ['TTA_API_KEY']}"}, timeout=60)
    data = r.json()
    if not data["success"]:
        sys.exit(f"error {data['error']['code']}: {data['error']['message']}")
    return data

def chunk(segments, max_chars):
    out, cur = [], None
    for s in segments:
        if cur and len(cur["text"]) + 1 + len(s["text"]) > max_chars:
            out.append(cur); cur = None
        if cur is None:
            cur = {"start": s["start"], "end": s["end"], "text": s["text"]}
        else:
            cur["end"] = s["end"]; cur["text"] += " " + s["text"]
    if cur:
        out.append(cur)
    return out

ap = argparse.ArgumentParser(); ap.add_argument("url"); ap.add_argument("--max-chars", type=int, default=220)
a = ap.parse_args()
d = fetch(a.url)
for i, c in enumerate(chunk(d["segments"], a.max_chars)):
    rec = {"id": f"{d['video_id']}:{i}", "video_id": d["video_id"], "start": c["start"], "end": c["end"],
           "source": d["transcript_source"], "text": c["text"]}
    print(json.dumps(rec))

A captured run

Run with --max-chars 90 on a five-segment fixture:

Captured output from the local mock (not the real API)

{"id": "7000000000000000001:0", "video_id": "7000000000000000001", "start": 0.5, "end": 5.1, "source": "captions", "text": "Okay, three tips for a small garden. Tip one: measure the sun before you buy seeds."}
{"id": "7000000000000000001:1", "video_id": "7000000000000000001", "start": 5.4, "end": 11.0, "source": "captions", "text": "Tip two: raised beds drain better. Tip three: water in the morning."}
{"id": "7000000000000000001:2", "video_id": "7000000000000000001", "start": 11.3, "end": 13.6, "source": "captions", "text": "Follow for the full guide."}

What can go wrong

A private or removed video exits with the typed error (see below).
Speech-to-text results may arrive as a job; this script does not poll one. The batch workflow does.
Segment text can be one or two words per line on some tracks; raise --max-chars.

A private video (mock)

error private_video: This video is private.

Get an API key to run it against the real service. Run against the local mock in tools/mock-api, which follows the assumed contract on these pages. It is not the real API, and the output is captured from that mock.

Frequently asked questions

How big should a chunk be?

It depends on your embedding model and question style. Start near one to three sentences and test retrieval on your own questions.

Can I link to the second in TikTok?

Store the start time. Whether TikTok links can open at a given second is not documented here, so check before you promise it in a product.

Does it embed the chunks?

No. It prints JSON lines you can send to any embedding service.