RAG Chunking, Embeddings, Evals & SSML: AI App Stack

Shipping an AI feature means sweating the unglamorous layers: how text is split before embedding, whether two vectors actually mean "similar", whether training data parses, whether the model retrieves under distraction, and whether speech markup validates. These five tools cover that stack: RAG Chunking Visualizer, Vector Similarity Calculator, Fine-tune JSONL Validator, Needle-in-a-Haystack Generator, and SSML Markup Builder.

Chunking is a retrieval decision, not a formatting one

chunkRAGText splits text with four strategies — fixed windows, sliding windows with overlap, sentence-aware splits, and paragraph splits — controlled by chunk size and overlap. Sliding windows advance by a stride of size − overlap, so a 500-character chunk with 100 characters of overlap steps 400 characters at a time and keeps boundary context alive across neighbors. Token counts use the explicitly approximate chars-per-4 rule, which is honest about being an estimate rather than a tokenizer. The visualizer shows where each chunk starts and ends, making it obvious when a table or definition gets sliced in half.

// 500-char chunks, 100 overlap → stride 400
chunkRAGText(doc, { strategy: "sliding", size: 500, overlap: 100 });

Start with sentence-aware chunks around 500 characters and only add overlap if recall on boundary-spanning questions is weak — overlap multiplies your index size for free.

Similarity scores and eval harnesses you can trust

cosineSimilarity normalizes out vector magnitude, which is what you want when comparing embedding directions; vectorDistance adds raw Euclidean distance and dot product for when magnitude matters. parseVectors accepts JSON arrays, comma-separated lists, or newline/semicolon-delimited rows, and projectVectors drops high-dimensional sets into a 2D scatter via first-two-dims or PCA so clusters become visible. Mismatched dimensions throw instead of silently broadcasting — a real footgun when mixing embedding models.

Question	Tool	Key function
Are these two chunks near-duplicates?	Similarity	cosineSimilarity(a, b)
Did my dataset parse and alternate roles?	Validator	validateFinetuneJsonl
Does retrieval survive long context?	Needle test	generateNeedleInHaystack

The validator checks OpenAI and Anthropic JSONL shapes — non-empty messages arrays, known roles (system, user, assistant, tool), valid role alternation — and reports per-line errors plus dataset statistics like role distribution and estimated tokens. The needle generator builds seeded, reproducible haystacks with an FNV-1a hash and LCG filler, placing a verification-code needle at a chosen depth percent so you can sweep 0–100% depth and plot the retrieval curve.

SSML that parses the first time

buildSsml assembles prosody (rate, pitch, volume), break (millisecond/second/nanosecond timing), emphasis levels, say-as interpretations (date, characters, cardinal, ordinal), and sub aliases into escaped, well-formed XML, while validateSsml flags unsupported tags and malformed attributes before they hit the TTS API. Everything is XML-escaped, so ampersands in your copy never corrupt the document. Draft the phrasing in plain text, then add one prosody tweak at a time — stacking rate, pitch, and emphasis changes together makes failures hard to attribute.

Try Them

• RAG Chunking Visualizer — compare fixed, sliding, sentence, and paragraph chunk strategies.
• Vector Similarity Calculator — cosine, Euclidean, and dot product with PCA scatter.
• Fine-tune JSONL Validator — check OpenAI and Anthropic datasets line by line.
• Needle-in-a-Haystack Generator — seeded context-window retrieval tests.
• SSML Markup Builder — build and validate prosody, breaks, and say-as markup.