- Why should I clean PDFs before generating vector embeddings for RAG?
- Raw PDFs contain repeated running headers, footers, hyphenated broken words, and page numbers that pollute vector embeddings and cause hallucinations during retrieval.
- What chunking strategies and token overlaps are available?
- Select from Token-based sliding window (128-1024 tokens), recursive character chunking, or semantic header splitting with customizable token overlap.
- Which vector databases and frameworks are directly supported?
- Exports clean JSON, JSONL, and Markdown compatible with LangChain, LlamaIndex, Pinecone, Chroma, Qdrant, Weaviate, and Milvus.
- Does this tool preserve Markdown tables and code snippets?
- Yes. Table structures are extracted and formatted as standard GitHub-flavored Markdown tables, maintaining column relationships for retrieval.
- Why do raw PDF files cause poor retrieval performance in RAG pipelines?
- PDFs store visual glyph coordinates rather than semantic text structure. Standard PDF extractors (like pypdf or pdfminer) output raw text with repeated running headers, footers, page numbers, hyphenated word splits (e.g. "opti-\ mization"), and arbitrary mid-sentence line breaks. In vector databases (Pinecone, Weaviate, pgvector, Chroma), this formatting noise distorts embedding vectors, wastes token context windows, and lowers retrieval recall by up to 32%.
- How does Cerilas PDF → RAG Cleaner eliminate noise without uploading files?
- Our engine runs 100% locally in your web browser using WebAssembly (pdfjs-dist). It analyzes the visual bounding box geometries and font sizes of each text item across pages. Repeated text strings at the top and bottom 8% of page boundaries are automatically identified as running headers/footers and stripped. Split hyphenated words are reconciled, whitespace is normalized, and visual headers are synthesized into clean Markdown (# H1, ## H2, ### H3).
- What chunking strategies are supported for vector embedding models?
- Cerilas provides three core RAG chunking algorithms: 1) Semantic Section Chunking (splits logically along Markdown headings to keep topical concepts unified), 2) Fixed Token Sliding Window (configurable target tokens like 500t or 1000t with 10% overlap to preserve boundary context for models like OpenAI text-embedding-3-small and Cohere Embed v3), and 3) Page-by-Page chunking with explicit page metadata.
- What is the optimal chunk size and overlap for OpenAI text-embedding-3-small?
- For OpenAI text-embedding-3-small and text-embedding-3-large, empirical RAG benchmarks show that a chunk window of 400 to 512 tokens with a 10% overlap (40-50 tokens) yields the highest precision recall. Smaller chunks (250 tokens) are ideal for factual Q&A and lookup tables, while larger chunks (800-1000 tokens) are preferred for complex legal and narrative summarization.
- How does this tool handle scientific and arXiv research papers with two columns?
- Multi-column academic PDFs (arXiv, IEEE, ACM) often cause naive Python extractors to read horizontally across column boundaries, mixing disparate paragraphs together. Cerilas processes visual X/Y spatial layout geometries, respecting column gutters to output continuous, linear Markdown in exact reading order.
- Can Cerilas PDF → RAG Cleaner parse financial 10-K and 10-Q reports?
- Yes. SEC filings, quarterly 10-Q reports, and financial prospectuses are notoriously laden with repetitive page headers, CIK stamps, and table splits. Cerilas strips running header artifacts while preserving tabular Markdown structures, allowing clean financial metrics ingestion into RAG knowledge bases.