Blockify® Performance Analysis Report for Big Four Consulting Firm

Blockify is a data optimization tool that takes messy, unstructured text, like hundreds of sales‑meeting transcripts or long proposals, and intelligently optimizes the data into small, easy‑to‑understand "IdeaBlocks." Each IdeaBlock is just a couple of sentences in length that capture one clear idea, plus a built‑in contextualized question and answer.

With this approach, Blockify improves accuracy of LLMs (Large Language Models) by an average aggregate 78X, while shrinking the original mountain of text to about 2.5% of its size while keeping (and even improving) the important information.

When Blockify's IdeaBlocks are compared with the usual method of breaking text into equal‑sized chunks, the results are dramatic. Answers pulled from the distilled IdeaBlocks are roughly 40X more accurate, and user searches return the right information about 52% more accurate. In short, Blockify lets you store less data, spend less on computing, and still get better answers- turning huge documents into a concise, high‑quality knowledge base that anyone can search quickly.

Blockify works by processing chunks of text to create structured data from an unstructured data source.

Showcased below are examples comparing the difference in data quality between the IdeaBlocks generated by the Blockify process, and the raw chunked text that would be used in a traditional chunk based ingestion process. Each example compares a user query with the corresponding IdeaBlock (left) and traditional text chunk (right).

The vector distance metric shows the relevance between the query and the returned content (lower is better). IdeaBlocks consistently demonstrate higher relevance (lower distance) compared to traditional chunking methods.

 

3.1 Purpose

This white paper is a decision tool for CIOs, data‑platform owners, and AI solution architects who are under pressure to scale Generative‑AI initiatives without compromising accuracy, security, or budget. After reading it you will be able to:

• Diagnose failure modes in today's "dump‑and‑chunk" RAG pipelines—duplicate data bloat, semantic fragmentation, stale versions, and permission leaks—and quantify the financial, security, and compliance risks they introduce.
• Compare legacy approaches with Blockify®, a patented ingestion, distillation, and governance stack that turns millions of inconsistent pages into a compact, governed "gold dataset" of IdeaBlocks.
• Validate headline claims such as ≈78× improvement in LLM RAG accuracy, ≈51% higher vector‑search precision, and ≈40× dataset reduction using an openly documented benchmark and step‑by‑step replication guide provided.
• Map Blockify's operating model (roles, review cadence, cost envelope, security posture) to your own enterprise architecture and regulatory landscape.
• Build an internal business case that links improved LLM accuracy to concrete outcomes: higher bid‑win rates, lower risk exposure, faster call‑center resolution, and demonstrable compliance with GDPR, CMMC, EU AI Act, and similar mandates.

This whitepaper supplies both the technical depth and the executive‑level rationale to decide whether Blockify should become the foundation of your production‑grade GenAI stack.

3.2 Definitions

• "Retrieval-Augmented Generation" (RAG): A method that enhances the generation capabilities of LLMs by retrieving relevant documents from a vast corpus to provide fact-based context to the answers.
• "Hallucination": An answer generated by an LLM that is not grounded in fact or conflicts with a known truth.
• "AirgapAI": A lightweight RAG + LLM chat application that operates on Intel® Core™ Ultra or comparable edge devices without internet connectivity.
• "Blockify®": A patented ingestion, distillation, and governance platform that resolves issues of data quality and duplication. It converts documents into smaller, manageable "IdeaBlocks" to improve the accuracy and efficiency of LLM outputs.
• "IdeaBlock": The smallest unit of curated knowledge in a data taxonomy associated with an Index, containing a Descriptive Name, Critical Question, Trusted Answer, and rich Metadata Tags and Keywords.

3.3 Industry Context

McKinsey expects GenAI to take over tasks that take up 60%–70% of people's working hours. "Gartner predicts that through 2026, organizations will abandon 60% of AI projects unsupported by AI-ready data.", due to poor data quality, inadequate risk controls, escalating costs or unclear business value. Blockify directly addresses this prediction by treating data quality as the prerequisite for any GenAI ROI.

https://www.mckinsey.com/capabilities/mckinsey-digital/our-insights/the-economic-potential-of-generative-ai-the-next-productivity-frontier#introduction

https://www.gartner.com/en/newsroom/press-releases/2025-02-26-lack-of-ai-ready-data-puts-ai-projects-at-risk

Blockify's patented data processing pipeline is flexible. Some of the interchangeable components if something other that the Iternal Technologies default is desired include:

• Document Parsing: Flexibility to use Unstructured.io, AWS Textract, or other document parsing solutions.
• Chunking Algorithm: Customizable chunking strategies to optimize for different document types and content structures.
• Open Source LLMs (Fine tuned): Support for various open source models that can be fine-tuned for specific Blockify® domains.
• Embeddings Model: Compatible with different embedding models including OpenAI, AWS Bedrock, Mistral, or open-source alternatives.
• Vector Search: Integration with multiple vector databases such as Azure AI Search, Pinecone, Milvus, and more.

Blockify can integrate into any RAG LLM Workflow to include Unstructured.io, AWS Bedrock, Azure AI Search, Google Vertex and more. Blockify is a data "preprocessing" step between parsing source documents and vectorizing them.

Similarly, with other architectures shown in the diagram, Blockify fits seamlessly between the initial document parsing and the vector database or retrieval layer—regardless of the tools and platforms used. For example:

• Unstructured IO and Azure AI Search Workflow: Parse documents using Unstructured.io to extract the text, and then apply the Blockify process to optimize the dataset. Once optimized with Blockify, the results can be passed into Azure's AI Search for use within the remainder of the pipeline.
• Gemini + Pinecone Workflow: Documents can first be processed using Gemini's LLM capabilities to extract content, after which Blockify transforms and optimizes the structured data. This enriched data is then sent to Pinecone for vector storage and similarity search, ultimately enhancing the accuracy and relevance of user-facing applications.
• Amazon Textract + Amazon Bedrock Workflow: When using Amazon Textract to parse and extract text from source documents, Blockify acts as an intermediate preprocessing stage to structure and segment the data efficiently. This makes the subsequent ingestion into Amazon Bedrock's AI-powered services more performant and reliable.

By inserting Blockify as a preprocessing step in these various pipelines—whether with Unstructured.io, Gemini, or Amazon Textract—organizations can ensure that the information entering their vector databases (Azure AI Search, Pinecone, Amazon Bedrock, and others) is properly segmented and optimized. This leads to improved retrieval, faster query times, and ultimately, better answers for end users. Blockify's modular approach allows it to integrate with widely used RAG (Retrieval-Augmented Generation) workflows, boosting the performance and scalability of knowledge-intensive applications.

5.1 Symptoms Observed in Legacy RAG Deployments

Many companies want to build AI search tools, but their document libraries fight back in several ways. Different versions of the same file sit side‑by‑side, so a query about a product might pull facts from version 15, 16, and even an unreleased 17 all at once. Salespeople often copy an old proposal, tweak a few lines, hit "save as" and upload it, so a three‑year‑old file looks brand‑new and slips past any "use only recent files" filter.

Even if only five percent of a 100,000‑document collection changes every six months, millions of pages would still need checking—far too much for people to manage by hand. Tech teams then slice each document into fixed‑size chunks to fit into a vector database, but the chunks are not granular or precise and this approach can split important ideas in half, mix in off‑topic sentences, and create many almost‑identical chunks that crowd out more relevant ones.

When the retrieval system feeds these messy snippets to the language model, the model tries to merge them and can end up inventing specs, prices, or legal clauses that never existed. Meanwhile, most vector stores cannot tag data with fine‑grained permissions like "export‑controlled" or "partner‑only," leaving security holes. All of this adds huge maintenance costs and makes leaders hesitate to roll AI systems into full production.

The following symptoms are consistently reported when enterprises rely on the "dump-and-chunk" approach (fixed-length chunking → vectorize → retrieve) without upstream data governance.

Data-Quality Drift

• Version Conflicts
  Example: An Computer Infrastructure provider's server specs return v15, v16, and unreleased v17 in a single answer because old proposal PDFs coexist with new product sheets.
• Stale Content Masquerading as Fresh
  "Save-As" behavior: sales staff copy a three-year-old proposal, tweak a few lines, and upload it. The file carries a "last-modified" date of last week, defeating any date-filter in the retrieval pipeline.
• Untrackable Change Rate
  Even with a modest 5% change to a 100k-document corpus every six months, more than millions of pages would require review annually—well beyond human capacity.

Semantic Fragmentation from Naïve Chunking

• Broken Concepts
  Fixed 1,000-character windows routinely split longer paragraphs with important context such as a product value proposition in half, degrading data quality and cosine similarity between query and chunk.
• Context Dilution
  Only 25%–40% of the information in a Naïve chunk may pertain to the user's intent; the rest introduce "vector noise," causing irrelevant chunks to score higher than precise ones.

Retrieval Noise & Hallucination Patterns

• Top-k Pollution
  Because duplicates appear with slightly different embeddings, they crowd out more relevant chunks; k = 3 may return three near-identical, outdated passages instead of the more current accurate one.
• Model Guess-work
  When conflicting chunks are fed to the LLM, it "hallucinates" a synthesis—often inventing specs, prices, or legal clauses that appear plausible but are unfounded.

Governance & Access-Control Gaps

• One-Size-Fits-All Index
  Standard vector stores lack robust tags for data permissioning, export-control, clearance, or partner-specific sharing (e.g., restrict who can access "classified" information).

Operational & Cost Burden

• Human Maintenance on Datasets is Impossible
  Locating and updating "paragraph 47 in document 59 of 1,000" for every single product update across every single document is infeasible in pilot scale, but also remember, it's not 1,000 documents in production, in reality it's 1,000,000 documents or more in production, causing stakeholders to freeze further AI rollout due to risk and update maintenance burdens.

5.2 Impact – Why "Just One Bad Answer" Can Cost Millions (or Lives)

Financial Repercussions

• Scenario 1 Mega‑Bid Meltdown: During a recent $2 billion infrastructure RFP, an LLM‑powered proposal assistant could mix legacy pricing (FY‑21) with current discount tables (FY‑24). The buyer flagged the inconsistency and disqualified the vendor on compliance grounds—a total write‑off of 18 months of pursuit costs and pipeline revenue.
• Scenario 2 Warranty & Recall Cascades: An electronics manufacturer published a chatbot‑generated BOM that incorporated an obsolete capacitor. Field failures triggered a $47 million recall and stiff penalties from downstream OEM partners.
• Scenario 3 Regulatory Fines: Under EU AI Act Article 10, companies must "demonstrate suitable data‑governance practices." Delivering a hallucinated clinical‑trial statistic led to a €5 million fine from the EMA and a forced product‑labelling change.

Operational & Safety Risks

• "Grounded Fleet" Scenario: A major Defense customer could experience that 4 of the top‑10 answers returned by a legacy RAG system referenced an outdated torque value for helicopter rotor bolts. Had the error propagated, every aircraft would have required emergency inspection—potentially stranding troops and costing $X million in downtime.
• Intelligence Failure: Conflicting country‑code names ("Operation SEA TURTLE" vs. "SEA SHIELD") in separate PDFs confused an analyst's threat‑brief, delaying a security response by X hours.

Strategic & Cultural Damage

• Erosion of Trust: Once users see a system hallucinate, uptake plummets; employees revert to manual searches, negating AI ROI.
• Innovation Freeze: Board‑level risk committees often impose a moratorium on GenAI roll‑outs after a single public mistake, stalling digital‑transformation road‑maps for quarters.
• Brand Hit: Social‑media virality of a bad chatbot answer can wipe out years of marketing investment.

5.3 Root Causes – The Mechanics Behind the Mayhem

Enterprise RAG initiatives fail less because of model quality and more because the underlying data pipeline lets chaos seep in at every stage. Product specs and policy language now change weekly, so even a modest 5% drift rate means one‑third of the knowledge base is obsolete within three years.

The same paragraph lives—slightly edited—in SharePoint, Jira, email, and vendor portals, while "save‑as syndrome" keeps stamping fresh timestamps onto stale proposals. With no single, governed taxonomy, subject‑matter experts can't "fix once, publish everywhere," and versioning scattered across silos makes audits or roll‑backs impossible.

The LLM is forced to guess its way through inconsistencies—amplifying hallucinations and potentially leaking classified text into public answers. Manually patching "paragraph 47 of document 59" across a million‑file corpus would take tens of thousands of labor hours, so errors linger, compound, and eventually freeze further AI roll‑outs.

A preprocessing and governance layer such as Blockify® is required to restore canonical truth, enforce permissions, and deliver context‑complete, drift‑resistant blocks to downstream LLMs.

Listed are some of the major challenges outlined in detail by category.

• Accelerating Data Drift
  • Product cadence keeps shrinking; SaaS revisions ship weekly, hardware every quarter. Even a "small" 5% drift every six months means that within three years roughly one‑third of a knowledge base is out‑of‑date.
  • Regulatory churn (GDPR, CMMC, FDA 21 CFR Part 11) forces frequent wording changes that legacy pipelines never re‑index.
• Content Proliferation Without Convergence
  • Same paragraph lives in SharePoint, Jira wikis, email chains, and vendor portals—each with slight edits.
  • "Save‑As Syndrome" (salespeople cloning old proposals) multiplies duplicates with misleading "last‑modified" timestamps.
• Absence of a Governed "Single Source of Truth"
  • No enterprise‑wide taxonomy linking key information to a master record; Subject Matter Experts (SMEs) cannot easily correct once, publish everywhere.
  • Versioning spread across disparate repositories prevents atomic roll‑back or audit.
• Naïve Chunk‑Based Indexing
  • Fixed‑length windows slice semantic units, destroying contextual coherence and crippling cosine similarity.
  • Chunks inherit the security label of the parent file (or none at all). A single mixed‑classification slide deck can surface "SECRET" paragraphs to a "PUBLIC" query.
• Vector Noise & Embedding Collisions
  • Near‑duplicate paragraphs occupy adjacent positions in vector space; retrieval engines return redundant or conflicting passages, increasing hallucination probability.
• Human‑Scale Maintenance Is Impossible
  • Updating "paragraph 47 of document 59" across 1 million files would require tens of thousands of labor hours—economically infeasible, so errors persist and compound.

These intertwined root causes explain why legacy RAG architectures plateau in pilot, why hallucination rates stay stubbornly high, and why enterprises urgently need a preprocessing and governance layer such as Blockify® to restore trust and unlock GenAI value.

Blockify® replaces the traditional "dump‑and‑chunk" approach with an end‑to‑end pipeline that cleans and organizes content before it ever hits a vector store.

Admins first define who should see what, then the system ingests any file type—Word, PDF, slides, images—inside public cloud, private cloud, or on‑prem. A ​context‑aware splitter finds natural breaks, and a series of specially developed Blockify LLM model turns each segment into a draft IdeaBlock.

Blockify identifies near‑duplicates so an LLM can merge them into a single, canonical IdeaBlock, while auto‑tagging assigns clearance, version, and product labels. Because the dataset is now thousands of blocks instead of millions of paragraphs, experts can validate the whole knowledge base in a quick pass once a quarter and export it to any vector DB or export as an air‑gapped JSON bundle for use with AirgapAI.

The payoff is a knowledge set roughly 40× smaller and more accurate, free of version conflicts and duplicate noise, and guarded by field‑level permissions that travel with every IdeaBlock, combined with improved search capabilities lead to a 78X improvement in LLM accuracy.

GenAI systems fed with this curated data return sharper answers, hallucinate far less, and comply with security policies out of the box.

The result: higher trust, lower operating cost, and a clear path to enterprise‑scale RAG without the cleanup headaches that stall most AI rollouts.

6.1 How It Works (Full Technical Flow)

• Step 0 – Scoping
  • Admins specify Index hierarchy (e.g., Org ▸ Business Unit ▸ Product ▸ Persona / Clearance).
• Step 1 – Document Ingestion
  • Accepted formats: DOCX, PDF, PPT, PNG/JPG, markdown, HTML.
  • Pipeline kicks off data ingestion in the cloud or inside a secure private cloud or on-prem cluster.
• Step 2 – Chunking & LLM Extraction
  • Adaptive windowing algorithm finds natural semantic breaks rather than fixed 500-char splits.
  • Use a specially developed Blockify Fine-tuned LLaMA 3 model to ingest and convert each text chunk into a collection of draft IdeaBlocks.
• Step 3 – Semantic Deduplication
  • Open-source Jina Embeddings (or customer-preferred model) generate embeddings.
  • Advanced clustering groups near-duplicates at user defined thresholds.
  • Specialized LLMs distill clusters of draft IdeaBlocks into canonical IdeaBlocks while preserving nuance.
  • Average reduction is 40 times smaller, reducing a dataset down to ~2.5% of original size.
• Step 4 – Taxonomy/Tagging
  • Auto-generated tags using specialized LLMs for data classification clearance level, source system, product line, version, NDA status, etc.
  • Admins may append manual tags (e.g., "NATO-restricted" vs "Five-Eyes-only").
• Step 5 – Human Validation
  • Because the dataset size is now much smaller and human manageable (thousands of IdeaBlocks vs millions of paragraphs), Subject Matter Experts can perform quarterly review in hours rather than years.
• Step 6 – Export & Integration
  • Option A: API push to customer's existing vector DB (Pinecone, Vertex Matching Engine, Azure AI Search, etc.).
  • Option B: Local embedding and output as a JSON-L file for offline-only environments.
• Step 7 – Use
  • Leverage Iternal's AirgapAI™ to perform RAG with Optimized datasets locally
  • Use Iternal's Turnkey AI Enterprise Platform of Apps with Blockify to improve accuracy of document creation, such as RFP response writing using Waypoint
  • Apply Blockified Dataset to an existing RAG workflow / pipeline already established by your IT teams.

In the "Blockify Vector Search Accuracy Improvement" component of this study, an A/B test scenario was devised to compare Blockify's vector search performance (Scenario A) against a conventional chunking method (Scenario B). The objective was to assess whether Blockify's specialized approach to generating IdeaBlocks and metadata-driven embeddings enhances search accuracy when users query a document.

The documents provided by the customer were segmented into IdeaBlocks using Blockify, which integrates distilled concept extraction and custom metadata (e.g., block names, critical questions, and trusted answers) in forming vector embeddings. This was contrasted with a traditional linear chunk-based approach, wherein approximately 1,000-character segments were extracted with a 100-character overlap and subsequently embedded. Average distances between each user query and its "best match" segment (or IdeaBlock) served as the primary performance measure for vector search accuracy.

Results highlight a marked improvement in accuracy when employing Blockify. Legacy chunking achieved an average distance of 0.3624, whereas Blockify's corresponding distance ranged from 0.1833 (undistilled IdeaBlocks) to 0.1585 (distilled IdeaBlocks). This translates to a 56.26% performance increase in search precision for distilled IdeaBlocks relative to the legacy approach. Notably, even after distillation, an optional process designed to remove duplicate or extraneous information and reduce text volume, change in the accuracy of the vector search negligible, with this occurrence being improved by an additional 13.55%.

These findings emphasize the potential of optimized, metadata-enriched embeddings to yield substantial gains in information retrieval tasks. By leveraging refined conceptual segmentation (IdeaBlocks), the Blockify pipeline not only mitigates redundancy but also enhances the granular representation of content. The net result is a more accurate and efficient vector search experience, indicating that intelligent data distillation can outperform conventional chunking strategies both in terms of precision and storage efficiency.

The percentage improvement is calculated by first determining the absolute reduction in time- subtracting the new method's 0.1585023275 distance from the legacy method's 0.3623982031 distance to get 0.20389588 units - and then calculating what fraction 0.20389588 is of the original 0.3623982031 distance. This fraction (0.20389588 / 0.3623982031) is multiplied by 100 to convert it into a percentage, resulting in a 56.26294% improvement in vector distance (accuracy).

We further determined that distilling the information using Blockify's Intelligent Distillation does not negatively impact vector search accuracy and instead improves vector search accuracy when comparing a Blockify Distilled dataset versus an Undistilled dataset by +13.54646% Improvement in Accuracy

To calculate the distance to the best match we use a script to determine the cosine similarity between two numerical vectors. It begins by converting each input vector from a string format into an array of numbers and verifies that both vectors have the same number of dimensions.

The cosine similarity is then computed by first determining the dot product of the two vectors (the sum of the products of each corresponding pair of elements) and independently calculating the Euclidean norm (or magnitude) of each vector by taking the square root of the sum of the squares of its components.

Finally, the cosine similarity is obtained by dividing the dot product by the product of the two magnitudes, yielding a value that measures the orientation similarity between the vectors regardless of their scale, as represented by the formula cos(θ) = (A · B) / (||A|| ||B||).

To provide an ultra conservative and non-subjective determination of vector accuracy improvement, the "Best Matching" Chunk DOES NOT mean it is the most textually accurate for the User Query. For impartiality, the same applies to the "Best Matching" IdeaBlock.

"Best Matching" only means it is the closest in vector search distance between the User Query and the chunk in question. This approach avoids subjectivity in choosing one chunk over another for more favorable results.

Note this would also mean that the LLM may receive less optimal data for its response synthesis which would also lower quality for the Legacy RAG response.

Additionally because we are conducting the test with a single document in the vector dataset, the results will be heavily idealized to the benefit of the chunked text because other documents could have made the best match even less relevant than when using a single document.

This process begins by analyzing the vector embeddings of the Chunks / IdeaBlocks and the vector embedding of the user's question where each row contains a question and an array of all the candidate answers along with their vector representations.

For every entry, the program uses the vector numerical arrays for mathematical comparison. The script calculates each question's cosine distance between the question's vector and every candidate answer's vector, which quantitatively measures their similarity.

The candidate answer with the smallest cosine distance (a sign of highest similarity) is selected as the best match, and that distance is output by the script.

We can then calculate the average distance for best match across all questions in the dataset to determine overall vector distance (which determines overall accuracy).

Legacy retrieval augmented generation, which uses the industry established process of uploading documents to a text extraction solution, the text is parsed linearly, the text is chunked into segments with a specific equal length (for example 1,000 characters per chunk). The chunks are truncated based on common punctuation that ends sentences such as (".","!","?"). The text chunks have a character overlap between each chunk to ensure important context between chunks is not lost.

Blockify is the patented solution from Iternal Technologies that includes a data ingestion pipeline of the following steps: Uploading documents to a text extraction solution, the text is parsed linearly, the text is chunked into segments with a specific equal length (for example 1,000 characters per chunk). The chunks are truncated based on common punctuation that ends sentences such as (".","!","?"). The text chunks have a character overlap between each chunk to ensure important context between chunks is not lost. The chunks are processed by a specially developed LLM that is designed to extract the IdeaBlocks from the source chunks. The IdeaBlocks are distilled and deduplicated via another series of LLMs and clustering similarity algorithms, the process is repeated a number of times for optimal dataset distillation. The IdeaBlocks are made available for optional human review.

An IdeaBlock is any self-contained concept or idea about a topic, for example your company, products, services, etc. An IdeaBlock is typically 2 - 3 sentences in length. It contains a Name, a Critical Question, a Trusted Answer, an Index, Tags, Keywords, and other metadata