Trillo Doc AI

Overview

Trillo Document AI is an intelligent documents management platform designed to revolutionize how companies handle insights from their vast collection of documents using semantic search, summarization, data extraction and NLP models. With our intelligent platform, you can effortlessly upload all your companyʼs physical and digital content, including images, text documents, forms, PDFs, structured data,

and unstructured data.

• Once uploaded, our system employs cutting edge Language and Machine Learning - (LLM) technology to enable powerful and human like querying capabilities.

Trillo DocAI Capabilities

It is a powerful tool for managing and organizing documents of all types and provides the following features:

• Document management: Create, edit, and manage documents of various types, including PDFs, Word documents, and images.

• Document summarization: Automatically summarize long documents, making it easy to quickly understand the key points.

• Data extraction: Extract data from documents, such as tables, charts, and images.

• Workflow automation: Create and execute workflows to automate tasks such as document review, approval, and publication.

• Search: Search for documents by keyword, full text, or metadata.

• Sharing: Share documents with others individually or with groups.

• Document ocr: Optical character recognition (OCR) to convert scanned documents into text format.

• Custom processors: The ability to create custom processors to process documents in specific ways.

• Batch processing: The ability to process multiple documents at once.
  

Document Processing Steps

Trillo Doc AI processing pipeline is a specific implementation of the generic document processing pipeline using GCP services such as Google Vertex AI (which includes Gen AI, LLMs APIs), Google Document AI (NLP based parsers for forms, tables, purchase orders, invoices, etc.), AlloyDB (vector database), Cloud SQL (managed relational database), Google Cloud Storage for buckets.

Document AI processing pipeline consists of these steps

• Organization

• Ingestion

• Processing

• Distribution

• Access control

Organization

Before any kind of processing, data has to be organized in the form of folders and sub folders. The Trillo DocAI pipeline processes data on folder level. Before processing, set the folder type as one of the following below:

• Text/Image

• AutoDetect

• Google provided processor types like form, OCRetc

• Schema based , where the schema will be discovered and generated on the fly

Ingestion

Data from multiple sources is brought into cloud storage bucket for further analysis and processing. It can be achieved in multiple ways:

• HTTPS

• SFTP

• RSync

• GSUtil

• Connectors

Processing

Document processing pipeline processes it using a set of NLP, LLM services, and open source libraries. It runs multiple substeps as follows:

1. Text and images are extracted from each document. The raw text and images are stored in new locations on buckets.

2. Optionally, PHI and PII data is redacted from text and images before storing.

3. Text is parsed to extract structured data from documents. The structured data extraction requires a multitude of steps - entity extraction, form fields extraction, and extraction of data in tables.

4. Text is chunked into smaller yet complete chunks (i.e. avoiding splitting a chunk such that the central idea of the chunk is not lost leading to less accurate semantic meaning).

5. The above two steps require the ingestion of domain knowledge into the pipeline.

6. Processed data is stored in a variety of databases - relational DBs, vector DBs, search indexing platforms.

Distribution

This can be achieved as follows:

• 1. Via UI
• 2. API
• 3. Export to a dump file or export to data in bucket

Access Control

Sometimes, the source document repositories provide access control information (such as enterprise content management systems). This access control information is stored and applied in the API gateway. The access control information can also be enriched or customized.

Trillo Doc AI Architecture

Trillo Doc AI processes documents using Large Language Models (LLMs) and Document AI processors. It reads the content of documents using OCR or libraries - such as PDF libraries. It classifies each document using an LLM based classifier. Based on the classification, it uses a processor to extract structured data from the content or processes it as a text document.

Text documents are summarized, chunked, and converted to vectors (text embeddings). These vectors are stored in a vector database for semantic matching.

Generic Architecture

Trillo Doc AI on GCP for Document Processing

Trillo Doc AI processing pipeline is a specific implementation of the generic document processing pipeline using GCP services such as Google Vertex AI (which includes Gen AI, LLMs APIs), Google Document AI (NLP based parsers for forms, tables, purchase orders, invoices, etc.), AlloyDB (vector database), Cloud SQL (managed relational database), Google Cloud Storage for buckets.

Trillo DocAI Features

Here is a list of features which Trillo DocAI has to offer

• Organize documents in a hierarchy of folders.

• Rbac rules at folder and file levels.

• Operations on files and folders using api or UI: Move, Copy, Rename, Delete, Download/ Upload/ Bulk Upload, Ingestion

• Semantic search using api or UI.

• Semantic matching to match a given documentʼs vectors against all available documents using cosine similarities.

• Question and answers using generative a i apis.

• PHI, Pii and pci data redaction.

Here is how Trillo DocAI works for various kinds of documents

Semantic Search and Q&A

Text document (structured data is not available or not required) -

Process using Text image type. Trillo Doc AI will index them (for semantic search and Q&A, will also summarize).

- Data extraction (structured / semi structured data) using GCP Document AI processor

The selected type should be one of the processors from the below list (shown when you create a folder). It requires that the corresponding Document AI processor is created using the GCP console. Name of processor should start with the name as in the JSON
processors list = [

• { name: "Text and Image", key: "Text image" },

• { name: "Form", key: "form" },

• { name: "Auto Detect", key: "autoDetect" },

• { name: "Purchase Order", key: "Purchase order" },

{ name: "Invoice", key: "Invoice" },

{ name: "W2", key: "W2" },

• { name: "Identity Document Proofing", key: "identity document proofing parser" },

• { name: "US Driver License", key: "us driver license parser" },

• { name: "US Passport", key: "us passport parser" },

• { name: "1003", key: "1003Parser" },

{ name: "1040", key: "1040Parser" },

• { name: "1040 Schedule C", key: "1040Schedule c" },

{ name: "1040 Schedule D", key: "1040ScheduleD" },

{ name: "1040 Schedule E", key: "1040Schedule e" },

{ name: "1099-DIV", key: "1099Div" },

{ name: "1099-G", key: "1099G" },

• { name: "1099-INT", key: "1099Int" },

• { name: "1099-MISC", key: "1099Misc" },
  { name: "1099-NEC", key: "1099Nec" },

{ name: "1099-R", key: "1099R" },

{ name: "1065", key: "1065" },

{ name: "1120", key: "1120" },

{ name: "1120S", key: "1120S" },

{ name: "Bank Statement", key: "bank statement" },

{ name: "HOA Statement", key: "hoa statement" },

{ name: "HUD-92900B", key: "hud92900B" },

{ name: "Lending Document Splitter & Classifier", key:

"lending document splitter classifier" },

{ name: "Mortgage Statement", key: "mortgage statement" },

{ name: "Pay Slip", key: "paySlip" },

{ name: "Retirement/Investment Statement", key:

"retirement investment statement" },

{ name: "SSA-89", key: "ssa89" },

{ name: "SSA-1099", key: "ssa1099" },

];

Processing a structured document using Generative AI

For the structured document types that are not supported by Google Document AI, Trillo Doc AI parses them using Generative AI and extracts data (example, Medical record, Commercial lease).

Chunking documents using Generative AI

This is needed when documents are to be matched against each other (example, resume with job, similarities in marketing materials, etc). This can be partially achieved using 3. This requires some custom configuration. If you have such a use case then let us discuss, we can assist you.

Features At A Glance

Doc AI Use Cases and Architecture

Use Cases

Trillo DocAI Architecture

The Document AI model can be divided into 4 subsections for ease of understanding.These are ordered as per logical dependencies. File Manager classes and Document Manager classes are at the same level. The file manager deals with the data ingestion and physical folder hierarchy. The document manager deals with the same files, treating them as documents. The document manager hierarchy is logical and not reflected on the cloud storage bucket. The document manager is responsible for text, image extraction from the documents (files) and storing in the Document AI Warehouse.

DocAI Architecture

Prompt Engineering and Schema Design

Trillo DocAI provides the flexibility of designing prompts based on specific use cases. These prompts can be further improved to achieve better responses. These prompts are designed as part of schema files , which are then processed by the DocAI .

Here is a sample schema file to extract Patient/Doctor information, ICD and CPT Codes from a sample medical transcription:

{{
  "name" : "invoices",
  "displayName" : "invoices",
  "identity attributes" : null,
  "fields" : [
    {
      "displayname" : "address",
      "name" : "address",
      "type" : "string"
    },
    {
      "displayname" : "apple",
      "name" : "apple",
      "type" : "string"
    },
    {
      "displayname" : "denny gunawan",
      "name" : "denny_gunawan",
      "type" : "string"
    },
    {
      "displayname" : "gst (10%)",
      "name" : "gst__10",
      "type" : "string"
    },
    {
      "displayname" : "invoice number",
      "name" : "invoice_number",
      "type" : "string"
    },
    {
      "displayname" : "quantity(kg)",
      "name" : "quantity_kg",
      "type" : "string"
    },
    {
      "displayname" : "subtotal",
      "name" : "subtotal",
      "type" : "string"
    },
    {
      "displayname" : "total",
      "name" : "total",
      "type" : "string"
    },
    {
      "displayname" : "watermelon",

      "name" : "watermelon",
      "type" : "string"
    }
  ]
}

UI Workflow

Login using your workbench credentials

Login and Landing Page

Login with your credentials

Landing Page

Document Upload

• Select browse documents on the top right corner of the homepage.

• Clickon "New folder"

• Enter an application domain

• Select the type of documents that will go into the folder (this tells the backend which processor to use)

• Navigate to newly created folder

• Click upload button

• Drag and drop or browse to select files.

• After upload completed, click close

Process Documents

Process Documents

1. Right click newly created folder (Processing is done at a folder level)

2. Check the “Process All” box if you want to process old and new files. Leave blank to process just new files.

3. Check Summarize and/or Extract Fields and Tables

4. Click “Execute Workflow”

5. Task submitted is displayed

Monitoring Processing Status

Check Processing Status

1. Click on Trillo Logo in upper left corner

2. Go to Tasks History

3. Select the task by clicking on the name (Folder (id=xx). Utilize User and Created/Updated fields to know which task to select.

4. Screen changes to Logs to view status of each file being processed

5. View Status field. Refresh browser to update Status. Task will show as Completed when finished.

Tasks History

Detailed Tasks Logs

Data Ingestion

1. Click on 3 dots on the top right corner and click on Ingestion

2. Create a Target folder under My Folders

3. Keeping the selected Target Folder, click on Select Source Folder

4. Select folder type, schema and Execute the workflow

Document Matching

Trillo DocAI also provides the document matching capabilities. For e.g Resumes to jobs matching with high accuracy for candidate search.

Match with a given Document/Folder or Match with Text

The execution of matching gives the following output

Verifying Results

DocAI pipeline works on multiple kinds of documents

• Text documents: Text and images are extracted from each document.

• Schema based : Entities are extracted from the documents. Text is parsed to extract structured data from documents. The structured data extraction requires a multitude of steps - entity extraction, form fields extraction, and extraction of data in tables.

Text Documents ( Summarization)

Follow the below steps to verify the results for Text/Image documents:

• Once the workflow is complete, navigate to the folder. The green tick near the file name provides the conformation that the processing of the file has been completed.

• Right click and select "Extracted file view"

• It will open up the side by side view of the document and the processed summary

• Navigate to respective tabs to see more information

Schema Based Documents(Data Extraction)

• For certain types of structured or semi structured documents, for examples lease documents/Invoices/POs, navigate to the the extracted entities as follows:

Technologies Used

Vertex AI (Document AI, Doc AI Warehouse)

Cloud Storage Bucket

Google Ku bernet es Engine

Data Loss Prevention API

Cloud SQL

Big Query(optional)

Looker/Studio (optional)

Trillo Document AI Vector db Setup

Prerequisites:

• A google cloud project with the cloud sql api enabled.

• A deployer vm instance where you have ssh access.

Steps:

SSH into Deployer VM:

• Connect to your deployer vm instance using ssh.

Create Setup Files:

• Create a directory named do cai.

• Inside do cai, create two files:

  • install.sh

  • vector embedding.sql

Populate install.sh:

• Paste the following code into install.sh, ensuring proper indentation:

set -e

INSTANCE_NAME=trillo-pg vector
REGION=us-central1
DB_TIER=db-custom-4-15360

# Create PostgreSQL instance (adjust REGION and DB_TIER as needed)
gcloud sql instances create $INSTANCE_NAME \
--database-version=POSTGRES_15 --edition=ENTERPRISE --tier=$DB_TIER \
--region=$REGION --availability-type ZONAL --no-assign-ip --
network=default

# Get instance IP address

IP_ADDRESS=$(gcloud sql instances describe $INSTANCE_NAME --
format='value(ip addresses.ip address)')

# Set PostgreSQL password (a random password will be generated)
export Pg password=$(openssl rand -base64 8)

gcloud sql users set-password postgres --instance=$INSTANCE_NAME --
password="$Pg password"

# Create database and install extensions
psql -h "$IP_ADDRESS" -U postgres -d postgres -c "CREATE DATABASE
pg vector;"
psql -h "$IP_ADDRESS" -U postgres -d pg vector -c "CREATE EXTENSION
vector;"

# Load data schema

psql -h "$IP_ADDRESS" -U postgres -d pg vector -f vector embedding.sql

# Connection details

echo "Database URL: jdbc:postgresql://$IP_ADDRESS:5432/pg vector"
echo "Database Username: postgres"
echo "Database Password: $Pg password"

Populate vector embedding.sql:

• Paste the provided sqlcode directly into vector embedding.sql

create table vector embedding_tbl
(
id varchar(255) not null
primary key,
created at bigint,
updated at bigint,
deleted boolean,
deleted at bigint,
folder id bigint,
docid bigint,
page number integer,
tenant id bigint,
document name varchar(255),
filename varchar(255),
content type varchar(100),
id of user bigint,
userid varchar(255),
author varchar(255),
gcs file url varchar(1024),
embedded image urls varchar(4096),
content text,
embedding vector(768)
);

create index _folder id_index_ on vector embedding_tbl (folder id);
create index _id of user_index_ on vector embedding_tbl (id of user);
create index vector embedding_tbl_embedding_idx on vector embedding_tbl
using hnsw (embedding vector_l2_ops);

Execute the Script:

• Run install.sh from within the doc a i directory: ./install.sh

• Important:Note down the generated database url, username, and password.

Workbench UI Configuration:

• Open the trillo document AI workbench UI.

• If a data source doesn'texist, create one using the connection details from step 5.

• Navigate to the appropriate section within the UI and enter the database url, username, and password.

• Save your changes.

Data Source connection