AI Pipeline & Patent: Summary of current plan PRD

Owned by Bart De Pelsmaeker (Unlicensed)
Last updated: Mar 13, 2024
3 min read

On this page, I am trying to summarize the current PRD for the AI pipeline.

The PRD underlying the current approach can be found here:

https://productmanagement.unifiedcompliance.com/LLmoRd4wwvUGjYUhilaz/automatic-taxonomic-hierarchies-and-visual-mapping/requirements

 

Screen Shot 2024-03-12 at 11.18.17 AM.png

 

 

 

Screen Shot 2024-03-12 at 11.27.54 AM.png

 

 

 

 

 

 

 

 

 

 

 

 

The following JSON snippet shows the label_value and children hierarchy for 45 CFR Part 164, which can be accessed here: https://www.ecfr.gov/api/versioner/v1/structure/2022-11-07/title-45.json

 

 

 

Source / Inspiration of AI Pipeline PRD

It is not mentioned in the PRD, but this PRD seems to draw heavily on an AWS document from Aug 15, 2022: Intelligent document processing with AWS AI services: Part 1 | Amazon Web Services

But the Amazon flow was developed to eg analyze invoices, receitps etc. In other words, PDFs or scans where data fields will be in pre-determined spots (like eg the total cost at the end of the receipt).

 

Patent Application document

 

 

If I read correctly, our patent is built upon the premise that we beat the F1 scores of the (then) leading LLM models.

The patent mentions a 74,38% F1 score.

We would urgently need to validate this premise that our approach is significantly superior. The advances in LLMs have been fast, and this benchmark may no longer be valid.

Attached are some instances for MWE matching with F1 scores of 80% and higher. With general available tools.

 

For example:

https://aclanthology.org/2021.emnlp-main.112.pdf

https://arxiv.org/pdf/2303.06623.pdf

Additionally, there is this one:

(PDF) Interpretable Unified Language Checking

 

Many LLMs are currently benchmarked on reasoning, knowledge as well: