AI Pipeline & Patent: Summary of current plan PRD

On this page, I am trying to summarize the current PRD for the AI pipeline.

The PRD underlying the current approach can be found here:

https://productmanagement.unifiedcompliance.com/LLmoRd4wwvUGjYUhilaz/automatic-taxonomic-hierarchies-and-visual-mapping/requirements

Screen Shot 2024-03-12 at 11.18.17 AM.png

The current approach as described in the above PRD:

Screen Shot 2024-03-12 at 11.27.54 AM.png

Screen Shot 2024-03-12 at 11.28.26 AM.png

Screen Shot 2024-03-12 at 11.28.42 AM.png

Screen Shot 2024-03-12 at 11.28.56 AM.png

Screen Shot 2024-03-12 at 11.29.11 AM.png

Screen Shot 2024-03-12 at 11.29.35 AM.png

The interface as proposed in the above PRD:

Screen Shot 2024-03-12 at 11.51.06 AM.png

Screen Shot 2024-03-12 at 11.51.42 AM.png

Performance requirements as outlined in the above PRD

Screen Shot 2024-03-12 at 11.53.37 AM.png

Extra information on the 'Corpus Structures'

Screen Shot 2024-03-12 at 11.54.30 AM.png

The PRD contains a lot of information about eCFR

Screen Shot 2024-03-12 at 11.55.45 AM.png

Screen Shot 2024-03-12 at 11.56.00 AM.png

Screen Shot 2024-03-12 at 11.56.16 AM.png

Screen Shot 2024-03-12 at 11.56.34 AM.png

The following JSON snippet shows the label_value and children hierarchy for 45 CFR Part 164, which can be accessed here: https://www.ecfr.gov/api/versioner/v1/structure/2022-11-07/title-45.json

Screen Shot 2024-03-12 at 11.57.13 AM.png

Screen Shot 2024-03-12 at 11.57.39 AM.png

Source / Inspiration of AI Pipeline PRD

It is not mentioned in the PRD, but this PRD seems to draw heavily on an AWS document from Aug 15, 2022: https://aws.amazon.com/blogs/machine-learning/part-1-intelligent-document-processing-with-aws-ai-services/

But the Amazon flow was developed to eg analyze invoices, receitps etc. In other words, PDFs or scans where data fields will be in pre-determined spots (like eg the total cost at the end of the receipt).

Patent Application doc

Screen Shot 2024-03-12 at 12.05.26 PM.png

If I read correctly, our patent is built upon the premise that we beat the F1 scores of the (then) leading LLM models.

The patent mentions a 74,38% F1 score.

We would urgently need to validate this premise that our approach is significantly superior. The advances in LLMs have been fast, and this benchmark may no longer be valid.

Attached are some instances for MWE matching with F1 scores of 80% and higher. With general available tools.

For example:

https://aclanthology.org/2021.emnlp-main.112.pdf

Screen Shot 2024-03-12 at 12.54.54 PM.png

https://arxiv.org/pdf/2303.06623.pdf

Screen Shot 2024-03-12 at 12.57.25 PM.png

Additionally, there is this one:

https://www.researchgate.net/publication/369912067_Interpretable_Unified_Language_Checking

Many LLMs are currently benchmarked on reasoning, knowledge as well: