...
Development Roadmap
Next steps would entail:
...
Over a period 0f 4-6 months: Going from a proof-of-concept to a production-ready platform:
Proposed way forward:
Refinement of Retrieval Mechanism: Enhance the AI's ability to accurately source and retrieve relevant data.
Re-visit Architecture
Evaluate vectorstores (Vectorstore/Knowledge Graph)
Set up Semantic Retrieval
Enrichment of Sources: Expand and diversify the data sources to improve the tool's comprehensiveness.
Fine-Tuning for Quality: Optimize the AI algorithms to ensure high-quality, accurate compliance term generation.
Set up evaluation RAG Triad with set of evaluation questions
Iterate with:
Re-rank,
Sentence-window and
Auto-merging Retrieval
Self-evaluation by LLM
...
Milestone after 3 months: Internal Beta Roll-out:
POC to Production in AWS: Transition the Proof of Concept (POC) into a full-scale production environment on AWS.
Integration with Mapper Team: Provide the tool to the mapper team for human validation, ensuring accuracy and reliability.
...
Milestone after 4-5 months: External Beta Roll-out:
Following successful internal use and quality benchmarks, release the tool externally as a feature of the Dictionary solution.
...
Strategically, this smallish genAI solution is a first step, a beachhead in our potential approach to meeting the growing demand for genAI solutions in the compliance industry.
Preliminary RICE evaluation
Reach & Impact
Small w/medium impact to large w/ small impact
Confidence:
Validated customer and internal need
Effort:
Small to Medium
Appendix:
Proof-of-concept
...
Accuracy Rate:
Definition Correctness: Percentage of terms where the generated definition accurately reflects the intended meaning.
Reference Relevance: Proportion of contextually relevant references to the generated terms.
Error Rate:
Misinterpretation Frequency: Track the frequency of incorrect interpretations or irrelevant definitions generated.
Inconsistency Detection: Measure instances where the tool provides varying quality across similar requests.
Response Time:
Generation Speed: Monitor the average time to generate a term and its definition, ensuring it meets efficiency standards.
Usage Metrics:
Adoption Rate: Track the number of active users and frequency of use, indicating the tool's perceived value.
Repeat Usage: Measure how often users return to the tool, indicating reliance and satisfaction.
Benchmarking:
Comparison with Manual Processes: Compare the quality of terms generated by the tool against those created manually.
Competitor Comparison: Regularly compare the tool's output quality against similar offerings in the market.
User feedback:
