Medical Device Regulatory Compliance Agentic RAG AI Chatbot

Introduction:

Regulatory compliance in the medical devices industry involves navigating complex regulations such as the Medical Devices Regulation (MDR) and In-Vitro Diagnostic Regulation (IVDR). 

Regulatory Globe, a Swiss-based compliance consulting firm, recognized the need for a transformative solution to simplify compliance queries globally. They approached us to develop an AI-powered chatbot capable of answering diverse regulatory questions with precision. The result is a state-of-the-art chatbot designed to handle regulatory compliance queries efficiently, both in Europe and worldwide.

The Challenge:

Upon initiation, the chatbot encountered several significant issues:

• Inadequate Contextual Retrieval: Up to 50% of queries resulted in the response, “Apologies, I cannot answer the question as it is not present in my content.”

• Poor Follow-Up Handling: The chatbot struggled with follow-up questions, often leading to disjointed responses.

• File-Specific Query Failures: The system was unable to effectively handle queries related to specific file names.

• Incorrect Citations: Frequent inaccuracies in citation undermined the chatbot’s reliability.

Solution Overview:

To address these challenges, we implemented a multi-faceted solution leveraging advanced technologies and methodologies:

1. Technological Stack:

• Web Framework: Flask - A lightweight Python web framework designed for quick and scalable application development.

• Database: Firebase - A real-time NoSQL cloud database offering easy integration with web and mobile applications.

• Vector Database: Pinecone - A high-performance vector database used for efficient similarity searches and managing large-scale vector data.

• Large Language Model (LLM): Anthropic - Provides advanced natural language processing for accurate understanding and generation.

• Embedding Model: Voyage - Converts textual data into numerical vectors for downstream tasks.

• Cloud Storage: AWS S3 - A scalable and secure service for storing large files, such as PDFs.

2. Enhanced Retrieval Mechanisms:

• Semantic Context Chunks with Metadata: We embedded data as semantic context chunks with metadata and full-page context chunks. This approach improved content relevance by including extracted keywords.

• Advanced Retrieval Techniques: Utilized multiple retrieval methods:

• Keyword Extraction: Filters vectors based on similarity and metadata keyword matches.

• Contextual Data Retrieval: Includes complete pages and partial chunks for enhanced context.

• Context Overlap: Added leading and trailing pages to improve response continuity.

3. Agentic Approach for Improved Query Handling:

• Custom Tools for Agents:

• Filename Retrieval and Ranking: Retrieves relevant filenames for accurate file-specific queries.

• GetFileSpecificContext: Retrieves file context based on filenames to enhance accuracy.

• GetContext for Open-Ended Questions: Handles open-ended queries without file filters.

• Mitigating Instruction Hallucination: The agentic approach adheres to a structured plan, reducing inaccuracies in responses.

• Enhanced Follow-Up Handling: Reconstructs follow-up questions and provides accurate responses by searching the context using custom tools.

Detailed Implementation:

1. Document Upload:

• Single Document Upload: Users and admins can upload single documents.

• Bulk Document Upload: Admins can upload multiple documents and documents via URLs. Metadata is programmatically fetched using Anthropic.

• Uploading Process: Files are first stored temporarily, then text is extracted using LLama-Parse or Unstructured. Content is chunked using SentenceSplitter and SemanticSplitter, with the best-performing technique being SentenceSplitting (chunk_size=1500, chunk_overlap=256). Metadata is extracted and documents are uploaded to AWS S3. Chunks are converted into embeddings and stored in Pinecone, while document information is saved in Firebase.

2. URL Upload:

• Process: Webpages are converted to PDFs using WeasyPrint and processed similarly to document uploads.

3. Chat Operation:

• Fetch Recent Messages: Retrieves the last 16 messages from the conversation.

• Query Expansion: Adds context to the user query.

• Semantic Search: Conducts searches in Pinecone to find relevant chunks.

• Re-ranking: Uses Sentence Transformers Cross Encoder for improved search result accuracy.

• Information Arrangement: Prepares information for the LLM, with accurate citations using a pattern (Refer ##) to include document names and URLs.

Critical Considerations and Outcomes:

• Issues in Traditional RAG:

• Lack of Appropriate Context: Often led to incomplete answers.

• Poor Follow-Up Handling: Resulted in disconnected responses.

• File-Specific Query Failures: Poor handling of file-specific questions.

• Incorrect Citations: Reduced credibility of responses.

• Our Enhanced Solution:

• Improved Retrieval Accuracy: Enhanced with semantic context chunks and advanced retrieval techniques.

• Agentic Approach: Improved handling of file-specific queries, follow-up questions, and adherence to instructions.

• Outcome: The system now reliably provides accurate, contextually relevant responses with correct citations and improved response quality.

Value Creation:

The advanced chatbot delivers substantial value by:

• Reducing Costs: Significantly cuts down on expenses associated with hiring consultants for compliance queries.

• Enhancing Efficiency: Provides instant, accurate information, improving operational efficiency.

• Scaling Globally: Handles diverse regulatory queries across different regions, making it a versatile tool for international compliance.

Conclusion:

Regulatory Globe’s AI-powered chatbot represents a significant leap forward in regulatory compliance management. Through the integration of advanced technologies and sophisticated retrieval techniques, the chatbot has transformed how companies access compliance information, offering a reliable, cost-effective, and scalable solution. This project not only highlights the potential of AI in streamlining complex processes but also showcases its immense value in global regulatory contexts.