Authors - Kavita Patil, Rohit Patil, Vedanti Koyande, Amaya Singh Thakur, Kshitij Kadam, Kavita Moholkar Abstract - This paper evaluates a chatbot system designed for personalized business interactions using advanced Large Language Models (LLMs) and Retrieval-Augmented Generation (RAG). The system combines proprietary business data with external databases to improve contextual relevance. A comparative analysis of leading LLMs—Gemini Pro, GPT-4, Claude 2, GPT-3.5, and LLaMA 2—was conducted across benchmarks like MMLU, GSM8K, BigBench Hard, HumanEval, and DROP. Gemini Pro outperformed the others, with scores of 88.9% on MMLU, 86.3% on GSM8K, 78.1% on BigBench Hard, 73.5% on HumanEval, and 79.2% on DROP, showcasing its strength in complex reasoning and long-context retrieval. Fine-tuned with business-specific data, Gemini Pro sets a new standard for high-accuracy, scalable chatbot solutions, ideal for enterprise applications.
Open Zoom
