In the world of AI, competition is fierce, and every new release is met with anticipation and excitement. Google recently introduced its highly anticipated AI model, Google Gemini, as a competitor to OpenAI GPT-3.5 Turbo. However, researchers from Carnegie Mellon University and BerriAI have found that Gemini Pro, the most powerful version available to consumers, lags behind GPT-3.5 Turbo in terms of performance across various tasks. Let’s dive into the details.
Table of contents
Gemini Pro vs GPT-3.5 Turbo: A Comparative Analysis
The researchers conducted an in-depth analysis of Gemini’s language abilities. Their findings, published on arXiv.org, revealed that Gemini’s Pro model lags behind OpenAI’s GPT-3.5 Turbo in most tasks.
Testing Methodology
The researchers tested four different language models: Google Gemini Pro, OpenAI GPT-3.5 Turbo, GPT-4 Turbo, and Mistral 8x7B. Using an AI aggregator site called LiteLLM, they ran these models through various prompts. Evaluation was done using the same prompts and methodology across models
Evaluation on Different Benchmarks
1. Knowledge-based Question Answering (MMLU)
Gemini Pro achieved slightly lower accuracy than GPT 3.5 Turbo on MMLU. Gemini Pro showed bias towards the last multiple-choice answer. It underperformed on tasks like formal logic and mathematics

2. General Reasoning (BIG-Bench Hard)
Again, Gemini Pro was slightly worse than GPT 3.5 Turbo overall. It struggled more on longer questions and tasks requiring state-tracking

3. Mathematical Reasoning
Gemini Pro underperformed GPT models on GSM8K, SVAMP and ASDIV. All models did well on MAWPS, but Gemini Pro was still slightly worse. It struggled more when answers had many digits.

4. Code Generation
Gemini Pro achieves a Pass@1 lower than GPT 3.5 Turbo and much lower than GPT 4 Turbo on both tasks. The results demonstrate that Gemini’s code generation capabilities still have room for improvement.

Research Results and Analysis
Across 10 tasks, Gemini Pro was consistently comparable but inferior to GPT 3.5 Turbo. GPT 4 Turbo performed the best overall, followed by GPT 3.5 Turbo. Gemini Pro scored 64.12% correct answers, trailing GPT-3.5 Turbo’s 67.75%. More concerningly, it lagged far behind GPT-4 Turbo’s leading 80.48% accuracy.

model is underlined.
A deeper analysis found Gemini displayed a bias towards final answers and declined questions in key domains like medicine due to its restrictions. This suggests opportunities to refine its training approach.
When prompted without options, Gemini again lags behind GPT-3.5 Turbo on most challenges, especially longer queries. Areas, where Gemini Pro excelled were long reasoning chains and language generation.
Overall Result
Despite months of development, Google’s latest model performs worse than OpenAI’s older, free model. This unexpected outcome raises questions about the effectiveness of Google’s AI product releases.
Anticipated Google Gemini Ultra Release
Naturally, the research findings must have come as a disappointment to Google and its dedicated team of researchers. However, Google maintains that their own research shows Gemini Pro outperforms GPT-3.5 and that an even more powerful version, Gemini Ultra, is set to be released in early 2024, surpassing the performance of GPT-4 in internal tests.
According to Google’s technical paper, Gemini Pro’s performance exceeds that of inference-optimized models like GPT-3.5. Additionally, Gemini Ultra showcases impressive reasoning and STEM competencies, opening up possibilities for advancements in the educational domain.
Implications and Future Possibilities
The comparative analysis between Google Gemini and GPT-3.5 Turbo highlights the challenges in developing and evaluating language models. The researchers acknowledged the need for more robust evaluation benchmarks to eliminate data contamination and ensure accurate results. Nevertheless, the findings provide valuable insights into the capabilities of these models and their potential impact on real-world tasks.
Final Takeaway
As AI continues to evolve, the competition between Google and OpenAI drives innovation and pushes the boundaries of what these language models can achieve. The ability to understand complex concepts and provide personalized learning experiences holds great promise for the future of AI in education. For now, independent assessment shows GPT-3.5 is the top choice based on these early tests as Google Gemini Lags Behind GPT-3.5 Turbo. Continued progress will be crucial for Gemini to compete at the cutting edge of conversational AI.
| Read More from Google:
- Google Imagen 2: A Game-Changing AI Tool That Takes Photorealism to New Heights
- MedLM by Google is Transforming the Healthcare Industry
- Google vs Epic Games: Google Play Store Monopoly Has Ended
- Google New Gemini 1.0 Model Outperforms GPT-4
- Google Bard Update: Get Instant, Detailed Responses from Any YouTube Video
- RIP Reporters? Google Unveils Genesis AI Writing Tool
- Google StyleDrop: A Game-Changing AI Image Generator
- AI in Google Workspace: Google Sheets, Slides, Docs, and Gmail
| Read More from OpenAI:
- OpenAI Released its Official ChatGPT Prompting Guide
- Has OpenAI Really Released GPT-4.5-Turbo?
- Hear This: OpenAI’s Whisper v3 is the Ultimate Speech Decoder
- OpenAI’s ChatGPT Voice Chat Feature is Now Available for Free!
- Custom GPTs: How to Create a Custom GPT in OpenAI ChatGPT
- What Is OpenAI GPT-4 Turbo?
- OpenAI DevDay All Key Highlights
- OpenAI’s GPT-4 Performance: A Deep Dive into Recent Studies






