Google recently unveiled its new generative AI system Gemini to power products like its ChatGPT rival Bard. However, early anecdotal testing suggests the model dubbed Gemini Pro still struggles with accuracy and reasoning compared to competitors like GPT-3.5 and ChatGPT.
Multiple users testing Bard, which runs on a basic version of Gemini, found the AI getting basic facts wrong, like 2023 Oscar winners and straightforward French vocabulary. It failed to produce functioning code in some cases and avoided summarizing current news topics, deferring users to search instead. One researcher managed to circumvent Gemini’s safety guardrails, getting inappropriate content generation related to theft and violence.
Comparatively, ChatGPT provided accurate award show and language information. It also readily produced news briefs and code. ChatGPT’s safety filters proved harder to bypass.
Google claims Gemini Pro meets or beats GPT-3.5 and ChatGPT capabilities in areas like summarization, brainstorming and content writing. But these early hands-on tests indicate otherwise. The model seems prone to confusion and hesitant to tackle certain prompts. This suggests it requires further training on reasoning, fact recall and safety protocols.
Fortunately, Gemini Pro is not the full version of Google’s Gemini AI. Gemini Ultra slated to launch in 2023 may demonstrate the advertised improvements in understanding and planning. However, Google faces pressure to enhance accuracy, content filtering and multilinguial support before wide deployment. Rushing Gemini into products risks reputational damage if core capabilities remain lacking.
For now, ChatGPT continues leading conversational AI on both performance and safety fronts. But the AI race remains in early stages. With refinement, Google’s Gemini may yet evolve into a top contender.