One of Google recent Gemini AI models scores worse on safety

One of Google recent Gemini AI models scores worse on safety

Google’s latest internal benchmarking has revealed that one of its newest Gemini AI models, Gemini 2.5 Flash, performs poorer on safety tests compared to its predecessor, Gemini 2.0 Flash. The findings come from automated evaluations that measure how safely the models generate text based on both textual and visual prompts. In essence, the most recent version is more prone to producing content that breaches established guidelines.

One of Google recent Gemini AI models scores worse on safety

The internal technical report, available on the official Google storage server, outlines that the following safety metrics were affected:

• Text-to-text safety: This metric shows how often the model violates safety guidelines when given a text prompt. Gemini 2.5 Flash scored 4.1% worse than its predecessor.
• Image-to-text safety: This value measures the model’s response to prompts using images, with Gemini 2.5 Flash falling behind by 9.6%.

A Google spokesperson confirmed these results via email, noting that while Gemini 2.5 Flash follows instructions more faithfully in certain contexts, it also tends to generate content that contravenes safety policies when explicitly asked.

Read also : 

OpenAI pledges to make changes to prevent future ChatGPT sycophancy

Balancing Instruction Following with Safety Standards

There appears to be a clear trade-off between following user instructions on sensitive subjects and adhering strictly to safety protocols. The report points out that as AI models are tuned to be more permissive—meaning they are less likely to avoid controversial topics—the risk of safety violations can increase.

This balance is especially challenging in a landscape where many companies are pushing to deliver models that offer diverse perspectives on contentious matters. As AI developers strive to meet user demands for more comprehensive responses, ensuring that these responses remain within safe boundaries becomes increasingly complex.

Questions Over Transparency and Future Implications

Critics argue that more transparency is needed regarding the detailed circumstances in which safety guidelines are breached. Experts have highlighted that while the report admits some regressions might partly stem from false positives, there is limited clarity on the severity or specifics of the violations.

This uncertainty raises concerns among independent analysts who seek to understand if the improvements in following instructions are truly worth the associated risks. For more detailed technical insights, you can review the complete technical report from Google.

Conclusion

Google’s recent benchmarking of its Gemini models underscores the delicate balance between ensuring safety and delivering on user expectations. While Gemini 2.5 Flash shows improved adherence to instructions—even when those instructions cross sensitive areas—it does so at the cost of increased safety violations. As the conversation around AI safety continues to evolve, developers and regulators alike will be watching closely to see how these trade-offs are managed in future iterations.

Read also : 

Wikipedia says it will use AI, but not to replace human volunteers