A new report is raising serious concerns about Google’s AI-powered search features and their impact on children. The study, conducted by Common Sense Media’s Youth AI Safety Institute, found that these tools are unsafe for young users in multiple ways.
The organization’s research tested Google’s AI Overview and AI Mode using accounts set up for 11 and 15-year-olds, and what they found was alarming.
When testers mimicked children in distress, the AI often failed to respond appropriately. It missed clear warnings of suicidal thoughts and even encouraged harmful behaviors. In one test, the AI validated purging as normal. In another, it celebrated drug use without any caution.
The AI reportedly gave dangerous advice about mental health. It reinforced manic thinking and provided outdated crisis resources, including a helpline that no longer works.
Google’s AI Mode also completed every homework assignment thrown at it. This raises questions about whether students are actually learning or just letting the tool do their work for them. A former education secretary involved with the institute said this undermines the whole purpose of schoolwork.
Answers were also said to be inconsistent. Nearly half the responses to history questions changed from one search to the next. And about a third of the citations came from social media, not trustworthy sources.
Unfortunately, parents and educators cannot turn off these AI features and the Overview appears automatically at the top of search results. The AI Mode works the same way. There are no controls to disable them while keeping regular search access.
As a result, the organization recommends Google to turn off these features by default for children’s accounts. It also asks for new controls that give families and schools more say over what kids see.
Google has responded by saying it could not confirm many of the test results. The company also argued that the queries used in the study were unusual and not typical of how people actually use Search. It defended its AI tools as helpful for learning and said safety measures are already in place.