Google’s new AI-Based Notes App Struggles With What’s True

Google is trying to sprinkle a bit of AI spice into the staid flavor of note-taking apps with NotebookLM, an AI assistant app focused on helping writers, educators, students, or other creative types.

After leaving the app in beta for a few months, Google released NotebookLM to the wider public Friday, along with a few new features and tweaks. The tool is meant to collect information from multiple documents and use it to synthesize information.

Google previously called it a “virtual research assistant” that can connect the dots across multiple documents. Say you take some of your own class notes from your most recent lesson, then combine them with a few assigned reading materials. You could ask NotebookLM to summarize the lesson’s main points or drill down into a few finer details of a particular subject.

Users can pull in multiple documents through their Google Drive or add in other PDFs or copied text. The app takes a few seconds to parse through the documents, but after that, you can use a simple prompt menu to ask questions about the material or even craft new documents based on the information contained within.

Or at least that’s the idea. Users can save some responses on a “Noteboard” and then use the “Add note” button to craft their own reminders or thoughts.

Google first showed off its NotebookLM concept back during Google I/O. This was when Google’s big AI announcement was its PaLM 2 model and not the newfangled Gemini AI. It was called Project Tailwind internally, but it was released as NotebookLM to a select group of U.S.-based writers to put the app through its paces. Now that it’s partially powered with Gemini, Google released the AI app to the public under its usual “experimental” qualifier, noting how it’s liable to make mistakes.

Well, in my time with it, I can’t say I’m impressed. NotebookLM lies, makes stuff up, and does it in some of the strangest ways possible. As an experiment, I plugged in a story I wrote a few years ago, along with a few supplementary documents, into NotebookLM and asked it to craft a query letter to send to literary agents.

It got some parts right about my story but then misinterpreted whole swathes of its structure and plot. It then tried to lie and say I was a graduate of the University of California Berkeley (I’m not) and that I’m a member of the Society of Children’s Book Writers and Illustrators. Mind you, I can’t remember the last time I even touched a children’s book.

The more information NotebookLM has, the better, but even then, it’s far from perfect. The system offers citations for its claims for users to check its work, but the highlighted text from the original documents does not always relate to the AI-generated text. Worse, I found the system misinterpreted whole aspects of the words I provided.

The system was insultingly bad at times, no less lazy than a grade-schooler trying to write a last-minute book report by skim-reading entire chapters. If any student thinks this will be the new AI tool they’ll use to cheat, know the tool’s liable to give you a big fat “F.”

Despite how many jotting apps are around, the best note-taking app is likely the app you’re currently using (though probably not Evernote if you’re using the free version). That is to say, there’s really not much of a difference between the various big-brand note-taking apps, especially if you only need to jot down fresh ideas or your shopping list.

The idea of an AI notetaking app sounds good enough, but NotebookLM still needs some heavy refinement and someone to check its work.