‘We now, as journalists, have a capability to search through data sets in ways that previously were not possible and really give our journalists a whole new superpower.’
As The New York Times’ first editorial director of artificial intelligence initiatives, Zach Seward leads a small multidisciplinary team called AI Issues, which applies AI tools across newsroom operations. “We’re a cross-functional team of journalists who bring other skills like machine learning, engineering, product design, and old fashioned editing to the table,” he explains.
Seward, who co-founded Quartz – where he served as CEO and editor in chief before joining NYT – says their focus includes everything from internal workflows and processes to prototyping, and exploring the potential use of AI “for how it might be consumed in the future.”
He shared how the legacy newspaper’s AI applications have matured beyond experimental to becoming institutionalised in their workflow – and what this means for investigative reporting – at WAN-IFRA’s recent Congress in Krakow.
In working with journalists to overcome their biggest challenge – unpacking massive reams of data – AI Issues found “repeatable patterns” that led to them building internal tooling to help support journalists, leading to a veritable investigative toolkit.
“LLMs are particularly valuable for analysing massive document or video collections that would be impossible for humans to review completely,” explains Seward. “We now, as journalists, have a capability to search through data sets in ways that previously were not possible and really give our journalists a whole new superpower.”
The AI Toolkit for Investigations stems from these four repeatable patterns that emerged while using AI in investigations: bias-based search, diving for pearls, augmenting datasets, and end-to-end verification.
Vibes-based search
Also known as semantic or vector search – or in the past, as natural language processing (NLP), vibes-based search uses vector embeddings to find semantically similar content beyond exact keyword matches. This enables journalists to discover connections and patterns that would be missed by traditional search, and is particularly valuable for identifying variations in terminology, says Seward.
“It’s not as simple as simply looking up one specific word in a data set… we would have found maybe one, two or three examples… By using semantic search, we were able to find a much, much wider swath of examples.”
How it works: The math behind semantic search
Semantic search works by encoding text as numerical vectors in multi-dimensional space:
- Text is converted into numerical representations (embeddings)
- Similar concepts cluster together in this mathematical space
- Distance calculations reveal semantic relationships between terms
- This enables “equations with text” – for example: [king] – [man] + [woman] ≈ [queen]
“What they’re doing, in essence, is encoding text or other types of media as huge arrays of numbers. And because you are creating numbers, you can start to do math with them,” explains Seward.
Diving for pearls
This data extraction tool applies AI to extract insights from overwhelming volumes of content by using journalist expertise to guide the AI through carefully crafted prompts.
It structures findings in spreadsheets organised by topics of interest. NYT, for example, analysed 500+ hours of video leaked from an election interference group.
“The first step that the AI helped with was just transcribing the videos into text, which was still 5 million spoken words – far more than we could deal with in the time allotted. But crucially, we didn’t just have the source material; we had two reporters who’ve been covering democracy and threats to elections in the US for collectively more than 18 years. So that proved to be a powerful combination.”
“The problem that our reporters typically have when they call in members of my team is… too much data. They’re sitting on tens of thousands of documents or hundreds of hours of video that are truly impossible for any journalist to go through themselves,” notes Seward.
Augmenting Datasets
Seward illustrated how NYT employs optical character reasoning (OCR) to analyse complex document sets, including handwritten notes.
“The newest foundational models from all of the major LLM developers have really taken OCR to the next level, and it’s now possible to do all sorts of really complex and messy analysis on all sorts of messy data sets,” he explains.
They also developed tools for monitoring “manosphere” content creators and generating daily summaries – and screen 10,000 individuals for a Puerto Rico tax investigation.
End-to-end verification
“It’s an axiom in our newsroom that you should never trust an LLM… Before any of that makes its way into a story, we’re always going all the way back to the original material,” says Seward, who advises newsrooms to design their own systems because: ”It’s crucial that we design these systems in a way that the original material is accessible and tied directly to the analysis.”
The NYT’s verification tool uses spreadsheet formats to link AI-generated insights back to primary sources, and requires journalists to review original material before publication.
The ‘Cheat Sheet’ tool
Seward stressed that “all AI-assisted reporting follows strict verification protocols, with journalists always returning to original sources.”
Hence, the NYT’s cheat sheet tool – still in early stages of development, this helps journalists make sense of large datasets because: “I can’t Google 10,000 people… but of course a machine can do that.”
The Cheat Sheet is essentially a spreadsheet-based interface accessible to all reporters that:
- Transforms unstructured data into structured formats
- Extracts quotes matching specific criteria
- Performs translations and summarisation
- Connects findings directly to source material for verification
“We’re really trying to connect the original material to whatever downstream analysis we’ve done, so that a reporter who wants to use a quote from a video can directly verify that this is exactly what was said, make sure no context is lost, and do all of the traditional verification that we would, for any other sort of reporting.”
