Study suggests GPT-4o may have accessed paywalled books for training

Identifying restricted content

A recent study has highlighted a significant difference in how various AI models recognize copyrighted, non-public materials. Researchers tested several models to see if they could identify specific paragraphs from a collection of 34 O'Reilly Media books. The goal was to determine if these models had been exposed to paywalled content during their training phase.

The study, titled Beyond Public Access in LLM Pre-Training Data, used a specific method called the DE-COP membership inference attack. This technique allows researchers to test whether a model recognizes material that it should not have had permission to access. By looking for these patterns, the study aimed to move beyond simply asking if the data was included to asking if the access rights were violated.

As The Cryptonomist reported, the results showed a marked disparity between OpenAI's different offerings. While some models performed similarly to random guessing, GPT-4o demonstrated a much higher ability to identify the restricted text. This suggests that the data used to train the flagship model might have included content that was not publicly available on the open internet.

How the models compared

The researchers measured the models using an AUROC score, which is a standard metric where 0.50 represents a result of random chance. GPT-4o achieved a score of 0.82, which is well above the threshold for random guessing. This indicates a high level of recognition for the specific paragraphs from the O'Reilly Media books used in the study.

In contrast, the other models tested performed much closer to the random chance mark. GPT-3.5 Turbo recorded a score of 0.54, and GPT-4o Mini recorded a score of 0.56. These figures suggest that these models did not show the same level of familiarity with the paywalled content as the primary GPT-4o model.

To ensure the results were accurate, the researcher conducted several checks. He used independent paraphrasing models, specifically Gemma and Qwen, to verify the findings. He also repeated the analysis while removing one book at a time to see if the results remained consistent across the entire collection.

Variables and limitations

The study accounted for several factors that could influence the results, such as the age of the models and their size. GPT-3.5 Turbo was trained two years earlier than the newer models, which could affect its performance. Additionally, GPT-4o Mini is a smaller model, which often results in different training compositions and architectures compared to its larger counterparts.

The researchers also looked at the training cutoff dates to see if changes in language over time played a role. Both GPT-4o and GPT-4o Mini share an October 2023 training cutoff. By comparing these two, the study attempted to isolate whether the results were due to the timing of the data or the specific way the models were built.

However, the study notes that these findings remain preliminary. The researcher identified several limitations, including the relatively small sample size of 34 books. He also noted the possibility of contamination from post-cutoff control data and the inherent differences in how various models are structured and trained.

Implications for copyright

The focus on non-public content raises serious questions about the provenance of the data used to build large language models. If a model can identify specific paragraphs from a paywalled book, it suggests the data may have been ingested without proper authorization. This creates a potential legal hurdle for AI companies who rely on massive datasets to improve their products.

The study suggests that the issue is not just about whether the data is present in the training set, but whether the models had the legal right to use it. This distinction is important for publishers and creators who want to ensure their work is not used to train competing AI products without compensation or consent.

The findings could lead to increased scrutiny regarding how OpenAI and other developers source their training materials. If models continue to show high recognition of restricted content, it may accelerate the push for clearer rules on what constitutes fair use in the context of AI training.

Calls for transparency

In response to these findings, researcher Ilan Strauss is calling for specific changes in how the AI industry operates. He suggests that there should be mandatory disclosure requirements that cover the provenance of pre-training data. This would mean companies would have to be more transparent about where they get the information used to teach their models.

Strauss also advocates for the creation of formal markets for licensing training content. Instead of scraping the internet or using non-public archives, companies could enter into structured agreements with content owners. This would provide a legal framework for using high-quality, copyrighted material while ensuring that the original creators are compensated.

The study serves as a starting point for a larger conversation about the ethics of AI data collection. As models become more capable, the methods used to train them will come under greater scrutiny from both the public and legal authorities. The goal is to balance the need for high-quality training data with the rights of the people who create that content.