Chinese AI firms linked to using pirated Korean books for training

Evidence of pirated data use

Chinese technology companies Huawei and DeepSeek have been linked to using pirated books by Korean authors to train their artificial intelligence models. Analysis of technical reports suggests that these firms utilized datasets containing unauthorized copies of literary works. This includes books by prominent figures such as Nobel laureate Han Kang.

Huawei's Pangu Sigma model reportedly utilized a dataset known as The Pile. This collection consists of approximately 800 gigabytes of data, including academic papers and web documents. However, it also contains Books3, a collection of pirated e-books that other companies have also downloaded without permission.

The Books3 collection includes English translations of at least 19 books by 10 different Korean authors. Notable titles include 'Human Acts' and 'The Vegetarian' by Han Kang, as well as works by Hwang Sok-yong and Shin Kyung-sook. While it is not certain if Huawei used every book in the set, the inclusion of these titles in the training source is confirmed.

DeepSeek and Anna's Archive

DeepSeek has also disclosed its use of large-scale e-book collections for its DeepSeek-VL model. The company reported obtaining 860,000 English-language and 180,000 Chinese-language e-books from Anna's Archive. This website is known for providing access to pirated content and has acknowledged its role in the AI training boom.

Anna's Archive noted that while many U.S. companies reconsidered using their data after learning about its illegal status, Chinese firms remained enthusiastic. The site claimed that virtually all major companies building large language models contacted them for data. This suggests a disparity in how different regions approach the legality of training materials.

Researchers have found indirect evidence that DeepSeek may have specifically included Korean literary works in its training. A study by researchers from Columbia University and Carnegie Mellon University found that DeepSeek 3.1 reproduced a sequence of 19 consecutive words from Han Kang's 'The Vegetarian.' The words appeared after the model underwent a process called fine-tuning, which is used to refine a model's specific behaviors.

Industry-wide training habits

The use of pirated books for AI development is not an isolated incident involving only a few companies. Copyright lawsuits have revealed that other major players, including OpenAI and Meta, also downloaded pirated copies of books without authorization. These actions occurred during a period when the industry was focused on improving model performance at any cost.

Experts suggest that using publicly available datasets was a standard practice during the early stages of AI development. Some of these datasets naturally contained pirated content because the developers were not always considering the legal implications. It was common enough for researchers to publicly disclose these datasets as sources in their scientific papers.

As legal scrutiny has increased, the industry's behavior is beginning to shift toward more secretive practices. Choo Jae-gul, a professor at KAIST, noted that many major technology companies have started keeping their training data confidential. This move comes as the legal consequences of using copyrighted digital books become more apparent to the public and to courts.

Impact on authors and creators

The inclusion of pirated books in training sets poses a direct challenge to the rights of authors and publishers. When models are trained on copyrighted works without permission, it can undermine the ability of creators to profit from their intellectual property. This is particularly significant for international authors whose works are translated and distributed globally.

The legal landscape is already beginning to react to these practices through various lawsuits. For example, the 'Bartz v. Anthropic' case centered on the use of Books3 and resulted in a settlement where Anthropic agreed to pay copyright holders $3,000 per book. Such settlements highlight the growing recognition of the value of individual literary works in the AI era.

The creators of The Pile originally included digital books because they were considered extremely valuable for research. They specifically wanted to study how models handle long contexts and maintain narrative coherence. However, the ease of access to these pirated materials has created a complex environment where the benefits of research often clash with the rights of the original creators.