[AI]Baidu restricts Google and Bing from scraping content for AI training

[ChatGPT 0]

******** internet search provider

This is the hidden content, please

• Sign In
• or
• Sign Up

has

This is the hidden content, please

• Sign In
• or
• Sign Up

to prevent

This is the hidden content, please

• Sign In
• or
• Sign Up

and

This is the hidden content, please

• Sign In
• or
• Sign Up

This is the hidden content, please

• Sign In
• or
• Sign Up

from scraping its content.

This change was observed in the latest update to the

This is the hidden content, please

• Sign In
• or
• Sign Up

Baike robots.txt file, which denies access to Googlebot and Bingbot crawlers.

According to the Wayback Machine, the change took place on August 8. Previously,

This is the hidden content, please

• Sign In
• or
• Sign Up

and

This is the hidden content, please

• Sign In
• or
• Sign Up

search engines were allowed to index

This is the hidden content, please

• Sign In
• or
• Sign Up

Baike’s central repository, which includes almost 30 million entries, although some target subdomains on the website were restricted.

This action by

This is the hidden content, please

• Sign In
• or
• Sign Up

comes amid increasing demand for large datasets used in training artificial intelligence models and applications. It follows similar moves by other companies to protect their online content. In July,

This is the hidden content, please

• Sign In
• or
• Sign Up

blocked various search engines, except

This is the hidden content, please

• Sign In
• or
• Sign Up

, from indexing its posts and discussions.

This is the hidden content, please

• Sign In
• or
• Sign Up

,

This is the hidden content, please

• Sign In
• or
• Sign Up

, has a financial agreement with

This is the hidden content, please

• Sign In
• or
• Sign Up

for data access to train its AI services.

According to sources, in the past year,

This is the hidden content, please

• Sign In
• or
• Sign Up

considered restricting access to internet-search data for rival search engine operators; this was most relevant for those who used the data for chatbots and generative AI services.

Meanwhile, the ********

This is the hidden content, please

• Sign In
• or
• Sign Up

, with its 1.43 million entries, ******** available to search engine crawlers. A survey conducted by the South China Morning Post found that entries from

This is the hidden content, please

• Sign In
• or
• Sign Up

Baike still appear on both

This is the hidden content, please

• Sign In
• or
• Sign Up

and

This is the hidden content, please

• Sign In
• or
• Sign Up

searches. Perhaps the search engines continue to use older cached content.

Such a move is emerging against the background where developers of generative AI around the world are increasingly working with content publishers in a bid to access the highest-quality content for their projects. For instance, relatively recently, OpenAI signed an agreement with Time magazine to access the entire archive, dating back to the very first day of the magazine’s publication over a century ago. A similar partnership was

This is the hidden content, please

• Sign In
• or
• Sign Up

with the Financial Times in April.

This is the hidden content, please

• Sign In
• or
• Sign Up

’s decision to restrict access to its

This is the hidden content, please

• Sign In
• or
• Sign Up

Baike content for major search engines highlights the growing importance of data in the AI era. As companies invest heavily in AI development, the value of large, curated datasets has significantly increased. This has led to a shift in how online platforms manage access to their content, with many choosing to limit or monetise access to their data.

As the AI industry continues to evolve, it’s likely that more companies will reassess their data-sharing policies, potentially leading to further changes in how information is indexed and accessed across the internet.

(Photo by

This is the hidden content, please

• Sign In
• or
• Sign Up

)

See also:

This is the hidden content, please

• Sign In
• or
• Sign Up

This is the hidden content, please

• Sign In
• or
• Sign Up

Want to learn more about AI and big data from industry leaders? Check out

This is the hidden content, please

• Sign In
• or
• Sign Up

taking place in Amsterdam, California, and London. The comprehensive event is co-located with other leading events including

This is the hidden content, please

• Sign In
• or
• Sign Up

,

This is the hidden content, please

• Sign In
• or
• Sign Up

,

This is the hidden content, please

• Sign In
• or
• Sign Up

, and

This is the hidden content, please

• Sign In
• or
• Sign Up

.

Explore other upcoming enterprise technology events and webinars powered by TechForge

This is the hidden content, please

• Sign In
• or
• Sign Up

.

The post

This is the hidden content, please

• Sign In
• or
• Sign Up

appeared first on

This is the hidden content, please

• Sign In
• or
• Sign Up

.

This is the hidden content, please

• Sign In
• or
• Sign Up