Baidu blocks Google, Bing from scraping content amid demand for data used on AI projects

[Pelican Press 0]

This is the hidden content, please

• Sign In
• or
• Sign Up

This is the hidden content, please

• Sign In
• or
• Sign Up

blocks

This is the hidden content, please

• Sign In
• or
• Sign Up

,

This is the hidden content, please

• Sign In
• or
• Sign Up

from scraping content amid demand for data used on AI projects

********

This is the hidden content, please

• Sign In
• or
• Sign Up

search giant

This is the hidden content, please

• Sign In
• or
• Sign Up

appears to have started blocking the online search engines of Alphabet’s

This is the hidden content, please

• Sign In
• or
• Sign Up

and

This is the hidden content, please

• Sign In
• or
• Sign Up

from scraping content derived out of the mainland firm’s

This is the hidden content, please

• Sign In
• or
• Sign Up

-style service, a Post survey found.

A recent update of

This is the hidden content, please

• Sign In
• or
• Sign Up

Baike’s robots.txt – a file that tells search engine crawlers which uniform resource locators, commonly known as web addresses, can be accessed from a site – has outright blocked the ability of the Googlebot and Bingbot crawlers to index content from the ******** platform.

That update appears to have been made some time on August 8, according to records on internet archive service the Wayback Machine. It also showed that earlier on the same day

This is the hidden content, please

• Sign In
• or
• Sign Up

Baike still allowed

This is the hidden content, please

• Sign In
• or
• Sign Up

and

This is the hidden content, please

• Sign In
• or
• Sign Up

to browse and index its online repository of nearly 30 million entries, with only part of its website designated as off limits.

Do you have questions about the biggest topics and trends from around the world? Get the answers with

This is the hidden content, please

• Sign In
• or
• Sign Up

, our new platform of curated content with explainers, FAQs, analyses and infographics brought to you by our award-winning team.

This initiative shows

This is the hidden content, please

• Sign In
• or
• Sign Up

-based

This is the hidden content, please

• Sign In
• or
• Sign Up

’s increased effort to safeguard its online assets, as demand for vast troves of data have increased for training and building

This is the hidden content, please

• Sign In
• or
• Sign Up

(AI) models and applications.

That followed US social news aggregation platform and forum

This is the hidden content, please

• Sign In
• or
• Sign Up

’s move in July, when it blocked various search engines, except

This is the hidden content, please

• Sign In
• or
• Sign Up

, from indexing its online posts and discussions.

This is the hidden content, please

• Sign In
• or
• Sign Up

has a multimillion dollar deal with

This is the hidden content, please

• Sign In
• or
• Sign Up

that gives it the right to scrape the social media platform for data to train its AI services.

data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///ywAAAAAAQABAAACAUwAOw==

Since OpenAI released ChatGPT on November 30, 2022, major search platforms

This is the hidden content, please

• Sign In
• or
• Sign Up

and

This is the hidden content, please

• Sign In
• or
• Sign Up

have sought to obtain more data for use in their own generative artificial intelligence systems. Photo: Shutterstock alt=Since OpenAI released ChatGPT on November 30, 2022, major search platforms

This is the hidden content, please

• Sign In
• or
• Sign Up

and

This is the hidden content, please

• Sign In
• or
• Sign Up

have sought to obtain more data for use in their own generative artificial intelligence systems. Photo: Shutterstock>

Even

This is the hidden content, please

• Sign In
• or
• Sign Up

to its internet-search data, which it licenses to rival search engine operators, if they did not stop using it as the basis for their chatbots and other

This is the hidden content, please

• Sign In
• or
• Sign Up

(GenAI) services, according to a Bloomberg report.

By comparison, the ******** version of online encyclopaedia

This is the hidden content, please

• Sign In
• or
• Sign Up

has 1.43 million entries to date, which are made accessible to search engine crawlers.

Story continues

Following

This is the hidden content, please

• Sign In
• or
• Sign Up

Baike’s robots.txt update, the Post’s survey of

This is the hidden content, please

• Sign In
• or
• Sign Up

and

This is the hidden content, please

• Sign In
• or
• Sign Up

on Friday found many entries – probably from older cached content – from the

This is the hidden content, please

• Sign In
• or
• Sign Up

-style service still come up in the US search platforms’ results.

Representatives from

This is the hidden content, please

• Sign In
• or
• Sign Up

,

This is the hidden content, please

• Sign In
• or
• Sign Up

and

This is the hidden content, please

• Sign In
• or
• Sign Up

did not immediately reply to requests for comment on Friday.

More than two years after the groundbreaking launch of

This is the hidden content, please

• Sign In
• or
• Sign Up

‘s

This is the hidden content, please

• Sign In
• or
• Sign Up

, many large AI developers around the world are striking deals with content publishers for access to quality content to for their GenAI projects.

GenAI refers to the algorithms and services, such as ChatGPT, that are used to create new content, including audio, code, images, text, simulations and videos.

OpenAI, for example, in June forged a deal with ********* news magazine Time that gives it access to all the archived content from more than 100 years of the publication’s history.

This article originally appeared in the

This is the hidden content, please

• Sign In
• or
• Sign Up

, the most authoritative voice reporting on China and Asia for more than a century. For more SCMP stories, please explore the

This is the hidden content, please

• Sign In
• or
• Sign Up

or visit the SCMP’s

This is the hidden content, please

• Sign In
• or
• Sign Up

and

This is the hidden content, please

• Sign In
• or
• Sign Up

pages. Copyright © 2024 South China Morning Post Publishers Ltd. All rights reserved.

Copyright (c) 2024. South China Morning Post Publishers Ltd. All rights reserved.

This is the hidden content, please

• Sign In
• or
• Sign Up

#

This is the hidden content, please

• Sign In
• or
• Sign Up

#blocks #

This is the hidden content, please

• Sign In
• or
• Sign Up

#

This is the hidden content, please

• Sign In
• or
• Sign Up

#scraping #content #demand #data #projects

This is the hidden content, please

• Sign In
• or
• Sign Up

This is the hidden content, please

• Sign In
• or
• Sign Up

For verified travel tips and real support, visit: https://hopzone.eu/