Microsoft has just announced a huge update to Bing that overhauls the search engine to put AI-powered answers first.
This means that when a search query is entered, the results page will pop up with a primary AI-generated answer detailing all the curated sources that have been tapped to get that result. You’ll still get the traditional search results on the Bing search page, but they will be presented to the side of the AI-generated material (in a smaller right-hand panel).
This change is currently rolling out to a small number of Bing users, but it’ll presumably become more widely available before too long. From what we can tell there’s no obvious way to turn off the AI results if you wanted to do so.
Crawling the web is expensive, and plenty of bigger sites have proprietary deals (Reddit serving exclusive results to Google for instance). Also, since actively hosting data costs money, lots of sites have archived or compressed their offerings. Others have set up higher and higher paywalls, to limit what anyone without a subscription can see.
The end result is a treasure trove of data that is inaccessible to modern crawlers and scrapers. If you’re not tapping into one of the big search engine catalogs, you’re going to miss a lot of the more attractive results.
Then there’s the problem of AI crap filling up lots of the spaces that used to be mineable for search results. This isn’t a problem unique to Bing. AI contamination is everywhere and crawlers can’t avoid it easily. What’s a modern search engine to do?