DIRS.INFO The most relevant business & tech news

☆ Like digital locusts, OpenAI and Anthropic AI bots cause havoc and raise costs for websites
▼ + stars: | 2024-09-19 | by ( Darius Rafieyan | ) www.businessinsider.com time to read: +11 min

One study estimated that the world's supply of usable AI training data could be depleted by 2032. An OpenAI spokesperson said the company's bot was querying Coates' website roughly twice per second. But hungry AI botnets scrape first, ask questions later. "Utterly sick"Coates says his Game UI Database is back up and running and he continues to add to it. But Coates' story is emblematic of a bigger question: When AI comes to change the world, who bears the cost?

Persons: —, Edd Coates, Coates, Jay Peet, We've, Joshua Gross, Gross, Jennifer Martinez, Anthropic, Senecal, botnets, robots.txt, Roberto Di Cosmo, Di Cosmo, Tania Cohen, Cohen Organizations: Service, Business, OpenAI, Google, Microsoft, Software Heritage, Software Locations: It's

☆ The Data That Powers A.I. Is Disappearing Fast
▼ + stars: | 2024-07-19 | by ( Kevin Roose | ) www.nytimes.com time to read: +1 min

Over the past year, many of the most important web sources used for training A.I. models have restricted the use of their data, according to a study published this week by the Data Provenance Initiative, an M.I.T.-led research group. The study, which looked at 14,000 web domains that are included in three commonly used A.I. training data sets, discovered an “emerging crisis in consent,” as publishers and online platforms have taken steps to prevent their data from being harvested. The researchers estimate that in the three data sets — called C4, RefinedWeb and Dolma — 5 percent of all data, and 25 percent of data from the highest-quality sources, has been restricted.

☆ OpenAI and Anthropic are ignoring an established rule that prevents bots scraping online content
▼ + stars: | 2024-06-21 | by ( Kali Hays | ) www.businessinsider.com time to read: +1 min

AdvertisementThe world's top two AI startups are ignoring requests by media publishers to stop scraping their web content for free model training data, Business Insider has learned. OpenAI and Anthropic have been found to be either ignoring or circumventing an established web rule, called robots.txt, that prevents automated scraping of websites. TollBit, a startup aiming to broker paid licensing deals between publishers and AI companies, found several AI companies are acting in this way and informed certain large publishers in a Friday letter, which was reported earlier by Reuters. The letter did not include the names of any of the AI companies accused of skirting the rule. This story is available exclusively to Business Insider subscribers.

Persons: — Organizations: Service, Business, Reuters

☆ Meta's AI chatbot says it was trained on millions of YouTube videos. Meta didn't deny it but said the bot could be inaccurate.
▼ + stars: | 2024-06-04 | by ( Kali Hays | ) www.businessinsider.com time to read: +5 min

Read previewThe Meta AI chatbot is more willing to share what data it was trained on than Meta is. It expanded Meta AI in April as a chat and image generator function across all its apps, including Instagram and WhatsApp. Meta AI told Business Insider that it was trained on large datasets of transcriptions from YouTube videos. Meta AI initially said its training data included a third-party dataset of 3.7 million transcribed YouTube videos. In responding to further queries about its YouTube training data, Meta AI said its training data included another, larger dataset of transcriptions from 6 million YouTube videos also compiled by a third party.

Persons: —, hasn't, Meta, OpenAI, Meta AI's, We'll, Meta's chatbot, Google's GoogleBot, Kali Hays Organizations: Service, Meta, Facebook, Business, TED, YouTube, NBC News, CNN, Financial Times, US Locations: khays@businessinsider.com

☆ Top websites block Google from training AI models on their data. Nowhere near as much as OpenAI, though.
▼ + stars: | 2024-03-14 | by ( Hugh Langley | ) www.businessinsider.com time to read: +4 min

Google launched a new tool that lets publishers opt out of training Google's AI models. It turns out that all this content has been stored in datasets that are the foundation for training powerful AI models, including those from OpenAI, Google, Meta, and others. Part of Google's response has been to launch a new tool that lets websites block the company from using their content for training AI models. BI asked Originality.ai CEO Jonathan Gillham why Google-Extended is being used less than other AI training data-blockers. It's unclear if the company will launch this fully in the future, or how much different it will be from the traditional Google search engine.

Persons: —, There's, Robots.txt, Jonathan Gillham, Gillham, Axel Springer Organizations: Google, Service, New York Times, CNN, BBC, Business Locations: Chicago

☆ OpenAI offers a way for creators to opt out of AI training data. It's so onerous that one artist called it 'enraging.'
▼ + stars: | 2023-09-29 | by ( Kali Hays | ) www.businessinsider.com time to read: +5 min

Artists and image owners can now ask OpenAI to remove their images from DALL-E training data. OpenAI recently unveiled a new form that image owners and creators can use to request that owned or copyrighted images be removed from DALL-E training data. AI models need high quality, and human generated training data to perform well. "Enraging"Toby Bartlett, an artist with a namesake consulting firm, wrote on Threads that OpenAI's DALL-E opt-out process is "enraging." Or, as OpenAI put it, its model will have "learned from their training data" and be able to "retain the concepts that they learned."

Persons: —, OpenAI, Toby Bartlett, OpenAI's, Greg Madhere, He's, it's, we've, We've, Kali Hays Organizations: Service, Georgia O'Keeffe Museum, US Copyright, Twitter Locations: khays@insider.com, @hayskali

☆ OpenAI's GPTBot and other AI web crawlers are being blocked by even more companies now
▼ + stars: | 2023-09-28 | by ( Kali Hays | ) www.businessinsider.com time to read: +7 min

Unique, high quality data, mainly scraped from the web, is vital to the performance of AI models. AdvertisementAdvertisementMore and more companies are trying to avoid having their data freely scraped and saved by web crawlers working for the benefit of AI models. Last month, OpenAI last revealed its own crawler, GPTBot, saying it would respect robots.txt, a decades-old method through which a website can tell a web crawler to ignore it. Many more companies are now also blocking CCBot, a web crawler used by Common Crawl. AdvertisementAdvertisementSee below for a full list of the biggest websites now blocking GPTBot and CCBot as of Sept. 22:Blocking GPTBotamazon.comquora.comnytimes.comtheguardian.comshutterstock.comwikihow.comcnn.comsciencedirect.comusatoday.comhealthline.comstackexchange.comalamy.comscribd.comwebmd.combusinessinsider.comdictionary.comreuters.comwashingtonpost.commedicalnewstoday.comnpr.orgcbsnews.comgoodhousekeeping.comamazon.co.uktumblr.comlatimes.cominsider.comglassdoor.comvocabulary.cominvestopedia.comslideshare.netamazon.decosmopolitan.comnbcnews.comindiamart.comstackoverflow.comhindustantimes.combloomberg.comcnbc.compeople.comtvtropes.orgamazon.invimeo.comverywellhealth.comikea.comespn.comindianexpress.comthesaurus.compbs.org123rf.comwattpad.comvariety.comtoday.compopsugar.comthespruce.comuol.com.bramazon.frgeeksforgeeks.orgelle.comeconomictimes.compcmag.comtheverge.comallrecipes.comthoughtco.comrollingstone.comwired.comnextdoor.comhollywoodreporter.comabc.net.auew.comamazon.canews18.comwomenshealthmag.comrateyourmusic.comamazon.co.jptechradar.comairbnb.comndtv.comlifewire.comtomsguide.comvulture.comeverydayhealth.compolygon.comtheconversation.comesquire.comprnewswire.combillboard.commenshealth.commetro.co.ukcountryliving.commashable.comgamesradar.comthehindu.comtimesofindia.comdeadline.comharpersbazaar.commedscape.comnymag.comrefinery29.comradiotimes.comcbssports.comtandfonline.comtheatlantic.comtrulia.comamazon.espinterest.esnationalgeographic.combhg.comeater.comsouthernliving.comhealthgrades.comvice.compicclick.combustle.comnewyorker.comeonline.comdigitalspy.comopentable.compinterest.dethepioneerwoman.comcaranddriver.combyrdie.comlivemint.commedicinenet.comteacherspayteachers.comcookpad.comthespruceeats.combizjournals.compagesjaunes.frliputan6.comdelish.commasterclass.comarchiveofourown.orgvox.comrealsimple.comaarp.orgfrancetvinfo.frpinterest.frkumparan.comtheathletic.comtravelandleisure.comvogue.comlivescience.comapartments.commarketwatch.comglamour.comamazon.itcinemablend.comthrillist.comamazon.com.brpinterest.co.ukangi.comalamy.esusmagazine.comdistractify.combbcgoodfood.comjagran.commercadolibre.com.mxandroidauthority.comcity-data.comfoodandwine.comhellomagazine.comamazon.com.augq.comingles.comamarujala.comieee.orgprevention.comstern.dekbb.comedmunds.commarthastewart.compcgamer.comjustanswer.comhealth.com20minutes.frfortune.comhomes.comscientificamerican.compopularmechanics.comverywellfit.comvanityfair.comchicagotribune.comverywellmind.comhousebeautiful.comcntraveler.comallure.comspanishdict.comneverbounce.comanswers.commoneycontrol.comarchitecturaldigest.comslate.comlonelyplanet.cominverse.comcorriere.itactu.frself.comtripsavvy.cominstyle.comeatingwell.comsuperuser.comwelt.despiegel.dewomansday.comseventeen.comhbr.orgoprahdaily.comautotrader.combonappetit.comsueddeutsche.deseriouseats.comliveabout.comseattletimes.comcoursera.orglivehindustan.comfrance24.comtownandcountrymag.comdotesports.comworldplaces.mefaz.netteenvogue.commotor1.comnj.comglamourmagazine.co.ukokdiario.combrides.comstylecaster.comalamyimages.frjagranjosh.comtheglobeandmail.comaxios.comfrancebleu.frtabelog.comthebalancemoney.comnydailynews.comsheknows.comnaomedical.comverywellfamily.comBlocking CCBot

Persons: —, OpenAI, GPTbot, Conde Nast, Masterclass, Kelly, robots.txt, verywellhealth.com, indianexpress.com Organizations: Service, Amazon, Guardian, NPR, CBS News, CBS Sports, NBC News, CNBC, Yorker, Hearst, New York Times Locations: USA, Europe, Originality.ai, androidauthority.com

☆ AI is killing the grand bargain at the heart of the web. 'We're in a different world.'
▼ + stars: | 2023-08-30 | by ( Kali Hays | Alistair Barr | ) www.businessinsider.com time to read: +14 min

AdvertisementAdvertisementAI is undermining the web's grand bargain, and a decades-old handshake agreement is the only thing standing in the way. Now, though, generative AI and large language models are changing the mission of web crawlers radically and rapidly. Without a supply of potential consumers, there's little incentive for content creators to let web crawlers continue to suck up free data online. It's also open to manipulation, especially given the voracious appetite for quality AI data. Because robots.txt is voluntary, web crawlers can also simply ignore the blocking instructions and siphon the information from a site anyway.

Persons: Microsoft's Bing, Joost de Valk, It's, de Valk, Nick Vincent, Valk, OpenAI, robots.txt, Jason Schultz, Catherine Stihler, Archie, NYU's Schultz, Steven Sinofsky, who's, Andreessen Horowitz, De Valk, Stihler Organizations: Big Tech, Google, Wordpress, NYU's Technology, Policy Clinic, AWS, Creative Commons, Creative, Microsoft, Nvidia, Star Wars, DC Comics, Warner Brothers, Marvel, Disney, Atlantic, Meta Locations: CCBot, EleutherAI

☆ The US Copyright Office just took a big step toward new rules for generative AI
▼ + stars: | 2023-08-30 | by ( Kali Hays | ) www.businessinsider.com time to read: +5 min

The US Copyright Office is taking a big step toward new rules for generative AI. AdvertisementAdvertisementThe US Copyright Office is inching closer to creating new rules and regulations around generative AI and how the technology uses the work of authors and other creators. In the government rule-making process, a public comment period typically happens before a final rule is proposed and adopted. The major tech companies behind these generative AI tools use the crawled data to train their models without paying the creators who produced the original content. More online businesses are slowly becoming aware of the degree to which the web is being scraped for the benefit of generative AI.

Persons: OpenAI's ChatGPT, Google Bard, Andreessen Horowitz, Bard Organizations: Morning, US, Google, Microsoft, Meta, New York Times, CNN, Office, Hollywood

☆ Major websites like Amazon and the New York Times are increasingly blocking OpenAI's web crawler GPTBot
▼ + stars: | 2023-08-24 | by ( Kali Hays | ) www.businessinsider.com time to read: +3 min

The top 100 sites blocking GPTBot include bloomberg.com, scribd.com, and reuters.com, as well as insider.com and businessinsider.com. Among the top 1,000 sites blocking the bot are ikea.com, airbnb.com, nextdoor.com, nymag.com, theatlantic.com, axios.com, usmagazine.com, lonelyplanet.com, and coursera.org. AdvertisementAdvertisement"GPTBot launched 14 days ago and the percentage of Top 1,000 sites blocking it has been steadily increasing," the analysis said. How these websites block GPTBot is relatively simple, even crude, depending on your perspective. When revealing the crawler, OpenAI said it would abide by robots.txt and GPTBot would not crawl websites that deploy it.

Persons: OpenAI, GPTBot, robots.txt, Stephen King, ChatGPT Organizations: Reuters, Amazon, The New York Times Locations: ChatGPT, robots.txt

☆ Adding one line of code can now prevent OpenAI from accessing a website's data to train ChatGPT
▼ + stars: | 2023-08-08 | by ( Kai Xiang Teo | ) www.businessinsider.com time to read: +3 min

OpenAI launched a new web crawler called GPTBot to browse the internet and collect information. However, adding one line of code to a website will block the crawler from accessing the site's data. Adding just one line of code to a website will now block OpenAI from using the site's data to train its AI models. A web crawler is a bot that browses the internet to collect information. Search engines like Google use web crawlers to collect information for their search results, while AI companies use these crawlers to collect data to train their models.

Persons: OpenAI, Michael Veale, ChatGPT —, James Patterson, Margaret Atwood — Organizations: Morning, University College London, MIT Technology, OpenAI

☆ OpenAI just admitted it has a bot that crawls the web to collect AI training data. If you don't block GPTbot, that's self-sabotage.
▼ + stars: | 2023-08-08 | by ( Alistair Barr | ) www.businessinsider.com time to read: +8 min

Some of these bots have been helpful because they send users to sources of original content online. The most active one is probably Googlebot, which automatically collects web information so Google can later rank and serve it up in Search results. It's called GPTbot and it's being used to scrape and collect online content for AI model training. So what is Clarke's advice for other online content creators when it comes to GPTbot? What is the incentive that OpenAI offers to have these content creators allow GPTbot to crawl and scrape their sites?

Persons: OpenAI, Prasad Dhumal, Neil Clarke, Clarkesworld, Clarke, I've, hasn't Organizations: Morning, Twitter, OpenAI, Associated Press

☆ Universal Music Group calls AI music a ‘fraud,’ wants it banned from streaming platforms. Experts say it’s not that easy
▼ + stars: | 2023-04-18 | by ( Vanessa Yurkevich | ) edition.cnn.com time to read: +5 min

New York CNN —Universal Music Group — the music company representing superstars including Sting, The Weeknd, Nicki Minaj and Ariana Grande — has a new Goliath to contend with: artificial intelligence. Artificial intelligence, and specifically AI music, learns by either training on existing works on the internet or through a library of music given to the AI by humans. That could possibly threaten UMG’s deep library of music and artists that generate billions of dollars in revenue. “However, the training of generative AI using our artists’ music … begs the question as to which side of history all stakeholders in the music ecosystem want to be on.”The company said AI that uses artists’ music violates UMG’s agreements and copyright law. Grammy-winning DJ and producer David Guetta proved in February just how easy it is to create new music using AI.

Search resuls for: "Robots.txt"

13 mentions found