Andy Reid@lemmy.world to Technology@lemmy.worldEnglish · 1 年前AI companies are violating a basic social contract of the web and and ignoring robots.txtwww.theverge.comexternal-linkmessage-square187fedilinkarrow-up11.07Kcross-posted to: technology@midwest.socialtechnology@beehaw.orgwolnyinternet@szmer.infotechnology@lemmy.zip
arrow-up11.07Kexternal-linkAI companies are violating a basic social contract of the web and and ignoring robots.txtwww.theverge.comAndy Reid@lemmy.world to Technology@lemmy.worldEnglish · 1 年前message-square187fedilinkcross-posted to: technology@midwest.socialtechnology@beehaw.orgwolnyinternet@szmer.infotechnology@lemmy.zip
minus-squarewise_pancake@lemmy.calinkfedilinkEnglisharrow-up57·edit-21 年前robots.txt is a file available in a standard location on web servers (example.com/robots.txt) which set guidelines for how scrapers should behave. That can range from saying “don’t bother indexing the login page” to “Googlebot go away”. IT’s also in the first paragraph of the article.
robots.txt is a file available in a standard location on web servers (example.com/robots.txt) which set guidelines for how scrapers should behave.
That can range from saying “don’t bother indexing the login page” to “Googlebot go away”.
IT’s also in the first paragraph of the article.