Luu Tuyen@lemmy.world to Technology@lemmy.worldEnglish · 3 months agoTikTok’s parent launched a web scraper that’s gobbling up the world’s online data 25-times faster than OpenAIfortune.comexternal-linkmessage-square110fedilinkarrow-up1555cross-posted to: technik@feddit.org
arrow-up1555external-linkTikTok’s parent launched a web scraper that’s gobbling up the world’s online data 25-times faster than OpenAIfortune.comLuu Tuyen@lemmy.world to Technology@lemmy.worldEnglish · 3 months agomessage-square110fedilinkcross-posted to: technik@feddit.org
minus-squarepurrtastic@lemmy.nzlinkfedilinkEnglisharrow-up47·3 months agoIt’s not fine. They are not archiving the internet. I had to ban their user agent after very aggressive scraping that would have taken down our servers. Fuck this shitty behaviour.
minus-squareMelvin_Ferd@lemmy.worldlinkfedilinkEnglisharrow-up4·3 months agoIsn’t there a way to limit requests so that traffic isn’t bringing down your servers
It’s not fine. They are not archiving the internet.
I had to ban their user agent after very aggressive scraping that would have taken down our servers. Fuck this shitty behaviour.
Isn’t there a way to limit requests so that traffic isn’t bringing down your servers