Warning: Some posts on this platform may contain adult material intended for mature audiences only. Viewer discretion is advised. By clicking ‘Continue’, you confirm that you are 18 years or older and consent to viewing explicit content.
yes but also there’s a point where it’s blatantly obvious. And i can’t imagine it’s hard to get rid of the obviously offending ones. Respectful crawlers are going to be imitating humans, so who cares, disrespectful crawlers will ddos your site, that can’t be that hard to implement.
Though if we’re talking “hey please dont scrape this particular data” Yeah nobody was ever respecting that lol.
yes but also there’s a point where it’s blatantly obvious. And i can’t imagine it’s hard to get rid of the obviously offending ones. Respectful crawlers are going to be imitating humans, so who cares, disrespectful crawlers will ddos your site, that can’t be that hard to implement.
Though if we’re talking “hey please dont scrape this particular data” Yeah nobody was ever respecting that lol.
Both paragraphs demonstrate gross ignorance