Warning: Some posts on this platform may contain adult material intended for mature audiences only. Viewer discretion is advised. By clicking ‘Continue’, you confirm that you are 18 years or older and consent to viewing explicit content.
But companies are crawling everything like mad - I’ve noticed a 400% upturn this year alone in bot traffic on a low traffic web forum and a few sites I host, so much so that I’m having to do some fairly heavy filtering upstream to keep them out. (They don’t resepect robots.txt, obviously)
When bot traffic outnumbers legitimate traffic at least 10x, it makes you wonder why you’re paying to host stuff.
“We’re shocked” - nobody.
But companies are crawling everything like mad - I’ve noticed a 400% upturn this year alone in bot traffic on a low traffic web forum and a few sites I host, so much so that I’m having to do some fairly heavy filtering upstream to keep them out. (They don’t resepect robots.txt, obviously)
When bot traffic outnumbers legitimate traffic at least 10x, it makes you wonder why you’re paying to host stuff.