Warning: Some posts on this platform may contain adult material intended for mature audiences only. Viewer discretion is advised. By clicking ‘Continue’, you confirm that you are 18 years or older and consent to viewing explicit content.
Sorry to say, archive.org is under a ddos attack. The data is not affected, but most services are unavailable.
We are working on it & will post updates in comments.
Brought to you by the Department of Erasing History.
TBH I can understand that it’s a problem for people who aren’t expecting it. If they disregard instructions not to index things then that’s also a problem. The only real way to prevent scrapers from replicating content is to place it behind a registration wall.
Can you elaborate on the last part?
TBH I can understand that it’s a problem for people who aren’t expecting it. If they disregard instructions not to index things then that’s also a problem. The only real way to prevent scrapers from replicating content is to place it behind a registration wall.
https://blog.archive.org/2017/04/17/robots-txt-meant-for-search-engines-dont-work-well-for-web-archives/
Does this answer the question?