Whonix forums not crawlable by archive.org

Website
1

I’ve tried saving the thread with Goldstein’s comments on Ethereum twice but at the end of the process a forum error page appears saying: oops private page or doesn’t exist.

Does anyone know if there is a robots.txt problem or something to do with discourse?

2

Is this reproducable? By coincidence I restarted Nginx twice in the last 5-10 minutes, wondered if you just had bad luck with timing.

3

Can’t see anything in the robots.txt that would cause any general issues crawling normal pages https://forums.whonix.org/robots.txt

4

Hopefully was just bad timing indeed while I was restarting a service for other reasons? Seemed to save OK just now

EDIT: ahh I see what you mean, it saves but the site throws a Discourse error.

I suspect this is something to do with archive.org’s ability to evaluate javascript. Likely a Discourse thing in general. Will investigate

https://web.archive.org/web/20180901221603/http://forums.dds6qkxpwdeubwucdiaord2xgbbeyds25rbsgr73tbfpqpt4a6vjwsyd.onion/t/is-mysterium-blockchain-based-vpn-as-good-as-its-claimed-to-be/4029

1 Like
5

Issue seems to be that the WayBack machine forges the User-Agent to be that of the user that sent the request to save the page in the WayBack machine. Then it therefore gets a page designed to be evaluated by javascript (like our browsers) but can’t do so.

Probably can’t be fixed by us.

3 Likes
6

Thanks for investigating @mig5