Whonix forums not crawlable by archive.org

I’ve tried saving the thread with Goldstein’s comments on Ethereum twice but at the end of the process a forum error page appears saying: oops private page or doesn’t exist.

Does anyone know if there is a robots.txt problem or something to do with discourse?

Is this reproducable? By coincidence I restarted Nginx twice in the last 5-10 minutes, wondered if you just had bad luck with timing.

Can’t see anything in the robots.txt that would cause any general issues crawling normal pages https://forums.whonix.org/robots.txt

Hopefully was just bad timing indeed while I was restarting a service for other reasons? Seemed to save OK just now

EDIT: ahh I see what you mean, it saves but the site throws a Discourse error.

I suspect this is something to do with archive.org’s ability to evaluate javascript. Likely a Discourse thing in general. Will investigate


1 Like

Issue seems to be that the WayBack machine forges the User-Agent to be that of the user that sent the request to save the page in the WayBack machine. Then it therefore gets a page designed to be evaluated by javascript (like our browsers) but can’t do so.

Probably can’t be fixed by us.


Thanks for investigating @mig5