abolishing Whonix phabricator issue tracker, moving issue tracking to forums, migrating phabricator.whonix.org to forums.whonix.org

Website
22

Been a while since ive done any meaningful sql work but something like that could work. Would just need to figure out how i can connect to db without having access to the server

1 Like
23

Could share a database dump.

24

If that is worth it. Not sure if mysql based extraction is going to be difficult?

How many tickets are broken? Maybe only few, maybe only unimportant ones that I could manually recover from the web archive perhaps.

25

Lets check with the API first. I will report back.

1 Like
26

Yet another option is to wait until upstream phorge resolves this.

27

Best I can tell, this issue does not affect the API. When I query the API I am getting 900+ issues without any errors

28

confirmed that API is fine for retrieving broken issues in UI…I get this back for T387

Id: 387
Title: Qubes-Whonix-Gateway as ClockVM
Description: __prerequisite knowledge__:

* The design goal is, that the host's [any VMs], Whonix-Gateway and any Whonix-Workstation's clock should slightly differ.
* Rationale: Prevent adversaries from linking anonymous and pseudonymous activity. Described in more detail on the [Dev/TimeSync](https://www.whonix.org/wiki/Dev/TimeSync) wiki page.

-----

__What this is not__:

* Therefore Whonix-Gateway should not "directly" be the ClockVM for all other VMs.
* Using a second instance of sdwdate.

-----

__task description__:

* Reusing Whonix-Gateway and sdwdate.
* Having the time securely provided by sdwdate.
* Useful to reuse these components, since sdwdate depends on Tor, that Whonix-Gateway provides. And Whonix-Gateway is also the right place for Tor configuration.
* Having a second instance of sdwdate running within Whonix-Gateway that provides time for dom0 and all non-Whonix VMs would be an improvement against time related attacks.
* Better than NTP.
* For those who are willing to use Tor.
Status: open
29

Status Update

  1. Migrator successfully parses all phabricator issues in to objects
  2. Migrator successfully parses all phabricator users in to objects

Todo

  1. Parse all phabricator transactions in to objects (these are “comments”)
  2. Link all this data in to a usable JSON structure
  3. Write all this data to a file
  4. Have the DiscoursePoster object parse the file, and prepare to post the issues in the Phabriactor Migration categor
  5. Create the Phabricator Migration category, as well as tags for issue status (open, closed, wontfix, etc)
  6. Create the Discourse User and API key for posting
  7. Test this functionality with a small version of the file (to ensure it works)
  8. Do it live

Note

If something fucks up, there is a handy admin feature to delete all user’s posts. So if this thing for some reason fails or does something horrible, @Patrick can easily delete all the phabricator migrator user’s posts with a single click

Source Code

1 Like
30

okay just got another bit of code pushed up. all comments are now objects and the project is restructured to be a little bit cleaner.

these steps remain

1 Like
31

The forum might also rate limit @phabricator-migrator user and prevent it from posting. To prevent that I adjusted the trust level just now but not sure that was sufficient.

True. Since all under 1 account, easily deleted.

1 Like
32

Can we change the name to @phabricator-migration-bot?

I think this is a little more clear, since not everything in phabricator is a bug. Also, for people who do not have the historical context coming here years later, they can easily see where the issues were migrated from.

Rate limiting is not a huge problem. I can simply put a sleep to slow down the posts if needed. The real question is how slow is rate limit. Currently there are about 950 issues and a bit less than 6000 comments. If rate limit is say, 1 request per second…well that is ~116 minute migration at the fastest.

Not a huge deal, but need to write some logic to log which objects post successfully. Then can pick the job back up from wherever it exited

1 Like
33

From the discourse team

Admin API rate limits

The administrative API keys can be generated via the yoursite.com/admin/api/keys page. These keys can operate on behalf of users, but require administrative privileges to generate. There is a limit of 60 requests per minute, shared between all keys.

Self-hosted users can change this in their app.yml file. Hosted customers will need to contact their hosting provider.

DISCOURSE_MAX_ADMIN_API_REQS_PER_MINUTE : 60

Source

Looks like we could easily add a setting to app.yml and run ./launcher rebuild app to crank up the limit to say, 1000 requests per minute. This would turn the ~2 hour job in to a ~7 minute job.

Risks are mostly crashing the forum because of bad yaml syntax, but can be reduced with yamllint. We want to make sure to create a backup of the existing app.yml so if something goes wrong, easily fixable

$ cd /var/www/discourse # or wherever it lives
$ cp app.yml phabricator_migration_backup.yml
$ vim app.yml # add DISCOURSE_MAX_API_REQS_PER_MINUTE: 1000
$ yamllint app.yml
$ ./launcher rebuild app

Revert steps from same dir

$ mv phabicator_migration_backup.yml app.yml
$ ./launcher rebuild app

No rush on the above work. It is not yet needed. Once we have everything else in place and we have tested it with a few issues, we can do it live with the above settings. Then once the deed is done, we can revoke the bot’s API key, reducing security risks of any shenanigans down the line with the API

1 Like
34

Sure we can change the same, but…

An error occurred: Username must be no more than 20 characters

Suggestions?

35

Updates

Today I added a testing framework to the phabricator migrator. The project has become a bit more complex than anticipated. Not writing tests felt like a shortcut and dishonorable to my brand of development. I merged 8 more commits today that add testing and fix some bugs.

Clearly, this was a good use of time because I wouldn’t have caught the bugs for a while otherwise.

As an annoying side note, for some reason I can’t open pull requests on my own repository I created in the Whonix org on Github. This probably has something to do with github settings/default permissions for me as a team member. Is there any chance you could take a look at this @Patrick when you get a chance? I really like having PRs as a historical source of data to track the development project and communicate information about the changes I am making. No rush on this.

Moving foward

Next, I plan to write all this data to a file that the program will parse and post to the Discourse API. I think I can have the first working iteration in two more focused programming blocks (usually I spend a couple hours of effort on the project when I get the time to do this).

@phabricator-migrator

exactly 20 chars and to the point :slight_smile:

1 Like
36

Okay @Patrick I managed to get a lot more done this evening. If you want to see the current status of how things work

  1. pull latest main branch of phabricator_migration
  2. go in to your phabricator account and create a conduit api key
  3. run ruby phabricator_export.rb
  4. enter your API key and phabricator.whonix.org when prompted
  5. inspect ./discourse_migration_data.json file to see the formatted dump of all our phabricator data

Note:

The test suite requires downloading a ruby gem, but to run it locally on your laptop, everything should work fine with the latest version of Ruby on apt…no gems required :slight_smile:

Once you verify that this data looks okay, we can go ahead and start posting it to discourse and wrap this project up

1 Like
37

@phabricator-migrator

Done.

DISCOURSE_MAX_API_REQS_PER_MINUTE: 1000

Done.

As an annoying side note, for some reason I can’t open pull requests on my own repository I created in the Whonix org on Github. This probably has something to do with github settings/default permissions for me as a team member. Is there any chance you could take a look at this @Patrick when you get a chance? I really like having PRs as a historical source of data to track the development project and communicate information about the changes I am making. No rush on this.

This is a difficult one. Under https://github.com/Whonix/phabricator_migration/settings/access there’s your name. Can you access that URL as well?

role: Admin

Can read, clone, and push to this repository. Can also manage issues, pull requests, and repository settings, including adding collaborators.

So I have no idea what the issue is. But since you’re admin for that repo can you look through, change other settings there?

38

Very late maybe too late feature request… What do we do with phabricator.whonix.org after migration? That is going to be a ton of broken links. An ok way would be to redirect the subdomain to whonix.org. Better than nothing.

Or archive all of it and make it a HTML only version?

The feature request… How difficult would it be to add the link to the forum topic by editing each ticket in phabricator after posting to the forums? That way before arching the HTML, users would know the new link to the forums.

Not trivial to get the API key. Needs some research. I’ve promoted your phabricator account to admin. Can you get the API key that way?

I guess this is something I should look at.

39

What if I created a key:value pair data structure that corellates phabricator issue to discourse thread…then we did some nginx shenanigans to put all those links in to redirect rules? If the issue doesnt match a discourse thread, redirect to the development forums?

Something like that :man_shrugging:

1 Like
40

Redirections would be even better!

How many tickets we got approximately? 1000(s)? Can nginx handle such a large number of redirections without performance penalty? Any experience with that? I guess if it is an issue a separate nginx instance could be run?

41

Just under 1000. 950ish. Cant remember the exact number.

The more I think about it, this is the simplest solution I can come up with

  1. Query the mediawiki API and search for all occurrences of phabricator links.
  2. Use the mediawiki api to programatically change all phabricator links to discourse links to the same issue.
  3. Delete phabricator

It seems logical to me. Way better than fuckin with 1000 nginx redirects.

Although, there may be a clever way to send phabricator links to discourse with a single nginx regex. Which would be the simplest solution.

1 Like