abolishing Whonix phabricator issue tracker, moving issue tracking to forums, migrating phabricator.whonix.org to forums.whonix.org

Website
1

phabricator is yet another webapp to maintain. Every now and then an update breaks it such as recently an update broke something which worked for a long time: the outbound mailer. Other issues with phabricator: we require 3 user accounts. Forums, wiki, and issue tracker. The latter is certainly the most underused where the fewest people sign up. Also other issues.

Often we have a forum subject with the “real” discussion and then a phabricator ticket as reminder to actually implement what was being discussed, some duplication. The main feature used by phabricator: issue tracking, open/close status and tags. So I was wondering if discourse is capable to replace phabricator.

I just noticed we have “only” 100 open tickets.
Correction: I just noticed we have “only” 220 open tickets.

https://phabricator.whonix.org/maniphest/query/open/

So even a manual migration is realistic. Wondering to just close the stale / unrealistic ones. The tracker could use a spring clean anyhow.

The good thing is the forum search allows to search for multiple tags which all have to be matched at the same time.

EDIT:

discourse does not yet support following which tags where changed

Added here just now:
Bug Reports, Software Development and Feature Requests

backlog 1 [archive] / backlog 2 [archive]

Previously there were no guidelines / rules on how to use discourse forum tags. Anyone could create them / add them / change them. [10]

Discourse forums tags are now only changeable by Whonix ™ discourse moderators [archive].

Search query examples:

You can create a comment in a separate post such as:

  • Please open.

  • Please close.

  • Please re-open.

  • Please add tag testers-wanted .

  • Please remove tag component_security .

If appropriate, the tag will be changed by a forum moderator. Such comments might be deleted if inappropriate, duplicated or for brevity of the discussion (such as in case the status was only not updated due to as mistake).

If a forum thread is not tagged with whonix-15 (the current stable release of Whonix ™) at time of writing to whonix-16 (the next Whonix ™ release) then it is not on the roadmap. In result chances the issue will be resolved (feature implemented, research/task done, or bug fixed) is lower. Please abstain from debating priorities which issue is more/less important or when it should be implemented. (reasons)

Tags should not be appended to every discussion or question about the related subject.

The goal of forum tags is to maximize Whonix ™ forum’s usefulness as a roadmap and issue tracker for Whonix ™ project developers.

2 Likes
2

Heavily updated above post just now.

Migration in progress now. Will probably take a long time.

Unless there’s a technical volunteer contributor who’d be capable of summarizing phabricator tickets and copying these over to forums.

3

There was no such option indeed. Changed now. Please test.

This discourse forum software supports “trust levels”. “level 4” is as far as I can see only manually given. Just now populated:

https://forums.whonix.org/g/trust_level_4

Added you.

Configured the forum software to allow all members of trust level 4 group to edit forum tags.

Anyone interested in triage of tickets can request access to this group.
Eligible is everyone where it’s reasonably certain that it’s a serious contributor. Such as people with significant, long term contributions such as research, code, documentation, user support.

Reason it’s not open for all accounts is that discourse does not have any easy way to show when tags where changed (only through post edit history) nor does notify about tag changes.

2 Likes
4

phabricator now deprecated upstream.

https://admin.phacility.com/phame/post/view/11/phacility_is_winding_down_operations/

phabricator has to be soon disabled. I recommend to archive any tickets anyone cares about. Otherwise these might be lost. Or even better, migrating those to this forum.

5

A post was merged into an existing topic: Long Wiki Edits Thread

6
7

need to check out:

8

I would be hesitant to integrate non official discourse supported plugins

reasoning is primarily:

  • potential security vulns in less scrutinized codebase
  • more depedency on external maintainers without official discourse support
1 Like
10

do we want to migrate only the open issues? Or do we want the resolved issues too for historical purposes? how do we want is structured in discourse? Maybe we could have a readonly forum for the closed issues or something?

I successfully have the issues loaded in to JSON, now I can simply programmatically post them to discourse…just need to figure out how we want it structured

1 Like
11

example of parsed response issue object. we need to decide what all we want to bring over

  {"PHID-TASK-tvxkao4tftxc2b65sm73"=>
    {"id"=>"990",
     "phid"=>"PHID-TASK-tvxkao4tftxc2b65sm73",
     "authorPHID"=>"PHID-USER-fdjzwnsuqpjbmss2qi6d",
     "ownerPHID"=>nil,
     "ccPHIDs"=>["PHID-USER-fdjzwnsuqpjbmss2qi6d", "PHID-USER-xbf6teehtqnwj3oqz4zn"],
     "status"=>"open",
     "statusName"=>"Open",
     "isClosed"=>false,
     "priority"=>"Normal",
     "priorityColor"=>"orange",
     "title"=>"whonixcheck tirdad module load",
     "description"=>"Non-Qubes-Whonix only.\n\n(Qubes-Whonix: https://forums.whonix.org/t/qubes-whonix-security-disadvantages-help-wanted/8581)",
     "projectPHIDs"=>["PHID-PROJ-o7u4jsmh2gq3fyxltby5", "PHID-PROJ-scuhlepkbzq6zhperadm", "PHID-PROJ-7ro5kt2nbndmev3eq4dh"],
     "uri"=>"https://phabricator.whonix.org/T990",
     "auxiliary"=>{"std:maniphest:Whonix:impact"=>"Whonix:normal"},
     "objectName"=>"T990",
     "dateCreated"=>"1589204010",
     "dateModified"=>"1589204010",
     "dependsOnTaskPHIDs"=>[]},

I am guessing the title can be the topic subject, and the description can be the post body. Then we just iterate through them all. My script is almost complete. Will post it here shortly

1 Like
12

and here is the script I wrote to get it done, saved as phabricator_migrate.rb

require 'json'
require 'uri'
require 'net/http'

PHAB_KEY=ARGV[0]
DISCOURSE_KEY=ARGV[1]

class Issue
  attr_reader :title, :description

  def initialize(issue_data)
    @title = issue_data["title"]
    @description = issue_data["description"]
  end
end

class PhabricatorParser
  attr_reader :api_key

  def initialize(api_key)
    @api_key = api_key
  end

  def get_raw_issues
    uri = URI("https://phabricator.whonix.org/api/maniphest.query")
    params = {
      "api.token": api_key,
      "status": "status-open"
    }
    uri.query = URI.encode_www_form(params)

    Net::HTTP.get_response(uri).body
  end
  
  def parse_issues
    raw = get_raw_issues
    parsed = JSON.parse(raw)

    parsed["result"].map do |id, issue|
      Issue.new(issue)
    end.compact
  end
end

class DiscoursePoster
  attr_reader :issues, :api_key
  
  def initialize(issues, api_key)
    @issues = issues
    @api_key = api_key

  end

  def post_issues
    issues.each do |issue|
      uri = URI('https://forums.discourse.org/posts.json')
      res = Net::HTTP.post_form(uri, 'title': issue.title, 'raw': issue.description, category: 8, tags: ["status_open_issue_todo", "phabricator_issue"])
      puts res.body  if res.is_a?(Net::HTTPSuccess)
    end

    return "Posting Completed"
  end
end

issues = PhabricatorParser.new(PHAB_KEY).parse_issues
DiscoursePoster.new(issues, DISCOURSE_KEY).post_issues

usage:

$ ruby phabricator_migrate.rb <phabricator_api_key> <discourse_api_key>

I was able to get a phabricator API key through the app, but am unable to get a discourse API key without @Patrick issuing it for me. It might be worth making a separate account to do the migration, so that not all the issues are posted from me.

1 Like
13

Unfortunately so.

14

Would be good if all phabricator links (or only tickets) (except spam if you can find more) would be web archived beforehand. That is so old links don’t break.

(Then maybe later an nginx redirection from phabriator.whonix.org to web.archive.org could be setup.)

Could you check with some simple “grep” query first if there is spam in any tickets before web archiving and re-creating tickets in the forums?

Good enough.

Better.

Otherwise auto archive them on the web archive?

Difficult, time consuming? Could even be two new sub forums. Archived tickets and active tickets.

readonly not needed. But sub forums would prevent the development forum getting flooded.

if yes then not too important. Should not become a blocker. Initially would be spammy but over time old forum threads (prior tickets) would fall down in history.

Difficult to persist the status of these tickets? A simple comment that the ticket was solved, resolved, won’t fix (just it was expressed textually in phabricator) would suffice. No need to close the discussion “for real” (using the forums function) so that nobody can reply anymore. Nothing controversial in the tickets. If that was required could be done manually.

Yes. That would be great. Could you create account name:
phabricator-migration
(or so)
please? Then I upgrade the account.

1 Like
15

I like this idea. Go ahead and create the forums if you dont mind.

Posting open issues and other status issues is not much more work. I can get it done quickly

No

Another idea…1 phabricator subforum. Seperate tag for each ticket status. Post open issues last so they appear near the top of the forum.

1 Like
16

(Can be beautified with descriptions, icon, maybe moved as sub forums of development etc. later on.)

17

Could you please test this in dry-mode or something beforehand? I.e. just show what the script would be doing instead running it for real.

Because if the script goes wild due to a bug could create lots of spam in the forums.

And maybe just for 1 (or a few) tickets first until doing the bulk for testing.

Feasible to add a bit metadata to each comment such as who wrote what and date?

18
  • who wrote what
  • date
  • web archived link?
    • A) simple: https://web.archive.org/web/https://phabricator.whonix.org/T509
    • B) better, maybe complicated: https://web.archive.org/web/20230529185755/https://phabricator.whonix.org/T509
19

Yes i was going to test like 3 topics first. Put a guard clause in and make sure it goes to the right subforum. Manually delete them. Then run it live

1 Like
20

Broken Maniphest ticket link:

phabricator migration might be currently blocked by this bug:

If it is only a very few of these cases we could go ahead and migrate the broken ones manually. Let’s see if / how fast the new upstream phorge replies.

21

I saw some of these annoying errors.

I might be able to pull it through the API. Will report back. Otherwise, we might be able to do some fancy SQL to generate a CSV and archive it

Pseudo code (dont know the exact data layout) but something like this:

SELECT * FROM issues
  INNER JOIN users.username AS issue_user ON issues.user_id = users.id
  INNER JOIN comments ON issues.id = comments.issue_id
  INNER JOIN users.username AS comment_user ON comments.user_id = users.id

Then just loop through it, format, and post to discourse API

1 Like