packages.debian.org APT package repository web interface for deb.whonix.org

Development
1

Maybe we can somewhat easily prettify deb.whonix.org. Making information for packages by Whonix and later Kicksecure (on own domain name kicksecure.com) much more accessible. Could better present what packages exist, what the dependencies are, version numbers, which repository has which version, changelogs and whatnot. Would be much better searchable since it would get indexed by search engines.

This is apparently what packages.debian.org is using:

https://packages.debian.org/about/

But looks rather involved to setup.

I would like to avoid re-introducing apache on whonix.org.

Is there any more lightweight alternative?

1 Like
2

Kali Linux Package Tracker - cifs-utils based on Debian QA / distro-tracker · GitLab. Similar:
apt - Debian Package Tracker. But not too great since it does not show package descriptions. Perhaps that is configurable? I haven’t found installation or configuration instructions yet.

https://pkginfo.devuan.org looks decent. Haven’t found source code and installation instructions yet.

3

https://lists.debian.org/debian-derivatives/2021/01/msg00000.html

4

https://lists.debian.org/debian-www/2021/01/msg00015.html

5

Not sure if it is correct, but #devuan on freenode mentioned these git repos:

http://5.196.69.93/?q=d1pkg

–
bye,
pabs

http://5.196.69.93/d1pkgweb/

http://5.196.69.93/d1pkgweb-query/

mpi

Maemo Package Indexer

Current instance is running at: Maemo Package Index

Fork of KatolaZ’s d1pkgweb tailored for Maemo Leste.

6

BunsenLabs have a pure-JavaScript based package index:

https://www.bunsenlabs.org/repoidx.html

https://github.com/BunsenLabs/bl-repo-index

7

Documentation for this is very spare. If anyone could contribute how to set this up with nginx that would help a ton in getting this done.

8
9

To get a local copy of the package repository:

The short summary of Hosting a Kicksecure ™ Mirror…

sudo apt install rsync

Get a local copy of the Kicksecure repository.

rsync-ssl --recursive --delete --times --perms rsync://kicksecure.com/kicksecure/developer-meta-files/internal/ ~/kicksecure-repository

(FYI: rsync://kicksecure.com/kicksecure is the same as https://download.kicksecure.com/.)

//cc @Mycobee

10

This functionality exists now

How it works

A Ruby application I wrote called package_parser parses the package repositories and generates markdown

The markdown is moved in to two seperate jekyll repositories that have been created and themed for Whonix and Kicksecure. These repositories are based on the just-the-docs theme, but modified with specific Whonix and Kicksecure branding

Jekyll generates a static site with search capabilities, and outputs them in to a folder. They will then be served via Nginx on our servers.

A high level shell script exists in package_parser to handle all of this functionality.

Usage

Setup

In one directory, clone the kicksecure-repository/ and whonix-repository/ to get the text files for all existing packages

In the same directory, clone kicksecure-packages, whonix-packages, and package_parser.

Dependencies

  • Install a working and secure version of Ruby 3
  • Install the latest version of bundler with gem install bundler

Script

From the root of the package_parser repo, run the shell script to generate static sites for the OS of your choosing.

./build-site.sh <os_type> <desired_static_output_path>

Examples

./build-site.sh whonix ../whonix-packages-html
./build-site.sh kicksecure ../kicksecure-packages-html
1 Like
11

It should be noted that MPI was not a good fit for our project. It uses Shell scripting to search for packages and needs a lot of customization to deploy it on our server. Because of this, I opted to build a solution for our needs.

@Patrick…I of course want all this code to be FLOSS, but there are some license things to consider that I would like your input on

The two Jekyll repos were cloned and have the MIT license in them because that was what was upstream. I am not sure if we should change, can change, or need to change this licenses. I am pretty uneducated about the “best” FLOSS licenses, and what is permissible.

For package_parser, I have not added a license yet. You should add the desired license of your choosing to that repo @Patrick

Existing MIT licenses from upstream fork

Repo that needs a license

1 Like
12

tl;dr - we have a packages site generator now and it works

1 Like
13
14

https://packages.whonix.org/ is now online for testing.
(Still noindex server side because I’ll split Kicksecure / Whonix package repositories to avoid confusing search engines.)

The link style is a bit non-ideal. Currently:

https://packages.whonix.org/docs/bullseye/main/binary-amd64.html#apparmor-profile-hexchat

A huge HTML file with all the packages is unfortunately not search engine friendly.

Would be better if each package had its own HTML file. For example this is how Debian does it:

https://packages.debian.org/bullseye/live-build

Then also the meta tags would be added per page.

There’s no need to create a HTML file per architecture. I.e. no need for:

  • binary-amd64.html
  • binary-kfreebsd-amd64.html

Architectures availability is much better to add to per package only.
(Again similar to https://packages.debian.org/bullseye/live-build.)

Also no need for the suite (main, contrib, non-free) to be inside the link. That’s also sufficient per package.

Probably due to the link style (too much content on one page), the reaction speed when clicking seems very low.

footer copyright notice could be misunderstood to apply to all packages.

footer please:

In 1 Line.

1 Like
15

Please drop .html suffix.

meta tags inside HTML head wishlist:

(Some lines are different for Kicksecure and Whonix.)

These are sourced from the mediawiki generated HTML head form Kicksecure or Whonix homepage. Maybe easier to see there than here in the forums.

General:

<meta name="generator" content="Kicksecure Package Parser"/>
<meta name="format-detection" content="telephone=no"/>
<meta name="description" content="package short description here"/>
<meta name="twitter:card" content="summary_large_image"/>

<meta property="og:title" content="package name here"/>
<meta property="og:description" content="package short description here"/>
<meta property="article:modified_time" content="2023-03-11T13:42:03Z"/>
<meta property="article:published_time" content="2023-03-11T13:42:03Z"/>
<meta property="og:type" content="website"/>

viewport I am not sure about and low priority for now.

<meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=yes, minimum-scale=0.25, maximum-scale=5.0"/>

Kicksecure:

<meta name="twitter:site" content="@Kicksecure"/>
<link rel="license" href="/wiki/Kicksecure:Copyrights"/>
<link rel="canonical" href="https://www.kicksecure.com"/full-link-here>
<meta property="og:site_name" content="Kicksecure"/>

<meta property="og:image" content="https://www.kicksecure.com/w/images/thumb/6/6b/Kicksecure-general.jpg/1200px-Kicksecure-general.jpg"/>
<meta property="og:image:width" content="1200"/>
<meta property="og:image:height" content="675"/>
<meta property="og:url" content="https://www.kicksecure.com/full-url-here"/>

Whonix:

<meta name="twitter:site" content="@Whonix"/>
<link rel="license" href="/wiki/Whonix:Copyrights"/>
<link rel="canonical" href="https://www.whonix.org/full-link-here">
<meta property="og:site_name" content="Whonix"/>

<meta property="og:image" content="https://www.whonix.org/w/images/thumb/3/36/Whonix-homepage-main-preview.jpg/1200px-Whonix-homepage-main-preview.jpg"/>
<meta property="og:image:width" content="1200"/>
<meta property="og:image:height" content="600"/>
<meta property="og:url" content="https://www.whonxi.org/full-url-here"/>

Kicksecure:

<link rel="icon" type="image/svg+xml" href="/favicon.svg" sizes="any">

<link rel="apple-touch-icon" sizes="180x180" href="/apple-touch-icon.png">
<link rel="icon" type="image/png" sizes="32x32" href="/favicon-32x32.png">
<link rel="icon" type="image/png" sizes="16x16" href="/favicon-16x16.png">
<link rel="manifest" href="/site.webmanifest">

<link rel="mask-icon" href="/safari-pinned-tab.svg" color="#5bbad5">
<meta name="theme-color" content="#ffffff">

<meta name="msapplication-TileColor" content="#da532c">

Whonix:

<link rel="icon" type="image/svg+xml" href="/favicon.svg" sizes="any">

<link rel="apple-touch-icon" sizes="180x180" href="/apple-touch-icon.png">
<link rel="icon" type="image/png" sizes="32x32" href="/favicon-32x32.png">
<link rel="icon" type="image/png" sizes="16x16" href="/favicon-16x16.png">
<link rel="manifest" href="/site.webmanifest">

<link rel="mask-icon" href="/safari-pinned-tab.svg" color="#5bbad5">
<meta name="theme-color" content="#ffffff">

<meta name="msapplication-TileColor" content="#b91d47">

The schema.org might be difficult? A lot stuff (copied from Kicksecure homepage). Unadjusted. And maybe therefore low priority. Might be difficult to auto generate?

<script type="application/ld+json">{"@context":"http:\/\/schema.org","@type":"website","name":"Kicksecure - Secure by Default Operating System","headline":"Kicksecure - Secure by Default Operating System","mainEntityOfPage":"Kicksecure - Secure by Default Operating System","identifier":"https:\/\/www.kicksecure.com\/wiki\/Homepage","url":"https:\/\/www.kicksecure.com\/wiki\/Homepage","description":"A secure by default operating system with the latest security research in place.","dateModified":"2023-03-11T13:42:03Z","datePublished":"2023-03-11T13:42:03Z","image":{"@type":"ImageObject","url":"https:\/\/www.kicksecure.com\/w\/images\/6\/6b\/Kicksecure-general.jpg?version=6fad8da500c824a53a3c05527fa2a6e7","width":1920,"height":1080},"author":{"@type":"Organization","name":"Kicksecure","url":"https:\/\/www.kicksecure.com","logo":{"@type":"ImageObject","url":"https:\/\/www.kicksecure.com\/w\/images\/3\/3d\/Kicksecure-basic-logo.svg","caption":"Kicksecure"}},"publisher":{"@type":"Organization","name":"Kicksecure","url":"https:\/\/www.kicksecure.com","logo":{"@type":"ImageObject","url":"https:\/\/www.kicksecure.com\/w\/images\/3\/3d\/Kicksecure-basic-logo.svg","caption":"Kicksecure"}},"potentialAction":{"@type":"SearchAction","target":"https:\/\/www.kicksecure.com\/w\/index.php?title=Special:Search&search={search_term}","query-input":"required name=search_term"}}</script>

The fields are all identical just with different contents sometimes. If some tag is specifically difficult to implement, we can drop it.

favicon relative links such as https://packages.whonix.org/favicon.svg I’ve already made available server-side. So for logos we don’t need these duplicated in any repositories.

1 Like
16

I have been thinking more about this @Patrick…still busy with other stuff for the next couple weeks, but will rework it

What do you think? Every branch of the tree is a page


├── bullseye
└── bullseye-developers
    └── main
    ├ ├──  package_1
    ├ └── package_2
    └── non_free
      └── package_1
      └── package_2
├── bullseye-proposed-updates
└── bullseye-testers

Each “package” page has the suite and architectures it is available for?

1 Like
17

Feature request: make the search bar appear on mobile devices without having to click the 3 lines hamburger icon.

3 Likes
18

https://packages.debian.org is already difficult to use. The implementation for Kicksecure / Whonix should be equally difficult to use or easier if possible.

Debian has only two levels. Example:

https://packages.debian.org/bullseye/nano

Syntax:

https://packages.debian.org/suite/package-name

Debian has only 2 levels.

Therefore I think 3 levels are too much.

main / contrib / nonfree by the way is called a component as per language used by Debian.

So in summary, I don’t think we should use the component inside the URL.

Here’s an example of Debian using the nonfree component.

https://packages.debian.org/bullseye/firmware-linux-nonfree

Being in the nonfree instead of the main component is visualized by Debian using a red colored link in the headline:

[non-free]

I was even wondering to drop the suite in the link.

suite examples are:

  • bullseye
  • bullseye-developers
  • bookworm

But not sure that’s doable. Each suite can have major differences:

  • short/long description
  • version number
  • dependencies

Also bullseye-developers might have a package which is not yet available in bullseye.

So simpler than the Debian 2 level https://packages.debian.org/bullseye/nano example is maybe not possible.

Much appreciated!

2 Likes
19

I will add that to the list :+1:

The existing structure was just the first pass. It will be iterated on and improved

2 Likes
20

I am probably going to move this site to hugo instead of jekyll…one depedency that is available via apt (no gems, no CI dependency)

This will be significantly more performant.

Originally, I was using jekyll to build each package its own html page, but this took hours to build, which is why the large payload page per arch was created.

I know this has much room to be improved, and I will improve it. I wanted to get something working though

2 Likes