write draft for local listener standard on debian-devel

phabricator-migrator · February 19, 2024, 5:43pm

Information

ID: 635
PHID: PHID-TASK-lk5dm27dnk3w3opis3cg
Author: Patrick
Status at Migration Time: resolved
Priority at Migration Time: Normal

Description

subjects

convention on listen port local or all network interfaces etc.

draft:

TODO:

~~improve draft~~ (done)
~~send to debian-devel mailing list~~ (done)
** convention on listen port local or all network interfaces etc.
** Re: convention on listen port local or all network interfaces etc. - revision 2
discuss on debian-devel

Comments

HulaHoop

2017-02-20 02:12:42 UTC

dau

2017-02-20 09:40:32 UTC

Patrick

2017-02-20 14:30:28 UTC

Patrick

2017-02-20 15:02:21 UTC

One thing is still too vague.

Does
# /etc/server-config.d/30_default.conf
listen_ip=0.0.0.0

# /etc/server-config.d/50_user.conf
listen_ip=127.0.0.1
mean

a) listen on 0.0.0.0 as well as on 127.0.0.1?

b) Or does the latter listen_ip=127.0.0.1 deactivate listen_ip=0.0.0.0?

a) is how systemd parses systemd unit files. In that case one would have to use listen_ip= to disable listeners by lower priority configuration files.
# /etc/server-config.d/30_default.conf
listen_ip=0.0.0.0

# /etc/server-config.d/50_user.conf
listen_ip=
listen_ip=127.0.0.1
b) is easier understand and easy to parse with bash (just use source as per the pseudo code in the draft). bash however is not so important here. A python parser could easily translate this for bash and most daemons will not be written in bash and not be using bash startup wrappers.

In other words… How to listen on multiple instances?

Like this…
listen_ip=127.0.0.1,10.152.152.10,UNIX-LISTEN:/path/to/<application-name>.sock
…?

Please take into consideration the difficulty of adhering this at the python implementation level. I mean, nothing is won if a superb convention is formulated that is then ignored by applications developers like you @dau because it looks too difficult to implement. We might want to bend the convention to suit python implementation difficulty.

Yet another style would be “lower priority configuration files are ignored, only highest priority configuration file is adhered”.

And if we don’t know the answer, we’ll just integrate these questions into the draft.

dau

2017-02-20 19:55:06 UTC

! In T635#12393, @Patrick wrote:
a) is how systemd parses systemd unit files. In that case one would have to use listen_ip= to disable listeners by lower priority configuration files.

That’s interesting. I did think that all files would be read and interpreted, but in cases where multiple listen_ip= would be found, the last one would override the previous. So indeed, it seems they would have to disabled.
In other words… How to listen on multiple instances?

Like this…
listen_ip=127.0.0.1,10.152.152.10,UNIX-LISTEN:/path/to/<application-name>.sock
…?
I like it and seems easy to parse.

Please take into consideration the difficulty of adhering this at the python implementation level. I mean, nothing is won if a superb convention is formulated that is then ignored by applications developers like you @dau because it looks too difficult to implement. We might want to bend the convention to suit python implementation difficulty.

As a python developer it would be great to easily get the ip that should be used. However, I do not agree with you on this point because a *nix standard is being proposed. I think that it should be consistent with how systems interpret these configurations and then it is up to each app to access them. That might actually be decisive if the proposal is accepted or not.

I am sure that if it is really hard to read the configs, let’s say in python, some package could be developed to parse the files and offer them in a very convenient way. I would just import it, tell which file to read or which specific config to get and do whatever I have to do with this object/dict/string returned. If the settings became environment variables I assume it would be pretty easy to use in any language, but if the file is too long then the environment might get polluted (?).

And if we don’t know the answer, we’ll just integrate these questions into the draft.

I agree. I think we could give them all the options we have and all questions that came up. The more information, the better.

Thanks!

Patrick

2017-02-20 23:25:47 UTC

Patrick

2017-02-20 23:30:13 UTC

dau

2017-02-21 00:11:04 UTC

! In T635#12405, @Patrick wrote:
Thank you for your feedback!

No problem. As I said, I do not have much experience/knowledge about this, so I am just giving opinions/ideas (not sure which ones make sense or are relevant).

I don’t dare to call this a standard, because then this could take ages and face lots of resistance. A volunteer convention seems nicer.

Hmm you are right, that’s true! At this point I think that Whonix will implement it regardless if it is adopted or not, right?

There is no standard on configuration file parsing. Many applications have many very different configuration languages. I personally like the way systemd goes about it. It allowed Whonix specific overrides to be implemented with a very few lines and cleanly.

Sounds good. If you are familiar with that and things work, then I’d say it’s probably the way to go.

! In T635#12406, @Patrick wrote:
Two more edits. It’s now all included based on your feedback which I hopefully interpreted correctly (btw being not a native English speaker).

Please have a look if it’s ready to go.

Cool! I cannot help you with that because I am not a native speaker as well

In my opinion it looks great, I think it is ready.

Patrick

2017-02-21 00:58:58 UTC

Great! Sent.

Not yet on the mailing list archive. I’ll share the link once available here.

debian-devel Feb 2017 by thread

! In T635#12409, @dau wrote:

! In T635#12405, @Patrick wrote:
Thank you for your feedback!

No problem. As I said, I do not have much experience/knowledge about this, so I am just giving opinions/ideas (not sure which ones make sense or are relevant).

Sure. That’s all I am looking for. Much appreciated!

I don’t dare to call this a standard, because then this could take ages and face lots of resistance. A volunteer convention seems nicer.

Hmm you are right, that’s true! At this point I think that Whonix will implement it regardless if it is adopted or not, right?

It is my hope to get some good feedback on debian-devel. One never knows what could have been overlooked. Lots of people reading there. The more developers following this convention, the better.

But yes. Even if only you representing unMessage like this convention, by all means, Whonix should go for it. I am however very positive that also onionshare, ricochet and ZeroNet would be open to it. Even if just the Tor ephemeral hidden services software packages adhere this convention that would be a huge win.

Patrick

2017-02-21 01:09:12 UTC

Patrick

2017-02-21 17:01:33 UTC

dau

2017-03-20 01:14:06 UTC

As we discussed in 0, I am updating the draft.

Based on Tomas Pospisek’s posts 1 and 2, I made the following changes (it’s attached). Following your reply 3 to Marco d’Itri’s post 4, I agree and think he did not fully understand our intentions, so I did not take that into consideration.

From my understanding, Tomas Pospisek would like to have additional configuration includes explicit and manually added to the main file so that it is clear what is supposed to happen instead of thinking “apparently placing customizations in /home/.config/server-config.d/ just work”. I do not fully agree as I am usually in favor of “convention over configuration” and either just adding configurations to some of the server-config.d dirs or customizing main.config still require you “to have a priory knowledge” (you need to know what you are doing and what files to change), but I do like the solution he presented by using include, which makes the file a lot “cleaner”.

From your reply 5 to his post:

Would you suggest just a single file /etc/server/main.config?

Or should there be also:

/usr/lib/server/main.config

/etc/server/main.config

/home/.config/server/main.config

I think that by naming that file main, he intended to have just a single file so that not much knowledge is required (otherwise you would have know about not only all the .config files, but also the .conf ones). Also, from his comments in 2 it would be a good idea to use multiple .config files and multiple .d subdirs for complex configs, but I do think it is the case here. I agree with that but one problem that we might face is having that single .config file “restored” from the template when using Qubes. What do you think? Maybe having a directory in main.config that is not restored solves that? (i.e., all customizations should be in that dir)

Should main.config file[s] only contain include statements and
comments and nothing else?

I am pasting /etc/logrotate.conf below, which contains more than include statements. I think that there is no need for that as this convention is very clear about what the related .conf files should be and their contents are consistent to each other. I do like that the other content in /etc/logrotate.conf is consistent to the content of the other files in /etc/logrotate.d/.
# see "man logrotate" for details
# rotate log files weekly
weekly

# keep 4 weeks worth of backlogs
rotate 4

# create new (empty) log files after rotating old ones
create

# uncomment this if you want your log files compressed
#compress

# packages drop log rotation information into this directory
include /etc/logrotate.d

# no packages own wtmp, or btmp -- we'll rotate them here
/var/log/wtmp {
    missingok
    monthly
    create 0664 root utmp
    rotate 1
}

/var/log/btmp {
    missingok
    monthly
    create 0660 root utmp
    rotate 1
}

# system-specific logs may be configured here
This new approach would not change the way the .conf files work when dealing with configurations from the previous files, right? If it is needed, they are able to to ignore the previous statements with, for example, listen_ip=. I am saying that because it looks like now main.config is allowed to change and just having include statements suggests that prefixing them with # is enough to ignore. That could work for non-Qubes systems, but as I mentioned above having main.config restored can be a problem.

Let me know what you think. Thanks!

{F283910}

Patrick

2017-03-20 14:13:39 UTC

If you are using git anyhow to generate the diff, could you please fork GitHub - adrelanos/proposals: https://phabricator.whonix.org/T635 | https://phabricator.whonix.org/T634 and commit? That would make the diff a lot easier to apply and read for me.

From your diff…

I think I agree for the most part.

+include /home/.config/server-config.d/*.conf

Looks like a user name is missing. There is no user .config.

Do you mean it would automatically interpreted as from

… /home/.config/server-config.d/*.conf

to /home/user1/.config/server-config.d/*.conf

to /home/user2/.config/server-config.d/*.conf

etc.? (Depending on the user under which the application is running?)

include vs home folder is not obvious yet in the draft? Perhaps the config should contain a variable $HOME` or so? And only be adhered by applications running as user such as unMessage? And ignored by applications run by the init system (such a web servers)?

! In T635#12783, @dau wrote:
As we discussed in 0, I am updating the draft.

Thanks!

Based on Tomas Pospisek’s posts 1 and 2, I made the following changes (it’s attached).

Following your reply 3 to Marco d’Itri’s post 4, I agree and think he did not fully understand our intentions, so I did not take that into consideration.

Ok.

From my understanding, Tomas Pospisek would like to have additional configuration includes explicit and manually added to the main file so that it is clear what is supposed to happen instead of thinking “apparently placing customizations in /home/.config/server-config.d/ just work”. I do not fully agree as I am usually in favor of “convention over configuration” and either just adding configurations to some of the server-config.d dirs or customizing main.config still require you “to have a priory knowledge” (you need to know what you are doing and what files to change), but I do like the solution he presented by using include, which makes the file a lot “cleaner”.

Ok.

From your reply 5 to his post:

Would you suggest just a single file /etc/server/main.config?

Or should there be also:

/usr/lib/server/main.config

/etc/server/main.config

/home/.config/server/main.config

From my current self, my question was a bit pointless. Of course only /etc/server/main.config and not /usr/lib/server/main.config etc. No need to make it overly complex.

I think that by naming that file main, he intended to have just a single file so that not much knowledge is required (otherwise you would have know about not only all the .config files, but also the .conf ones).

Yes.

Also, from his comments in 2 it would be a good idea to use multiple .config files and multiple .d subdirs for complex configs, but I do think it is the case here.

Yes.

I agree with that but one problem that we might face is having that single .config file “restored” from the template when using Qubes. What do you think? Maybe having a directory in main.config that is not restored solves that? (i.e., all customizations should be in that dir)

Not sure I understand. I guess you are referring to Qubes template implementation (and Qubes bind-dirs and such)? Yes, indeed! How could I forget about this.

So let’s sneak into the proposal…?
include /usr/local/etc/server-config.d/*.conf
/usr/local is persistent in Qubes TemplateBasedVMs. Then users would not have to modify their template, but could for example also persistently change settings in their TemplateBased AppVMs.
Should main.config file[s] only contain include statements and
comments and nothing else?

I am pasting /etc/logrotate.conf below, which contains more than include statements. I think that there is no need for that as this convention is very clear about what the related .conf files should be and their contents are consistent to each other. I do like that the other content in /etc/logrotate.conf is consistent to the content of the other files in /etc/logrotate.d/.
# see "man logrotate" for details
# rotate log files weekly
weekly

# keep 4 weeks worth of backlogs
rotate 4

# create new (empty) log files after rotating old ones
create

# uncomment this if you want your log files compressed
#compress

# packages drop log rotation information into this directory
include /etc/logrotate.d

# no packages own wtmp, or btmp -- we'll rotate them here
/var/log/wtmp {
    missingok
    monthly
    create 0664 root utmp
    rotate 1
}

/var/log/btmp {
    missingok
    monthly
    create 0660 root utmp
    rotate 1
}

# system-specific logs may be configured here
I like that.

This new approach would not change the way the .conf files work when dealing with configurations from the previous files, right? If it is needed, they are able to to ignore the previous statements with, for example, listen_ip=.

Yes.

I am saying that because it looks like now main.config is allowed to change and just having include statements suggests that prefixing them with # is enough to ignore.

Yes.

That could work for non-Qubes systems, but as I mentioned above having main.config restored can be a problem.

Yes. Let’s see if you like my suggestion to cover this using /usr/local/etc/server-config.d/*.conf as mentioned above.

dau

2017-03-21 00:42:35 UTC

! In T635#12787, @Patrick wrote:
If you are using git anyhow to generate the diff, could you please fork GitHub - adrelanos/proposals: https://phabricator.whonix.org/T635 | https://phabricator.whonix.org/T634 and commit? That would make the diff a lot easier to apply and read for me.

Great, there is a repo! I’ll send a PR then.

Looks like a user name is missing. There is no user .config.

Indeed /home/.config/ is wrong. I believe I did not copy that properly from the draft. Sorry!

include vs home folder is not obvious yet in the draft? Perhaps the config should contain a variable $HOME` or so?

That would be a good idea. Is that how other conventions reference ~?

And only be adhered by applications running as user such as unMessage? And ignored by applications run by the init system (such a web servers)?

That’s also a good idea. Configs in ~ are probably the most specific we can get. I am not sure though if in the end that will be useful/used. For example, listen_ip is customized likely due to some network/system scenario (in our case, Whonix’ workstation/gateway setup), but do you think that would change for different users of the same network and system? Maybe it’s useful when you do not have privileges to change the other files?
Not sure I understand. I guess you are referring to Qubes template implementation (and Qubes bind-dirs and such)? Yes, indeed! How could I forget about this.

So let’s sneak into the proposal…?
include /usr/local/etc/server-config.d/*.conf
/usr/local is persistent in Qubes TemplateBasedVMs. Then users would not have to modify their template, but could for example also persistently change settings in their TemplateBased AppVMs.
That certainly solves the problem!

Yes. Let’s see if you like my suggestion to cover this using /usr/local/etc/server-config.d/*.conf as mentioned above.

Completely agree with that.

I can send the changes. The only thing I wanted to clarify before that is whether we are going to have ~/.config/server-config.d/ by default or not, and how we are going to reference ~ in main.config if so.

Thanks!

Patrick

2017-03-21 12:37:24 UTC

include vs home folder is not obvious yet in the draft? Perhaps the config should contain a variable $HOME` or so?

That would be a good idea. Is that how other conventions reference ~?

I am not aware of any other conventions using $HOME. ~ might be shell specific. $HOME is being used in /etc a few times. I think $HOME is a good choice, because it’s a very common environment variable in user’s environments. (Not sure stuff even breaks without it.)

And only be adhered by applications running as user such as unMessage? And ignored by applications run by the init system (such a web servers)?

That’s also a good idea. Configs in ~ are probably the most specific we can get. I am not sure though if in the end that will be useful/used. For example, listen_ip is customized likely due to some network/system scenario (in our case, Whonix’ workstation/gateway setup), but do you think that would change for different users of the same network and system? Maybe it’s useful when you do not have privileges to change the other files?

With parsing config files in $HOME we may over complicate the proposal with a rule like this… When using include $HOME/server-config.d/*.conf,

if $HOME is set, actually do include $HOME/server-config.d/*.conf (this is when applications runs under a user account, not run by the init system)

if $HOME is unset, include $HOME/server-config.d/*.conf has no effect (happens most likely when applications are run by the init system)

I am also getting uncertain of the usefulness of parsing config files in $HOME.

Perhaps there are cases where they don’t have privileges to change /etc but install as user? But why at the same time they want to run a server software on a system they don’t have write access to /etc? Strange case.

The Whonix use case which started with “let’s make applications usually listening on 127.0.0.1 such as onionshare, ricochet, unMessage and ZeroNet actually listen on eth0 without having all these applications having to provide individual config files and parse these”. For Whonix use case, parsing of config files in $HOME is not required.

we’d pre-configure that for the user by shipping a config file with Whonix

other changes could be done by the user (perhaps VPN users should change it)

If we don’t find a convincing argument for $HOME config file parsing for this proposal, I guess we better leave it out. We can listen for feedback and if a compelling use case is provided at a later time, we could still upgrade to a higher version of the proposal at a later point.
Not sure I understand. I guess you are referring to Qubes template implementation (and Qubes bind-dirs and such)? Yes, indeed! How could I forget about this.

So let’s sneak into the proposal…?
include /usr/local/etc/server-config.d/*.conf
/usr/local is persistent in Qubes TemplateBasedVMs. Then users would not have to modify their template, but could for example also persistently change settings in their TemplateBased AppVMs.
That certainly solves the problem!
Great!

Yes. Let’s see if you like my suggestion to cover this using /usr/local/etc/server-config.d/*.conf as mentioned above.

Completely agree with that.

I can send the changes. The only thing I wanted to clarify before that is whether we are going to have ~/.config/server-config.d/ by default or not, and how we are going to reference ~ in main.config if so.

Sure thing. Please go ahead.

dau

2017-03-21 20:18:09 UTC

Alright, I just made the PR with the changes we discussed. Also made minor changes to make it a bit more friendlier.

! In T635#12791, @Patrick wrote:
With parsing config files in $HOME we may over complicate the proposal with a rule like this… When using include $HOME/server-config.d/*.conf,

if $HOME is set, actually do include $HOME/server-config.d/*.conf (this is when applications runs under a user account, not run by the init system)

if $HOME is unset, include $HOME/server-config.d/*.conf has no effect (happens most likely when applications are run by the init system)

I am also getting uncertain of the usefulness of parsing config files in $HOME.

Perhaps there are cases where they don’t have privileges to change /etc but install as user? But why at the same time they want to run a server software on a system they don’t have write access to /etc? Strange case.

The Whonix use case which started with “let’s make applications usually listening on 127.0.0.1 such as onionshare, ricochet, unMessage and ZeroNet actually listen on eth0 without having all these applications having to provide individual config files and parse these”. For Whonix use case, parsing of config files in $HOME is not required.

we’d pre-configure that for the user by shipping a config file with Whonix

other changes could be done by the user (perhaps VPN users should change it)

If we don’t find a convincing argument for $HOME config file parsing for this proposal, I guess we better leave it out. We can listen for feedback and if a compelling use case is provided at a later time, we could still upgrade to a higher version of the proposal at a later point.

Completely agree. Removed the ~ entry but added a note about using home for the configurations.

One thing I noticed: Tomas’ suggestions used /etc/server/main.config for the main file and server/config.d/ for the directories. I think that makes sense and makes things more consistent. I’m just not sure if server is the best name because that might be too generic. What do you think?

By the way, I think it is a good idea to use VCS for this. Would it be possible to create a proposals repository? Similar to the one Tor uses.

Patrick

2017-03-21 22:08:02 UTC

dau (Felipe Dau):

dau added a comment.

Alright, I just made the PR with the changes we discussed. Also made minor changes to make it a bit more friendlier.

Thanks, merged!

In ⚓ T635 write draft for local listener standard on debian-devel, @Patrick wrote:

With parsing config files in $HOME we may over complicate the proposal with a rule like this… When using include $HOME/server-config.d/*.conf,

if $HOME is set, actually do include $HOME/server-config.d/*.conf (this is when applications runs under a user account, not run by the init system)

if $HOME is unset, include $HOME/server-config.d/*.conf has no effect (happens most likely when applications are run by the init system)

I am also getting uncertain of the usefulness of parsing config files in $HOME.

Perhaps there are cases where they don’t have privileges to change /etc but install as user? But why at the same time they want to run a server software on a system they don’t have write access to /etc? Strange case.

The Whonix use case which started with “let’s make applications usually listening on 127.0.0.1 such as onionshare, ricochet, unMessage and ZeroNet actually listen on eth0 without having all these applications having to provide individual config files and parse these”. For Whonix use case, parsing of config files in $HOME is not required.

we’d pre-configure that for the user by shipping a config file with Whonix

other changes could be done by the user (perhaps VPN users should change it)

If we don’t find a convincing argument for $HOME config file parsing for this proposal, I guess we better leave it out. We can listen for feedback and if a compelling use case is provided at a later time, we could still upgrade to a higher version of the proposal at a later point.

Completely agree. Removed the ~ entry but added a note about using home for the configurations.

Great!

One thing I noticed: Tomas’ suggestions used /etc/server/main.config for the main file and server/config.d/ for the directories. I think that makes sense and makes things more consistent. I’m just not sure if server is the best name because that might be too generic. What do you think?

Neither /etc/server-config.d nor /etc/server look like ideal choices
for folder names to me. I was hoping for better suggestions. =)

/etc/server/main.config is not so pretty. A folder with a single
config file. Why a folder then.

(I also had in mind /etc/listen/main.conf as well as
/etc/listen/conf.d/*.conf. That would be similar to /etc/dpkg. But
/etc/listen/conf.d/ would be less consistent with /usr/lib/listen.d/.)

What about as name, listen? Is this specific enough only on the topic
of listen or will it likely expand to also covering “server”?

Let’s go with the following?

/etc/listen.conf

/etc/listen.d/*.conf

/usr/local/etc/listen.d/*.conf

/usr/lib/listen.d/*.conf

By the way, I think it is a good idea to use VCS for this. Would it be possible to create a proposals repository? Similar to the one Tor uses.

Good idea!

Patrick

2017-03-21 22:13:09 UTC

Patrick

2017-03-21 22:34:19 UTC

dau

2017-03-21 23:59:08 UTC

! In T635#12809, @Patrick wrote:
Neither /etc/server-config.d nor /etc/server look like ideal choices
for folder names to me. I was hoping for better suggestions. =)

/etc/server/main.config is not so pretty. A folder with a single
config file. Why a folder then.

(I also had in mind /etc/listen/main.conf as well as
/etc/listen/conf.d/*.conf. That would be similar to /etc/dpkg. But
/etc/listen/conf.d/ would be less consistent with /usr/lib/listen.d/.)

What about as name, listen? Is this specific enough only on the topic
of listen or will it likely expand to also covering “server”?

Let’s go with the following?

/etc/listen.conf

/etc/listen.d/*.conf

/usr/local/etc/listen.d/*.conf

/usr/lib/listen.d/*.conf

Sure, sounds good. I am really bad at names (specially in English), so I think it looks good and if this thing starts to grow with more general server content and people start to support it, they will either change to a better name or just get used to it.

! In T635#12816, @Patrick wrote:
Maybe a bit late, but perhaps now that the proposal is available in git, we could even mention something like this…?

This proposal is available in git [xx], pull requests are welcome.

[xx] https://github.com/Whonix/proposals/blob/master/listen-port-convention.txt

Good idea!

Btw, can I use numbers to index the references instead of x, xx, etc?

I think sometimes a shorter comment plus pull request would be less ambiguous, quicker to implement fixes, and if someone is up for that, more power to them.

I agree. I just really hope the discussions do not get split between github and phabricator. Hopefully one day github adds support to other trackers?

What do you think? If that is good as is or modified, could you add it please in your next pull request?

Sure, I will. But before that, can I make another suggestion?

I think that once the proposals start to be created we might get confused to reference all of them, so what if we prefixed them just like Tor does, but using the ticket numbers instead? That way we not only have an ID for each one of them, but which is related to a ticket and can be easily found/referenced/etc.

That means the current proposals would become:

634-stackable-wrappers.txt

635-listen-port-convention.txt

What do you think?

Patrick

2017-03-22 01:19:50 UTC

Maybe a bit late, but perhaps now that the proposal is available in git, we could even mention something like this…?

This proposal is available in git [xx], pull requests are welcome.

[xx] https://github.com/Whonix/proposals/blob/master/listen-port-convention.txt

Good idea!

Btw, can I use numbers to index the references instead of x, xx, etc?

Sure.

I think sometimes a shorter comment plus pull request would be less ambiguous, quicker to implement fixes, and if someone is up for that, more power to them.

I agree. I just really hope the discussions do not get split between github and phabricator.

Indeed an issue.

Hopefully one day github adds support to other trackers?

I’d hope so. Sadly I have little hope for that. Too closed / commercial.
More likely that some open source github clone takes off.

What do you think? If that is good as is or modified, could you add it please in your next pull request?

Sure, I will. But before that, can I make another suggestion?

Sure!

I think that once the proposals start to be created we might get confused to reference all of them, so what if we prefixed them just like Tor does, but using the ticket numbers instead? That way we not only have an ID for each one of them, but which is related to a ticket and can be easily found/referenced/etc.

That means the current proposals would become:

634-stackable-wrappers.txt

635-listen-port-convention.txt

What do you think?

Good idea. Please add. And if you find it useful, perhaps the subject
and link to the ticket would be useful for each proposal / draft as well.

dau

2017-03-22 02:40:08 UTC

Patrick

2017-03-22 03:25:55 UTC

! In T635#12820, @dau wrote:
Let me know what you think of the new PR.

Should we add “(Proposal X)” (or something like that) to the title of the documents as well?

Can do.

Perhaps like this. A dividing marker… (To keep things simple for copy and paste, not to mess that up when posting it to the final posting destination.)
subject here

##################################################

text here
! In T635#12809, @Patrick wrote:
Let’s go with the following?

/etc/listen.conf

/etc/listen.d/*.conf

/usr/local/etc/listen.d/*.conf

/usr/lib/listen.d/*.conf

After making the changes, I am not sure if with the above you actually meant to set a new order to the parsing of these. Was that the case? Because I changed the order to that.

Otherwise I can move back to how they were such as :

/usr/lib/listen.d/*.conf

/usr/local/etc/listen.d/*.conf

/etc/listen.d/*.conf

Thanks!

The latter is better. (Similar on how systemd does it.)

(I messed that up. Wanted to show how /etc/listen.conf looks nice vs /etc/listen.d/*.conf.)

Patrick

2017-03-22 03:29:17 UTC

Patrick

2017-03-22 03:34:01 UTC

dau

2017-03-22 10:11:12 UTC

! In T635#12823, @Patrick wrote:
Right. Could you please add that text the Description part? I mean, the Description part is the part that gets actually copied over to the Debian mailing list in this case.

I’d suggest to add the link to the ticket and repo rather as a bonus. To be added somewhere at the end. We are mainly addressing the people on the mailing list. Beginning the text with two external links, I mean with…

Ticket: ⚓ T635 write draft for local listener standard on debian-devel

File: proposals/635-listen-port-convention.txt at master · Kicksecure/proposals · GitHub

might deter readers? Or am I over thinking this? I don’t want to start painting the proverbial bikeshed.

Right, that’s much better. That first section is important just for the ones familiar with Whonix’ proposals and that feedback note would not make sense there. I moved to the very end of the documents.

Just made another update. Let me know what you think!

Thanks!

Patrick

2017-03-23 11:13:50 UTC

Patrick

2017-03-23 11:14:35 UTC

dau

2017-03-23 19:35:04 UTC

Patrick

2017-03-26 11:23:19 UTC

dau

2017-03-26 17:12:21 UTC

Patrick

2017-03-26 17:42:31 UTC

dau

2017-03-28 00:30:39 UTC

Re: convention on listen port local or all network interfaces etc. - revision 2
Hi Patrick,

On Sun, Mar 26, 2017 at 11:22:00AM +0000, Patrick Schleizer wrote:
> A convention on listen port local or all network interfaces etc. would
> be desirable.

I agree, though I'm not sure we can get consensus on this…

However, I also think that you should restart this discussion once stretch has
been released. Now it's not the best time to have such discussions…

> Any questions? Any suggestions? What do you think?
> 
> Feedback on the ticket [0] and pull requests on the repository [1] are
> welcome.
 
(I haven't looked at neither…) but I'd suggest you file a bug against the
"general" pseudo-package in the Debian BTS as that's a more proper place to
discuss changes in Debian.

Thanks!


-- 
cheers,
	Holger
Now that I am thinking about it, there is something that is not clear to me: is this convention a change to Debian or a “good practice” that any *nix system could/should do?

I am asking that because requesting a change to Debian seems to take a really long time, but proposing it like a “suggestion” which some (Debian) people agree to might not. Does that make sense?

Patrick

2017-03-28 11:29:54 UTC

Holger Levsen:

[…]

However, I also think that you should restart this discussion once stretch has been released.

[…]

suggest you file a bug against the “general” pseudo-package in the Debian BTS

[…]

[zzz] Well, why not. For the fun and learning experience by it, let’s wait until #debian_stretch gets released and then open a bug against general on Debian BTS.

! In T635#12846, @dau wrote:
Re: convention on listen port local or all network interfaces etc. - revision 2

Thank you for monitoring for replies!

Now that I am thinking about it, there is something that is not clear to me: is this convention a change to Debian or a “good practice” that any *nix system could/should do?

I did not dare to suggest a change to Debian. Did not have this in mind originally.

I am asking that because requesting a change to Debian seems to take a really long time, but proposing it like a “suggestion” which some (Debian) people agree to might not. Does that make sense?

Right. That’s what I had in mind.

I think [zzz] is a bonus. No need to wait for it or to let it block our progress here. We got some great suggestions for our initial draft which we then implemented. I think in this case it’s fair to interpret the absence of criticism and praise as an okay signal to move forward. This is probably also how things work. Some people create some convention, then just do it. If it then proves to be useful and working, it might become an official Debian standard. Neither Debian nor freedesktop want to be the gatekeepers of innovation.

( This is similar to Bug 1367 – define bash non-login shell snippet drop-in folder /etc/bash.bashrc.d/ in FHS. )

( If you like to go further down into the rabbit hole on that one… See https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=675008. )

Meaning, if you’d like to develop a python lib for parsing this convention and/or adhered this convention in unMessage, that’d be awesome! I would also be suggesting other application developers to implement this convention.

What do you think?

dau

2017-03-28 19:12:13 UTC

! In T635#12847, @Patrick wrote:
I think [zzz] is a bonus. No need to wait for it or to let it block our progress here. We got some great suggestions for our initial draft which we then implemented. I think in this case it’s fair to interpret the absence of criticism and praise as an okay signal to move forward. This is probably also how things work. Some people create some convention, then just do it. If it then proves to be useful and working, it might become an official Debian standard. Neither Debian nor freedesktop want to be the gatekeepers of innovation.

Cool!

( This is similar to Bug 1367 – define bash non-login shell snippet drop-in folder /etc/bash.bashrc.d/ in FHS. )

( If you like to go further down into the rabbit hole on that one… See https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=675008. )

Meaning, if you’d like to develop a python lib for parsing this convention and/or adhered this convention in unMessage, that’d be awesome! I would also be suggesting other application developers to implement this convention.

What do you think?

Alright, I should be able to take a look into that earlier next week.

It would be great to have a generic tool that parses these directories. For example parse('listen') would go through all those directories, merging the configs.

Could we make some kind of Python binding for an existing solution?

Patrick

2017-03-28 19:51:05 UTC

dau

2017-03-28 20:11:18 UTC

Patrick

2017-03-28 22:58:23 UTC

dau

2017-03-29 00:32:29 UTC

dau

2017-04-08 23:25:39 UTC

This class is reasonably flexible so that people are able to customize how files are found and parsed. I pretty much just wrote the API, but here is what I have in mind:
import argparse


class Parser:
    @classmethod
    def parse_global(cls, filepath):
        """Read `filepath`, parse it and return all directory paths found.

        The directories in the list are in the order they are intended to be
        read.
        """

    @classmethod
    def check_filename(cls, filename):
        """Return `True` if `filename` is a config file. `False` otherwise."""

    @classmethod
    def find_files(cls, dirpath):
        """Walk through the directory and return all file paths found.

        Depending on `check_filename`, each file is prepended by the
        directory's path and added to the list. The files in the list are in
        the order they are intended to be read.
        """

    @classmethod
    def parse_file(cls, filepath, configs=None):
        """Read `filepath`, parse it and return a `config: value` dict.

        If no configs are provided, a new dict is created. As the file is read,
        values are added to the configs with
        `add_config(configs, config, value)`."""

    @classmethod
    def add_config(cls, configs, config, value):
        """Add `config: value` to the configs dict and return it.

        In case the config did not exist, it is added to the dict. Otherwise,
        it replaces or merges to the existing one.
        """

    @classmethod
    def parse(cls, name):
        """Read the name of the config and return a `config: value` dict.

        The name defaults to a `/etc/<name>.conf` file path but can optionally
        specify a different path and extension. Its format should be something
        like `[/custom/path/]name[.custom]`.
        """
        cfgs = dict()

        for dirpath in cls.parse_global(name):
            for filepath in cls.find_files(dirpath):
                cfgs = cls.parse_file(filepath, cfgs)

        return cfgs

    @classmethod
    def serialize(cls, configs):
        """Serialize the configs dict into a convenient format."""

    @classmethod
    def parse_to_serialized(cls, name):
        """Parse configs and return them serialized."""
        return cls.serialize(cls.parse(name))


def call_parse():
    """Read a config name and print the serialized configs.

    This function should be accessed through an entry point. The name is passed
    via a command-line call and returns the configs in a format that
    applications are able to read in the language they use.
    """
    argparser = argparse.ArgumentParser()
    argparser.add_argument('name')
    args = argparser.parse_args()

    print Parser.parse_to_serialized(args.name)
It could be used with:
import Parser

cfgs = Parser.parse('listen')
listen_ip = cfgs['listen_ip']

# bind on `listen_ip`
Based on man systemd.unit “The syntax is inspired by XDG Desktop Entry Specification1.desktop files, which are in turn inspired by Microsoft Windows .ini files.”, which also inspired Python’s ConfigParser. I am going to take a look into systemd’s code, but I think using Python’s own config parser might be a better idea.

Once this is implemented, any non-Python app can make a command-line call to get a config parsed and serialized (maybe to json?).

If this becomes used by other people, they may implement their own file parser for their own syntax and override its methods. If other parsers become popular, we can implement them as well so that users can specify which parser they would like to use. Depending on all the customizations we allow and provide in this package, it might be easier to move from class methods to instance methods.

I chose to update the configs dict for every config parsing, for every file parsing. The processing increases but memory usage decreases. Then you do not have to merge multiple (possibly big) config dictionaries in the end as they are merged during the process.

We need to pick another name other than Parser or ConfigParser.

What do you think?

P.S. I noticed the link in the description needs to be updated. Would you change to proposals/635-listen-port-convention.txt at master · Kicksecure/proposals · GitHub?

Patrick

2017-04-12 12:15:23 UTC

Patrick

2017-04-12 12:18:07 UTC

dau

2017-04-12 20:55:56 UTC

! In T635#12941, @Patrick wrote:
Perhaps
check_filename → is_config_file
find_files → find_config_files
?

On the other hand… I personally prefer the very at the end and to have some consistency, i.e. config_read, config_check and so forth.

I like your suggestions. Especially is_config_file - It makes much more sense. Thanks!

Do you know how other popular libraries do it? Perhaps there is even a convention on that?

I do not know and I am not aware of such convention, but it’s worth to check.

json is not trivially parsed in shell scripts. For shell scripts, something that can be simply evaluated would be nice. (Just echo listen_ip=xxx, so listen_ip=xxx gets set.)

Hmm, maybe I was not very clear on the explanation. I did not suggest changing the configs format (the .conf files). They should still look like something evaluable.

What I suggested was the use of json to return the parsed configs to applications. As a Python user (and as this would be a Python package) I would expect to get a dict with the configs to use on my application. For non-Python applications, that dict would be serialized to json because I believe that is one of the most popular formats these days with support from many languages. Also, we can offer multiple “serializers” if needed.

Was that your concern?

I may not be the best person to comment on that. Could you ask your mentor or otherwise please for more input on this one?

Sure, I will talk to him. And can you clarify what exactly you are unsure about?

! In T635#12942, @Patrick wrote:
Sorry, I cannot find the mistake. Could you send a PR please?

Can I send PRs to phabricator? The error is in the description of this ticket (at the top of the page):

[…]
draft:

GitHub - adrelanos/proposals: https://phabricator.whonix.org/T635 | https://phabricator.whonix.org/T634

TODO:
[…]

Thanks!

Patrick

2017-04-13 08:35:17 UTC

json is not trivially parsed in shell scripts. For shell scripts, something that can be simply evaluated would be nice. (Just echo listen_ip=xxx, so listen_ip=xxx gets set.)

Hmm, maybe I was not very clear on the explanation. I did not suggest changing the configs format (the .conf files). They should still look like something evaluable.

I did not interpret your suggestion as a change in the conf files.

What I suggested was the use of json to return the parsed configs to applications.

Yes, got that. That’s why I wanted to point out, that json is hard to parse in shell scripts.

As a Python user (and as this would be a Python package) I would expect to get a dict with the configs to use on my application. For non-Python applications, that dict would be serialized to json because I believe that is one of the most popular formats these days with support from many languages. Also, we can offer multiple “serializers” if needed.

Yes, something that works for shell scripts would be nice.

Was that your concern?

Yes.

I may not be the best person to comment on that. Could you ask your mentor or otherwise please for more input on this one?

Sure, I will talk to him. And can you clarify what exactly you are unsure about?

Just generally.

! In T635#12942, @Patrick wrote:
Sorry, I cannot find the mistake. Could you send a PR please?

Can I send PRs to phabricator?

No.

The error is in the description of this ticket (at the top of the page):

Ah, now fixed.

dau

2017-04-13 09:36:51 UTC

! In T635#12944, @Patrick wrote:

json is not trivially parsed in shell scripts. For shell scripts, something that can be simply evaluated would be nice. (Just echo listen_ip=xxx, so listen_ip=xxx gets set.)

Hmm, maybe I was not very clear on the explanation. I did not suggest changing the configs format (the .conf files). They should still look like something evaluable.

I did not interpret your suggestion as a change in the conf files.

What I suggested was the use of json to return the parsed configs to applications.

Yes, got that. That’s why I wanted to point out, that json is hard to parse in shell scripts.

As a Python user (and as this would be a Python package) I would expect to get a dict with the configs to use on my application. For non-Python applications, that dict would be serialized to json because I believe that is one of the most popular formats these days with support from many languages. Also, we can offer multiple “serializers” if needed.

Yes, something that works for shell scripts would be nice.

Was that your concern?

Yes.

Ohh, now I got it! Sure, it should be possible to offer multiple serializers and also allow users to implement their own.

I may not be the best person to comment on that. Could you ask your mentor or otherwise please for more input on this one?

Sure, I will talk to him. And can you clarify what exactly you are unsure about?

Just generally.

Alright!

! In T635#12942, @Patrick wrote:
Sorry, I cannot find the mistake. Could you send a PR please?

Can I send PRs to phabricator?

No.

The error is in the description of this ticket (at the top of the page):

Ah, now fixed.

Thanks!

Patrick

2017-04-15 09:39:52 UTC

dau

2017-04-17 22:35:07 UTC

dau

2017-04-17 22:39:23 UTC

Patrick

2017-04-17 23:03:02 UTC

dau

2017-04-24 03:32:34 UTC

Patrick

2017-04-28 14:32:58 UTC