msgdispatcher_dispatch_x lacks new lines for output from console applications

[phabricator-migrator]

Information

ID: 97
PHID: PHID-TASK-xv5wmicz6qznmnr45kd2
Author: Patrick
Status at Migration Time: resolved
Priority at Migration Time: Normal

Description

Sometimes it is useful for whonixcheck to add stuff like this to the output.

sudo -u debian-tor tor --verify-config

Naturally, lines aren’t prepended or terminated with <br></br>. Therefore new lines are ignored. They end all up in one huge line.

https://github.com/Whonix/msgcollector/blob/master/usr/lib/msgcollector/msgdispatcher_dispatch_x

Comments

---

Patrick

2015-01-17 00:16:03 UTC

Example. Result…

Jan 17 00:15:03.916 [notice] Tor v0.2.5.10 (git-43a5f3d91e726291) running on Linux with Libevent 2.0.19-stable, OpenSSL 1.0.1e and Zlib 1.2.7. Jan 17 00:15:03.916 [notice] Tor can't help you if you use it wrong! Learn how to be safe at https://www.torproject.org/download/download#warning Jan 17 00:15:03.916 [notice] Read configuration file "/etc/tor/torrc". Jan 17 00:15:03.918 [warn] /var/run/tor is not owned by this user (root, 0) but by debian-tor (106). Perhaps you are running Tor as the wrong user? Jan 17 00:15:03.918 [warn] Failed to parse/validate config: Couldn't access/create private data directory "/var/run/tor" Jan 17 00:15:03.918 [err] Reading config failed--see warnings above.

Desired:

Jan 17 00:15:03.916 [notice] Tor v0.2.5.10 (git-43a5f3d91e726291) running on Linux with Libevent 2.0.19-stable, OpenSSL 1.0.1e and Zlib 1.2.7.
Jan 17 00:15:03.916 [notice] Tor can't help you if you use it wrong! Learn how to be safe at https://www.torproject.org/download/download#warning
Jan 17 00:15:03.916 [notice] Read configuration file "/etc/tor/torrc".
Jan 17 00:15:03.918 [warn] /var/run/tor is not owned by this user (root, 0) but by debian-tor (106). Perhaps you are running Tor as the wrong user?
Jan 17 00:15:03.918 [warn] Failed to parse/validate config: Couldn't access/create private data directory "/var/run/tor"
Jan 17 00:15:03.918 [err] Reading config failed--see warnings above.

---

Patrick

2015-01-17 00:17:48 UTC

@troubadour, do you think there is any good way to do this on the python level? (Without requiring us to do lots of formating changes…)

Otherwise I guess I could also inject the <br></br> at the end of each line using a bash function.

---

troubadour

2015-01-17 21:28:46 UTC

There is no easy way to do this in msgdispatcher_dispatch_x. It just shows what it receives. The formatting of the messages is all done in whonixcheck and /usr/lib/msgcollector/msgcollector (pretty_type_x), and I don’t think it makes sense to even try to do it at python level.

---

Patrick

2015-01-18 04:36:51 UTC

We indeed shouldn’t mix this into msgdispatcher_dispatch_x.

---

Are you any good at string manipulation stuff?

Maybe you could write a small python tool that does the trick?

For testing purposes, suppose any input as this.

x="$(telnet -h 2>&1)"

Variable $x will now contain the following content.

telnet: invalid option -- 'h'
Usage: telnet [-4] [-6] [-8] [-E] [-L] [-a] [-d] [-e char] [-l user]
        [-n tracefile] [ -b addr ] [-r] [host-name [port]]

Let’s say I was running something like this…

x="$(telnet -h 2>&1)" ; ./some-python-script "$x"

Could you come up with an output that looks like this?

telnet: invalid option -- 'h'<br></br>
Usage: telnet [-4] [-6] [-8] [-E] [-L] [-a] [-d] [-e char] [-l user]<br></br>
        [-n tracefile] [ -b addr ] [-r] [host-name [port]]

In other words, can you append <br></br> at the end of every line but the last line?

Don’t put too much effort into this.

---

troubadour

2015-01-22 21:10:41 UTC

It’s not that much a question of string manipulation than of finding a common mark in each line to place the extra html. Do we want a universal script?

In this:
Jan 17 00:15:03.916 [notice] Tor v0.2.5.10 (git-43a5f3d91e726291) running on Linux with Libevent 2.0.19-stable, OpenSSL 1.0.1e and Zlib 1.2.7. Jan 17 00:15:03.916 [notice] Tor can’t help you if you use it wrong! Learn how to be safe at Tor Project | Download Jan 17 00:15:03.916 [notice] Read configuration file “/etc/tor/torrc”. Jan 17 00:15:03.918 [warn] /var/run/tor is not owned by this user (root, 0) but by debian-tor (106). Perhaps you are running Tor as the wrong user? Jan 17 00:15:03.918 [warn] Failed to parse/validate config: Couldn’t access/create private data directory “/var/run/tor” Jan 17 00:15:03.918 [err] Reading config failed–see warnings above.
if there is no newlines, we could take the colon, providing the preceding characters(mmm dd hh) have always the same format/length.

In the telnet output example, in python, we would first have to know that we want to grab stderr, and replace \n with ,br>

repr(err)
‘“telnet: invalid option – 'y'\nUsage: telnet [-4] [-6] [-8] [-E] [-L] [-a] [-d] [-e char] [-l user]\n\t[-n tracefile] [ -b addr ] [-r] [host-name [port]]\n”’

That’s a lot of conditions.

---

troubadour

2015-01-22 21:14:23 UTC

Was carired a little. ./some-python-script “$x” is possible if we have new lines

---

Patrick

2015-01-22 21:38:30 UTC

Do we want a universal script?

Yes, universal. To terminate any kind of terminal output

common mark in each line

The common mark is \n, the newline character. Before (or after?) that, we need to inject <br></br>. Except for the last line.

---

Patrick

2015-02-04 20:42:23 UTC

How to append a string at the end of each line of a string in python?:
How to append a string at the end of each line of a string in python? - Stack Overflow

---

troubadour

2015-02-04 20:55:27 UTC

Forgot that one. Yeah, the solution is there. Only, do we expect a \n at the end of the last line?

---

Patrick

2015-02-04 21:08:56 UTC

Good question. Dunno. Is it important?

Damn. Looks like I messed up that question highlighting “except last line”.

---

troubadour

2015-02-04 21:15:57 UTC

Dunno either. The question is, if it is there, do we want to strip it?

---

Patrick

2015-02-04 21:31:04 UTC

We probably don’t want to strip any \n at all?

---

troubadour

2015-02-25 20:53:30 UTC

I believe I made it complicated from the beginning. The following should do the job.

import sys
in_ = sys.argv[1]
out = '<br />'.join(in_.split('\n'))
print out```

Note the `'<br />'`. Just found it.

Example:
```x="$(ls -l)"
python add_html.py "$x"```

---

Patrick

2015-02-25 21:33:37 UTC

Looks good. Works fine!

Perhaps change that line to.

out = '<br />\n'.join(in_.split('\n'))

Then it looks a bit better when running scripts using xtrace for debugging.

Could you commit that to usr/lib/msgcollector/br_add or so? Not sure. Perhaps you have a better name in mind.

---

Patrick

2015-02-25 21:34:26 UTC

Ah. Overlooked add_html. Fine either way.

---

troubadour

2015-02-26 21:12:38 UTC

! In T97#2559, @Patrick wrote:
Looks good. Works fine!

Perhaps change that line to.

out = '<br />\n'.join(in_.split('\n'))

Then it looks a bit better when running scripts using xtrace for debugging.
The problem is that the \n will be displayed literally in the html, not interpreted as new line.

---

Patrick

2015-02-26 21:32:37 UTC

I don’t think so. It works for me. Perfectly. Example:

(Stored the modified script as “a” for testing.)

x="$(cat --help)"

br_added="$(./a "$x")"

/usr/lib/msgcollector/msgdispatcher_dispatch_x "info" "test title" "$br_added" "" ""

echo "$br_added"
Usage: cat [OPTION]... [FILE]...<br />
Concatenate FILE(s), or standard input, to standard output.<br />
<br />
  -A, --show-all           equivalent to -vET<br />
  -b, --number-nonblank    number nonempty output lines, overrides -n<br />
  -e                       equivalent to -vE<br />
  -E, --show-ends          display $ at end of each line<br />
  -n, --number             number all output lines<br />
  -s, --squeeze-blank      suppress repeated empty output lines<br />
  -t                       equivalent to -vT<br />
  -T, --show-tabs          display TAB characters as ^I<br />
  -u                       (ignored)<br />
  -v, --show-nonprinting   use ^ and M- notation, except for LFD and TAB<br />
      --help     display this help and exit<br />
      --version  output version information and exit<br />
<br />
With no FILE, or when FILE is -, read standard input.<br />
<br />
Examples:<br />
  cat f - g  Output f's contents, then standard input, then g's contents.<br />
  cat        Copy standard input to standard output.<br />
<br />
Report cat bugs to bug-coreutils@gnu.org<br />
GNU coreutils home page: <http://www.gnu.org/software/coreutils/><br />
General help using GNU software: <http://www.gnu.org/gethelp/><br />
For complete documentation, run: info coreutils 'cat invocation'

The \n is not inside $br_added, so it won’t be visible in the html either.

---

troubadour

2015-02-26 22:03:13 UTC

Nice demonstration. Was testing by adding \n randomly in the text.

Added file br_add to msgcollector.
new file br_add: add html line break to messages containing \n · troubadoour/msgcollector@c68878a · GitHub

---

Patrick

2015-02-27 00:22:59 UTC

Merged.

Can you please add a license header on top and a newline at the end?

---

troubadour

2015-02-27 20:01:27 UTC

Done.
license · troubadoour/msgcollector@2e2c464 · GitHub

---

Patrick

2015-02-27 20:07:51 UTC

Thanks! Merged. And added a minor fix on top (right order for license header and shebang).
fix · Kicksecure/msgcollector@ef03b5d · GitHub

---

Patrick

2015-03-05 15:58:09 UTC

Created a follow up task: T216

Anything left to do here or closable, @troubadour?

---

troubadour

2015-03-05 19:26:49 UTC

We can close it.