Stream Isolation and Python Requests module

Development
1

I have read the documentation referring to stream isolation support. I have tested it with while true; do curl https://check.torproject.org | grep strong; done;

Stream isolation is confirmed by viewing the exit nodes changing for each request. I tried this with the simple requests module from python, but it is always the same IP address. At first I thought maybe I could open socket with 127.0.0.1:9051 and send the SIGNAL NEWNYM to it, but that never worked. Would it be better to keep digging down this path or to just wrap calls to ‘curl’ for when I need to programmatically interact with a website?

2

“You need to help Tor to help you.”

Tor needs to “know” that it’s another invocation supposed to be stream isolated.

Tor Project: manual

IsolateSOCKSAuth

Don’t share circuits with streams for which different SOCKS authentication was provided.

(For HTTPTunnelPort connections, this option looks at the Proxy-Authorization and X-Tor-Stream-Isolation headers. On by default; […])

I.e. set a socks user name or see manual quote above.

That would be difficult. SIGNAL NEWNYM is a non-blocking command. You’d need a Tor controller and wait for the event when Tor actually finished doing that. Complex and unneeded.

1 Like
4

Sorry I withdrew my previous comment because I do not think it was replying to yours.

Prior to reacting, thank you kindly for your diligent efforts Patrick! One day maybe I might acquire aptitude in systems administration, operating system’s, etcetera, and loan a hand on Whonix’s undertaking. The work done here is greatly appreciated.

set a socks user name or see manual quote above

I have perused the man page you linked and sadly I don’t think I’m all around read enough to comprehend the necessities here. The man page makes reference to tor as an executable or script that has many command line args. One of them is IsolateSOCKSAuth. On the Whonix-WS am I expected to invoke tor with these args for each stream isolated instance?

(For HTTPTunnelPort connections, this option looks at the Proxy-Authorization and X-Tor-Stream-Isolation headers. On by default; […])

When this says headers, does it mean in the HTTP header portion? Like when using curl -H "Connection: keep-alive", or does it mean something else?

I do not expect you to read through this and answer every question. Is there some more preliminary reading topics I should investigate further? I don’t think I know enough about SOCKS either, so much of this is going over my head.

5

You don’t need to start Tor. No Tor config changes required.

A proxy (Tor or any) can accept username / password for authentication.
Tor “abuses” this to get hints when to use stream isolation.
What is required is setting a different socks user name per request which should be stream isolated.
This can be done with curl from Whonix-Workstation. Syntax:

curl --proxy socks5h://user:password@ip:port

The user could be anything. Probably randomly generated. Dunno the maximum. Password can be anything as it would be ignored by Tor as far as I know. Examples:

UWT_DEV_PASSTHROUGH=1 curl --proxy socks5h://random1:password@10.152.152.10:9152 https:/check.torproject.org/api/ip
UWT_DEV_PASSTHROUGH=1 curl --proxy socks5h://random2:password@10.152.152.10:9152 https:/check.torproject.org/api/ip

and also with python requests.

Please post a working example if you figure this out.

6

I have sorted out some way to get stream isolation working, you can imitate it utilizing this content. The final lines of the script incorporates a brief printout of the details.

username == password, but what matters is the unique authentication.

import requests
from bs4 import BeautifulSoup
checkTorUrl = "https://check.torproject.org"
counter = 0
import secrets, string, signal
keepScraping = True
def signal_handler(sig,frame):
        global keepScraping
        print('Handling Ctrl+C')
        keepScraping = False
signal.signal(signal.SIGINT,signal_handler)
IPList = {}
def generateUID(low=10,high=20):
        UID_Set = string.ascii_lowercase + string.ascii_uppercase + string.digits
        return ''.join(secrets.choice(UID_Set) for i in range(secrets.choice(range(low,high))))
while keepScraping:
        counter += 1
        with requests.Session() as s:
                # randomly generate 'username:password' pair for each session
                # 'username:password'
                creds = '{}:{}'.format(generateUID(),generateUID())
                s.proxies = {'http': 'socks5h://{}@localhost:9050'.format(creds), 'https': 'socks5h://{}@localhost:9050'.format(creds)}
                r = s.get(checkTorUrl)
                soup = BeautifulSoup(r.text,'html.parser')
                IPLogged = soup.select('strong')[0].string
                if(IPLogged in IPList):
                        IPList[IPLogged] += 1
                else:
                        IPList[IPLogged] = 1
                print("check torproject shows exit node IP = {}".format(IPLogged))
avg = 0
usesCount = {}
for IP in IPList:
        if(IPList[IP] in usesCount): # IPList[IP] returns number of times a request was made with that IP
                usesCount[IPList[IP]] += 1
        else:
                usesCount[IPList[IP]] = 1
        avg += IPList[IP]
avg/=counter
print("Number of GET requests made to https://check.torproject.org = {}".format(counter))
print("Total unique IPs used = {}".format(len(IPList)))
print("Average uses per IP address = {}".format(avg))
print("Median uses per IP address = {}".format(max(zip(usesCount.values(),usesCount.keys()))[1])) # returns the highest amount of frequency bin
print("Max used IP address ({}) = {}".format(max(zip(IPList.values(),IPList.keys()))[1],max(IPList.values())))

Importantly the username must remain unique between all instances executing the script. Otherwise it shares the tor circuit of when it was first generated. Have not tested for VM resetting and username sharing.

1 Like