Perl and Python through a proxy server

laptop-firewallWriting scripts from inside a corporate environment to retrieve URLs or to call APIs, you’re bound to have to negotiate a web proxy server/firewall. Whilst it’s not too difficult to set the necessary environment variables, it often means having to wrap your script in another script, which entails deploying more files, writing more instructions, etc. It’s much neater to set up the proxy and associated authentication from inside the script.

Here is the bare bones code I’ve found most successful for Perl and Python, written on my Windows 7 Enterprise work laptop.

Perl 5.18 (Strawberry Perl )

#!/usr/bin/perl -w

use LWP::UserAgent;
use HTTP::Request;
use HTTP::Response;
use Data::Dumper;

# *** PUT YOUR VALUES HERE ***
use constant {
  PROXY      => "xxx.xxx.xxx.xxx",
  PROXY_PORT => "80",
  YOUR_ID    => "id-goes-here",
  YOUR_PW    => "password-goes-here",
  URL        => "http://www.google.co.uk",
};


my $ua = new LWP::UserAgent;
$ua->proxy('http', 'http://'.PROXY.':'.PROXY_PORT.'/');

my $request = HTTP::Request->new(GET => URL);
$request->proxy_authorization_basic( YOUR_ID, YOUR_PW );

# Make the call
my $response = $ua->request($request);

print Dumper $response;

if ($response->is_success) {
  print "SUCCESS\n"
} else {
  print "FAILURE\n"
}

Python 3.4

#!/usr/bin/python

import urllib.request
from urllib.error import URLError

# *** PUT YOUR VALUES HERE ***
PROXY      = "xxx.xxx.xxx.xxx"
PROXY_PORT = "80"
YOUR_ID    = "id-goes-here"
YOUR_PW    = "password-goes-here"
URL        = "http://www.google.co.uk"


proxy = urllib.request.ProxyHandler({
    "http":
    ''.join(["http://", YOUR_ID, ":", YOUR_PW, "@", PROXY, ":", PROXY_PORT])
})

auth = urllib.request.HTTPBasicAuthHandler()

opener = urllib.request.build_opener(proxy, auth, urllib.request.HTTPHandler)
urllib.request.install_opener(opener)

# Make the call
try:
    connection = urllib.request.urlopen(URL)
    print(connection.read())
    print("SUCCESS")
except URLError as e:
    print("FAILURE: %s" % e)

Building Mosquitto 1.4

A mosquitoHow to build Mosquitto

Why are we doing this?

I wanted to use MQTT to interact with a browser-based application in order to deliver real-time interactions such as notifications. Having read Oriel Ruis’ instructions, my initial approach was to put Lighttpd in front of Mosquitto, and tunnel websockets, but it was perplexing how we were going to secure it.

I asked a question on StackOverflow and, in mid-July 2014, Mosquitto got websockets. This will allow us to more easily use existing security models for MQTT.

However, since (at the time of writing) it was still pre-release, we need to compile v1.4 ourselves.

How to do it

These instructions were successful on a clean build of Mint Linux 17, i.e. Ubuntu 14.04.

The Brass Moustaches section towards the end of this post lists the additional steps I took due to missing dependencies.

Your mileage may vary…

Install libwebsockets

Option 1: build instructions for an newer version…

sudo apt-get install cmake libssl-dev
cd <SRC>   # i.e. your source code home
wget http://git.warmcat.com/cgi-bin/cgit/libwebsockets/snapshot/libwebsockets-1.3-chrome37-firefox30.tar.gz
tar -xzvf libwebsockets-1.3-chrome37-firefox30.tar.gz
cd libwebsockets-1.3-chrome37-firefox30/
mkdir build
cd build
cmake .. -DOPENSSL_ROOT_DIR=/usr/bin/openssl
make
sudo make install

Option 2: build instructions for an older version…

cd <SRC>   # i.e. your source code home
wget http://git.warmcat.com/cgi-bin/cgit/libwebsockets/snapshot/libwebsockets-1.22-chrome26-firefox18.tar.gz
tar -xzvf libwebsockets-1.22-chrome26-firefox18.tar.gz
sudo apt-get install autoconf
cd libwebsockets-1.22-chrome26-firefox18/
./autogen.sh
./configure
make
sudo make install

Install Mercurial tools

sudo apt-get install mercurial

Clone the Mosquitto repo and switch to the 1.4 branch

cd <SRC>   # i.e. your source code home
hg clone ssh://hg@bitbucket.org/oojah/mosquitto
cd mosquitto/
hg pull && hg update 1.4

make it

First edit config.mk and ensure that the websockets option is set to “yes”.

WITH_WEBSOCKETS:=yes

Then install pre-reqs:-

sudo apt-get install uuid-dev xsltproc docbook-xsl

And then…

make
make test
sudo make install

If need be, edit or create your config file and create the service user ID.

sudo vi /etc/mosquitto/mosquitto.conf
sudo useradd -r -m -d /var/lib/mosquitto -s /usr/sbin/nologin -g nogroup mosquitto

Then start it up…

sudo /usr/local/sbin/mosquitto -c /etc/mosquitto/mosquitto.conf

And you’re done.

Brass Moustaches

zlib

If you get an error about missing zlib, do this:-

cd <SRC>   # i.e. your source code home
wget http://zlib.net/zlib-1.2.8.tar.gz
tar xvf zlib-1.2.8.tar.gz
cd zlib-1.2.8
./configure
make
make test
sudo make install

ares.h

If you get an error about missing ares.h, do this:-

cd <SRC>   # i.e. your source code home
wget http://c-ares.haxx.se/download/c-ares-1.10.0.tar.gz
tar xvf c-ares-1.10.0.tar.gz
cd c-ares-1.10.0
./configure
make
sudo make install

make test and libwebsockets

make test failed for me. As per https://answers.launchpad.net/mosquitto/+question/252173, it might be that the websockets build has not put the library on the shared library path.

To diagnose this, just try bringing the test broker up directly:-

cd <SRC>/mosquitto/test/broker
../../src/mosquitto -p 1888

If it’s the shared library problem, you’ll see this immediately. To resolve this, do something like:-

find /usr -name libwebsockets.so

Mine was in /usr/local/lib64/libwebsockets.so

sudo vi /etc/ld.so.conf.d/lib64c.conf

…and add the following lines:-

# lib64c default configuration
/usr/local/lib64

…and then:-

sudo ldconfig

…and try the test again.

Public key error when cloning Bitbucket repo

If you get an error when cloning the Bitbucket repository, such as:-

remote: Permission denied (publickey).
abort: no suitable response from remote hg!

…then you might need to load your public key into Bitbucket.

Simple And Secure Passwords

…recovered from old blog (originally posted 26th April 2012)

 

After a pre(r)amble, this post is intended to provide a system through which you can generate passwords for your various on-line accounts that are:-

  • memorable
  • (adequately) secure
  • (generally) unique

This will be wonderfully redundant once we’ve sorted out federated identity management, biometrics and associated whatnots … but I recommend you don’t hold your breath.

Click here to cut to the chase

Preamble

I’m sure no one reading this has a password based on their pet’s name or containing their birthday, or has the same password across multiple accounts.  I’m sure everyone reading this has a mixture of lower and upper case letters and numbers in their passwords.  You’ve obviously never written a password down on a post-it note and put it in your desk drawer.  Of course, you also have all of your photos backed up on secure media kept somewhere other than in your house.  You’ve probably never left your wallet on top of the car or gone away for the weekend leaving the front door open.  And you brush your teeth twice a day, every day.  Don’t you?

A couple of months ago, my email account was hacked and a number of friends and colleagues sent spam.  The same thing seems to happen to someone else in my address book every few weeks.  Beyond the annoyance and having to apologise to everyone to whom you’ve just advertised drain-cleaner-based pharmaceuticals, there are many concerns here.  If it is your main email account, then it’s undoubtedly the one you use to authenticate you or to safeguard all manner of other services – Facebook, Twitter, WordPress, LinkedIn etc etc.  Your reputation could easily be trashed or, if it’s related to your Paypal account, you could easily find yourself significantly out of pocket.

In my case, my password was all lower case and was made up of concatenated dictionary words (I’m not going to tell you what it was but, believe me, it was childish and made me smirk).  These sorts of passwords are susceptible to simple brute force attacks – the miscreant starts at “A” and tries every combination of words in the dictionary plus all of the passwords published on sites like this.  One similar to mine can allegedly be cracked in around thirteen minutes (NB see footnote 2).

Furthermore, I used to use the same password on more than one site.  I had a small handful of different passwords and would use one for shopping sites, one for social sites, etc.  The Gnosis hack of Gawker showed how dangerous this can be – break one and someone will try it against one of your other accounts.

But we can’t remember unique passwords for every site, can we?  I work in IT and have seen cases where users have been expected to manage up to forty different log-ons.  Obviously, this was within an IT security framework demanding each was unique, non-dictionary, mixed upper and lower case, etc etc etc.  It’s nothing more than a recipe to force people to write the passwords down.

On that topic, someone close to me (who will remain nameless) was struggling to log into Amazon.  He was adamant that he was typing his password in correctly – he’d even checked it in his filofax (yes, a filofax – how quaint).  I said, “so it’s not this word written next to the word ‘Amazon’ here, is it?”

“No”, he said.

But it was.

So, what is the alternative?  You could get a password safe on your mobile.  Or you could try the following…

The Algorithm

  1. Take your favourite song lyric, song title, nursery rhyme, television programme, etc.  Something of around four or five words.
  2. Write down the initial letters, capitalising the last one
  3. Add a number, any number that has relevance to you.  Something around two or three digits.
  4. Take the initial letter of the application or site for which you’re generating a password.  Add it to the end.
  5. That’s it.  You now have a basically gibberish but memorable password of around eight or nine characters, mixing upper and lower case with numbers, thereby satisfying 99% of applications’ security requirements, and which is relatively unique across your many services and accounts.

So, if I take “Mary had a little lamb” and the year of my birth, my WordPress password would be:-

  • mhalL72W

All you need remember is the mnenonic and the number.  Nothing needs to be written down and, if we put it through the same password strength tester, we have ten days before a brute force attack will be successful. This is more than enough for our delinquent hacker to give up and go elsewhere.

Remember, you don’t need to outrun the bear – you just need to run faster than someone else.  Speaking of which, can I suggest a step (6)?

  • Add your own spice to the rules above.  Perhaps take the first and third letters of the app/site’s name, reverse the capitalisation or add an ampersand to the end; I’m sure you get the idea…


Footnotes:

  1. In the spirit of transparency, the algorithm wasn’t my idea.  I can’t recall where I came across it but, if I do, I will happily give all due credit.
  2. Don’t put your real password into a password strength testing website.
  3. No, this won’t really sort out our work passwords that need changing every 90 days.  We’ll have to live with those. Personally, I wonder whether policies requiring a password to be changed periodically materially improve system security.
Follow

Get every new post delivered to your Inbox.