Transitioning to more secure passwords

by Martin Westin in

With all the news of hacked databases (mostly at Sony) and the clear-text or poorly hashed passwords in their datasets, I thought I might offer my standard trick for transitioning to a more secure form of hashing. I think some sites don't change passwords security for fear of annoying users or the workload involved in managing a transition. This simple technique is completely invisible to the user and very low maintenance for the developer.

I will be giving examples from the Devise library for Rails apps, since I recently implemented it there.

The technique is very very simple

You configure your authentication to check passwords against both the old and the new form of hashed password. And when you find a match for the old hash you update your database with the version of the password encoded using the new hash. You keep this dual check in place until all (or most likely most) of your users have logged in and had their passwords changed. The unlucky few can use your password recovery feature if you have one.

Metacode of the basic principle:

if new_hash(password) == stored_password
  if old_hash(password) == stored_password

How to implement this transition in Devise

I implemented this by overriding the method valid_password? injected into your User model.

class User

  def valid_password?(incoming_password)
    result = super incoming_password
    if !result
      # try old encryptor during transition
      digest = Devise::Encryptors::LegacyEncryptor.digest(incoming_password, self.class.stretches, self.password_salt, self.class.pepper)
      result = Devise.secure_compare(digest, self.encrypted_password)
      if result
        # update password to use new encryptor when there is a match
        self.password = incoming_password


Fairly simple. You may need to hard-code some parameters (salt, stretching, pepper) if they cause problems.

If you are changing from, say, sha1 to sha256, you can easily check the character lengths of the passwords in your database to check the "adoption rate" of the new hashes.

Implications on Security

You should realize that you ARE lowering your security level slightly by effectively allowing 2 different password checks. In reality this problem is small and only really matters if you have plain passwords you are transitioning from (and you really shouldn't have). The problem then becomes real since I could login using a stolen new (supposedly) secure hash as the given password. In this case I would definitely disallow any password of the same length as, or simple reg-ex match for, your new hashing system to avoid this hole.

You will also not fully benefit from the new hashing system until you remove the "dual check" after a reasonable period of time.

If you can live with that to gain the benefits of a clean migration for you and your users this is a nice technique. I know from reading and talking to developers that I am far from the only of the first to come up with something like this. Many apps and sites have used and continue to use this kind of technique to beef-up password hash-strength without bothering users.

Am I Fierce?

by Martin Westin in

I love Dropbox and use it every day. I love Flipboard and use it every day. So imagine my reaction to seeing my (and my co-founders') little company named as one of the 15 coolest young companies in the world next to all these great companies. That was really special.

Fierce Wireless about Great Connection

They didn't get all the details right but if you want to know what I am up to most of the time I have seen worse descriptions. ;)

Minimalist Wordpress Deployment

by Martin Westin in

So, my "sugar daddy" datacenter is kicking me out. This is where I have been hosting absolutely free for a decade so I have nothing bad to say about them... at all. :-) My 4U quad-core2duo, 4GB RAM, 8TB SATA beast will soon become homeless.... Did I mention they let me host that monster for free?

This website was a minor task for that server but last Sunday I moved it onto a minuscule "cloud" VPS at Gelsys. Running PHP and MySQL on 256MB of RAM (and no swap) and expecting a stable server under load did require some tinkering. I am not going to detail every little thing. I'll only outline the basic setup.

I have Wordpress spitting out static html files which is what is being served for 99% of all requests. PHP is only being called on to re-generate the files when the caches expire and to serve up the admin interface. In this way my mini-VPS can handle a pretty decent amount of traffic. I don't generally get a lot of traffic but I am curious to see how the server behaves during a spike.

Wordpress caching is handled by the excellent W3 Total Cache plugin which recently got official support for Nginx. Yay! no more custom rewrite rules. My only problem getting it running this time was that it refused to cache the index page, which is kind-of bad when it is one of the heaviest pages to render. Turned out it was a simple precedence issue. "/" is considered a folder which I caught in a rule before the caching rules had a chance to do their thing.

Nginx is the web server. I have been running it for a few years already but for this it is absolutely essential to save on ram and get the best performance serving static files.

PHP is called from Nginx over an unix socket and served by php-fpm. PHP is compiled with my favorite accelerator eaccelerator. It is just the simplest to get going and has shown good performance for my apps. PHP has been scaled down a lot in php.ini, reducing mainly memory usage and timeouts.

MySQL loves memory so I had to do the equivalent here. Basically taking the standard performance tips and doing the opposite. I know I will only server Wordpress and I know I don't have any enormous datasets so MySQl should be fine on very low memory. Also since I limit the number of php processes I consequently limit the maximum number of client connections MySQL will need to keep track of.

The only other service running is ssh and I was surprised to find that sftp was one of the biggest memory hogs. Uploading a bunch of files of sftp and the server would immediately come dangerously close to running our of memory. In fact it did a bunch of times during my tweaking.

I have been thinking about switching from Wordpress to some static website generator but that is for another time. This setup looks fairly smooth at the moment.

Graylog2 on Mac OS X

by Martin Westin in

I have been playing with Graylog2 on my Mac today. Since the setup guides are all for Debian and not fully compatible with Mac OS X I thought I'd mention the changes I needed to make to get thing rolling smoothly. The guides are good, so go read them in the wikis on Github. I won't re-iterate them, only point out the minor changes and tweaks I had to make.

Graylog2 comes in two main parts. The server and the web interface. I'll start with the server component.

Install The Server

Mac OS X has java bundled with the OS (for now). There is no need to install anything. The configuration file needs one non-obvious tweak.

mongodb_host = # localhost

Java resolves localhost to the strangest thing. It tries to connect to the Bonjour name and external IP (e.g. Martin's Mac/ instead of which is what you want. Instead of opening MongoDB up to external access I changed the configuration to point to the loopback IP directly.

Starting The Server

I didn't get the daemon script to start and did not investigate is since I run Graylog2 for evaluation and development and like seeing the output. Starting by running the jar file requires that you sudo.

sudo java -jar graylog2-server.jar debug

That gets Graylog2 running and spitting out a lot of fun info so you know you are logging thing as you expect.

Installing The Web Interface

You can follow most of those steps if you don't have rails and Bundler and that stuff installed. For testing and development, I would suggest running the interface using Passenger Standalone instead of Apache. And, you install Passenger as a gem and not apt, of-course.

The cool thing about installing passenger standalone is that it will compile and run itself the first time you call passenger start. It will take a few minutes that first time but after that it will start instantly.

Logging from your Rails app

In the Rails app I want to log from I installed Graylog2 Exceptions. It is a small Rack middleware with practically no configuration. Only problem is that it has not been updated to comply with the current version of the Graylog2 server. Until it is updated, you have to modify the source for it. A very small mod. For me it is ok as long as I am still on my Mac and not a server.


> cd /to/my/app/dir
> bundle open graylog2_exceptions

This should get you the installed gem open in your editor. In the file lib/graylog2_exceptions.rb you need to add the version parameter to the notification message. Possibly this should be added to the gelf gem instead. I am not sure how that version string is supposed to be used.

Here is the modified method that does the actual notification:

  def send_to_graylog2 err
      notifier =[:hostname], @args[:port])
      puts notifier.notify!(
        :version => "1.0",
        :short_message => err.message, # <- this line is new!!!
        :full_message => err.backtrace.join("\n"),
        :level => @args[:level],
        :host => @args[:local_app_name],
        :file => err.backtrace[0].split(":")[0],
        :line => err.backtrace[0].split(":")[1]
    rescue => i_err
      puts "Graylog2 Exception logger. Could not send message: " + i_err.message

So, that is it. Finally I get all my exceptions in Graylog2. To try it out you can just raise some dummy exception – raise "Dummy Exception Error" – here and there and see them pop up in Graylog2.

Rails migration of indexes

by Martin Westin in ,

A small gotcha when changing indexes in a migration. To change an index one has to first remove it and then add it again. Removing an index is the tricky part. The documentation states: remove_index(table_name, index_name): Removes the index specified by index_name.

This is not strictly true as it turns out. The docs should probably say: remove_index(table_name, column_name)

The crux is that one cannot use this syntax to remove a named index. Rails assumes the index is named something like "tablename_columnname_index" or something similar.

To remove a named index one has to use the block syntax afaik:

change_table :tablename do |t|
  t.remove_index :name => :indexname
  t.index ["columnname"], :name => "indexname", :unique => true

Offensively lazy web developers

by Martin Westin in ,

There exist many websites and web applications with elements of poor usability, engineering, design and so on. Some exhibit features so poor I can only attribute them to laziness. Some actually make me feel offended that I have to jump through hoops to accommodate their laziness. Top of my list are form fields for postcodes, phone numbers, dates, times or any similar type of data. Simple numeric data. Easily validated and normalized. Yet I often see requirements to enter the data in a very specific format, often at odds with how humans customarily write such data. To most people (who are not developers) it can be very confusing to enter data in a perfectly normal way and have a computer tell them it is invalid. For me it is offensive since I know it is practically always the result of data not being normalized. A developer that does not normalize input data before validating it can only be described as lazy or, if you prefer, incompetent.


Nothing can be simpler, right. In many countries it is simply a few digits. Some through a few alphabetic characters in there. I'll focus on local Swedish websites and Swedish postcodes. We have 5 digit postcodes. They are typically written "123 45". So Why would any Swedish website validate that input and claim it is invalid. As a developer I know it is probably that a space is not a numeric character and that adding it also make the string 6 characters long and not 5.

Phone numbers

A phone number is a string of numeric characters. Anything else: spaces, parenthesis, dashes and other chrome, is just that: chrome. None of that is part of the data. No server at ATT, Telia, Vodafone or any other network carrier reads these thing and need them in order to route a call. Quite the opposite. Any phone network and particularly cellular networks require a very specific internationally standardized format. Guess what? It is all numeric. A Swedish cellphone might be typed "0701 - 23 45 67" in my address book but the phone sends 46701234567 to the network anytime I make a call.

Dates and Times

These are a bit more complex than the above. But the same principles apply. Then again, most web-focused programming languages have functions to parse a myriad of ways one could type a date or time and create a proper date or time object or data type. If you still find it does not work for you then, in this case, you should provide something other than a blank test field. Try googling for datepicker or timepicker. The problem will be one of choosing your favorite rather than finding anything at all.

Being constructive

I have ranted a bit now so I thought I'd put my code where my mouth is. Since I am not allowed to share my phone number normalizer code I wrote at work I thought I'd at least share the secret to all the normalizing tasks above... It's regex. Regular Expressions.

Getting rid of whitespace

The following will "match" whitespace characters. By substituting the matches with nothing you will simply remove all whitespace from a string. /\s/

Remove anything that is not a number

And the following will remove non digits if you substitute the matches with nothing. /\D/

A very very very simple example in Ruby

num = "(0)701 - 23 45 67".gsub(/\D/,'')

Seriously. That is all it takes to turn an offensive web form into a more humane one.

SOAP with Attachments in Ruby

by Martin Westin in

I found myself once again facing SOAP. This abomination of a protocol they even have the nerve to call "web services" is not my favorite type of API to interface with (how did you guess?). I think probably the only language with any decent support is Java and possibly .net. Neither rank among my favorite languages either. Funny that. My bigger problem is that the service I am interfacing with is noting as simple as sending an integer and getting an integer back. It requires that I post a multipart/mime SOAP message (aka SOAP with Attachments afaik). This is something that most SOAP libraries are not too keen on supporting.

What are multipart SOAP messages?

In short they are encoded a kind-of like email messages and their attachments but sent using http to a SOAP endpoint. The normal SOAP message becomes one of the mime parts and any other parts are called attachments and usually referenced from inside the SOAP message.

A little history

A few years ago in PHP I was stuck using NuSOAP and ended up basically bypassing most of NuSOAP and encoding the attachments and doing all that myself. The code was a real mess.

Last week I got to do it all over again. This time in Ruby. At work, we are porting our entire platform to Ruby, but detailing that process might be a post in itself. I was so happy when I found that soap4r has support for mime messages. Then I tried to use soap4r. Long story short. I liked it so much I chose to go with Savon instead... which has no mime support.

What I ended up with

The results of my efforts is not pretty by Ruby standards but a lot better than my old code in php. I patched Savon in two places. One to enable any namespace on the SOAP body (which is otherwise hard-coded to "wsdl") and has little to do with mime messages.

The other place was to intercept the output and check if the SOAP object had any attachments (parts) added to it. If so, it will take the intended output and encode that as a mime part and then encode the other parts and put it all together as a nice big http packet ready for posting.

I think it best if I just show the code now.

Any questions posted to the gist or here will be adressed to the best of my abilities.

Kanye West - 808s & Heartbreak

by Martin Westin in

Part of the series Milestones in Music


This album blew my mind completely. It not only shows a personal side to Kanye as a song-writer. The coolest thing is that the album could more or less have been produced in any bedroom studio in the world. We all have the tools at home to make this album... we are just not as good as Kanye. Damn!

That is the last milestone for now...

Burial - Burial

by Martin Westin in

Part of the series Milestones in Music


I stumbled upon this album on by mistake. What a find. So dirty. So atmospheric. Nothing like "normal" Dub Step. Really in a genre of his own. It also helped that he (William Bevan) remained very anonymous for a very long time. No interviews, no real name, nothing. The lines spoken by Benicio del Toro and Forrest Whittaker (I believe they are real samples and not re-creations) help set the spooky mood of the album.

Cliff Martinez - Solaris OST

by Martin Westin in

Part of the series Milestones in Music


This soundtrack is a pure flash of genius. I'd say it has a lot to do with making the film a pretty good film and on its own the music is just as good. Watching Solaris for the first time I could not wait for the credits so I could see who had made this magical music. I love walking with this music in my headphones. It turn the most boring walk into a surreal experience.