Not The Wizard

Oz Solomon's Blog

Page 2 of 2

Multi-Generational Flaws in iOS, Part 1

I’ve been an iPhone user since the iPhone 3G went on sale in Canada.  I love my iPhone, but I also love to hate it.

Years ago I started feeling that iOS is showing its age: Features and UX design decisions that were made when the OS and it’s ecosystem were brand new just didn’t make sense anymore.

I wrote down a list of feature requests for iOS in 2011, intending to write a blog post.  While I was taking my time (if you can call three year “taking your time”), I watched two major releases of iOS (6, 7) come and go, but not a single item from the original list had been addressed!  At this point I no longer consider them feature requests, but rather flaws in iOS.  And just what are those flaws?  Read on.

Continue reading

See the Actual “To” Address in Outlook

My Outlook is connected to an Exchange account that accepts emails for multiple domains.  When I open up the emails, Outlook always displays my name in the To field, but doesn’t show which email (i.e. which domain) the mail was actually sent to.

I decided that manually checking the mail headers for this information was a dumb way to do things, so I wrote a little macro to automate this little task.  Hopefully some of you out there will find it useful.

Installation Instructions

  1. Press Alt+F11
  2. On the left hand side, expand the tree until you see ThisOutlookSession.  Make sure ThisOutlookSession is highlighted.
  3. Paste the code from below
  4. Close the Visual Basic window
  5. Attach the new macro to a toolbar item.  The way to do this varies with your version of Outlook.  For example, in Outlook 2007:
    1. Open any email
    2. Press the down arrow on the right side of the quick access toolbar
    3. Select “More Commands”
    4. In the “Choose Commands From” dropdown, select “Macros”
    5. Add the macro to the right hand side
    6. (Optional) Change the icon by highlighting the macro and selecting Modify.
  6. You can add the macro either to the “view email” screen as I explained above, or to the toolbar of the main inbox screen.



Added Dec 17: To ensure the macro isn’t blocked by the Outlook macro security settings, follow the instructions in this article to sign your own macros.

Multimedia Keyboards – How to Prevent Multiple Outlook Instances From Opening

Multimedia KeyboardIf you have a modern keyboard, it most likely has a Mail key on the top row which will launch your mail client.  I prefer to use this key not only as a means to launch my mail client, but also as a shortcut for switching to it.

Unfortunately, by default, if Outlook is your mail client, hitting the Mail key repeatedly opens multiple copies of Outlook.  Fortunately, the fix is easy:

  1. Launch the Registry Editor by hitting Win+R and typing regedit in the dialog box.
  2. When the Registry Editor opens, navigate to Computer\HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Explorer\AppKey\15. (You should see the following value there: Association=mailto)
  3. From the Edit menu pick New -> String Value
  4. In Name, type ShellExecute
  5. In Data, type outlook.exe /recycle

That’s it!  You can now hit the Mail key multiple times.

I use Microsoft keyboards but this should hopefully work with other manufacturers’ keyboards as well.

Tip: Don’t install the special Microsoft Keyboard software that came with your keyboard or that is offered to you via Windows Update – it will mess things up.

How to Destroy Your Marketing Efficiency

What's wrong with this picture?

Last week I got an newsletter email from Future Shop promoting their latest sale.  (To my American readers – Future Shop is a brand of Best Buy stores in Canada).

Aside from the fact that half the images in the email didn’t load (a sure way in itself to destroy your marketing efficiency), I was more bothered by a much much more idiotic use of the email’s pixel space.

Come Back Daily For … Yeah Right

This newsletter had a format I’m sure you’re familiar with.  It boasted a “deal of the day” and hinted that I should come back daily, over the next 8 days to see what that day’s deal would be

How stupid can they be?

(I don’t mean to pick specifically on Future Shop.  Other companies do this too.  Dell’s 12 Days of Deals comes to mind immediately though I’m sure there are many others.)

I’m sure that like me, most people subscribed to their newsletters receive many emails every day.  The time we spend going over these emails is increasingly short.  That’s why marketers try to optimize open rates (how many people bother opening the email).  Get that?  They’re thrilled if you just opened it, because that’s a huge hurdle.  Then what do they do?  They waste their pixel on telling me to come back again tomorrow to open their email again.  They’re trying to create suspense and anticipation by hiding the next days’ deals, but they’re losing out because nobody cares enough to check again tomorrow.

A better way

Marketers, please pay attention: Instead of trying to create fake anticipation, I have a better plan for you.  In that newsletter, the one I actually bothered to open, tell me what’s going to be on sale every day.  Show me a little calendar view with a picture of a laptop today, a tablet tomorrow and some worthless anti-virus a week from Tuesday.

Now that I know what’s coming, I can mentally flag to come back to your site that day to check out the deal I’m actually interested in.  If you’re deal speaks to me, I’ll be sure to visit your site, even if it’s a few days from now.

Another offender

Lessons in Choosing Hosting Providers

We’ll make this right for you.

— Corinne Moore, our Peer1 Account Manager

Since I built Status Shuffle 4.5 years ago, I’ve used five different hosting providers to run the core application and three other providers for auxiliary servers.  My experiences ranged from truly horrible (as with Peak Web Hosting where we terminated service after 1 week) all the way to excellent.  On the Excellent corner we have Peer1 and SoftLayer.  Let me tell you why.

The Five Pillars of an Excellent Hosting Experience

When I started out, I had no money so cost was the #1 driving factor.  Low cost doesn’t necessarily mean lower value, just as higher cost doesn’t automatically mean better value.  In fact, I can say that over the year I’ve gotten damn good value for my money with various budget providers (and vice verse).  Nowadays I look for the best blend of the following:

  • Cost: Still important.  You don’t want to overpay, you want to pay just enough to maximize the next four.
  • Physical location: The physical location of the server matter for a variety of reasons:
    • Distance from your end users: The closer the server to your end users (as judged by millisecond latency) the better their experience is.  If you’re building a game server for people in Australia, then it better not be located in Florida.  Beyond the physical distance, your provider’s peering agreement can also mean a short route to users nearby and a long one.  I’ve seen traceroutes to smaller providers go from Toronto to Chicago and back just to hit servers that were 5 miles away from me.  Always test!
    • Distance from an API provider: If the purpose of your server is to work with an API, then the latency to the API provider really matters.  For example, if your server needs to make a lot of requests to Facebook, then you better be near Facebook’s servers.  If your server is part of Google real-time-bidding exchange, then you your best bet is the next rack over.
    • Distance from you: The closer the server is to you (latency wise), the better your personal experience is when you do “stuff” on your server.  I always like my stuff to be done faster.  But more importantly, if the server is close to you physically, you can do some of your own maintenance as opposed to relying on 3rd parties.  Those 3rd parties may be capable, but they don’t know your infrastructure nearly as well as you do.  Personally, knowing that I don’t have to get on a plane to access my mission critical servers helps me to sleep better at night.
  • Uptime: What percentage of the time are your servers working and reachable?  With one provider, it seemed like there was always something happening.  I knew things were really bad when I noticed that I knew their tech support number by heart.
  • Time-to-fix: When something goes wrong (eventually something always goes wrong), what’s their ability to diagnose and recover from the problem quickly?  And I don’t just mean network connectivity: If your servers are managed, the provider’s ability to upgrade/fix parts quickly and efficiently are a big part of the story.  I had one provider where uptime was great, but when the shit hit the fan, it really hit the fan.  It got to the point that if I needed upgrades to the server I was scared to death of prolonged downtime (and rightfully so).  That same provider at some point had a big power outage and it took them 36 hours to get my server online after power came back.  Adiós1.
  • Quality customer support: You server is down.  Customers are calling you every two seconds.  Your sales partners are chewing your ear off.  Everyone wants answers.  You want answers, damn it!  You call the support line, all nervous and panicked.  This is money time, and customer support can blow it in so many ways.  For example, they can:
    • Ask you to “file a ticket” and refuse to let you talk to the tech on the floor even though you have pertinent information knowing full well that ticket won’t be seen until everything is long over
    • Refuse to provide, or have no access to, any useful information about the state of things
    • Blame you first, investigate later
    • Call you in an attempt to save your business only after you call to terminate your hosting due to any of the above.  (Note to all service providers: Customer retention efforts are best when preventative, not post-mortem).

A Shout-out

I’ve lived through every negative example listed above, but never ever with the following two companies.  Here is that positive feedback I promised in the last post.


I’ve used Peer1 for a long time.  I can only recall two times where the network became unavailable and in both cases it was resolved in under 10 minutes.  (Uptime: Check, Time-to-fix: Check).  A couple of years ago I wanted to move servers from LA to Toronto so they would be physically closer to our base of operation.  Luckily Peer1 is in both cities (Physical location: Check).  Pricing was reasonable (Check.  Though what’s up with the 1 year minimum commitment?  Please stop with BS).  But what really won me over was the customer service.

Peer1 shipped my servers from LA to Toronto.  When they arrived, I drove down to rack them.  I opened the boxes and to my horror I was staring at two dented servers.  Whoever packed them in LA did a horrible job, and although four servers were fine, two looked like they were hit on the side with a hammer.  The UPS looked like I dug it up from a dumpster.  I don’t need to tell you that servers don’t like getting smashed.

I literally froze, thinking about the time and money cost of replacing those servers.  Then I called my customer rep.  Now, there are many things she could have said.  99% of those things would have caused Peer1 to lose me as a customer for life.  But what did she say? “We’ll make this right for you.”  That’s it.  Pure and simple.  They fucked up, and they will fix it.  If anybody at Peer1’s reading this, you need to give Corinne a raise.  I’m still paying you thousands of dollars a month, you can use some of that.

Customer service: Check!

In the end the servers were miraculously fine, and Peer1 paid for the damaged UPS.


I was referred to SoftLayer by a friend.  Like anything else in life, it’s easier to choose when you get a recommendation from someone you trust.  (If you’re evaluating a provider and don’t personally know anybody who’s using them, try searching Web Hosting Talk forums, but be prepared for negativity bias and obvious astroturfing).

As with Peer1, network uptime is pretty good, and recovery is very fast when there are problems.  Reaching technical support is easy and they are very friendly.

One nice surprise came during a hardware upgrade I scheduled for 4am.  I was trying to call tech support, but accidentally called the sales line.  And someone picked up.  At 4am on a weekend.  “So what?” you say.  I’ll tell you what: With every other managed hosting provider I worked with, when you’re having problems at 4am and need some sort of hardware change (e.g. more RAM, more hard disks) they may refuse to do the work until Sales come in on Monday and price it out.  I can still remember myself screaming “I’ll work out the fucking price with Sales on Monday just get the server working now!” at a previous provider and believe me it didn’t sound that nice when I was actually screaming it.  So obviously, I like the fact that SoftLayer has their act together.

To top it all off, the prices are great (through you do have to call in and negotiate), no minimum commitments (yey!) and they get you up and running super fast.  That 4am hardware upgrade didn’t cost me a cent, even though it involved a tech off-hours.  All servers have remove KVM built in which can be a life saver, and you don’t pay for traffic flowing to your server (only traffic flowing out).

Highly recommended.



1 In reality moving providers is a very painful task, involving a search for a new provider, migration planning and testing, setting up new servers and the possibility of downtime when moving.  So “adiós” not only means you’re leaving, it also means you’re screwed.

A Story About Positive Feedback

Let’s play a little social psychology guessing game, and see how well you do.

Imagine you have an app on Facebook where users can hit a Contact Us link and write you anything that’s on their mind.  Further imagine that you get between 7,000 and 10,000 such contact requests a year.

How many of those users do you think clicked on Contact Us to say something nice?

Before I tell you the answer, let me be clear that all the numbers are real.  Status Shuffle has a lot of users, and those users write us that much every year.  In fact, I’m surprised they don’t write us more given the sheer usage volume Status Shuffle has.

The Shocking Truth

The answer is one.  One person a year writing something nice.  That’s less than 0.02% of contacts.  That’s a statistical anomaly.  We actually got one last week, and I can’t even remember the last time it happened before that.

When positive feedback arrives at our inbox, it’s such a celebration that it immediately gets bubbled up he same escalation path reserved for those OMG-we’re-down-and-the-whole-company-needs-to-know tickets.  And I doubt our users or product are unique.  I remember the same exact thing happening at a previous employer.

It’s All About the Effort

I don’t think people lack positive feelings about products they use, but it’s as though it’s not worth the effort to go out of one’s way to say so.

Point at hand: As soon as we got the aforementioned nice email, I posted it on our Facebook page and quickly got 5,000 likes and many positive comments.  Thumbs up is easy, so people do it.

Interestingly, inside Status Shuffle users are given the opportunity to thumbs up or down content.  It’s the same amount of effort.  We consistently see 10 thumbs up for every thumb down.  So given equal feedback mechanisms, both effortless, people tend to be positive.

Why are people are more inclined to contact you in order to complain? Maybe it’s because we get nothing in return for just saying something nice, but if we complain we have a vested interest: We might get the company to change it’s product.  I’m sure this guy or his friends have research about this.

What Have We Learned?

Wouldn’t it be nice if right about now I had an actionable recommendation for you?  Well, I have two.  The first is a bit weak and the second you won’t like.

My first advice is to make it as frictionless as possible for your customers to show their love for you.  It may be as simple as popping up a single question dialog: Do you love us?  Yes?  No?  Aggregate the result and share with your team.  Hopefully they’ll smile at the results.

My second advice is to go out and spread the love.  Your love.  When was the last time you wrote a nice blog post about a positive experience?  When was the last time you left a nice review for a restaurant you frequent or a book you enjoyed?  We all need to help the world lose the negativity bias.

As for myself, I’m as guilty as anyone1.  That, I’m going to fix with the next post.


1 In fact, I plan to use this very blog as a platform to complain and rant about many things.

nginx Reverse Proxy Can Cause IE to Fail

Some Background

For years I had Apache serving up Status Shuffle.  It wasn’t perfect, but it worked.  In fact, it worked for so well that it handled a million users a day on one box with plenty of room to spare.  However, in late 2011, Facebook started requiring HTTPS support from all it’s publishers, us included.  We bought an SSL certificate, made the necessary configuration changes, then restarted Apache.  It all seemed to work as planned.

Over the next few days we watched our server logs closely and discovered that our error rate has gone up.  It seemed that the extra overhead caused by the SSL negotiation step was enough to dramatically increase the failure rate for some of our users (probably those with unstable Internet connections).  Ideally, we would use HTTP keep-alives to allow everyone to open the connection once and make multiple requests through it, thereby offsetting the SSL negotiation overhead.  Alas, trying to hold thousands of connections with Apache’s pre-fork MDM was a sure way to eat up all the RAM in our box.  Instead, I decided to put up an nginx box in front of Apache in a reverse proxy setup.

The idea is simple and well documented.  You use nginx to handle connections with the end users.  nginx in turn calls Apache to actually do the work, then finally hands off the data to the end user.  nginx can hold thousands of connections open, thus keep-alives are no problem.  (Note: Apache 2.4 was just released and it can nativity do all of this using “event” MPM.  Alas, “event” doesn’t support SSL connections so we’ll be sticking with this nginx setup for now).

Redmond, We Have a Problem

At first we didn’t realize anything was wrong.  As I mentioned above, the amount of connection issues we were logging went down dramatically so we were very happy.  But then the complaints started: “It doesn’t work” and “When I load Status Shuffle it looks funny”.  When more and more complaints started piling up, we couldn’t ignore them anymore, even though the app worked perfectly on every machine and browser we could get our hands on.

All we know was that all the people complaining were using Internet Explorer (versions, 6, 7, 8 and 9).  I added extra client side logging, and it appeared that IE was failing to execute our JavaScript files, while spitting out “Invalid character” errors.

A Google search regarding the “Invalid character” error failed to add clarity.  Some said this error would be returned if IE failed to download the file at all.  Some implicated an old (now fixed) bug in IE 6 where it wasn’t properly decoding gzip’ed web pages properly.  And that couldn’t be it because we were seeing the issue in Internet Explorer versions up to 9 (the latest version at the time of writing).

User Visible Symptoms

We finally caught a break when a user who reported the issue agreed to remoting session with us.  Over the course of an hour we poked around her computer, trying to figure out what’s wrong.  The strangest thing of all was that she shared the computer with her husband, and when she logged in through his account, the problem didn’t exist.  I concluded that there must be a corruption somewhere in her local user registry settings, but couldn’t tell what it was in the time I had with her.

As Dan and I poked around the user’s computer, we noticed IE was in fact downloading the JavaScript files.  In fact we were able to save them on to her desktop through IE, and they looked fine.  However, when we viewed them in the context of the application using the developer toolbar, they looked like a garbled binary stream.  It was obvious the browser wasn’t decoding the compressed response properly.

Piecing It Together

I had a theory that nginx’s proxy was causing the issue.  We quickly confirmed this by temporarily taking nginx out of the loop and hitting Apache directly.  The errors stopped coming in.

Scanning the nginx documentation for the proxy_cache directive revealed this innocuous looking sentence:

nginx does not handle “Vary” headers when caching.

The Vary header is used to tell caching proxies that a response is tied to a particular request header format.  For example, when your browser requests a web page, it will tell the web server that it will accept (understand) compressed results by using this request header:

The web server will then happily compress the page and return it with (at least) the following two response headers:

The first means that the response is compressed using gzip.  The second means: This response is only valid for requests that have the same Accept-Encoding value that you just sent me.

As stated in the documentation, nginx doesn’t handle the Vary response header.  The sensible thing to expect from nginx is that it would not cache responses that contain Vary.  Instead, what nginx does is cache the result of the first request and serve it to everyone, even if they don’t have the same request header.

As an experiment, I make a normal request through nginx using Firefox and got a compressed response as expected.  I then modified FireFox by changing the about:config setting network.http.accept-encoding to blank and reissued my request.  By setting network.http.accept-encoding to blank, I’m advertising to the server that I don’t support gzip or deflate encoding.  Surely enough, nginx served me the compressed cached copy and FireFox showed binary garbage instead of my JavaScript file.

I assume this is what was happening with IE for all those people.  I estimate that 0.5%-1% of our Internet Explorer users were affected.  If Status Shuffle didn’t have the massive request volume that it does, we would have probably never caught on to this.

Fixing It

I definitly consider this to be an nginx bug.  But at least there is an easy fix.  We moved all compression away from Apache and into nginx.

This means that in Apache, we removed:

And in nginx we added:

We now have no more errors and no more angry users.


Update Feb 29, 2012: I filed a bug against nginx 1.0.12 on the nginx bug tracker.  Maxim Dounin of nginx offered two interesting ways of working around the issue, so please follow the bug report link to read his comment.

Newer posts »

© 2024 Not The Wizard

Theme by Anders NorenUp ↑