Thursday, July 01, 2010

This morning's intermittent downtime

This morning we saw some downtime that affected many of our customers. All scheduler functions were intermittently available/non available for a few minutes at a time. The downtime was unusual in that way as it was unavailable for 30 minutes total but over the course of about 2 hours before it was fully resolved.

This is the time it took to reliably diagnose the cause of the issue and then find a temporary, then longer term fix, and then implement it. The bad news is this issue could not be avoided or anticipated but the good news is, should the same issue happen again, we will be able to resolve it right away with almost unnoticeable downtime. There are also some steps we will be taking soon to even further reduce the likelihood of this particular problem happening.

As a result of work we did in late January we have experienced about 56 minutes of downtime in 151 days (including today) with about half of it happening on various nights (in the US) and the other half this morning. This is rather low downtime (or an uptime of 99.97% if you will) and well below some major multi hour outages that affected some huge companies's paid online services recently.

Unfortunately, downtime can and will happen in the future, sometimes at inconvenient times. Just like the power can fail, the internet connection may go down or your computer may die on you unexpectedly. When things do go wrong, we always diligently get to work to try to bring the site back up as quickly as possible. We usually can't tell how long it's going to last or can't help out with most customer inquiries. This is a very stressful moment for us as we understand this is a crucial service that needs to be up at all times ideally.

In order to mitigate the impact on your business we suggest you take the following steps:
  • subscribe to your calendar feed in a desktop calendar application such as Outlook 2007+, Windows Vista Calendar, the free Sunbird, Apple's iCal or even from your iPhone or a different web application such as Google Calendar. This will ensure that you have offline access to your upcoming appointments (including client contact details) as they will be automatically updated by your calendar software from our site. This is even useful if the site is still up but you have an internet outage. Go to "my settings" > "links" to find out more about the calendar feeds we offer.
  • download our standard CSV files which are Excel compatible and will also provide offline access to the appointments in addition to your own backups for your local records. This needs to be manually downloaded by clicking a link and saving the resulting file to your computer. Go to "my settings" > "links" to find out more about the CSV files we offer.

This way, should the scheduler be unavailable at the time you can still at least get to your appointment list until we bring it back up or your internet access is restored.