Email notifications tribulations

Some of you may have noticed fewer email notifications than usual lately. This seems to be due mostly to 2 separate issues which happened coincidentally.. one of which took much longer to be corrected than the other.

An honest mistake

Even though we had made no changes to the live version of our email engine, our delivery rate (% of emails making it to the intended recipient's inbox) went down all of a sudden last month.  We quickly found out that we had been mistakenly put on a spam list operated by Spamhaus. This is a reputable company and their lists are being used by many email providers worldwide.

Remedial action: we informed them of the mistake and were able to get removed from the list within hours of noticing the issue.

Unfortunately, even though it was very effective in bringing our delivery rates back to normal levels, this was short lived. A number of our emails were now being dropped again.

A change of policy

Many of our emails were not being put in a spam folder for the most part, but simply silently dropped when received, and some business customers were affected more than others. After investigation we determined that the problem was affecting Yahoo! email addresses mostly and set out to find out why.

It turns out Yahoo! changed their global email policy last month and started rejecting all emails sent by third party servers (like ours) on behalf of Yahoo! users. So any email sent from a Yahoo! email address to anyone (even outside Yahoo!)), but not sent by Yahoo! itself, would be rejected. Since we have a significant portion of our business users and their clients using Yahoo! email addresses, it quickly became problematic as we generally send our notifications using staff and client email addresses to make it easier for people to reply to each other. But this is no longer allowed for Yahoo! email addresses.. and now for AOL's too. We expect other big name email providers to follow suite and eventually we will not be able to send emails on behalf of anyone anymore.

Remedial action: we then started transitioning our email infrastructure to no longer send emails on behalf of other people at all, and instead use our own no_reply email address. From now on, all messages will come from "Online scheduler" using our no_reply email address, but we are also specifying a "Reply-To" header, instructing email softwares to reply to you (when your clients receive a notification) or to the client (when staff receives a notification). So even though you will now always see all emails come from us, you should still be able to reply and the message to go to whoever should get it... just as before, but in a different way.

For more information about the change and how it affected lots of online services, you can type "yahoo email policy change" in Google (good articles here, here and here).. and straight from the horse's mouth here and here.

Going forward

In order to better stay on top of things in the future, we are now hiring outside help (email delivery experts) to handle most of our email traffic going forward. Email delivery has become a complicated science requiring expert knowledge and infrastructure. We could have avoided the first issue (being mistakenly put on a spam list) by negotiating to be put on whitelists instead and we could have mitigated the severity of the Yahoo! policy change by having a better monitoring system alerting us to the specific problem sooner.

In an ideal world, all spam messages would be blocked and none of the legitimate messages would. And ideally, we wouldn't need to worry about how to make our messages not look like spam to the computers sorting them. Instead, emails are getting increasingly expensive and difficult to manage.

Reliable email communications are vital to you, vital to us, and we are going to invest more to ensure that we are among the least affected when things outside our control change or break all of a sudden.

Settings menu disabled for non admin users

It has recently come to our attention that some people may mistakenly delete appointment types or schedules from the system, thereby leading to an irreversible loss of data.

To address this problem, we have done two things:

1. the "settings" menu is now off limits to non admin users.
2. the "time off" sub menu was moved out of the "settings" menu and is still accessible to non admin users from links in the "management" section.

We are also working on a new preference option allowing administrators to let other users access all settings or keep them hidden. We will fully support this in our next admin beta update. In the meantime, only admins can now make changes to appointment types, schedules or access the "links" page.

This morning's intermittent downtime

This morning we saw some downtime that affected many of our customers. All scheduler functions were intermittently available/non available for a few minutes at a time. The downtime was unusual in that way as it was unavailable for 30 minutes total but over the course of about 2 hours before it was fully resolved.

This is the time it took to reliably diagnose the cause of the issue and then find a temporary, then longer term fix, and then implement it. The bad news is this issue could not be avoided or anticipated but the good news is, should the same issue happen again, we will be able to resolve it right away with almost unnoticeable downtime. There are also some steps we will be taking soon to even further reduce the likelihood of this particular problem happening.

As a result of work we did in late January we have experienced about 56 minutes of downtime in 151 days (including today) with about half of it happening on various nights (in the US) and the other half this morning. This is rather low downtime (or an uptime of 99.97% if you will) and well below some major multi hour outages that affected some huge companies's paid online services recently.

Unfortunately, downtime can and will happen in the future, sometimes at inconvenient times. Just like the power can fail, the internet connection may go down or your computer may die on you unexpectedly. When things do go wrong, we always diligently get to work to try to bring the site back up as quickly as possible. We usually can't tell how long it's going to last or can't help out with most customer inquiries. This is a very stressful moment for us as we understand this is a crucial service that needs to be up at all times ideally.

In order to mitigate the impact on your business we suggest you take the following steps:

• subscribe to your calendar feed in a desktop calendar application such as Outlook 2007+, Windows Vista Calendar, the free Sunbird, Apple's iCal or even from your iPhone or a different web application such as Google Calendar. This will ensure that you have offline access to your upcoming appointments (including client contact details) as they will be automatically updated by your calendar software from our site. This is even useful if the site is still up but you have an internet outage. Go to "my settings" > "links" to find out more about the calendar feeds we offer.
• download our standard CSV files which are Excel compatible and will also provide offline access to the appointments in addition to your own backups for your local records. This needs to be manually downloaded by clicking a link and saving the resulting file to your computer. Go to "my settings" > "links" to find out more about the CSV files we offer.

This way, should the scheduler be unavailable at the time you can still at least get to your appointment list until we bring it back up or your internet access is restored.