Recurly had some very bad news for its clients this morning but at least initial fixes are now on the way. A cascading hardware failure erased some end customer recurring billing info, preventing them from processing payments and essentially requiring them to ask customers to re-sign up for their subscriptions.

Considering people often forget about the subscriptions they pay for, they might just ignore the request to resubmit billing info, effectively canceling their subscriptions. Thankfully, now Recurly says it plans to have data to some clients restored by midnight. It’s been a painful moment to watch for a startup aiming to offer “Subscription Billing Pain Relief”.

Here’s what happened. Recurly lets business easily offer subscription billing, and handles credit card storage, customer upgrades and downgrades, communication, plus it provides analytics to improve retention. Monday night at 3:30am PST, Recurly experienced a hardware failure in its primary encryption device, which cascaded to the backup.

This currupted the encryption keys that access the stored credit cards that it uses to process subscription payments for its clients. Unfortunately, these encryption keys were purposefully designed to hinder information retrieval in order to thwart hackers trying to access the stored credit cards. That’s making recovery difficult.

For now, new customer signups can continue but all existing recurring transactions are paused. Some data has already been recovered, but other clients may have to endure disrupted service for a while, and worst of all, some have had their data erased permanently.

Here’s the really sad part: “Some customers will be required to reach out to (some or all) of their customers to have them re-enter billing information. If and when this is necessary, we will provide support and tools to make this easy.” Tools are already being made available, including a PCI-compliant ‘Billing Information Upate Form’ that can be embedded on clients’ websites.

I spoke with Recurly CEO Dan Burkhart who says his company snapped into action, quickly quelling confusion by issuing an alert that something had gone wrong, and then following up with a blog post explaining the problem. He notes “Our support team has been active 24/7 attending to ~1100 customers in many timezones”, and commended Braintree for responding with exceptional speed. Though we don’t have details on exactly who was affected how badly, Recurly’s clients include Adobe, LinkedIn, and Livestream.

This evening Recurly released an update explaining that “As of midnight tonight, we expect to have merchants who have been processing via Braintree Payments, PayPal Payments Pro, and Wirecard to be restored and operational. Over the coming days, we expect to have merchants who have been processing with Cybersource, Intuit Payment Solutions, Litle & Co., and Merchant e-Solutions restored with replenished customer billing information as well.”

What comes next is less clear, though. Burkhart wouldn’t specify if Recurly would offer compensation to those who suffered permanent data loss from the failure. It may also be tougher convincing additional merchants to come on board. But outages and failures are an inescapable part of the technology business, and hopefully potential clients will consider Recurly’s diligent response to what could have been much uglier had it left those it serves in the dark.

Resources
Post Your Resume to 65+ Job Sites
Resume Service

Tweet This Post