Zello network upgrade tonight

By Alexey Gavrilov on May 30, 2012

In effort to keep up with the growth, improve Zello service stability and speed we’ll perform a planned upgrade of our infrastructure tonight. We’ll start at 4am Central Time and finish by 6am on May 31, 2012.

Since last major outage we’ve been busy adding new servers and re-writing server side code to support millions of online users. Now this work is completed and we’ll need to take the network down briefly to replace currently running server code with the new release.

As we are deploying the update the service might be unavailable or slow. Follow us on Twitter to get real-time updates on our progress and service availability alerts.

Zello@Work uses separate infrastructure so won’t be affected.

Photo credit: torkildr

April 25 outage postmortem

By Alexey Gavrilov on April 29, 2012

Zello has been growing like crazy recently — with number of active users increasing 23% on average each week since iPhone release. April 25 was starting great setting a new record of 222k people online at the same time and all systems running smoothly.

We first noticed the problem at 9:30 AM Pacific — signing into network was taking long time and the number of online user started to drop. Quick check confirmed that some of the supernodes handling user connections were down. Normally this wouldn’t be a problem because if one server dies, the users will automatically reconnect to a different one. It wasn’t happening though — the users couldn’t re-login creating a huge backlog, traffic and CPU load on the login server. At Zello we use membase to store data, which require low-latency access. The failed server was also a part of a membase cluster, we created recently to handle increased load. This failure however had catastrophic effect making membase unresponsive and preventing new users from signing in.

Read the rest of this entry »

We’re back

By Alexey Gavrilov on March 23, 2009

Maintenance works are completed and all services are running normally again.

Scheduled maintainance (Monday, March 23, 1AM-2AM EST)

By Alexey Gavrilov on March 23, 2009

This Monday, March 23 from 1 AM till 2 AM EST we are performing scheduled database maintenance. During that time Loudtalks accounts registration won’t be available and a some users might not be able to log in to their accounts. We’ll post an notice in the blog as soon as the work is completed.

If you have any questions or concerns please contact us at support@loudtalks.com

echo is back online

By Alexey Gavrilov on December 18, 2008

So is voices. Unlike echo, voices contact is not added by default — you have to manually add it via Tools > Add a contact… menu. Once added you can send messages to it (as short as empty) and hear back the messages from other people, who talked to voices before along with their username.

echo robot is broken

By Alexey Gavrilov on December 17, 2008

We’ve discovered “echo” robot, which is added to the new users’ contacts by default has been broken for a few days now. We are now working to isolate the issue and bring it back online. For now if you want to test Loudtalks briefly please feel free to add me (“alex”) to your contacts.

I’ll post a separate note, when echo is back online.