US Audio MartUS Audio Mart
It is currently Mon Dec 10, 2018 6:28 am

All times are UTC - 8 hours

Post new topic Reply to topic  [ 1 post ] 
Author Message
PostPosted: Thu May 17, 2018 12:07 pm 
Site Admin
User avatar

Joined: Wed Apr 21, 2004 6:03 pm
Posts: 291
Location: Vancouver, BC, CA
We just wanted you to know that last night at around 2:07AM PST (5:07AM EST) CAM and USAM went down and remained down for around 6hrs, until 8am PST/11am EST. This type of extended non-planned downtime is painful, but thankfully not too frequent. In this particular case, one of our systems that help with scam and abuse prevention went off the rails and took up more resources than it should have, while a series of checks and balances against that type of thing failed to alert us in time.

I believe we've brought the sites back up to full operation now, and we will monitor the situation further. In addition to this, we may schedule some additional maintenance tonight past midnight EST in order to clean up additional collateral consequences of the downtime.

I'm also auditing some of our processes and checks to try and minimize this type and length of downtime, as one of our priorities has always been a fast running and stable site.

Last night's downtime is particularly frustrating because Wednesday morning we had a brief 20minute outage due to routine maintenance gone awry. Please rest assured that we take the site's stability and availability seriously and invest significant time and resources towards keeping the site running smoothly and reliably, and we hope that after last night and after additional work is identified that we can return to the level of reliable service you've come to expect from CAM/USAM.

Thanks for your patience when we encounter these hiccups, and sorry for the inconvenience this has caused.

Reply with quote  
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 1 post ] 

All times are UTC - 8 hours

Who is online

Users browsing this forum: No registered users and 0 guests

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB® Forum Software © phpBB Group