Ask questions via twitter! Message any question to @answers on twitter. We'll publish the question and send you a reply each time there's a new answer.
Next Question

Answered Question

jasoncala... jasoncala...
 
M$1 January 23, 2009 06:30 AM

UPDATE ON SITE PROBLEMS: Why we're down.

We've had our worst day ever in terms of downtime. We've been down from 10AM to 6PM, and we're only slowly getting back. We're investigating what happened, and it seems that today we were crushed with traffic (5-10x what we normally have) and we've found some problems keeping up. For the past month we have been expanding our infrastructure to keep up with our growth, but we never expected this much extra traffic. As such we're now rebuilding our servers to handle 20x our peak traffic. We are going to be down from 11pm PST until 4am to put new database servers online. Tomorrow should be much better. Know that the developers are not sleeping until we get this resolved. So, and inspirational video would be great below. :-)You can do it!
Interesting Question?  Yes (8)   No (0)   
RSS
 
 

Best Answer  Chosen by Asker

easyeboy easyeboy
 
 January 23, 2009 06:52 AM
Nick Vujicic, No Arms, No Legs, No Worries
http://www.youtube.com/watch?v=yo_24_qTNac

Billy Ocean - When The Going Gets Tough The Tough Get Going
http://www.youtube.com/watch?v=35YivWEs3zw


Helpful Answer?  (1)   (0)    Tip easyeboy for this answer
Permalink | Report
   Reply  
 
 
nyssa nyssa
 
January 23, 2009 07:15 AM
Now how can anyone top your first video? can't

Report
 
 
peconi peconi
 
January 23, 2009 07:46 PM
Amazing first video! Thanks for pointing Nick out... He's amazing.

Report
 
 

Other Answers (10)

Sort By
jefflauras... jefflauras...
 
January 23, 2009 06:40 AM
I can' think of anything  more inspirational.


Helpful Answer?  (1)   (0)    Tip jefflauras78 for this answer
Permalink | Report
   Reply  
 
 
markjeffre... markjeffre...
 
January 23, 2009 06:48 AM
*Profuse* apologies to everyone for the downtime.

Oddly, we had just spent the last several days overbuilding our infrastructure for a traffic spike exactly like the one we experienced today, but when it arrived, it was several times larger than we had planned for. Basically, we prepared for a 3.0-4.0 traffic-quake and were hit with an 8.0. We have since adjusted our plans ... so, sorry for the big bump.

The dev is team is here working furiously both at our NOC (Network Operations Center) and Mahalo HQ. Some people have not slept for nearly 48 hours.

We're on it. Sorry for the trouble, but know we care deeply about great service and feel very bad that we haven't delivered it today --- and we're doing everything we can to get back to where we need to be.

- Mark

Helpful Answer?  (7)   (0)    Tip markjeffrey for this answer
Permalink | Report
   Reply  
 
 
nyssa nyssa
 
January 23, 2009 07:18 AM
We have a saying around here: No Blood, No Foul. At least you folks were working towards a solution before it happened. Keep it up the great work you do. Good things come to those who wait. I'm waiting and patiently at that.

Report
 
 
philipy philipy
 
January 23, 2009 02:07 PM
Thanks for keeping us informed.

The "Mahalo is taking a quick break" screen was not very helpful, I have to say. Good to know what's actually going on now.

You guys have earned our goodwill and patience.

At this time, I still can't access my profile, though the Q&A; seems to be working well.

CORRECTION - After a reload, I can access my profile. Thanks guys, good to be back!

Report
 
 
nyssa nyssa
 
January 23, 2009 07:36 AM
Well in a way this is good news. People know Mahalo is around. I know bad news too if they find out you exist and then don't work quite right. But hey it happens. If it happens regularly that's when folks really get upset. Actually, in a way I'm glad you were having the problems because Now I know it is not me :) and my puter. (See I blame myself before I blame others and my puter is cluttered). Well I don't have any videos sorry. Definitely can't top easyeboy's there. I'm hanging around for Mahalo. The idea that keeps on helping. Peace, folks.

Helpful Answer?  (0)   (0)    Tip nyssa for this answer
Permalink | Report
   Reply  
 
 
bugsy bugsy
 
January 23, 2009 08:22 AM
"So, and inspirational video would be great below. :-)You can do it!"
Put your back into it?

http://www.youtube.com/watch?v=cuBM_YdHy-A

Helpful Answer?  (0)   (1)    Tip bugsy for this answer
Permalink | Report
   Reply  
 
 
mshubin mshubin
 
January 23, 2009 09:20 AM
I tried something different this time around and the database stuff went much faster. Some people are still having some issues, which a reload usually fixes, but mahalo.com is back up.


Helpful Answer?  (1)   (0)    Tip mshubin for this answer
Permalink | Report
   Reply  
 
 
bugsy bugsy
 
January 23, 2009 09:26 AM
Yeah, I'm still having problems with avatars, like 30% of avatars won't show up.

Report
 
 
markjeffre... markjeffre...
 
January 23, 2009 09:59 AM
We're working on that right now -- sorry about that

Report
 
 
bugsy bugsy
 
January 23, 2009 11:11 AM
Works excellent now.

Report
 
 
bugsy bugsy
 
January 23, 2009 04:42 PM
Avatar problems again (and that "You just viewed your blablath page today" slows down the page even more).

Report
 
 
robbrown robbrown
 
January 23, 2009 10:48 AM
Wow - good on you guys for the transparency of the problem!!!

Thanks.

Hey, not that you guys really need the help, but it sure would be awesome to have some higher level tech questions thrown at us.

I know that I would have been all over a "what is the best way to handle traffic spikes" question. Even if its contrived and you know the answer, It sure would be fun to research out the BGP routed, load balanced, dns failover answer that would ensue.

Helpful Answer?  (3)   (0)    Tip robbrown for this answer
Permalink | Report
   Reply  
 
 
mshubin mshubin
 
January 23, 2009 11:18 AM
Without going into too much boring detail about our network, I'll say that we are headed in the direction. We've recently purchased some new Cisco hardware and we'll be implementing BGP in the very near future.

Report
 
 
robbrown robbrown
 
January 23, 2009 11:24 AM
Oh ya... I'm sure you guys have got it... don't get me wrong.

I'm just selfishly looking for some tougher tech questions is all :)

Again, you guys rock for letting us know whats up and working so hard.

Report
 
 
beni beni
 
January 23, 2009 11:04 AM
Even though eight hours of downtime are unpleasant, I'm so sure and confident you will figure out how to handle the problems and optimize the infrastructure. After all, I understand that Mahalo Answers is still in Beta. The earlier these issues are resolved, the better. I hope you will have to and be capable of dealing with 1000x the traffic within a couple of months.

Bottom line is: the site is growing, the traffic volume higher than expected. And ain't that exactly what Mahalo Answers needs? I think it's great news! 

Just keep on trying, hook up more servers and keep on debugging!! We appreciate it very much! And @markjeffrey & team try to get some sleep, too! :-)

Sometimes, it just takes more than one attempt...

as in this video... although the kid failed twice, we all know she will finally manage, probably already bythe following day..


Helpful Answer?  (1)   (0)    Tip beni for this answer
Permalink | Report
   Reply  
 
 
markjeffre... markjeffre...
 
January 23, 2009 12:17 PM
Ok --
We think we've got everything back to where it was before -- plus a huge amount of extra capacity. We're still monitoring but things look ok again. If you see weirdness send to feedback@mahalo.com, we're reading that.

Thanks again for patience, all --- you've been very supportive, we appreciate that!
-- @markjeffrey

Report
 
 
bugsy bugsy
 
January 23, 2009 02:11 PM
This is the first time it works flawlessly. Leave it this way. :)

Report
 
 
nativenerd nativenerd
 
January 23, 2009 03:19 PM
I might be showing my age..but...

http://www.youtube.com/watch?v=hVKG5dtU5I4

Helpful Answer?  (0)   (0)    Tip nativenerd for this answer
Permalink | Report
   Reply  
 
 
rosshann rosshann
 
January 23, 2009 04:35 PM
happy to the site functional once again, and here is something to inspire you, if you know what I mean!! it is for humor, take it with a grain of salt, ok


Helpful Answer?  (0)   (0)    Tip rosshann for this answer
Permalink | Report
   Reply  
 
 
albanian albanian
 
January 23, 2009 06:26 PM
It looks as if going down for a day solved your capacity problem two ways. Not only did you upgrade, but few folks came back to ask new questions.

Helpful Answer?  (0)   (0)    Tip albanian for this answer
Permalink | Report
   Reply  
 
 

Answer this Question

How tips and payments work

This question has already been resolved. You may add an answer to it but you will not be eligible to win best answer or any associated tips.

Related Questions

No questions found.
View All Mahalo Answers Community Questions

Ask a Question


140 characters left
Top of Page
Buy Mahalo Dollars with Credit Card or PayPal

Top Members

This Week All Time
  • cfinke
    cfinke
    2nd Degree Black Belt
    27513 Points
    M$29.75 Earned
  • bunnyphuph...
    bunnyphuph...
    2nd Degree Black Belt
    21533 Points
    M$771.39 Earned
  • opher
    opher
    Purple Belt with a Brown Tip
    6320 Points
    M$243.74 Earned
   See All
 

Most Popular Tags

mahalo(1804)
twitter(982)
music(505)
iphone(486)
google(394)
online(372)
food(344)
internet(314)
beer(290)
movies(288)
money(286)
facebook(277)
apple(260)
answers(247)
aotd(235)
free(233)
health(231)
video(225)
dog(219)
windows(211)
   See All
 

Categories

Welcome New Members


 
 
Mahalo Dollars are the currency of Mahalo Answers.

Each Mahalo Dollar costs $1.

Once you earn more than 40 Mahalo Dollars, you can request to be paid via PayPal. Each Mahalo Dollar is currently worth $0.75 when paid out via PayPal. Learn More

 
 

Please log in to use this function.

Answers Home | Ask a Question | Earn Points | How Tips Work | Widget