Facebook on the outage: we changed some settings, our bad

avalon-0540032945
As someone who does not use Facebook except for business reasons (we post our stories there, follow us if you want), I tend to underestimate the number of people who use it. My mom and dad have also given up Facebook despite the fact that it’s the only way many of their friends and family keep in touch. Facebook’s six hour outage on Monday, which also hit the other sites and apps they own, Instagram and Whatsapp, affected so many people. There are over 3 billion users of Facebook alone and many people overseas use Whatsapp as their main text application. The outage was welcome to some people, who found they were less distracted, and upsetting to others, who were cut off from both business and personal contacts. Facebook has issued a statement about what happened. Apparently they changed some settings on the routers, which locked access in various ways and made them unable to get in to fix it. This explains why their key cards stopped working too.

Facebook has apologized for the mass outage that left billions of users unable to access Facebook, Instagram, WhatsApp and Messenger for several hours.

“To all the people and businesses around the world who depend on us, we are sorry for the inconvenience caused by today’s outage across our platforms,” said Santosh Janardhan, Facebook’s vice president of infrastructure, in a blogpost late Monday.

The outage, which prevented users from refreshing their feeds or sending messages, was caused by “configuration changes on the backbone routers,” Janardhan said, without specifying exactly what the changes were.

The changes caused “issues” that interrupted the flow of traffic between routers in Facebook’s data centers around the world, he added.

“This disruption to network traffic had a cascading effect on the way our data centers communicate, bringing our services to a halt,” Janardhan said.

Facebook, Instagram and WhatsApp stopped working shortly before noon ET, when the websites and apps for Facebook’s services were responding with server errors.

Just after 7 p.m. ET, around six hours after the platforms went offline, Facebook CEO Mark Zuckerberg wrote on his Facebook page: “Facebook, Instagram, WhatsApp and Messenger are coming back online now.”

He added, “Sorry for the disruption today – I know how much you rely on our services to stay connected with the people you care about.”

[From CNBC]

Many people thought this was too much of a coincidence given that the Facebook whistleblower, Frances Haugen, just had her segment on 60 Minutes air the day before and was testifying in front of the Senate this week. It seemed like a targeted attack or like Facebook was trying to purge evidence that could be used against them. I sound like a Facebook user with those theories, don’t I? While Facebook has allowed countless people to connect and keep in touch with loved ones, it’s also become too much of a tech giant and has helped brainwash large segments of the world’s population. I don’t say that lightly, watch The Social Dilemma, Facebook has allowed misinformation to spread by favoring profit over facts and humanity. This is what Haugen said on 60 Minutes too and she was part of their now defunct civic integrity department.

Hopefully this outage will help light a fire under congress to finally regulate Facebook and other social networks. I believe this explanation, that Facebook tried to change some things which had a cascade effect and disabled their access to fix them. It really fits with how back-asswards and short-sighted that company is. I’m sure multiple engineers tried to bring this to higher ups’ attention before it happened and were told it would be fine. I’ve worked in so many companies like that! Here’s a thread on Twitter explaining how it happened.

Oh and Mark Zuckerberg wrote a long ass post apologizing for the outage and denying Haugen’s claims. I’m not going to excerpt it because it’s a lot of bullsh-t and excuses. He’s being a defensive little sh-t instead of admitting to problems, which is what he always does. I really hope he’s made to step down.

avalon-0563594424

photos credit: Avalon.red

Related stories

You can follow any responses to this entry through the RSS 2.0 feed.

22 Responses to “Facebook on the outage: we changed some settings, our bad”

Comments are Closed

We close comments on older posts to fight comment spam.

  1. Merricat says:

    He is a repugnant, avaricious attempt at being human.

  2. Snuffles says:

    I 1000% believe Frances Haugen. Mark Zuckerberg is an amoral lying little shit. I would laugh my ass off if the outage yesterday was a result of them trying to cover their ass and changes the settings and backdate them to make it look like they kept the protections on they had during the election. I’m sure they are shitting their pants at being connected to the January 6th insurrection.

  3. Willow says:

    Can I have ‘who’s shredding documents’ for one million dollars Alex?

  4. Moira's Rose's Garden says:

    I swear he and Jared are related. Repugnant is the nicest thing that can be said about him. And I won’t even address Harvard giving him the honorary JD.

    ugh

    • Lionel says:

      Zuck and Jared do share a look, don’t they? Same waxy skin and dead eyes.

      • Golly Gee says:

        They were both made at the same Japanese man-boy robot company.

        And if Zuckerberg weren’t such a liar, he could be Pinocchio come to life. His weird haircut reminds me of a dummy ventriloquist puppet.

  5. BOOGIE says:

    Business I get but personal use- if people are working how on earth do they have time to scroll Instagram and Facebook and do their job at the same time? I guess I can understand some here and there. But it’s not like this happened during the weekend! If it’s during the work day you could just… work.

    • MrsRobinson says:

      I run a business account for a large nonprofit and it surprises me how many employees are engaging with posts during the day.
      I don’t post on FB on my personal account, but have to use it to access the business account (a weird work/personal hybrid). FB started Reels this week (all repurposed Tiktok content) and I got sucked into it 3 times already. Somehow it works there better than IG, unfortunately.

    • khaveman says:

      If you think people don’t make time for social media time-outs from job overwork and stress, you’re kidding yourself. It’s the tech version of the sanctioned smoke break.

  6. sunny says:

    I’m not surprised he is admitting to nothing. Years ago when I was in business school, I took a crisis management class. One of our tests was a simulation of a crisis in a tech company(a data breach) that became covered up by fraud. the actual test was to build a crisis response and create a press statement(that our prof brought journalists in to evaluate). The class was 70% men, and 30% women. The prof notes that the majority of the female students (almost all of them) built a crisis response that 1. Admitted to the fraud 2. Apologized. 3. Took steps to address it. Less that 10% of the men took the first 2 steps instead they went straight into changes. What was fascinating about this is one of the first things you learn is that a crisis evolves- it never stays the same type of crisis.

    It is sad and interesting to see how bad management and leadership help a crisis spin out of control.

  7. SpankFD says:

    I’m calling B.S. on Zuckerberg’s “Oops, our bad!” explanation.

    My thinking is that Facebook either 1. got hacked (likely due to the recent whistleblower revelations) or 2. fell victim to an inside job, i.e. a disgruntled employee took vengeance. I’m leaning towards the former due to the timing of the incident.

    After all doesn’t FB have the $$$ to employ top notch security?

    Side note: I believe Zuckerberg would lie about the cause in either case in order to limit the damage to FB’s reputation. He would think it is better to be seen as incompetent than victimized. Lest he incur further victimization.

    Bonus: Zuckerberg personally lost over $6 billion yesterday. May it never return.

    • BrainFog 💉💉😷 says:

      As someone with 20+ years experience in IT I feel that their explanation makes sense. I agree that the timing is sus, and I totally do not trust a word that Suckerberg says, but from a technical point of view I buy their story. It’s also not about employing top notch security. They had TOO MUCH security. They locked themselves out, basically. I have seen things like this happen way too often to doubt the story.

      • ExpatInTheUK says:

        +1 to this.
        The conspiracy theories are fun but sadly, it’s actually not uncommon for major network outages to happen. I am in a field where I work with network engineers and I’ve seen this happen with big infrastructure every so often.
        One engineer brought down his company’s internal network by just plugging in a router that caused a packet storm. If you have infrastructure as large as Facebook’s it may take time to isolate the exact cause of a fault if you don’t have the right monitoring tools in place.

    • SpankFD says:

      Hi BrainFog, ExpatInTheUK

      I agree, BrainFog, that FB blamed the outage on a plausible scenario. They gave us an explanation that is vague enough to pass muster while sidestepping the timing issue. MZ isn’t a moron; he’s just evil. The timing is still sus…

      But I am fascinated by the suggestion —via ExpatInTheUK —-that FB has skimped on its network monitoring tools. Wow. That’s a huge vulnerability and an embarrassing oversight on FB’s part. Worthy of a 7% stockprice drop. I wonder whether FB’s network engineers recognized the vulnerability or were blindsided?

      Thanks for sharing the observation, ExpatInTheUK.

  8. steph says:

    We wiped our servers, our bad.

  9. JEM says:

    Facebook got Trump elected and imperiled democracy, with no signs of stopping. It gives the antivaxxers a loud platform and white supremacists a direct line to the morons who gobble it up. Zuckerberg served us all up on a platter to Putin and we’re too stupid to notice because we’re too busy screaming at each other. Meanwhile everyone I know continues to post pics of their dinners and tell gushing lies about how happy they are. It all makes me sick.

    • Ann says:

      I blame Twitter more for that, and the media in general for pointing a camera at him for the entire primary. Facebook is on the hook too but Twitter was his biggest mouthpiece. Anyway, I do use Facebook but I only check it once a day or so to see if a friend has posted anything interesting, or to post pictures from my garden. But yes, Zuckerberg is a s**t.

      • Golly Gee says:

        Facebook is just as culpable as Twitter. Think of the thousands of people who have died of Covid because they believed in conspiracy theories fed to them through both platforms. Jack Dorsey and Zuckerberg both have blood on their hands.

  10. Kkat says:

    They either got hacked or they were trying to get rid of evidence and screwed up or created the outage as a cover to get rid of stuff and backdate.

    • khaveman says:

      I think it was hacking b/c they probably already did a bunch of things internally prior to the testimony. I mean, they wouldn’t lock their own doors to themselves. Also I am betting they would be erasing the Russia ties if they exist.

  11. Deering24 says:

    “…she was part of their now defunct civic integrity department.”

    They _had_ one?!!!?🙄