Outages this AM

[rivermobster]

It's all good here where I'm at!

Looks like it should be fixed soon...

Global IT outage causes chaos, disrupting airlines, banks, media, telecoms

Cybersecurity firm CrowdStrike says outage caused by ‘defect’ in software update, rules out ‘cyberattack’.

www.aljazeera.com

 

[angiebaby]

iPhone users should be used to this from every time their phone updates it's a shitshow.

I probably will jinx myself, but I don't recall having any issues with my iPhone updates. However, I don't auto-update. I wait until it is forced, which is usually a couple of months after launch.

 

[just_floatin]

I probably will jinx myself, but I don't recall having any issues with my iPhone updates. However, I don't auto-update. I wait until it is forced, which is usually a couple of months after launch.

Linux and Mac users are not impacted. Operating Systems using Microsoft are impacted. Companies with untested Disaster Recovery servers failed this exercise and are down worldwide. Busy day for all operating system engineers everywhere.

 

[Ladsm]

Your packages coming from the "Jungle Company" are not leaving the loading dock until this is resolved

 

[Ladsm]

The V-Red Gas station in BHC just posted they are down till this outage is fixed.

 

[wallnutz]

Your packages coming from the "Jungle Company" are not leaving the loading dock until this is resolved

So much for my Prime day delivery that is already a day late.

 

[was thatguy]

I probably will jinx myself, but I don't recall having any issues with my iPhone updates. However, I don't auto-update. I wait until it is forced, which is usually a couple of months after launch.

You’re not jinxing yourself. I do the same thing.
The anti Apple crowd just can’t stand it.
The mere sight of an IPhone makes them hyperventilate.

 

[mbrown2]

Linux and Mac users are not impacted. Operating Systems using Microsoft are impacted. Companies with untested Disaster Recovery servers failed this exercise and are down worldwide. Busy day for all operating system engineers everywhere.

Unless your DR servers are airgapped and not allowing crowdstrike updates, they would have been impacted too. Most DR systems sit in a warm state and actually have n-1 or n-2 patch cycles... This particular channel content update from Crowdstrike did not care about your n-x posture so all versions got it...meaning your DR servers got it as well.

 

[renodaytona]

Massive outage at my company this AM and over half the IT team is on vacation.

 

[mbrown2]

Massive outage at my company this AM and over half the IT team is on vacation.

Actually you are on vacation now too if it is a massive outage!

 

[was thatguy]

Actually you are on vacation now too if it is a massive outage!

Overall, is this a plug in fix or update fix?

 

[DarkHorseRacing]

Massive outage at my company this AM and over half the IT team is on vacation.

If you are running crowdstrike you could try to boot windows into safe mode, go into a crowdstrike directory and deleted a specific sys file and then reboot the system.

Google search for that specific fix, or I can try to copy it in from my work which is dealing with it now.

 

[hallett21]

You and Jeff def stay prepared.

I forget how many meals missed is estimated before roving crowds start to riot. Something along the lines of a couple of days.

If I have to I will eat beans and rice everyday no problem.

Its true they have made fun of preppers for years in the media. There are extreme examples of course that I think goes too far but who am I to judge?

Everybody should have enough resources to get by for a few weeks minimum IMO.

I think it’s 3 missed meals.

 

[renodaytona]

If you are running crowdstrike you could try to boot windows into safe mode, go into a crowdstrike directory and deleted a specific sys file and then reboot the system.

Google search for that specific fix, or I can try to copy it in from my work which is dealing with it now.

IT just came by and got my PC dialed in.

 

[Singleton]

This will stop folks from auto-updates, which could start other issues.

 

[stephenkatsea]

Is there ANY chance . . . This is what Kamala has been trying to describe to us ? ? Maybe, this can unburden the passage of time, maybe?

 

[brecht]

Not only do we use Crowdstrike at work, which mostly just affected our Aussie team, but we are doing a company offsite in Hawaii. Half the team is trying to fly out right now lol its a mess

 

[mbrown2]

Overall, is this a plug in fix or update fix?

They already fixed the issue but the workaround/patch they provided was to clean the state some computers were left disabled due to the faulty change. A good percentage of computers exhibited bsod and required the workaround 'rebooting in safe mode, making a couple of changes" to get back to a healthy state. The original change has been fixed but the yard sale of shit the original change caused, there was quiet a bit of cleanup needed.

 

[Outdrive1]

I don’t understand how a cybersecurity program makes the entire OS go down. Seems way intrusive to me. I have a MacBook though, and have never installed any security software on it.

 

[just_floatin]

Unless your DR servers are airgapped and not allowing crowdstrike updates, they would have been impacted too. Most DR systems sit in a warm state and actually have n-1 or n-2 patch cycles... This particular channel content update from Crowdstrike did not care about your n-x posture so all versions got it...meaning your DR servers got it as well.

Our FinTech company BCP (Business Continuity Plannig) operates in a “cold” (nonprod) environment. All patching cycles are managed and tested (UAT) to determine the RTO (Recovery Time Objective) before ever getting released to production (consumers). This outage impacted some internal employees (VDI users) in the cloud (Azure a MS product). Our helpdesk instructed these folks to remove the file above in safe mode and reboot.

 

[4Waters]

All good here

 

[Her454]

I think it’s just a test. Maybe they’ll test again around election time

 

[4Waters]

They got the sphere, BSOD

 

[C-2]

Sounds like it took out a couple of State servers for sites/databases we regularly use. But no BSOD's on desktops/laptops.

I'm the only dumbass in my office with my own redundancy program. Dumbass because everybody else is like "free day to fawk off."

 

[mbrown2]

I don’t understand how a cybersecurity program makes the entire OS go down. Seems way intrusive to me. I have a MacBook though, and have never installed any security software on it.

These sorts of technologies run at the kernal level level so its liteally at the OS layer... Why is that important; spam, malware, ect try and attack those same levels. These embedded security solutions are able to allow a person to click on a malicious link and almost handle it in a sandbox sense where it will play out the malicious payload and isolate the negative impacts from the end user. That is a simple explanation... from a MAC perspective they run a very closed OS and as such tend be more protected. Crowdstrike also has IOS/Mac, and linux agents but it was only the windows update that had the problem.

 

[petie6464]

Time to selling Generators again..

 

[Cole 392]

 

[havasuhusker]

I've been dealing with the fallout all morning. Trying to get servers/apps back online. I need a beer, or two, or 12!

 

[DWC]

All good here.

Playing as a single with 3 IT guys. They don’t seem too worried about what’s going on….

 

[jetboatperformance]

Marking myself safe in Atascadero

 

[EarpRider]

Isn't this how the movie "Leave The World Behind," starts?

 

[paradise]

I can‘t believe all these companies have auto-update activated.
We run everything and I mean everything through a 2 week test cycle.

Even antivirus? Usually AV updates are rolled out as soon as possible to stop zero day attacks. Most CS questionnaires we fill out for clients specifically ask if they are auto deployed. I assume that practice will be changing... Which will only help the bad guys...

 

[C-2]

These sorts of technologies run at the kernal level level so its liteally at the OS layer... Why is that important; spam, malware, ect try and attack those same levels. These embedded security solutions are able to allow a person to click on a malicious link and almost handle it in a sandbox sense where it will play out the malicious payload and isolate the negative impacts from the end user. That is a simple explanation... from a MAC perspective they run a very closed OS and as such tend be more protected. Crowdstrike also has IOS/Mac, and linux agents but it was only the windows update that had the problem.

Nice explanation, great job.

Kernel - just as the name suggests; calls all the shots.

Hey keep the cat poop out of the sandbox

 

[was thatguy]

Nice explanation, great job.

Kernel - just as the name suggests; calls all the shots.

Hey keep the cat poop out of the sandbox

Kernel/ colonel?

 

[regor]

I think it’s just a test. Maybe they’ll test again around election time

https://twitter.com/i/web/status/1814303695728783550

 

[DC-88]

Marking myself safe in Atascadero

Just got back from Chicago Grade Dump and the kids in the shack were panicked due to their system being crashed. The scale worked but they were having to weigh in and out and do the math on their I phones. Pretty sure I got a good bro deal due to their math . It's scary how dependent on it everything and everyone is ... especially the younger gen .

 

[Flyinbowtie]

I am sure it unrelated...but my 5 day old from new HP laptop blank screened yesterday morning. I wound up driving down the hill to return it.the guy at best buy just took one try to fix it and sent me a new one.
We are headed to Reno tomorrow for a Celebration of Life for my old partner. I hope all this stuff is sorted by then. This is gonna be hard enough to get through without screw ups at the hotel or the venue for the even

We are far too dependent on tech these days. Far too vulnerable. There is no shame in being prepared.

 

[Singleton]

Even antivirus? Usually AV updates are rolled out as soon as possible to stop zero day attacks. Most CS questionnaires we fill out for clients specifically ask if they are auto deployed. I assume that practice will be changing... Which will only help the bad guys...

AV has a shorter window (usually a day or 2).

 

[boatpi]

Slots in Vegas are down at some casinos.

 

[DILLIGAF]

I think it’s just a test. Maybe they’ll test again around election time

If I understood correctly...they are based in Austin TX and their servers are in Ulkraine. The system going down is also the platform for our friggin elections. Deep ties to the Clintons and Obama.

Nothing to see here....fuckers

 

[Her454]

I am sure it unrelated...but my 5 day old from new HP laptop blank screened yesterday morning. I wound up driving down the hill to return it.the guy at best buy just took one try to fix it and sent me a new one.
We are headed to Reno tomorrow for a Celebration of Life for my old partner. I hope all this stuff is sorted by then. This is gonna be hard enough to get through without screw ups at the hotel or the venue for the even

We are far too dependent on tech these days. Far too vulnerable. There is no shame in being prepared.

Sorry to hear about your partner Jeff, sending prayers to you all. I would offer to help with your laptop, but after 23 years at HP things have changed so much and I’m pretty much out of the loop now. And I 100% agree with you that we are all far too dependent on the Internet and the newest technologies.

 

[eand28]

 

[River Dirt 2]

My daughters current view, she works at a very, very large oil company.

 

[Done-it-again]

Sure made for a easy commute to work... Should happen more often me thinks.....

 

[regor]

 

[liquid addiction]

I just got on the plane. Been here since 7am est

 

[Paradox]

Just went to the Chase bank here in Havasu. Closed. Only one ATM machine working and there was a fairly long line. I suspect it will be out of money shorty.

 

[jetboatperformance]

The preacher man says it's the end of time
And the Mississippi River, she's a-goin' dry
The interest is up and the stock market's down
And you only get mugged if you go downtown
I live back in the woods you see
My woman and the kids and the dogs and me
I got a shotgun, a rifle and a four-wheel drive
And a country boy can survive
Country folks can survive

 

[Flyinbowtie]

The preacher man says it's the end of time
And the Mississippi River, she's a-goin' dry
The interest is up and the stock market's down
And you only get mugged if you go downtown
I live back in the woods you see
My woman and the kids and the dogs and me
I got a shotgun, a rifle and a four-wheel drive
And a country boy can survive
Country folks can survive

Been my ringtone for years.

 

[Badchoices03]

So did the world end yet or we still holding?