New York City Mayor Eric Adams (D) held a press briefing on Friday to discuss the Microsoft outage.
Fuel your success with Forbes. Gain unlimited access to premium journalism, including breaking news, groundbreaking in-depth reported stories, daily digests and more. Plus, members get a front-row seat at members-only events with leading thinkers and doers, access to premium video that can help you get ahead, an ad-light experience, early access to select products including NFT drops and more:
https://account.forbes.com/membership/?utm_source=youtube&utm_medium=display&utm_campaign=growth_non-sub_paid_subscribe_ytdescript
Stay Connected
Forbes on Facebook: http://fb.com/forbes
Forbes Video on Twitter: http://www.twitter.com/forbes
Forbes Video on Instagram: http://instagram.com/forbes
More From Forbes: http://forbes.com
Fuel your success with Forbes. Gain unlimited access to premium journalism, including breaking news, groundbreaking in-depth reported stories, daily digests and more. Plus, members get a front-row seat at members-only events with leading thinkers and doers, access to premium video that can help you get ahead, an ad-light experience, early access to select products including NFT drops and more:
https://account.forbes.com/membership/?utm_source=youtube&utm_medium=display&utm_campaign=growth_non-sub_paid_subscribe_ytdescript
Stay Connected
Forbes on Facebook: http://fb.com/forbes
Forbes Video on Twitter: http://www.twitter.com/forbes
Forbes Video on Instagram: http://instagram.com/forbes
More From Forbes: http://forbes.com
Category
🗞
NewsTranscript
00:00Good morning, everybody.
00:08My name is Fabian Levy, and I serve as Deputy Mayor for Communications for the City of New
00:11York.
00:13Early this morning, our office became aware of a global IT outage impacting city governments,
00:19companies and other organizations around the world.
00:22We first became aware of the situation at 12.40 a.m., and at 1 a.m. in the morning,
00:26our team started convening together to deliver for New Yorkers.
00:29New Yorkers need to know that their city government moved quickly to assess the incident and protect
00:34critical resources.
00:36Chief Technology Officer Matt Fraser even conducted a series of television and radio
00:40interviews throughout the morning to keep New Yorkers informed about the outage, starting
00:44at 5.40 a.m.
00:46While we are continuing to monitor the situation, emergency operations and crucial services,
00:53including our 911 system, are working properly.
00:56To tell you more about the incident, we are joined this morning by Mayor Eric Adams, Chief
01:01of Staff Camille Joseph-Warlick, Deputy Mayor for Public Safety Phil Banks, Deputy Mayor
01:05for Operations Mira Joshi, Deputy Mayor for Strategic Initiatives Ana Almanzar, New York
01:10City Chief Technology Officer Matt Fraser, New York City Fire Department Commissioner
01:15Laura Kavanaugh, New York City Health and Hospitals President and CEO Dr. Mitch Katz,
01:21New York City Department of Citywide Administrative Services Commissioner Luis Molina, New York
01:25City Department of Environmental Protection Commissioner Rit Agrawal, First Deputy Commissioner
01:30of New York City Emergency Management Christina Farrell, New York City Police Department Chief
01:34of Information Technology Ruben Beltran, New York City Department of Education Deputy Chancellor
01:40for Family and Community Engagement Melissa Villas-Ramos, MTA Authority Chair Jano Lieber,
01:46and Con Edison Director of Regional and Community Affairs Ruth Fosshold.
01:50So without further delay, I'll turn it over to Mayor Adams.
01:56Thanks so much, DM Levy, and I often recall, although this was not, it was not a cyber
02:03attack, I recall when I was briefed by the former mayor, he says one of the major concerns
02:11is dealing with any form of IT outage, including if it is a cyber attack, we need to be prepared.
02:19And with that in mind, Chief of Staff Camille Joseph-Volok has been hosting a series of drills.
02:29We just recently did one to deal with any type of IT outage or slash cyber attack.
02:37And so it was good to see how quickly the team coordinated.
02:43CTO Smalls, Frazier, called me around a little after midnight last night and gave me a preliminary
02:55briefing that what was taking place, and reached out to the other team members and everything
03:00kicked in gear, and you see the results of that from not only the notifications he made
03:06this morning, but also the coming together of the team.
03:10The blueprint was already in place.
03:12All we had to do was to execute on it, and I just want to thank all these city agencies
03:19as well as our partners in other agencies like the MTA.
03:24And we have, we're here to update New Yorkers on this global IT outage involving CrowdStrike.
03:32CrowdStrike is a cybersecurity software prevalent in computers across the globe.
03:40Their goal, as the Chief Technology Officer would explain, is to identify if there is
03:46a problem and to immediately take action.
03:50And early this morning, they sent out a software update that inadvertently took systems offline.
03:58They sent out a software update that inadvertently took systems offline.
04:03This was not a cyber attack that was a hit or a cyber hit on our city infrastructure.
04:12Our city's IT and security teams led by the Office of Technology and Innovation have been
04:19working nonstop to troubleshoot problems and restore services.
04:24The CTO, Matt Frazier, has been directly in touch with CrowdStrike to get real-time updates
04:32and identify the fastest path to getting all systems up and running as quickly as possible.
04:37And I just really want to emphasize this was not a cyber attack.
04:42This was an update that took systems offline.
04:46And this is why the preparation is so important that the Chief of Staff is in charge of doing
04:53to make sure that we can immediately respond if there is an IT disruption of this magnitude
05:00or if there is a cyber attack.
05:02We have to be prepared, and this is what the drills are for.
05:06There are no calls are being held or missed, and there is no backlog at the FDNY.
05:14And FDNY is reviewing to ensure this remains the case.
05:18We want to ensure New Yorkers that 9-11's core systems have not been impacted, our infrastructure
05:24and emergency operations, they're all in place, and we're going to continue to do that.
05:30Life-saving complaints that come through 311 or city agencies are being prioritized.
05:37Traffic signals are working and are fully operational, as is the Staten Island Ferry.
05:42Our water systems are fully operational, and our summarizing program will continue as scheduled,
05:50including bus service.
05:52Although we want to tell parents you may see disruption on the bus tracking app, the systems
05:57are still operating.
05:59We expect to continue to see cascading effects of the outage throughout the day, but New
06:06York City Office of Technology will be doing everything possible to minimize potential
06:12service disruptions and restore service to all systems as soon as possible.
06:18Our city agency's IT department will be prioritizing this work today to get things back up and
06:24operating online based on the priority of those particular systems.
06:29We want to make sure New Yorkers are safe and they can rely on the city services they
06:33need.
06:34And this is a developing situation.
06:39We are assured by the coordination that's taking place from all the agencies involved,
06:44as well as the entities that are responsible, such as CrowdStrike, to make sure we can get
06:51this under hand without a minimum amount of interruptions in our city.
07:00Thank you, Mayor.
07:01Next we'll hear from Chief Technology Officer and Commissioner of the Office of Technology
07:06and Innovation, Matt Frazier.
07:10So good morning, everyone.
07:12So to give everyone a brief detail in terms of what happened and how we responded, at
07:16approximately 1220, between 1220 and 1230, we started to see impacts from systems across
07:21the city manifesting in different places.
07:24And like usual, when you see something like this, this is one of the reasons why we have
07:27New York City Cyber Command and we have a citywide operation center, so we can detect
07:31these things as quickly as possible and dispel what's a threat versus what's not a threat.
07:35So as we started to see the symptoms manifest, we contacted our partners, both at Microsoft
07:40and CrowdStrike, and subsequently we got down to the root of the issue.
07:44The thing with technologies like this is, in order to be safe and be able to respond
07:48to threats that evolve in a continuous basis, you need tools that are capable of being updated
07:54in that way.
07:55As a result, a tool like CrowdStrike, which gets updated real time, when they push a patch,
08:00if that patch goes wrong, this is a perfect example of how bad that can be.
08:05So one patch goes out at about 12 o'clock a.m. eastern time, and it's pushed between
08:1112 and 1.30 a.m. eastern time.
08:13And in that period of time, we had a number of workstations, a number of computers and
08:18servers that were impacted.
08:20After notifying CrowdStrike and working with them, by 1.30, they had stopped deploying
08:24that patch, and since then, we've been in a state where we haven't seen any additional
08:28machines impacted.
08:29We're now in a state where we're doing, where we're going through the recovery.
08:32Unfortunately, the recovery at this moment is a fairly manual process that requires physical
08:37touch to most of the machines.
08:40And we have a team of people that are working across the city in real time, trying to get
08:44things back up and running as quickly as possible.
08:46As Amir said, most of our critical services, all of our life safety-related services remain
08:51up and running.
08:52There's no impact to 911, no impact to 311, no impact to police, fire or EMS, radio or
08:58dispatch operations.
09:00Everything is running as expected.
09:02In addition to that, our operational technology networks and our SCADA networks, things that
09:06supply our water systems, things that operate our traffic lights, none of those systems
09:11are also impacted.
09:12Today is a good example of how some of the planning that we've done in advance helps
09:16protect the city and ensures continuity of our critical services.
09:20And as we've seen across the globe, this is not an issue that's limited just to New York
09:24City.
09:25We have everyone from the airlines to financial institutions that are experiencing the same
09:29challenges.
09:30The good news is that despite the challenge that appeared today, our most critical services
09:36remain up and running, and we have a pathway to recovery to bring everything else up.
09:42Thank you, Matt.
09:43And thank you for joining so many TV stations and radios this morning to keep New Yorkers
09:46informed.
09:47Next, we'll hear from MTA Chair Jan O'Leaber to hear about what's going on in the subways,
09:51trains and buses.
09:57So the MTA activated our incident command system and our emergency management procedures
10:04in the late morning hours.
10:08And we established quickly that there is no impact to subway service, no impact to the
10:14bus system, no impact to commuter rails, and no impact to the paratransit operations.
10:21The impacts are limited so far that are customer-facing, are limited to one portion of our countdown
10:32clocks on the B Division, which is the letter trains, are not operating.
10:37But as old New Yorkers know, the trains are coming.
10:39They're coming on regular schedules, and you can count on it.
10:43You just may not have the countdown clock available.
10:45Otherwise, the tolls are being collected.
10:47Bad news for some people, I know.
10:50But every single operating system is functioning.
10:53You may not have up-to-date information, just like the countdown clocks are not 100%.
10:58You may not have up-to-date information on exactly when commuter trains are arriving,
11:03but all of that is coming back online pretty promptly.
11:08And as I said, all of the customer-facing impacts have already been evaluated and mitigated,
11:13and all of our systems are running.
11:14I want to thank the mayor and the entire New York City team for their cooperation.
11:18I've also been in touch with the governor's team and briefed her on everything that's
11:23going on.
11:24Bottom line, the airline system may be in disarray, but New York City's public transit
11:31system is going full speed.
11:33Thank you, everybody.
11:34Thank you very much, Geno.
11:37Next we'll hear from Dr. Mitch Katz from H&H.
11:41Good morning, everyone.
11:43Dr. Mitch Katz, I'm the CEO and president of New York City Health and Hospitals, and
11:48I'm happy to report that not only are all of our hospitals functional, but all of the
11:54hospitals across New York City are functional.
11:57Every hospital is required to have a plan and to practice what would happen if computer
12:03systems went down, because we deal with life and death and have to make instant decisions
12:08about prescribing a medicine, running a test.
12:12And so every hospital has to be able to run without computers, and some of us are even
12:18old enough to remember when everything ran without computers.
12:22So we know how to do it.
12:24There is a manual override for a process for when you have to not use the computer.
12:30You go to paper.
12:32But throughout the city, because most of the hospitals, including ours, use an electronic
12:37health record called EPIC, which did not go down, overwhelmingly all of the systems are
12:43running just as they should.
12:46Patients should keep their appointments.
12:47They might experience delays because some of the individual workstations have to be
12:56brought up manually, as you've heard.
12:58And so not all of them are brought up, so people should have a little bit extra patience
13:03with their doctor or nurse, but people should not neglect their care.
13:08The hospitals are all running fine and able to do their function.
13:13Thank you, Dr. Katz.
13:16We'll take some on-topic questions now.
13:18So about the system update and how it caused the problem the city's experiencing now, is
13:29it something in the software update, the patches you were talking about, or was it how it was
13:37implemented?
13:38Could you just give us a little more detail on how things went awry there?
13:43Yeah, sure.
13:45The way these technologies work is that there's a control file that gets deployed into the
13:50agent that runs on every computer and server.
13:53When that control file got deployed, part of the control file was corrupted.
13:57So when it went to execute and apply the update, the Windows system itself panicked, which
14:03is what causes the blue screen that you see that says that they commonly refer to as the
14:08blue screen of death.
14:09So that update that went in, that came as a control file into the agent, is the thing
14:14that triggered the symptoms that we're seeing.
14:17That corrupt file is on the CrowdStrike end of things?
14:22That is correct.
14:23That is correct.
14:24So the way these agents work, because we have malicious actors that work 24-7 around the
14:30clock.
14:31They're new signatures that appear every day.
14:33These tools stay connected consistently so that as new updates come, they update so that
14:39our systems can be as safe as possible.
14:41Now, part of that means that those control files get updated sometimes multiple times
14:46a day, on a daily basis, weekly basis, so on and so forth.
14:50So as those files come in, it's normal to see that type of update.
14:53What's not normal is the code that was corrupted as part of it.
14:57You mentioned some workstations went down.
15:02What types of workstations, meaning is this, you know, processing vouchers on an administrative
15:07level, for example?
15:08I'm not saying that's what happened, but what types of workstations went down, and is there
15:13some sort of bifurcation to ensure the critical services didn't go down, or was it just luck?
15:20So in this case, there's no luck.
15:22It's good planning and practice, which kept our emergency services and our lifeline services
15:27up.
15:28So the way that New York City controls its infrastructure, our most critical services,
15:31911-related, 311-related, we isolate and keep it in a separate environment, and we control
15:37what updates get pushed into that segment of the environment to ensure that in a situation
15:42like this, when something happens, it doesn't take down our most critical systems.
15:46So there's no automatic updating on those critical services?
15:49There's automatic updating, but it's controlled so that it only happens within certain periods
15:53of time.
15:54And we have a sandbox where we test those updates to ensure that when they get applied,
15:58if it impacts, it impacts something that's in a test environment, not in a production
16:02environment.
16:03In the other cases, for the machines that have access out to the internet and the basic
16:09workstations that many people interact with, those are updated in more real time because
16:13the risks to those machines are much greater than the ones that are in the public safety
16:18side of the world.
16:19And what kind of workstations were those?
16:20What type of jobs will New York City impact on?
16:23Yeah, so what I'd say is that there's going to be a myriad of impact, which we are currently
16:27assessing.
16:29And those workstations are in places like TLC, DOB, they're just regular computers that
16:34are out in most of the agencies.
16:41Just to clarify that complaints as they come in through either 311 or to the agencies are
16:46always prioritized.
16:48Those high prioritized complaints are being addressed.
16:52So even if there was a period of time when the workstation is down or there is a lack
16:57of interface because of the outage, we are manually going in and ensuring we have all
17:03of those priority complaints and they're being addressed across the city, especially with
17:07the first responder agencies.
17:08Does CrowdStrike know how that file got corrupted or that the implementation got corrupted?
17:09Are they still trying to figure it out?
17:10Yeah, so after we discovered the incident, we did a lot of work on that.
17:11And we started to see the impact.
17:29With working with CrowdStrike, they were able to correlate the impacts that not only that
17:34we saw but the global community saw with an update that they pushed and they found it
17:39with timing.
17:40Now, what they said was they tested the update in their test environment and none of these
17:46symptoms manifested, but something happened somewhere in between where when they went
17:51to test it and they went to deploy it, something changed or something got corrupted, which
17:55is why we see what we see right now.
17:57So CrowdStrike has confirmed that the issue is related to their control file that was
18:01updated and they said that which is why they stopped deploying it and they rolled back
18:07to the previous version.
18:08It sounds like they're still trying to pinpoint like what went wrong in a way.
18:13Yeah, I'm sure they're looking in their quality control process, but I would defer to CrowdStrike
18:17to have them answer what's going on in their internal process.
18:19Just to be clear, are there any impacts to fire or police services at all?
18:26No.
18:27So emergency services, meaning if you dial 911 or you call for support, there are no
18:32impacts to calls for support.
18:34911 operations is working, dispatch operations are working, and emergency communication via
18:38radio are also working.
18:39And one more.
18:40As New Yorkers go about their day, is there anything that they can expect to see any specific
18:41disruptions?
18:42I'm sorry, can you repeat that one?
18:43As New Yorkers go about their day, are they going to see any specific disruptions?
18:51So as we mentioned, the essential services and life safety related services are not impacted.
18:55However, there are other services that are in the city that may be impacted.
18:59So if you go in to file for a permit or you go in to request some other service, you go
19:03to pay a bill online like a water bill or a parking ticket, you might find yourself
19:07in a position where those services may be offline temporarily.
19:11But as I said, we're moving through and we have the pathway to recovery.
19:15And we're confident that soon we'll be back to a normal operating state for the services
19:20that may feel slightly impacted.
19:27Water bills are fine.
19:28Water bills are fine.
19:29You can still pay your water bill.
19:32And when we have situations like that, we automatically go to what are the emergency
19:37services that impact New Yorkers on an everyday basis.
19:39I think Chief Fraser actually explained it properly.
19:43When it comes to the FDNY and when it comes to the NYPD, there are some back issues that
19:47we're dealing with now that won't have the immediate impact on the day-to-day services.
19:52There's some arrest processing issues that's taken place that we're on top of.
19:55There's some camera issues that are affecting DOC, DEP, NYPD.
20:00The average, 99% of them won't be impacted by it, and we're pretty relatively confident
20:05that the impact that we will have will be cleared up before it does have an operational
20:10impact on, in fact, the everyday public.
20:13But we're looking on this.
20:14We're working on it now.
20:15While we're here meeting, the mayor has a whole team, all the agencies are all talking
20:20coordinated together.
20:21So we're pretty confident, in fact, that we jumped on top of this.
20:23So as the mayor said earlier, I just want to reiterate, and the chief of staff, a series
20:28of meetings.
20:29So when this came up, and we went into mold, this was something that we did, I think, two
20:33or three weeks ago.
20:34It was almost like it was an exercise that we had kind of planned for.
20:37It came out and packed, and that's why we're going to see minimal disruptions to New York
20:42City.
20:43And it was, the team is working well.
20:46Mike.
20:47MTA related, which is why I wanted to get it in.
20:52Is there any sort of update on when the somewhat minor impacts, ETAs and whatnot, will be resolved
20:59by MTA?
21:00When are New Yorkers expecting those?
21:02It's all related to the same technological issues that the chief technology officer outlined
21:07in detail.
21:08We're anticipating when CrowdStrike and Microsoft get their resolution, it'll translate through
21:15pretty quickly.
21:16But in the meantime, New Yorkers know where to go.
21:20Regular bus service, regular subway service, regular commuter rail, on schedule.
21:25We're keeping to that, and they can count on the service being provided.