Last week, Mayor Eric Adams announced that he was instituting a 90-day waiting period for testing weapons scanners on New York City’s subway system.
The scanners he demonstrated, supplied by Evolv Technology, aim to mix physical detection technology with artificial intelligence to detect when someone is carrying a gun or knife. , and has been used in city hospitals before, Adams pointed out at the press conference. Fulton Street Station.
“We’re going to make sure, based on our numbers, hit ratio, false hits, we’re going to do that based on our data,” Adams told reporters. “What I’m hearing from our corporations, from our hospitals, from others, they’re saying it’s meeting our expectations.”
What Adams didn’t mention was that the city already has some data, and it’s not good. Evolv’s scanners were installed at Jacobi Medical Center in the Bronx in 2022, and according to pilot results obtained by Hell Gate through a public records request filed with NYC Health + Hospitals, there were a large number of false positives. was activated.
During the seven months that Jacobi’s pilot was active, 194,000 people passed through Jacobi’s scanners, and in only 50,000 of those cases, the scanners raised an alarm—an incidence rate of about 26 percent, or one in four. Someone went through the scanner. Of those 50,000 alarms, about 43,800, or just over 85 percent, were false positives. Of the alarms, 7,027, or 14 percent, were law enforcement officers possibly carrying their service weapons. And only 295 alarms, or 0.57 percent, were determined to be a non-law enforcement person carrying either a knife, a gun, or a threat simply labeled “other,” including bats. Other weapons are included.
Notably, Evolv’s scanners didn’t become more accurate as the pilot progressed—there wasn’t a single month where the alarm-to-visitor ratio was below 25 percent. In September 2022, the final month of the seven-month pilot, 27,900 visitors passed through the scanners, and about 7,000 set off alarms. Of those 7,000 alarms, only 345 were identified as potential threats—a false positive rate of 95 percent, with only 0.45 percent of alerts being non-law enforcement threats. Throughout the pilot, the alerts resulted in the detection of 24 guns, 139 knives, and 132 other potential threats out of 50,000 alarms.
“It’s basically a coincidence,” said Daniel Schwarz, senior privacy and technology strategist at the New York Civil Liberties Union, who reviewed the pilot data at Hellgate’s request, referring to the small number of weapons. which were actually found. “They’re looking for all these people and then they accidentally get something on them.”
The NYCLU has been vocal in its opposition not only to these scanners, but also to various NYPD surveillance technologies, not only on the grounds that they infringe on the civil liberties of New Yorkers, but also because they are particularly well-documented. do not work, and their use may produce unintended consequences. On subways, the concern is that not only will the tech potentially cost millions — each scanner is a service the company offers for about $2,500 a month, and police officers will have to monitor them — But shut down a system that relies on speed and ease of use and keeps policemen on constant alert for confrontation.
“On the scale of the NYC subway system, if we imagine four million riders a day, there would be a million false flags or alerts, which would bring the entire system to a standstill,” Schwarz said. Evolv CEO Peter George himself said in an investor call earlier this month that “subways in particular are not a space that we think is a good use case for us.” (Evolv did not respond to a request for comment.)
Broadly, this system works like a metal detector but is specifically aimed at detecting weapons as opposed to metal objects. It has sensitivity settings that are set by the user. At higher sensitivity, it will go off more often, detecting additional guns and knives but also producing more false positives. At lower sensitivity settings, it will stop playing as many bogus alarms, but will also miss out on more real weapons. In one infamous case, a school district in Illinois found that tech was consistently labeling Chromebook laptops as guns. It is not clear what sensitivity settings were set in the Jacobi system.
Although it’s been two years since the Evolve pilot at Jacobi, there are good indications that the technology has not yet developed to the point where it is reliable. Evolv’s own shareholders filed a class-action lawsuit against the company this month, arguing that the company’s marketing materials and public announcements exaggerated the technology’s effectiveness. This follows the opening of investigations by both the Federal Trade and Securities and Exchange Commission. There are individual accounts of the technology failing to detect weapons in schools across the country, while throwing off false alarms for innocent objects.
Meyer gave mixed messages on whether Evolv itself would ultimately be the vendor of the latest scanner effort. During his March 28 announcement, Adams said the NYPD “will work to identify all vendors with efficient technology and expertise,” and later said in response to a question that the bidding process would be “highly competitive.” Yes, the best product is going to be used in our city.” Nonetheless, he used an Evolv scanner to demonstrate the technology, saying “the company we’re partnering with and announcing today is Evolv.”
In response to our questions about whether the Jacobi data raised any concerns and why an additional pilot might be necessary, a City Hall spokesperson wrote that “last week’s announcement was never about any specific company, it was about our The intention was to use every available tool… During this 90-day waiting period, we will engage with as many tech companies that will work in this space to see if the pilot program will What is our best option? Our pilot program will then test what works, what doesn’t.”
Evolv has a handful of competitors, including companies like ZeroEyes and Patriot One, which enjoyed increased attention and funding after mass shootings around 2018 and 2019. Yet, no weapon scanner technology has been conclusively proven to be reliably accurate.
In addition to Jacobi Hospital, Evolve scanners were also briefly installed at City Hall. We filed a Freedom of Information request for the results of the 2022 pilot, but the NYPD denied it on the dubious claim that the data would “reveal unusual techniques and procedures.”
“We have no evidence that this technology is effective in finding guns,” Schwarz said. “The only evidence we have is that it isn’t.”