NIST staff revolts against expected appointment of ‘effectively altruistic’ AI researcher at US AI Safety Institute

The National Institute of Standards and Technology (NIST) is facing an internal crisis as staff members and scientists vacate a key position at the agency’s newly formed USAI Safety Institute (AISI). Threatened to resign over Cristiano’s expected appointment. According to at least two sources with direct knowledge of the situation, who asked to remain anonymous.

Cristiano, who is known for his ties to the Effective Altruism (EA) movement and its offshoot, long-termism (a view that prioritizes humanity’s long-term future, popularized by philosopher William MacAskill), allegedly had gone through the recruitment process without anyone. To date, said one of the sources.

Cristiano’s appointment, which was said to come directly from Secretary of Commerce Gina Raimondo (NIST is part of the US Department of Commerce), caused consternation among NIST employees. has sparked anger among those who fear that Cristiano’s association with the EA and the long-term could compromise the institute’s objectivity and integrity. .

Many argue that EA—defined by the Center for Effective Altruism as “an intellectual project using evidence and reason to determine how to maximize the benefit of others. ” – has evolved into a cult-like group of extremely influential and wealthy followers (made famous by FTX founder and jailbird Sam Bankman Freud) whose primary concern is preventing future AI disasters from destroying humanity. Spins around. Critics of EA focus on this existential risk, or “x-risk,” saying it’s happening to the detriment of necessary attention to existing, measurable AI risks — including bias, misinformation, high-risk applications. cations and traditional cyber security.

The USAI Safety Institute was established in November 2023.

AISI was established in November 2023 under the AI ​​Executive Order to “support responsibilities assigned to the Department of Commerce.” Earlier today, US Senate Majority Leader Chuck Schumer (D-NY) announced that NIST will receive up to $10 million to establish the US AI Safety Institute.

Last month, VentureBeat reported on criticism of NIST’s lack of transparency surrounding AISI: In mid-December, lawmakers on the House Science Committee from both parties sent a letter to NIST in which Politico reported that “Agency penalized for lack of transparency and failure to announce. A competitive process for planned research grants related to new USAI Safety Institute. Lack of transparency centers around potential grant to RAND Corporation, “An influential think tank connected to tech billionaires, the AI ​​industry and effective altruism.” (VentureBeat points to a “wide web” of effective altruism followers in AI ‘safety’ and security circles, including RAND and leading LLM model company Anthropic. Also reported about).

‘It is important to ensure that NIST’s mission is not compromised’

Divyansh Kaushik, associate director of emerging technologies and national security at the Federation of American Scientists, told VentureBeat that regardless of who is appointed to the AISI role, “it is important to ensure that NIST’s mission is Don’t get affected.”

NIST has always done “methodological measurement research that is very grounded,” he said. “We are in uncharted territory with trying to establish an assessment program for hypothetical threats from general-purpose models. I fully understand the concerns NIST scientists will have and it is up to the Commerce leadership to Ensure that any appointments are consistent with science and NIST’s mission and procedures.

If some of the appointments lead to the departure of NIST’s “superstar scientists,” he added, “that would be a shame — in that case, I would certainly hope that the secretary would see that What will she lose by appointing one person and reconsider her choice.”

He pointed out that congressional committees are already paying close attention to the work of AISI. “I hope they’ll be able to work well with good people who are excellent measurement scientists and do it in a way that builds on NIST’s strengths rather than trying to recreate NIST’s mission.” It does,” he said. .

VentureBeat reached out to Secretary of Commerce Gina Raimondo, AISI Director Elizabeth Kelly, and Paul Cristiano. We’ll update if and when we hear back.