AI trained to be inspired by images, not imitate them.

Powerful new artificial intelligence models sometimes, quite famously, get things wrong—whether spoofing false information or remembering the work of others and passing it off as their own. To address the latter, researchers led by a team at the University of Texas at Austin have developed a framework to train AI models on images corrupted beyond recognition.

DALL-E, Midjourney and Stable Diffusion are among the text-to-image diffusion generative AI models that can transform arbitrary user text into highly realistic images. All three are now facing lawsuits from artists who allege that the designs produced duplicate their work. Trained on billions of image-text pairs that are not publicly available, these models are capable of capturing high-quality images from textual cues but can capture the copyrighted images they mimic.

The new proposed framework, called Ambient Diffusion, solves this problem by training diffusion models by only accessing corrupted image-based data. Initial efforts show that the framework is able to generate high-quality samples without seeing anything that is recognizable as the original source images.

Ambient Diffusion was originally presented in 2023 at NeurIPS, a machine learning conference, and has since been adapted and extended. A follow-up paper, “Continuous Diffusion Meets Tweedy,” is available at arXiv Preprint server, accepted at the 2024 International Conference on Machine Learning. Together with Constantinos Daskalakis of the Massachusetts Institute of Technology, the team extended the framework to train diffusion models on image datasets corrupted by other types of noise, rather than just masking pixels.

“The framework could also be useful for scientific and medical applications,” said computer science professor Adam Clevans, who was involved in the work. “This would be essentially true for any research where it would be expensive or impossible to obtain a complete set of uncorrupted data, from black hole imaging to certain types of MRI scans.”

Clevance; Alex Demax, professor of electrical and computer engineering; and other collaborators at the multi-institution Institute for Foundations of Machine Learning, led by two UT faculty members, first experimented by training a diffusion model on a set of 3,000 images of celebrities, then using the model to generate new samples. used.

In the experiment, the diffusion model trained on clean data accurately replicated the training examples. But when the researchers corrupted the training data, randomly masked up to 90% of the individual pixels in an image, and retrained the model with their new approach, the generated samples remained of high quality but looked very different. Came. The model can still generate human faces, but the generated training images are quite different.

“Our framework allows us to control the trade-off between memorability and performance,” said Giannis Darras, a computer science graduate student who led the work. “As corruption is encountered during training, the memory of the training set decreases.”

The researchers said that this points to a solution that, although it may change the performance, will never produce noise. The framework provides an example of how academic researchers are advancing artificial intelligence to meet societal needs, a key theme this year at the University of Texas at Austin, which has designated 2024 as the “Year of AI.” has declared

The research team included members from the University of California, Berkeley and MIT.