Higher than JPEG? Researcher discovers that Secure Diffusion can compress photos

Final week, Swiss software program engineer Matthias Bühlmann found that the favored picture synthesis mannequin Secure Diffusion might compress current bitmapped photos with fewer visible artifacts than JPEG or WebP at excessive compression ratios, although there are vital caveats.

Secure Diffusion is an AI picture synthesis mannequin that sometimes generates photos primarily based on textual content descriptions (known as “prompts”). The AI mannequin realized this skill by finding out tens of millions of photos pulled from the Web. In the course of the coaching course of, the mannequin makes statistical associations between photos and associated phrases, making a a lot smaller illustration of key details about every picture and storing them as “weights,” that are mathematical values that signify what the AI picture mannequin is aware of, so to talk.

When Secure Diffusion analyzes and “compresses” photos into weight kind, they reside in what researchers name “latent house,” which is a manner of claiming that they exist as a kind of fuzzy potential that may be realized into photos as soon as they’re decoded. With Secure Diffusion 1.4, the weights file is roughly 4GB, however it represents information about lots of of tens of millions of photos.

Whereas most individuals use Secure Diffusion with textual content prompts, Bühlmann minimize out the textual content encoder and as a substitute pressured his photos by way of Secure Diffusion’s picture encoder course of, which takes a low-precision 512×512 picture and turns it right into a higher-precision 64×64 latent house illustration. At this level, the picture exists at a a lot smaller knowledge dimension than the unique, however it will probably nonetheless be expanded (decoded) again right into a 512×512 picture with pretty good outcomes.

Whereas operating exams, Bühlmann discovered that photos compressed with Secure Diffusion regarded subjectively higher at increased compression ratios (smaller file dimension) than JPEG or WebP. In a single instance, he exhibits a photograph of a sweet store that’s compressed down to five.68KB utilizing JPEG, 5.71KB utilizing WebP, and 4.98KB utilizing Secure Diffusion. The Secure Diffusion picture seems to have extra resolved particulars and fewer apparent compression artifacts than these compressed within the different codecs.

Bühlmann’s technique presently comes with vital limitations, nonetheless: It isn’t good with faces or textual content, and in some instances, it will probably really hallucinate detailed options within the decoded picture that weren’t current within the supply picture. (You most likely don’t need your picture compressor inventing particulars in a picture that do not exist.) Additionally, decoding requires the 4GB Secure Diffusion weights file and further decoding time.

Whereas this use of Secure Diffusion is unconventional and extra of a enjoyable hack than a sensible answer, it might probably level to a novel future use of picture synthesis fashions. Bühlmann’s code might be discovered on Google Colab, and you will find extra technical particulars about his experiment in his submit on In direction of AI.