Highlights:

  • The Stability AI’s popularity lies in its image generation models, namely Stable Diffusion and Stable Doodle, designed to interpret text instructions and user-provided doodles as inputs.
  • Stability AI trained U-Net on over 800,000 audio files from AudioSparx, a stock music provider.

Stability AI Ltd. unveiled Stable Audio, a software platform employing a latent diffusion model to produce audio based on user text prompts.

The platform can produce audio clips lasting up to 95 seconds, spanning diverse music genres. Stability AI also notes that Stable Audio can be utilized for crafting various audio forms, including sound effects.

Stability AI, headquartered in London, has secured over USD 100 million in venture funding. The company’s popularity lies in its image generation models, namely Stable Diffusion and Stable Doodle, designed to interpret text instructions and user-provided doodles as inputs. Stability AI has also introduced an open-source language model capable of generating code and text.

Typically, artificial intelligence audio generators, such as Stability AI’s recently launched Stable Studio platform, employ what are known as diffusion models. These neural networks are built using a training dataset deliberately added with errors. These Gaussian noise errors instruct a diffusion model to analyze the audio files within its training dataset and autonomously produce similar files.

Stability AI contends that AI systems of this nature possess two primary limitations.

According to the company, diffusion models typically have a limitation in generating audio snippets of a fixed length. For instance, an AI trained on 30-second sound snippets cannot generate files of 40 or 20 seconds in duration. Moreover, clips produced by such models frequently commence in the middle or toward the end of a musical phrase, negatively impacting their overall quality.

Stable Audio employs a specialized variant of diffusion models called a latent diffusion model to surmount these constraints. What distinguishes these models from the conventional ones is that they are invariably employed in tandem with a second neural network known as an autoencoder.

An autoencoder is an AI system capable of taking a data input and eliminating extraneous information. For instance, such a model could take in an audio file that includes background noise and effectively filter out that. The autoencoder stores the retained information within a mathematical construct called latent space.

A standard diffusion model is developed using raw training datasets. In contrast, a latent diffusion model is constructed using an enhanced version of the same training datasets, from which an autoencoder has eliminated superfluous information. Due to the improved quality of the refined datasets, the latent diffusion model trained on them can produce higher-quality output.

Stability AI’s innovative Stable Audio platform comprises not just one but three neural networks. At its core is U-Net, a latent diffusion model boasting an impressive 907 million parameters. It represents an improved iteration of a pre-existing neural network, Moûsai, introduced earlier this year.

Stability AI trained U-Net on over 800,000 audio files from AudioSparx, a stock music provider. The company reports that these files contain approximately 19,500 hours of audio. Stability AI incorporated text-based metadata or contextual information to optimize the AI training process.

Stable Audio incorporates U-Net and two additional neural networks. One is an autoencoder, while the other translates user prompts describing the audio to be generated into a format U-Net can comprehend.

When running on an A100 graphics processing unit, the platform can produce 95 seconds of audio with a sample rate of 44.1 kHz in less than one second. The A100 was Nvidia Corporation’s flagship data center GPU until the H100 replaced it last year.

Stability AI intends to improve both its audio generation models and the dataset utilized for training them in the future. Additionally, the organization will release open-source models based on Stable Audio.