Highlights:

  • Galileo CEO Vikram Chatterji reported that businesses must be able to assess hundreds, if not thousands, of AI solutions in near real-time for issues including toxicity, security threats, and hallucinations.
  • The startup tested Luna EFMs through a sequence of benchmarks to compare their performance with other AI assessment tools.

Galileo’s latest evaluation foundation models are specifically designed to assess the effectiveness of large language models, like Google LLC’s Gemini Pro and OpenAI’s GPT-4o. The generative AI evaluation startup has introduced industry’s first family of evaluation foundation models.  

Galileo created the Luna EFM models, essentially LLMs, in response to the AI industry’s experiments in using AI for AI evaluation. The business stated that there has been encouraging development in the last few years in the amount of research published on the applicability of employing models like GPT-4 to evaluate the responses of different LLMs.

Galileo reasoned that considering these advancements, it could be more sensible to design a group of specialized LLMs that were explicitly trained to assess the outputs of other generative AI models. The outcome of this effort is the Luna EFM family.

The business notes in a study posted on Arxiv that each of its Luna EFMs has been customized to carry out a particular evaluation task, such as identifying instances of artificial intelligence systems fabricating replies or hallucinations. Others are made to identify fraudulent prompts, context quality issues, and data leaks.

Galileo, a company that creates tools to improve the accuracy of AI models, asserts that its Luna EFM models can provide businesses with the assurance they need to implement generative AI chatbots at scale because they are faster, more affordable, and more accurate than either GPT-4 or conventional human vibe checks.

Galileo CEO Vikram Chatterji reported that businesses must be able to assess hundreds, if not thousands, of AI solutions in near real-time for issues including toxicity, security threats, and hallucinations. He stated that after collaborating with numerous enterprises to address this issue, the organization concluded that standard LLM-based reviews and human assessments were excessively costly and time-consuming.

“We set out to solve that, and with Galileo Luna we’re setting new benchmarks for speed, accuracy and cost efficiency. Luna can evaluate millions of responses per month 97% cheaper, 11x faster, and 18% more accurately than evaluating using OpenAI GPT3.5,” Chatterji claimed.

The startup tested the Luna EFMs through a sequence of benchmarks to see how well they performed compared to other AI assessment tools. The findings were quite encouraging.

Chatterji claims that the Luna family outperformed all other evaluation models by as much as 20% in terms of overall accuracy, including surpassing the business’s own Chainpoll LLM, which is intended to identify hallucinations.

Furthermore, the testing demonstrated how much less expensive the Luna EFMs are, with assessment compute costs reportedly up to 30 times lower than those of GPT-3.5. With data provided in a matter of milliseconds, the evaluations are also far faster. The CEO added that because the Luna EFMs can be easily adjusted to identify particular issues with generative AI outputs, they are also far more flexible than other systems.

Furthermore, the Luna EFMs excel in explainability by offering users an explanation of their assessments. According to the startup, doing this can facilitate the debugging and root-cause analysis processes.

The early adopters of the Luna EFMs have endorsed Galileo’s benchmarks.

According to Alex Klug, Head of product, data science, and AI at HP Inc., the industry leader in personal computers, producing safe, dependable, and production-grade AI systems requires precise model evaluation tools. “Until now, existing evaluation methods such as human evaluations or using LLMs as a judge have been very costly and slow. With Luna, Galileo is overcoming enterprise teams’ biggest evaluation hurdles,” added Klug.

The company added that many Fortune ten banks and Fortune 50 organizations are already using the Luna EFMs, which are available in its Galileo Project and Galileo Evaluate platforms.