Highlights:
- OpenAI trained CriticGPT on a dataset with intentional bugs to teach it to identify and flag common coding errors.
- OpenAI asserts that CriticGPT can detect even the most nuanced mistakes that humans often overlook, even with thorough evaluations.
Recently, OpenAI introduced CriticGPT, which is designed to identify bugs and errors in the outputs of artificial intelligence models. This development aims to ensure that AI systems operate according to their creators’ intentions.
Traditionally, AI developers use a method called Reinforcement Learning from Human Feedback (RLHF) to assist human reviewers in evaluating and improving the accuracy of large language models’ outputs. However, OpenAI believes that LLMs themselves can aid in this process, which primarily involves critiquing the outputs of AI models.
In their research paper titled “LLM Critics Help Catch LLM Bugs,” OpenAI researchers outlined how CriticGPT was developed to aid human reviewers in assessing code produced by ChatGPT. Built using the GPT-4 LLM, CriticGPT demonstrated promising skills in analyzing code and identifying errors, helping human reviewers detect AI “hallucinations” that might otherwise go unnoticed.
CriticGPT was trained on a dataset of intentionally flawed code samples, according to OpenAI researchers, so it could identify and highlight the various coding errors that frequently occur in software.
OpenAI explained that during the training process, human developers were tasked with altering code generated by ChatGPT by introducing various errors and providing sample feedback, just as they would if they encountered genuine bugs. This approach trained CriticGPT to recognize both frequent and less typical coding errors.
After training CriticGPT, OpenAI tested its performance with impressive results. CriticGPT showed greater competence than the average human code reviewer, with its critiques being preferred by human trainers over those written by humans in 63% of cases. OpenAI attributes this success partly to CriticGPT generating fewer unhelpful “nitpicks” and fewer false positives.
To advance their research, OpenAI’s team developed a new technique called “Force Sampling Beam Search,” allowing CriticGPT to produce more detailed critiques of AI-generated code. This method also provided greater flexibility, enabling human teachers to adjust CriticGPT’s thoroughness in bug detection and better control its occasional tendency to “hallucinate” or flag non-existent errors.
CriticGPT’s thoroughness enabled it to outperform humans significantly. The researchers applied CriticGPT to ChatGPT training datasets previously marked as “flawless” by human annotators, indicating they should be free of any bugs. Despite this, CriticGPT identified bugs and errors in 24% of these datasets, which human reviewers later confirmed.
According to OpenAI, this demonstrates that CriticGPT can detect even the most subtle mistakes that humans typically overlook, even during exhaustive evaluations.
However, it is important to note that CriticGPT, like the supposedly flawless training datasets, has its limitations. For instance, it was trained on relatively short responses from ChatGPT, which could hinder its ability to evaluate longer and more complex tasks, representing the next stage of generative AI evolution. Additionally, CriticGPT cannot catch every error and occasionally hallucinates, creating false positives that might lead human annotators to make mistakes when labeling data.
One challenge facing CriticGPT is its effectiveness in detecting inaccuracies stemming from errors in a single piece of code. However, it struggles more with AI hallucinations caused by errors scattered across multiple code segments, making it challenging for CriticGPT to pinpoint the source of the issue.
Nevertheless, OpenAI remains optimistic about the progress made thus far and intends to integrate CriticGPT into its RLHF pipeline. This move means that human trainers will have their own generative AI assistant to assist them in reviewing outputs from generative AI models.