There are an increasing number of new EDA companies these days, and this is great news. The industry needs some new vitality, new thought leaders, people willing to take risks. The same goes for established companies that cannot go outside their existing comfort zone while meeting next quarter’s financial goals. But sometimes, new startups forget the risks associated with semiconductor design. While they may be able to point to an improvement in time to market, or meeting better performance goals, they fail to understand how small that particular step is in the whole process.
Today, a number of companies want us to believe that AI can do everything. Some point the way to automating design, others to verification, and some even claim it can do both at the same time, from the same source material. No, it can’t. While some of it may be possible in the future, it is far from that point today.
What many of them fail to do is show proof. One example, one design, one piece of IP is not proof. When verifying any new tool or methodology, you always need to hold back a test set to apply at the end. In this case, with AI, it has to be something it has never seen before, never been trained on. That is the only thing that can provide a fair, unbiased result. As soon as you tweak your tool to do well on that test case, it is no longer a true test.
This is why I was delighted to receive communications from Vikash Kumar, senior verification architect at Arm. He has 15+ years of semiconductor industry experience and is the author of the book “AI-Assisted Hardware Verification: Cognitive Verification Architecture with the VEGA Framework.” What he shared with me was his independent work on the quality of existing LLMs for the creation of testbenches.
More than 20 years ago, Certess introduced the industry to mutation analysis. If you inject a bug into the design, does the testbench find it? They used the traditional stuck-at fault model and sophisticated statistical analysis to reduce the total number of faults required. The results were a slap across the face to many companies that thought they were doing a good job, only to find out how poor their testbenches were.
Vikash’s approach is different. “The setup was simple — hand AI a spec and the RTL, tell it to verify and report back when it is confident the design is clean, and check what it actually found. I hand-seeded realistic, design-representative functional bugs, which keeps the ground truth clean. I know exactly which bugs were planted, so detection is measured against a known set rather than an impression.”
I asked Vikash why he chose to insert bugs rather than using the stuck-at model, which has been the industry go-to since the ’80s. “The whole question with AI verification is whether it comprehends design intent well enough to catch the subtle, intent-level bugs, and that is exactly where it struggled — the misses clustered in data-path, encoding logic, and functional corners, not in anything a stuck-at pattern would have flagged.”
He started with 25 hand-coded designs spanning bus protocols, DMA engines, ECC memory, control and peripheral logic, and concurrency scenarios. The RTL has never been made public. Into those he inserted 806 bugs. The models were given a specification based on published protocol standards. Three frontier AI models were chosen: Claude Opus 5, GPT-5.4, and GLM-5.2.
“Without human intervention or predefined time limits, each model generated testbenches, developed coverage goals, executed simulations, identified and corrected RTL defects, and iteratively refined verification until independently declaring sign-off. All defects were seeded upfront, although some became observable only after earlier corrections.”
Here are his key findings:
Claude Opus 5 detected 79.2% of seeded defects, GPT-5.4 detected 76.2%, and GLM-5.2 detected 62.8%.
Even the highest-performing model missed 20.8% of seeded defects, which is approximately 168 bugs.
All three models reached 100% of their self-defined verification goals and declared sign-off despite remaining functional defects.
In this audit, particularly high miss rates were observed in ECC memory, GPIO, and AXI DMA designs.
Some missed defects involved ambiguous specifications. The agents proceeded without requesting clarification and ultimately declared verification complete.
Even the best model, Claude Opus 5, did not detect roughly 168 defects. GPT-5.4 missed about 192, and GLM-5.2 missed close to 300. Perhaps even worse is that Vikash ran each model three times, and each run produced different results. But he is very optimistic about the results. “The impressive part is how much AI caught on its own,” he says. “The important part is how much it missed, and that’s where the engineer still matters.”
In a similar manner, ChipAgents just released its new build-a-chip-bench tool. It goes from a workload specification to GSD. The company ranked five models. The best, Claude Fable 5.1 achieved a score of 16.6%, whilst the bottom of the pack was GLM-5.3 with a score of zero. These results look dire, but the biggest failure was their lack of understanding of performance constraints.
I believe this will be an even bigger problem when it comes to writing software that has to be cognizant of the underlying hardware and its performance beyond raw speed. Real-time systems deployed within physical AI need predictable latency, at low power, over a broad range of operating conditions. None of this data is available for training today.
I will finish with another comment Vikash made: “You can automate the work of verification. You cannot automate the responsibility for it.”
The post Putting AI To The Test appeared first on Semiconductor Engineering.