Can LLMs Really Check Code?

In recent years, large language models (LLMs) have taken the programming world by storm. Today they not only write code but are also starting to check it. How does it actually work? In practice, LLMs can act as assistants for software verification, meaning they help spot bugs, vulnerabilities, or unexpected behavior. Some researchers have linked them to traditional static analysis tools, like linters, or to formal verifiers that mathematically prove a program's correctness. The trick lies in "prompt engineering": you ask the model a precise question, for example "find all possible overflows in this function," and the model returns suggestions or even corrected code.

Another crucial aspect is verifying code generated by the LLMs themselves. When a model writes a function, it’s essential to know if it’s safe. Some studies have built pipelines that, right after generation, run the code through vulnerability scanners or automatic patch systems. In this way, the LLM becomes both author and reviewer.

Research shows that hybrid solutions—LLM plus traditional tools—perform better than each part alone. Yet challenges remain: models can invent bugs, their answers aren’t always explainable, and scaling to huge projects is still an issue. The future seems to point toward a tight collaboration between AI and software engineering, aiming to make code safer and more reliable.

Large language models (LLMs) such as GPT‑4 or Claude are built on transformer architectures trained on massive text corpora that also contain code snippets. This exposure enables the model to learn syntactic and semantic patterns typical of programming languages, as well as deeper relationships between specific APIs and expected behaviors. When an LLM is used for verification, its probabilistic reasoning capacity is leveraged to generate error hypotheses or suggest fixes. However, the lack of a formal logical foundation makes integration with static analysis tools (e.g., clang‑tidy, SonarQube) or formal verifiers (e.g., Coq, Z3) necessary. A typical pipeline includes: (1) extracting the code to be analyzed; (2) crafting prompts that ask the model to identify specific vulnerabilities (buffer overflow, injection, race condition); (3) parsing the LLM's response to locate concrete suggestions; (4) automatically validating those suggestions with a traditional analysis engine. This hybrid approach reduces false positives typical of LLM‑only solutions, thanks to formal confirmation, while still exploiting the model's flexibility to cover cases not captured by predefined static rules. Regarding verification of LLM‑generated code, common errors include: (a) missing input validation; (b) misuse of unsafe libraries; (c) ambiguous conditional logic. Some groups have proposed "self‑verification loops," where the model generates code, passes it through a second verification pass (often with a smaller or specialized model), and iterates until convergence on an acceptable solution. Another research direction involves formalizing prompts using DSLs (Domain‑Specific Languages) that unambiguously describe security properties, allowing the model to translate these specifications into unit tests or invariants verified with tools like Dafny. Finally, scalability remains a challenge: analyzing large projects requires slicing and incremental analysis techniques to limit computational load for both the LLM and traditional verifiers. Ongoing interaction between machine learning and formal methods promises to enhance software robustness in the future.