What question did this study set out to answer?

The aim is to evaluate the effectiveness of large language models in finding security vulnerabilities in code.

April 22, 2026Open Access

Benchmarking Large Language Models for Automated Code Security Analysis

Key Points

The aim is to evaluate the effectiveness of large language models in finding security vulnerabilities in code.
Developed CodeSight, an open-source CLI tool for code analysis.
Tested five large language models on three known-vulnerable codebases.
Compared detection rates of LLMs with a traditional tool, Semgrep.
LLMs identified approximately twice as many vulnerabilities as pattern matching tools.
Significant improvements were noted in detecting authentication bypasses and business logic bugs.

Abstract

Static analysis tools match patterns and miss logic bugs. This paper presents CodeSight, an open-source CLI that sends code to LLMs and gets back security findings. We tested five models (GPT-5.4, Claude Opus 4.7, Gemini 3.1 Pro, Llama 4 Maverick, DeepSeek V3.2) on three known-vulnerable codebases and compared detection rates with Semgrep. LLMs found roughly twice as many vulnerabilities as pattern matching, with the biggest gains in authentication bypasses and business logic bugs. CodeSight is MIT licensed and available at https://github.com/AvixoSec/codesight.

Read Full Paperexternally

AIに質問

Bookmark

View Full Paper