March 3, 20260 citationsOpen Access

Learning to Detect Hardcoded Secrets

Key Points

The best performance in secret detection is achieved with context features surrounding the secret value, leading to higher accuracy.
CodeBERT shows superior results with an MCC score of 88% and an F1-score of 89% in controlled conditions and datasets.
Non-generative models outperform generative ones, though both experience high false positive rates during tests.
Manual investigations indicate that large language models (LLMs) flag potential secrets cautiously, indicating room for improvement.

Abstract

Numerous tools have been developed to uncover secrets and credentials in source code. However, they often rely on predefined rules that yield a high rate of false positives. We conduct an in-depth investigation into the performance of different AI models in secret detection. In non-generative models, we found that the choice of feature set greatly influences the performance in secret detection. These models demonstrated the best performance using the context feature, i.e., code surrounding the secret value. Particularly, CodeBERT outperforms other models with an MCC and F1-scores of 88% and 89%, respectively.When we applied the models to an unseen dataset of secrets, the top models were RF and CodeBERT, achieving an MCC score of around 65%. In generative models, we observed moderate performance (a maximum MCC of 63\%) and a high number of false positives, most of which occurred in the test path files. Our manual investigation of incorrect predictions revealed that LLMs tend to be cautious, flagging potential secrets to promote secure coding practices.

问 AI

Bookmark

View Full Paper