Re-running a Vulnerability Detection Benchmark, More Carefully

Overview

A widely cited graph neural network for detecting vulnerable code is reproduced from scratch on the authors’ own released data, then put under three kinds of scrutiny: does the reported accuracy actually replicate, is the benchmark itself measuring what it claims to measure, and is the model’s own architecture doing what its designers intended?

Method

The reproduction is trained and evaluated exactly as described in the original paper. Two further checks are then layered on top: a data-leakage audit that traces which functions in the test set share a code change (“commit”) with functions in the training set, and a diagnostic pass that inspects the model’s internal readout layer at initialization to see whether it can actually learn in the first place. A replacement readout layer is then proposed and tested against the original.

Results

The reproduction lands close to other independent replications, noticeably below the original paper’s own number. The bigger finding is structural: roughly two-thirds of the benchmark’s test examples share a code change with something the model already saw in training, and once that overlap is removed, performance drops to barely above random guessing — meaning a large share of the benchmark’s apparent difficulty was already solved by memorizing which change something came from. Separately, the original architecture’s readout layer turns out to be poorly conditioned at initialization in a way that stalls learning; a replacement pooling method trains in a healthier regime and, as a side benefit, points to which specific lines of code the model considers suspicious.

2026

Back to top ↑

2025

Toward a Field-Based View of LLM Backdoors

1 minute read

Overview Existing LLM backdoor scanners mostly search for an object: a trigger string, a target string, or a memorized leaked example. This project explores ...

Back to top ↑

2024

Osmium Project Tasks

1 minute read

Overview In this project, we developed an Android application to estimate the location of cellular network cells using Received Signal Strength Indicator (RS...

Back to top ↑

2023

Back to top ↑

2021

Sonic pi

less than 1 minute read

My sonic pi project : . این پست من مربوط به پوروژه سونیک پای بنده است

Back to top ↑

2020

مصاحبه

1 minute read

My works and wishes Success secret مصاحبه با جناب اقا پارسا: ایشون بسیار ادم سخت کوشی بودن و گفتن که حتی در دوران دانشجویی شون درس هم میدادن 1 به موار...

My works and wishes

less than 1 minute read

Its my favorite university This is oxford: In England The best unniversity. this university is first inthe world. this university have best descover...

Back to top ↑