Measurement · Tool comparison

cloc vs CodeDelta: what a line counter measures and what it misses

cloc is excellent at its job — counting lines, quickly, in hundreds of languages. The trouble starts when its diff numbers get quoted as churn. We ran both tools over six and a half years of Erlang/OTP on identical commit hashes; this is what each one saw.

August 2026 · full measurement report with per-window output: the Erlang/OTP study

Different questions, different tools

cloc answers "how many lines of code are here?" — and does it fast, across hundreds of languages. CodeDelta answers "what changed between these two versions, at the level of statements?" Those sound adjacent. They are not, and teams that quote cloc diff output as churn inherit the gap.

What the side-by-side showed

Both tools ran on the same public commit hashes of Erlang/OTP — a two-million-statement platform with a C/C++ virtual machine, machine-generated JIT tables, and deliberately hostile test suites. Reproducible findings:

Three of seven diff windows aborted outright in cloc 2.10 — legitimate repository content (Unicode-named test directories) defeats its git-archive step unless you know an obscure git setting.
4,046 files were set aside silently. No coverage line says what fraction of the tree was measured; you only learn it by asking for the ignore log. CodeDelta states coverage in every report (93.0% on OTP, remainder itemised).
Real Erlang defeats its comment detection. cloc's Erlang is two regular expressions; on a 16-line file with known-by-construction truth (14 code, 6 comment lines) it reads 14 code / 2 comment, and it classifies documentation blocks in OTP's own source as comments.
A pure reformat produced 80 lines of "churn" and exactly 0 changed statements. Line-diffing cannot tell an edited statement from a re-wrapped one; statement-level measurement can.

The honest scorecard

Counting lines in a tree: use cloc; it's the standard for a reason. Measuring change — churn that gates merges, feeds trend databases, or goes in front of an auditor — needs statement identity, whole-tree coverage with the coverage stated, and a parser that reads each language as written. That is the job churn measurement exists for. The full report publishes every window hash, cloc's per-window output, and the two defects OTP exposed in our own engine along the way — including the withdrawn first scan. Comparisons you can re-run beat comparisons you're asked to believe.