added source code review to methodology section

This commit is contained in:
2026-02-24 19:03:31 +01:00
parent 2e2d8880c0
commit c08df6b885
3 changed files with 110 additions and 14 deletions

View File

@@ -4,11 +4,14 @@
\label{Methodology}
This chapter describes the methodology used to benchmark peer-to-peer
mesh VPN implementations. The experimental design prioritizes
reproducibility at every layer; from dependency management to network
conditions; enabling independent verification of results and
facilitating future comparative studies.
This chapter describes the methodology used to benchmark and analyze
peer-to-peer mesh VPN implementations. The evaluation combines
performance benchmarking under controlled network conditions with a
structured source code analysis of each implementation. The
benchmarking framework prioritizes reproducibility at every layer;
from pinned dependencies and declarative system configuration to
automated test orchestration; enabling independent verification of
results and facilitating future comparative studies.
\section{Experimental Setup}
@@ -277,6 +280,60 @@ confidence intervals, which is appropriate for benchmark data that
may not follow a normal distribution. The nix-cache test (via hyperfine)
additionally reports standard deviation.
\section{Source Code Analysis}
To complement the performance benchmarks with architectural
understanding, a structured source code analysis was conducted for
all ten VPN implementations. The analysis followed three phases.
\subsection{Repository Collection and LLM-Assisted Overview}
The latest main branch of each VPN's git repository was cloned,
together with key dependencies that implement core functionality
outside the main repository. For example, Yggdrasil delegates its
routing and cryptographic operations to the Ironwood library, which
was analyzed alongside the main codebase.
Ten LLM agents (Claude Code) were then spawned in parallel, one per
VPN. Each agent was instructed to read the full source tree and
produce an \texttt{overview.md} file documenting the following
aspects:
\begin{itemize}
\item Wire protocol and message framing
\item Encryption scheme and key exchange
\item Packet handling and performance
\item NAT traversal mechanism
\item Local routing and peer discovery
\item Security features and access control
\item Resilience / Central Point of Failure
\end{itemize}
Every claim in the generated overview was required to reference the
specific file and line range in the repository that supports it,
enabling direct verification.
\subsection{Manual Verification}
The LLM-generated overviews served as a navigational aid rather than
a trusted source. The most important code paths identified in each
overview were manually read and verified against the actual source
code, correcting inaccuracies and deepening the analysis where the
automated summaries remained superficial.
\subsection{Feature Matrix and Maintainer Review}
The findings from both the automated and manual analysis were
consolidated into a comprehensive feature matrix cataloguing 131
features across all ten VPN implementations. The matrix covers
protocol characteristics, cryptographic primitives, NAT traversal
strategies, routing behavior, and security properties.
The completed feature matrix was published and sent to the respective
VPN maintainers for review. Maintainer feedback was incorporated as
corrections and clarifications, improving the accuracy of the final
classification.
\section{Reproducibility}
Reproducibility is ensured at every layer of the experimental stack.