added source code review to methodology section
This commit is contained in:
@@ -4,11 +4,14 @@
|
||||
|
||||
\label{Methodology}
|
||||
|
||||
This chapter describes the methodology used to benchmark peer-to-peer
|
||||
mesh VPN implementations. The experimental design prioritizes
|
||||
reproducibility at every layer; from dependency management to network
|
||||
conditions; enabling independent verification of results and
|
||||
facilitating future comparative studies.
|
||||
This chapter describes the methodology used to benchmark and analyze
|
||||
peer-to-peer mesh VPN implementations. The evaluation combines
|
||||
performance benchmarking under controlled network conditions with a
|
||||
structured source code analysis of each implementation. The
|
||||
benchmarking framework prioritizes reproducibility at every layer;
|
||||
from pinned dependencies and declarative system configuration to
|
||||
automated test orchestration; enabling independent verification of
|
||||
results and facilitating future comparative studies.
|
||||
|
||||
\section{Experimental Setup}
|
||||
|
||||
@@ -277,6 +280,60 @@ confidence intervals, which is appropriate for benchmark data that
|
||||
may not follow a normal distribution. The nix-cache test (via hyperfine)
|
||||
additionally reports standard deviation.
|
||||
|
||||
\section{Source Code Analysis}
|
||||
|
||||
To complement the performance benchmarks with architectural
|
||||
understanding, a structured source code analysis was conducted for
|
||||
all ten VPN implementations. The analysis followed three phases.
|
||||
|
||||
\subsection{Repository Collection and LLM-Assisted Overview}
|
||||
|
||||
The latest main branch of each VPN's git repository was cloned,
|
||||
together with key dependencies that implement core functionality
|
||||
outside the main repository. For example, Yggdrasil delegates its
|
||||
routing and cryptographic operations to the Ironwood library, which
|
||||
was analyzed alongside the main codebase.
|
||||
|
||||
Ten LLM agents (Claude Code) were then spawned in parallel, one per
|
||||
VPN. Each agent was instructed to read the full source tree and
|
||||
produce an \texttt{overview.md} file documenting the following
|
||||
aspects:
|
||||
|
||||
\begin{itemize}
|
||||
\item Wire protocol and message framing
|
||||
\item Encryption scheme and key exchange
|
||||
\item Packet handling and performance
|
||||
\item NAT traversal mechanism
|
||||
\item Local routing and peer discovery
|
||||
\item Security features and access control
|
||||
\item Resilience / Central Point of Failure
|
||||
\end{itemize}
|
||||
|
||||
Every claim in the generated overview was required to reference the
|
||||
specific file and line range in the repository that supports it,
|
||||
enabling direct verification.
|
||||
|
||||
\subsection{Manual Verification}
|
||||
|
||||
The LLM-generated overviews served as a navigational aid rather than
|
||||
a trusted source. The most important code paths identified in each
|
||||
overview were manually read and verified against the actual source
|
||||
code, correcting inaccuracies and deepening the analysis where the
|
||||
automated summaries remained superficial.
|
||||
|
||||
\subsection{Feature Matrix and Maintainer Review}
|
||||
|
||||
The findings from both the automated and manual analysis were
|
||||
consolidated into a comprehensive feature matrix cataloguing 131
|
||||
features across all ten VPN implementations. The matrix covers
|
||||
protocol characteristics, cryptographic primitives, NAT traversal
|
||||
strategies, routing behavior, and security properties.
|
||||
|
||||
The completed feature matrix was published and sent to the respective
|
||||
VPN maintainers for review. Maintainer feedback was incorporated as
|
||||
corrections and clarifications, improving the accuracy of the final
|
||||
classification.
|
||||
|
||||
\section{Reproducibility}
|
||||
|
||||
Reproducibility is ensured at every layer of the experimental stack.
|
||||
|
||||
Reference in New Issue
Block a user