From c08df6b8858bb50805cd0057203752542f58a650 Mon Sep 17 00:00:00 2001 From: Qubasa Date: Tue, 24 Feb 2026 19:03:31 +0100 Subject: [PATCH] added source code review to methodology section --- Chapters/Introduction.tex | 56 ++++++++++++++++++++++++++----- Chapters/Methodology.tex | 67 +++++++++++++++++++++++++++++++++++--- Chapters/Preliminaries.tex | 1 - 3 files changed, 110 insertions(+), 14 deletions(-) diff --git a/Chapters/Introduction.tex b/Chapters/Introduction.tex index 414628a..002cff8 100644 --- a/Chapters/Introduction.tex +++ b/Chapters/Introduction.tex @@ -2,9 +2,23 @@ \label{Introduction} -TODO +Peer-to-peer overlay VPNs promise to restore genuine decentralization +by enabling direct connectivity between nodes regardless of NAT or +firewall restrictions. Yet practitioners choosing among the growing +number of mesh VPN implementations must rely largely on anecdotal +evidence: systematic, reproducible comparisons under realistic +conditions are scarce. -\subsection{Motivation} +This thesis addresses that gap. We benchmark ten peer-to-peer VPN +implementations across seven workloads and four network impairment +profiles, yielding over 300 unique measurements. We complement these +performance benchmarks with a source code analysis of each +implementation, verified through direct engagement with the respective +maintainers. The entire experimental framework is built on Nix, NixOS, +and the Clan deployment system, making every result independently +reproducible. + +\section{Motivation} Peer-to-peer architectures promise censorship-resistant, fault-tolerant infrastructure by eliminating single points of failure @@ -176,13 +190,39 @@ in pursuit of top rankings. \label{fig:vision-stages} \end{figure} -\subsection{Research Contribution} +\section{Research Contribution} -TODO +This thesis makes the following contributions: -\subsection{Related Work} +\begin{enumerate} + \item A comprehensive benchmark of ten peer-to-peer VPN + implementations across seven workloads. Including real-world + video streaming and package downloads; and four network + impairment profiles, producing over 300 unique measurements. + \item A source code analysis of all ten VPN implementations, + combining manual code review with LLM-assisted analysis, + followed by verification through direct engagement with the + respective maintainers on GitHub. + \item A fully reproducible experimental framework built on + Nix, NixOS, and the Clan deployment system, with pinned + dependencies, declarative system configuration, and + deterministic cryptographic material generation, enabling + independent replication of all results. + \item A performance analysis demonstrating that Tailscale + outperforms the Linux kernel's default networking stack under + degraded conditions, and that kernel parameter tuning; Reno + congestion control in place of CUBIC, with RACK + disabled; yields measurable throughput improvements. + \item The discovery of several security vulnerabilities across + the evaluated VPN implementations. + \item An automated benchmarking framework designed for public + leaderboard generation, intended to encourage ongoing + optimization by VPN developers. +\end{enumerate} -\subsubsection{A Comparative Study on Virtual Private Networks} +\section{Related Work} + +\subsection{A Comparative Study on Virtual Private Networks} Lackorzynski et al.\ \cite{lackorzynski_comparative_2019} evaluate VPN protocols in the context of industrial communication systems (Industry 4.0), @@ -192,7 +232,7 @@ latency, and CPU overhead; rather than overlay network behavior. In contrast, this thesis evaluates VPNs that provide a full data plane with peer-to-peer connectivity, NAT traversal, and dynamic peer discovery. -\subsubsection{Full-Mesh VPN Performance Evaluation} +\subsection{Full-Mesh VPN Performance Evaluation} Kjorveziroski et al.\ \cite{kjorveziroski_full-mesh_2024} provide a comprehensive evaluation of full-mesh VPN solutions for distributed @@ -210,5 +250,5 @@ This thesis extends their work in several ways: \item Fully reproducible experimental framework via Nix/NixOS/Clan \end{itemize} -\subsubsection{Performance Evaluation of TCP over QUIC Tunnels} +\subsection{Performance Evaluation of TCP over QUIC Tunnels} TODO \cite{guo_implementation_2025} diff --git a/Chapters/Methodology.tex b/Chapters/Methodology.tex index f8c77de..44c7cc4 100755 --- a/Chapters/Methodology.tex +++ b/Chapters/Methodology.tex @@ -4,11 +4,14 @@ \label{Methodology} -This chapter describes the methodology used to benchmark peer-to-peer -mesh VPN implementations. The experimental design prioritizes -reproducibility at every layer; from dependency management to network -conditions; enabling independent verification of results and -facilitating future comparative studies. +This chapter describes the methodology used to benchmark and analyze +peer-to-peer mesh VPN implementations. The evaluation combines +performance benchmarking under controlled network conditions with a +structured source code analysis of each implementation. The +benchmarking framework prioritizes reproducibility at every layer; +from pinned dependencies and declarative system configuration to +automated test orchestration; enabling independent verification of +results and facilitating future comparative studies. \section{Experimental Setup} @@ -277,6 +280,60 @@ confidence intervals, which is appropriate for benchmark data that may not follow a normal distribution. The nix-cache test (via hyperfine) additionally reports standard deviation. +\section{Source Code Analysis} + +To complement the performance benchmarks with architectural +understanding, a structured source code analysis was conducted for +all ten VPN implementations. The analysis followed three phases. + +\subsection{Repository Collection and LLM-Assisted Overview} + +The latest main branch of each VPN's git repository was cloned, +together with key dependencies that implement core functionality +outside the main repository. For example, Yggdrasil delegates its +routing and cryptographic operations to the Ironwood library, which +was analyzed alongside the main codebase. + +Ten LLM agents (Claude Code) were then spawned in parallel, one per +VPN. Each agent was instructed to read the full source tree and +produce an \texttt{overview.md} file documenting the following +aspects: + +\begin{itemize} + \item Wire protocol and message framing + \item Encryption scheme and key exchange + \item Packet handling and performance + \item NAT traversal mechanism + \item Local routing and peer discovery + \item Security features and access control + \item Resilience / Central Point of Failure +\end{itemize} + +Every claim in the generated overview was required to reference the +specific file and line range in the repository that supports it, +enabling direct verification. + +\subsection{Manual Verification} + +The LLM-generated overviews served as a navigational aid rather than +a trusted source. The most important code paths identified in each +overview were manually read and verified against the actual source +code, correcting inaccuracies and deepening the analysis where the +automated summaries remained superficial. + +\subsection{Feature Matrix and Maintainer Review} + +The findings from both the automated and manual analysis were +consolidated into a comprehensive feature matrix cataloguing 131 +features across all ten VPN implementations. The matrix covers +protocol characteristics, cryptographic primitives, NAT traversal +strategies, routing behavior, and security properties. + +The completed feature matrix was published and sent to the respective +VPN maintainers for review. Maintainer feedback was incorporated as +corrections and clarifications, improving the accuracy of the final +classification. + \section{Reproducibility} Reproducibility is ensured at every layer of the experimental stack. diff --git a/Chapters/Preliminaries.tex b/Chapters/Preliminaries.tex index f0441b2..514e820 100644 --- a/Chapters/Preliminaries.tex +++ b/Chapters/Preliminaries.tex @@ -19,7 +19,6 @@ isolated from stateful interactions typical in imperative package management. This property is essential for ensuring identical test environments across benchmark runs. - \subsubsection{UDP NAT and Firewall Puncturing in the Wild} Halkes and Pouwelse~\cite{halkes_udp_2011} measure UDP hole punching