From ec32331bae03372978544e4967eeff6de2d8a2ff Mon Sep 17 00:00:00 2001 From: Qubasa Date: Sat, 28 Feb 2026 19:36:17 +0100 Subject: [PATCH] added empty chapter for structure --- Chapters/Conclusion.tex | 4 ++ Chapters/Discussion.tex | 4 ++ Chapters/Introduction.tex | 137 +++++++++++++++---------------------- Chapters/Methodology.tex | 71 +++++++++++++++---- Chapters/Preliminaries.tex | 11 +-- Chapters/Results.tex | 98 ++++++++++++++++++++++++++ main.tex | 4 ++ 7 files changed, 233 insertions(+), 96 deletions(-) create mode 100644 Chapters/Conclusion.tex create mode 100644 Chapters/Discussion.tex create mode 100644 Chapters/Results.tex diff --git a/Chapters/Conclusion.tex b/Chapters/Conclusion.tex new file mode 100644 index 0000000..682f619 --- /dev/null +++ b/Chapters/Conclusion.tex @@ -0,0 +1,4 @@ + +\chapter{Conclusion} % Main chapter title + +\label{Conclusion} \ No newline at end of file diff --git a/Chapters/Discussion.tex b/Chapters/Discussion.tex new file mode 100644 index 0000000..b520f91 --- /dev/null +++ b/Chapters/Discussion.tex @@ -0,0 +1,4 @@ + +\chapter{Discussion} % Main chapter title + +\label{Discussion} \ No newline at end of file diff --git a/Chapters/Introduction.tex b/Chapters/Introduction.tex index 002cff8..38cda50 100644 --- a/Chapters/Introduction.tex +++ b/Chapters/Introduction.tex @@ -97,25 +97,64 @@ rather than systematic evaluation. This observation revealed a clear need for rigorous, evidence-based comparison of peer-to-peer overlay VPN implementations. -Existing research on this topic remains sparse. -One notable work from 2024, ``Full-mesh VPN performance evaluation -for a secure edge-cloud continuum'' \cite{kjorveziroski_full-mesh_2024}, -benchmarks a subset of mesh VPNs but focuses primarily -on solutions with a central point of failure. -In contrast, this thesis evaluates more widely adopted mesh VPNs -with an emphasis on fully decentralized architectures. -Furthermore, that study relied exclusively on iperf3 for performance -measurement, -whereas our benchmark suite includes real-world workloads -to better reflect practical usage patterns. +\subsection{Related Work} -A further motivation was to create a fully automated benchmarking framework -capable of generating a public leaderboard, -similar in spirit to the js-framework-benchmark -(see Figure~\ref{fig:js-framework-benchmark}). -By providing an accessible web interface with regularly updated results, -we hope to encourage P2P VPN developers to optimize their implementations -in pursuit of top rankings. +Existing research offers only partial coverage of this space. +Lackorzynski et al.\ \cite{lackorzynski_comparative_2019} benchmark +OpenVPN, IPSec, Tinc, Freelan, MACsec, and WireGuard in the context +of industrial communication systems, measuring point-to-point +throughput, latency, and CPU overhead. Their work does not address +overlay network behavior such as NAT traversal or dynamic peer discovery. +The most closely related study by Kjorveziroski et al.\ +\cite{kjorveziroski_full-mesh_2024} evaluates full-mesh VPN solutions +for distributed systems, analyzing throughput, reliability under packet +loss, and relay behavior for VPNs including ZeroTier. However, it +focuses primarily on solutions with a central point of failure and +limits its workloads to synthetic iperf3 tests. This thesis extends +that foundation by evaluating a broader set of VPN implementations +with emphasis on fully decentralized architectures, exercising them +under real-world workloads such as video streaming and package +downloads, applying multiple network impairment profiles, and +providing a fully reproducible experimental framework built on +Nix, NixOS, and Clan. + +Beyond filling this research gap, a further goal was to create a fully +automated benchmarking framework capable of generating a public +leaderboard, similar in spirit to the js-framework-benchmark +(see Figure~\ref{fig:js-framework-benchmark}). By providing an +accessible web interface with regularly updated results, we hope to +encourage P2P VPN developers to optimize their implementations in +pursuit of top rankings. + +\section{Research Contribution} + +This thesis makes the following contributions: + +\begin{enumerate} + \item A comprehensive benchmark of ten peer-to-peer VPN + implementations across seven workloads. Including real-world + video streaming and package downloads; and four network + impairment profiles, producing over 300 unique measurements. + \item A source code analysis of all ten VPN implementations, + combining manual code review with LLM-assisted analysis, + followed by verification through direct engagement with the + respective maintainers on GitHub. + \item A fully reproducible experimental framework built on + Nix, NixOS, and the Clan deployment system, with pinned + dependencies, declarative system configuration, and + deterministic cryptographic material generation, enabling + independent replication of all results. + \item A performance analysis demonstrating that Tailscale + outperforms the Linux kernel's default networking stack under + degraded conditions, and that kernel parameter tuning; Reno + congestion control in place of CUBIC, with RACK + disabled; yields measurable throughput improvements. + \item The discovery of several security vulnerabilities across + the evaluated VPN implementations. + \item An automated benchmarking framework designed for public + leaderboard generation, intended to encourage ongoing + optimization by VPN developers. +\end{enumerate} \begin{figure}[H] \centering @@ -190,65 +229,3 @@ in pursuit of top rankings. \label{fig:vision-stages} \end{figure} -\section{Research Contribution} - -This thesis makes the following contributions: - -\begin{enumerate} - \item A comprehensive benchmark of ten peer-to-peer VPN - implementations across seven workloads. Including real-world - video streaming and package downloads; and four network - impairment profiles, producing over 300 unique measurements. - \item A source code analysis of all ten VPN implementations, - combining manual code review with LLM-assisted analysis, - followed by verification through direct engagement with the - respective maintainers on GitHub. - \item A fully reproducible experimental framework built on - Nix, NixOS, and the Clan deployment system, with pinned - dependencies, declarative system configuration, and - deterministic cryptographic material generation, enabling - independent replication of all results. - \item A performance analysis demonstrating that Tailscale - outperforms the Linux kernel's default networking stack under - degraded conditions, and that kernel parameter tuning; Reno - congestion control in place of CUBIC, with RACK - disabled; yields measurable throughput improvements. - \item The discovery of several security vulnerabilities across - the evaluated VPN implementations. - \item An automated benchmarking framework designed for public - leaderboard generation, intended to encourage ongoing - optimization by VPN developers. -\end{enumerate} - -\section{Related Work} - -\subsection{A Comparative Study on Virtual Private Networks} - -Lackorzynski et al.\ \cite{lackorzynski_comparative_2019} evaluate -VPN protocols in the context of industrial communication systems (Industry 4.0), -benchmarking OpenVPN, IPSec, Tinc, Freelan, MACsec, and WireGuard. -Their analysis focuses on point-to-point protocol performance; throughput, -latency, and CPU overhead; rather than overlay network behavior. -In contrast, this thesis evaluates VPNs that provide a full data plane -with peer-to-peer connectivity, NAT traversal, and dynamic peer discovery. - -\subsection{Full-Mesh VPN Performance Evaluation} - -Kjorveziroski et al.\ \cite{kjorveziroski_full-mesh_2024} provide a -comprehensive evaluation of full-mesh VPN solutions for distributed -systems. Their benchmarks analyze throughput, reliability under packet -loss, and relay behavior for VPNs including ZeroTier. - -This thesis extends their work in several ways: -\begin{itemize} - \item Broader VPN selection with emphasis on fully decentralized - architectures - \item Real-world workloads (video streaming, package downloads) - beyond synthetic iperf3 tests - \item Multiple impairment profiles to characterize behavior under - varying network conditions - \item Fully reproducible experimental framework via Nix/NixOS/Clan -\end{itemize} - -\subsection{Performance Evaluation of TCP over QUIC Tunnels} -TODO \cite{guo_implementation_2025} diff --git a/Chapters/Methodology.tex b/Chapters/Methodology.tex index 8f6bf9b..e1f4841 100755 --- a/Chapters/Methodology.tex +++ b/Chapters/Methodology.tex @@ -510,17 +510,64 @@ benchmark directories. This prevents cross-contamination between tests. \subsection{Data Provenance} -Every test result includes metadata recording: +Results are organized in the four-level directory hierarchy shown in +Figure~\ref{fig:result-tree}. Each VPN directory stores a +\texttt{layout.json} capturing the machine topology used for that run. +Each impairment profile directory records the exact \texttt{tc} +parameters in \texttt{tc\_settings.json} and per-phase durations in +\texttt{timing\_breakdown.json}. Individual benchmark results are +stored in one subdirectory per machine pair. -\begin{itemize} - \item Wall-clock duration - \item Number of attempts (1 = first try succeeded) - \item VPN restart attempts and duration - \item Connectivity wait duration - \item Source and target machine names - \item Service logs (on failure) -\end{itemize} +\begin{figure}[ht] + \centering + \begin{forest} + for tree={ + font=\ttfamily\small, + grow'=0, + folder, + s sep=2pt, + inner xsep=3pt, + inner ysep=2pt, + } + [date/ + [vpn/ + [layout.json] + [profile/ + [tc\_settings.json] + [timing\_breakdown.json] + [parallel\_tcp\_iperf3.json] + [\textnormal{\textit{\{pos\}\_\{peer\}}}/ + [ping.json] + [tcp\_iperf3.json] + [udp\_iperf3.json] + [qperf.json] + [rist\_stream.json] + [nix\_cache.json] + [connection\_timings.json] + ] + ] + ] + [General/ + [hardware.json] + [comparison/ + [cross\_profile\_*.json] + [profile/ + [benchmark\_stats.json] + [per-benchmark .json files] + ] + ] + ] + ] + \end{forest} + \caption{Directory hierarchy of benchmark results. Each run produces + per-VPN and per-profile directories alongside a \texttt{General/} + directory with cross-VPN comparison data.} + \label{fig:result-tree} +\end{figure} -Results are organized hierarchically by VPN, TC profile, and machine -pair. Each profile directory contains a \texttt{tc\_settings.json} -snapshot of the exact impairment parameters applied. +Every benchmark result file uses a uniform JSON envelope with a +\texttt{status} field, a \texttt{data} object holding the +test-specific payload, and a \texttt{meta} object recording +wall-clock duration, number of attempts, VPN restart count and +duration, connectivity wait time, source and target machine names, +and on failure, the relevant service logs. diff --git a/Chapters/Preliminaries.tex b/Chapters/Preliminaries.tex index 514e820..ae466a7 100644 --- a/Chapters/Preliminaries.tex +++ b/Chapters/Preliminaries.tex @@ -2,7 +2,7 @@ \label{Preliminaries} -\subsubsection{Nix: A Safe and Policy-Free System for Software Deployment} +\subsection{Nix: A Safe and Policy-Free System for Software Deployment} Nix addresses significant issues in software deployment by utilizing cryptographic hashes to ensure unique paths for component instances @@ -11,7 +11,7 @@ multiple versions, atomic upgrades, and safe garbage collection make Nix a flexible deployment system. This work uses Nix to ensure that all VPN builds and system configurations are deterministic. -\subsubsection{NixOS: A Purely Functional Linux Distribution} +\subsection{NixOS: A Purely Functional Linux Distribution} NixOS extends Nix principles to Linux system configuration \cite{dolstra_nixos_2008}. System configurations are reproducible and @@ -19,7 +19,7 @@ isolated from stateful interactions typical in imperative package management. This property is essential for ensuring identical test environments across benchmark runs. -\subsubsection{UDP NAT and Firewall Puncturing in the Wild} +\subsection{UDP NAT and Firewall Puncturing in the Wild} Halkes and Pouwelse~\cite{halkes_udp_2011} measure UDP hole punching efficacy on a live P2P network using the Tribler BitTorrent client. @@ -37,5 +37,8 @@ the 80\% success rate sets a baseline expectation, while the 55-second timeout informs analysis of each implementation's keep-alive behavior during source code review. -\subsubsection{An Overview of Packet Reordering in TCP} +\subsection{An Overview of Packet Reordering in TCP} TODO \cite{leung_overview_2007} + +\subsection{Performance Evaluation of TCP over QUIC Tunnels} +TODO \cite{guo_implementation_2025} diff --git a/Chapters/Results.tex b/Chapters/Results.tex new file mode 100644 index 0000000..9e8ae7a --- /dev/null +++ b/Chapters/Results.tex @@ -0,0 +1,98 @@ +% Chapter Template + +\chapter{Results} % Main chapter title + +\label{Results} + +This chapter presents the results of the benchmark suite across all +ten VPN implementations and the internal baseline. Results are +organized by first establishing overhead under ideal conditions, then +examining how each VPN performs under increasing network impairment. +The chapter concludes with findings from the source code analysis. + +\section{Baseline Performance} + +Under the baseline impairment profile (no added latency, loss, or +reordering), the overhead introduced by each VPN relative to the +internal (no VPN) baseline and WireGuard can be measured in isolation. + +\subsection{Throughput Overhead} + +% TCP and UDP iperf3 results at baseline profile. +% Compare all VPNs against internal and WireGuard. +% Consider a bar chart or grouped table. + +\subsection{Latency Overhead} + +% Ping RTT results at baseline profile. +% Show min/avg/max/mdev per VPN. + +\section{Impact of Network Impairment} + +This section examines how each VPN responds to the Low, Medium, and +High impairment profiles defined in Chapter~\ref{Methodology}. + +\subsection{Ping} + +% RTT and packet loss across impairment profiles. + +\subsection{TCP Throughput} + +% TCP iperf3: throughput, retransmits, congestion window. + +\subsection{UDP Throughput} + +% UDP iperf3: throughput, jitter, packet loss. + +\subsection{Parallel TCP} + +% Parallel iperf3: throughput under contention (A->B, B->C, C->A). + +\subsection{QUIC Performance} + +% qperf: bandwidth, TTFB, connection establishment time. + +\subsection{Video Streaming} + +% RIST: bitrate, dropped frames, packets recovered, quality score. + +\subsection{Application-Level Download} + +% Nix cache: download duration for Firefox package. + +\section{Tailscale Under Degraded Conditions} + +% The central finding: Tailscale outperforming the raw Linux +% networking stack under impairment. + +\subsection{Observed Anomaly} + +% Present the data showing Tailscale exceeding internal baseline +% throughput under Medium/High impairment. + +\subsection{Congestion Control Analysis} + +% Reno vs CUBIC, RACK disabled to avoid spurious retransmits +% under reordering. + +\subsection{Tuned Kernel Parameters} + +% Re-run results with tuned buffer sizes and congestion control +% on the internal baseline, showing the gap closes. + +\section{Source Code Analysis} + +\subsection{Feature Matrix Overview} + +% Summary of the 131-feature matrix across all ten VPNs. +% Highlight key architectural differences that explain +% performance results. + +\subsection{Security Vulnerabilities} + +% Vulnerabilities discovered during source code review. + +\section{Summary of Findings} + +% Brief summary table or ranking of VPNs by key metrics. +% Save deeper interpretation for a Discussion chapter. \ No newline at end of file diff --git a/main.tex b/main.tex index 13f0841..86e09f8 100644 --- a/main.tex +++ b/main.tex @@ -61,6 +61,7 @@ \usepackage{subcaption} % For subfigures \usepackage{tikz} \usetikzlibrary{shapes.geometric} +\usepackage[edges]{forest} \usepackage[backend=bibtex,style=numeric,natbib=true]{biblatex} % % Use the bibtex backend with the authoryear citation style (which @@ -340,6 +341,9 @@ and Management}} % Your department's name and URL, this is used in \include{Chapters/Introduction} \include{Chapters/Preliminaries} \include{Chapters/Methodology} +\include{Chapters/Results} +\include{Chapters/Discussion} +\include{Chapters/Conclusion} %\include{Chapters/Chapter1} %\include{Chapters/Chapter2}