added empty chapter for structure
This commit is contained in:
4
Chapters/Conclusion.tex
Normal file
4
Chapters/Conclusion.tex
Normal file
@@ -0,0 +1,4 @@
|
||||
|
||||
\chapter{Conclusion} % Main chapter title
|
||||
|
||||
\label{Conclusion}
|
||||
4
Chapters/Discussion.tex
Normal file
4
Chapters/Discussion.tex
Normal file
@@ -0,0 +1,4 @@
|
||||
|
||||
\chapter{Discussion} % Main chapter title
|
||||
|
||||
\label{Discussion}
|
||||
@@ -97,25 +97,64 @@ rather than systematic evaluation.
|
||||
This observation revealed a clear need for rigorous,
|
||||
evidence-based comparison of peer-to-peer overlay VPN implementations.
|
||||
|
||||
Existing research on this topic remains sparse.
|
||||
One notable work from 2024, ``Full-mesh VPN performance evaluation
|
||||
for a secure edge-cloud continuum'' \cite{kjorveziroski_full-mesh_2024},
|
||||
benchmarks a subset of mesh VPNs but focuses primarily
|
||||
on solutions with a central point of failure.
|
||||
In contrast, this thesis evaluates more widely adopted mesh VPNs
|
||||
with an emphasis on fully decentralized architectures.
|
||||
Furthermore, that study relied exclusively on iperf3 for performance
|
||||
measurement,
|
||||
whereas our benchmark suite includes real-world workloads
|
||||
to better reflect practical usage patterns.
|
||||
\subsection{Related Work}
|
||||
|
||||
A further motivation was to create a fully automated benchmarking framework
|
||||
capable of generating a public leaderboard,
|
||||
similar in spirit to the js-framework-benchmark
|
||||
(see Figure~\ref{fig:js-framework-benchmark}).
|
||||
By providing an accessible web interface with regularly updated results,
|
||||
we hope to encourage P2P VPN developers to optimize their implementations
|
||||
in pursuit of top rankings.
|
||||
Existing research offers only partial coverage of this space.
|
||||
Lackorzynski et al.\ \cite{lackorzynski_comparative_2019} benchmark
|
||||
OpenVPN, IPSec, Tinc, Freelan, MACsec, and WireGuard in the context
|
||||
of industrial communication systems, measuring point-to-point
|
||||
throughput, latency, and CPU overhead. Their work does not address
|
||||
overlay network behavior such as NAT traversal or dynamic peer discovery.
|
||||
The most closely related study by Kjorveziroski et al.\
|
||||
\cite{kjorveziroski_full-mesh_2024} evaluates full-mesh VPN solutions
|
||||
for distributed systems, analyzing throughput, reliability under packet
|
||||
loss, and relay behavior for VPNs including ZeroTier. However, it
|
||||
focuses primarily on solutions with a central point of failure and
|
||||
limits its workloads to synthetic iperf3 tests. This thesis extends
|
||||
that foundation by evaluating a broader set of VPN implementations
|
||||
with emphasis on fully decentralized architectures, exercising them
|
||||
under real-world workloads such as video streaming and package
|
||||
downloads, applying multiple network impairment profiles, and
|
||||
providing a fully reproducible experimental framework built on
|
||||
Nix, NixOS, and Clan.
|
||||
|
||||
Beyond filling this research gap, a further goal was to create a fully
|
||||
automated benchmarking framework capable of generating a public
|
||||
leaderboard, similar in spirit to the js-framework-benchmark
|
||||
(see Figure~\ref{fig:js-framework-benchmark}). By providing an
|
||||
accessible web interface with regularly updated results, we hope to
|
||||
encourage P2P VPN developers to optimize their implementations in
|
||||
pursuit of top rankings.
|
||||
|
||||
\section{Research Contribution}
|
||||
|
||||
This thesis makes the following contributions:
|
||||
|
||||
\begin{enumerate}
|
||||
\item A comprehensive benchmark of ten peer-to-peer VPN
|
||||
implementations across seven workloads. Including real-world
|
||||
video streaming and package downloads; and four network
|
||||
impairment profiles, producing over 300 unique measurements.
|
||||
\item A source code analysis of all ten VPN implementations,
|
||||
combining manual code review with LLM-assisted analysis,
|
||||
followed by verification through direct engagement with the
|
||||
respective maintainers on GitHub.
|
||||
\item A fully reproducible experimental framework built on
|
||||
Nix, NixOS, and the Clan deployment system, with pinned
|
||||
dependencies, declarative system configuration, and
|
||||
deterministic cryptographic material generation, enabling
|
||||
independent replication of all results.
|
||||
\item A performance analysis demonstrating that Tailscale
|
||||
outperforms the Linux kernel's default networking stack under
|
||||
degraded conditions, and that kernel parameter tuning; Reno
|
||||
congestion control in place of CUBIC, with RACK
|
||||
disabled; yields measurable throughput improvements.
|
||||
\item The discovery of several security vulnerabilities across
|
||||
the evaluated VPN implementations.
|
||||
\item An automated benchmarking framework designed for public
|
||||
leaderboard generation, intended to encourage ongoing
|
||||
optimization by VPN developers.
|
||||
\end{enumerate}
|
||||
|
||||
\begin{figure}[H]
|
||||
\centering
|
||||
@@ -190,65 +229,3 @@ in pursuit of top rankings.
|
||||
\label{fig:vision-stages}
|
||||
\end{figure}
|
||||
|
||||
\section{Research Contribution}
|
||||
|
||||
This thesis makes the following contributions:
|
||||
|
||||
\begin{enumerate}
|
||||
\item A comprehensive benchmark of ten peer-to-peer VPN
|
||||
implementations across seven workloads. Including real-world
|
||||
video streaming and package downloads; and four network
|
||||
impairment profiles, producing over 300 unique measurements.
|
||||
\item A source code analysis of all ten VPN implementations,
|
||||
combining manual code review with LLM-assisted analysis,
|
||||
followed by verification through direct engagement with the
|
||||
respective maintainers on GitHub.
|
||||
\item A fully reproducible experimental framework built on
|
||||
Nix, NixOS, and the Clan deployment system, with pinned
|
||||
dependencies, declarative system configuration, and
|
||||
deterministic cryptographic material generation, enabling
|
||||
independent replication of all results.
|
||||
\item A performance analysis demonstrating that Tailscale
|
||||
outperforms the Linux kernel's default networking stack under
|
||||
degraded conditions, and that kernel parameter tuning; Reno
|
||||
congestion control in place of CUBIC, with RACK
|
||||
disabled; yields measurable throughput improvements.
|
||||
\item The discovery of several security vulnerabilities across
|
||||
the evaluated VPN implementations.
|
||||
\item An automated benchmarking framework designed for public
|
||||
leaderboard generation, intended to encourage ongoing
|
||||
optimization by VPN developers.
|
||||
\end{enumerate}
|
||||
|
||||
\section{Related Work}
|
||||
|
||||
\subsection{A Comparative Study on Virtual Private Networks}
|
||||
|
||||
Lackorzynski et al.\ \cite{lackorzynski_comparative_2019} evaluate
|
||||
VPN protocols in the context of industrial communication systems (Industry 4.0),
|
||||
benchmarking OpenVPN, IPSec, Tinc, Freelan, MACsec, and WireGuard.
|
||||
Their analysis focuses on point-to-point protocol performance; throughput,
|
||||
latency, and CPU overhead; rather than overlay network behavior.
|
||||
In contrast, this thesis evaluates VPNs that provide a full data plane
|
||||
with peer-to-peer connectivity, NAT traversal, and dynamic peer discovery.
|
||||
|
||||
\subsection{Full-Mesh VPN Performance Evaluation}
|
||||
|
||||
Kjorveziroski et al.\ \cite{kjorveziroski_full-mesh_2024} provide a
|
||||
comprehensive evaluation of full-mesh VPN solutions for distributed
|
||||
systems. Their benchmarks analyze throughput, reliability under packet
|
||||
loss, and relay behavior for VPNs including ZeroTier.
|
||||
|
||||
This thesis extends their work in several ways:
|
||||
\begin{itemize}
|
||||
\item Broader VPN selection with emphasis on fully decentralized
|
||||
architectures
|
||||
\item Real-world workloads (video streaming, package downloads)
|
||||
beyond synthetic iperf3 tests
|
||||
\item Multiple impairment profiles to characterize behavior under
|
||||
varying network conditions
|
||||
\item Fully reproducible experimental framework via Nix/NixOS/Clan
|
||||
\end{itemize}
|
||||
|
||||
\subsection{Performance Evaluation of TCP over QUIC Tunnels}
|
||||
TODO \cite{guo_implementation_2025}
|
||||
|
||||
@@ -510,17 +510,64 @@ benchmark directories. This prevents cross-contamination between tests.
|
||||
|
||||
\subsection{Data Provenance}
|
||||
|
||||
Every test result includes metadata recording:
|
||||
Results are organized in the four-level directory hierarchy shown in
|
||||
Figure~\ref{fig:result-tree}. Each VPN directory stores a
|
||||
\texttt{layout.json} capturing the machine topology used for that run.
|
||||
Each impairment profile directory records the exact \texttt{tc}
|
||||
parameters in \texttt{tc\_settings.json} and per-phase durations in
|
||||
\texttt{timing\_breakdown.json}. Individual benchmark results are
|
||||
stored in one subdirectory per machine pair.
|
||||
|
||||
\begin{itemize}
|
||||
\item Wall-clock duration
|
||||
\item Number of attempts (1 = first try succeeded)
|
||||
\item VPN restart attempts and duration
|
||||
\item Connectivity wait duration
|
||||
\item Source and target machine names
|
||||
\item Service logs (on failure)
|
||||
\end{itemize}
|
||||
\begin{figure}[ht]
|
||||
\centering
|
||||
\begin{forest}
|
||||
for tree={
|
||||
font=\ttfamily\small,
|
||||
grow'=0,
|
||||
folder,
|
||||
s sep=2pt,
|
||||
inner xsep=3pt,
|
||||
inner ysep=2pt,
|
||||
}
|
||||
[date/
|
||||
[vpn/
|
||||
[layout.json]
|
||||
[profile/
|
||||
[tc\_settings.json]
|
||||
[timing\_breakdown.json]
|
||||
[parallel\_tcp\_iperf3.json]
|
||||
[\textnormal{\textit{\{pos\}\_\{peer\}}}/
|
||||
[ping.json]
|
||||
[tcp\_iperf3.json]
|
||||
[udp\_iperf3.json]
|
||||
[qperf.json]
|
||||
[rist\_stream.json]
|
||||
[nix\_cache.json]
|
||||
[connection\_timings.json]
|
||||
]
|
||||
]
|
||||
]
|
||||
[General/
|
||||
[hardware.json]
|
||||
[comparison/
|
||||
[cross\_profile\_*.json]
|
||||
[profile/
|
||||
[benchmark\_stats.json]
|
||||
[per-benchmark .json files]
|
||||
]
|
||||
]
|
||||
]
|
||||
]
|
||||
\end{forest}
|
||||
\caption{Directory hierarchy of benchmark results. Each run produces
|
||||
per-VPN and per-profile directories alongside a \texttt{General/}
|
||||
directory with cross-VPN comparison data.}
|
||||
\label{fig:result-tree}
|
||||
\end{figure}
|
||||
|
||||
Results are organized hierarchically by VPN, TC profile, and machine
|
||||
pair. Each profile directory contains a \texttt{tc\_settings.json}
|
||||
snapshot of the exact impairment parameters applied.
|
||||
Every benchmark result file uses a uniform JSON envelope with a
|
||||
\texttt{status} field, a \texttt{data} object holding the
|
||||
test-specific payload, and a \texttt{meta} object recording
|
||||
wall-clock duration, number of attempts, VPN restart count and
|
||||
duration, connectivity wait time, source and target machine names,
|
||||
and on failure, the relevant service logs.
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
\label{Preliminaries}
|
||||
|
||||
\subsubsection{Nix: A Safe and Policy-Free System for Software Deployment}
|
||||
\subsection{Nix: A Safe and Policy-Free System for Software Deployment}
|
||||
|
||||
Nix addresses significant issues in software deployment by utilizing
|
||||
cryptographic hashes to ensure unique paths for component instances
|
||||
@@ -11,7 +11,7 @@ multiple versions, atomic upgrades, and safe garbage collection make
|
||||
Nix a flexible deployment system. This work uses Nix to ensure that
|
||||
all VPN builds and system configurations are deterministic.
|
||||
|
||||
\subsubsection{NixOS: A Purely Functional Linux Distribution}
|
||||
\subsection{NixOS: A Purely Functional Linux Distribution}
|
||||
|
||||
NixOS extends Nix principles to Linux system configuration
|
||||
\cite{dolstra_nixos_2008}. System configurations are reproducible and
|
||||
@@ -19,7 +19,7 @@ isolated from stateful interactions typical in imperative package
|
||||
management. This property is essential for ensuring identical test
|
||||
environments across benchmark runs.
|
||||
|
||||
\subsubsection{UDP NAT and Firewall Puncturing in the Wild}
|
||||
\subsection{UDP NAT and Firewall Puncturing in the Wild}
|
||||
|
||||
Halkes and Pouwelse~\cite{halkes_udp_2011} measure UDP hole punching
|
||||
efficacy on a live P2P network using the Tribler BitTorrent client.
|
||||
@@ -37,5 +37,8 @@ the 80\% success rate sets a baseline expectation, while the 55-second
|
||||
timeout informs analysis of each implementation's keep-alive behavior
|
||||
during source code review.
|
||||
|
||||
\subsubsection{An Overview of Packet Reordering in TCP}
|
||||
\subsection{An Overview of Packet Reordering in TCP}
|
||||
TODO \cite{leung_overview_2007}
|
||||
|
||||
\subsection{Performance Evaluation of TCP over QUIC Tunnels}
|
||||
TODO \cite{guo_implementation_2025}
|
||||
|
||||
98
Chapters/Results.tex
Normal file
98
Chapters/Results.tex
Normal file
@@ -0,0 +1,98 @@
|
||||
% Chapter Template
|
||||
|
||||
\chapter{Results} % Main chapter title
|
||||
|
||||
\label{Results}
|
||||
|
||||
This chapter presents the results of the benchmark suite across all
|
||||
ten VPN implementations and the internal baseline. Results are
|
||||
organized by first establishing overhead under ideal conditions, then
|
||||
examining how each VPN performs under increasing network impairment.
|
||||
The chapter concludes with findings from the source code analysis.
|
||||
|
||||
\section{Baseline Performance}
|
||||
|
||||
Under the baseline impairment profile (no added latency, loss, or
|
||||
reordering), the overhead introduced by each VPN relative to the
|
||||
internal (no VPN) baseline and WireGuard can be measured in isolation.
|
||||
|
||||
\subsection{Throughput Overhead}
|
||||
|
||||
% TCP and UDP iperf3 results at baseline profile.
|
||||
% Compare all VPNs against internal and WireGuard.
|
||||
% Consider a bar chart or grouped table.
|
||||
|
||||
\subsection{Latency Overhead}
|
||||
|
||||
% Ping RTT results at baseline profile.
|
||||
% Show min/avg/max/mdev per VPN.
|
||||
|
||||
\section{Impact of Network Impairment}
|
||||
|
||||
This section examines how each VPN responds to the Low, Medium, and
|
||||
High impairment profiles defined in Chapter~\ref{Methodology}.
|
||||
|
||||
\subsection{Ping}
|
||||
|
||||
% RTT and packet loss across impairment profiles.
|
||||
|
||||
\subsection{TCP Throughput}
|
||||
|
||||
% TCP iperf3: throughput, retransmits, congestion window.
|
||||
|
||||
\subsection{UDP Throughput}
|
||||
|
||||
% UDP iperf3: throughput, jitter, packet loss.
|
||||
|
||||
\subsection{Parallel TCP}
|
||||
|
||||
% Parallel iperf3: throughput under contention (A->B, B->C, C->A).
|
||||
|
||||
\subsection{QUIC Performance}
|
||||
|
||||
% qperf: bandwidth, TTFB, connection establishment time.
|
||||
|
||||
\subsection{Video Streaming}
|
||||
|
||||
% RIST: bitrate, dropped frames, packets recovered, quality score.
|
||||
|
||||
\subsection{Application-Level Download}
|
||||
|
||||
% Nix cache: download duration for Firefox package.
|
||||
|
||||
\section{Tailscale Under Degraded Conditions}
|
||||
|
||||
% The central finding: Tailscale outperforming the raw Linux
|
||||
% networking stack under impairment.
|
||||
|
||||
\subsection{Observed Anomaly}
|
||||
|
||||
% Present the data showing Tailscale exceeding internal baseline
|
||||
% throughput under Medium/High impairment.
|
||||
|
||||
\subsection{Congestion Control Analysis}
|
||||
|
||||
% Reno vs CUBIC, RACK disabled to avoid spurious retransmits
|
||||
% under reordering.
|
||||
|
||||
\subsection{Tuned Kernel Parameters}
|
||||
|
||||
% Re-run results with tuned buffer sizes and congestion control
|
||||
% on the internal baseline, showing the gap closes.
|
||||
|
||||
\section{Source Code Analysis}
|
||||
|
||||
\subsection{Feature Matrix Overview}
|
||||
|
||||
% Summary of the 131-feature matrix across all ten VPNs.
|
||||
% Highlight key architectural differences that explain
|
||||
% performance results.
|
||||
|
||||
\subsection{Security Vulnerabilities}
|
||||
|
||||
% Vulnerabilities discovered during source code review.
|
||||
|
||||
\section{Summary of Findings}
|
||||
|
||||
% Brief summary table or ranking of VPNs by key metrics.
|
||||
% Save deeper interpretation for a Discussion chapter.
|
||||
Reference in New Issue
Block a user