15 KiB
Data-Mesher Specification
The Data-Mesher is an application that runs on every node in a peer-to-peer network. It functions as a database that eventually synchronizes across all nodes, using a special data structure called a CRDT (Conflict-Free Replicated Data Type) to resolve conflicts. What sets it apart from other databases is that even untrusted nodes can add data without compromising data added by others. Its primary use is for announcing hostnames and application settings, though it is flexible enough to support other use cases as well.
Below is a more detailed explanation of how it works:
What is a "dm-network"?
A dm-network is a group of hosts (computers/nodes) and settings that are grouped under a single key in a JSON file. This key is a public ed25519 key.
- Settings:
- The settings in a dm-network are protected by a digital signature to prevent tampering.
- The admin users are the only ones with the private key needed to change these settings.
- However, any node in the network can update the settings as long as they sign the change correctly (with the admin's private key). This ensures there isn’t just one "admin" node, allowing settings to be changed on any local node and then pushed to other nodes in the network.
Hosts in the dm-network
A dm-network also includes a hosts attribute that stores information about hostnames for DNS lookups.
- Each node can add its own hostnames to this list.
- Every node has its own pair of private and public keys to sign the hostnames it adds.
- In case multiple nodes try to use the same hostname, the one with the oldest signed entry (earliest timestamp) will be chosen.
Preventing clock manipulation:
To avoid cheating with time (for example, backdating a hostname entry), the dm-network relies on trusted timestamp attestation servers. We can use the Time-Stamp Protocol RFC3161 which allows sending a payload and getting a signed sha256 with a timestamp back. We can use the FreeTSA servers for that.
Or we could use the Opentimestamp specification and their free to use servers.
Data Synchronization Between Nodes
When two nodes communicate, they exchange their entire set of data.
After ensuring the data is merged, both nodes should end up with the same data.json file. The merge function ensures that both nodes arrive at the same result, no matter the order or timing.
currently the merge function is quite primite: for settings it checks if the signature is valid and afterwards the bigger last_update timestamp wins.
Handling Invalid or Missing Timestamps
-
If a hostname entry doesn't have a valid timestamp, it will still be shared with other nodes, but it won’t be active or used yet.
-
The entry stays inactive until it reaches a trusted dm-node that also acts as a timestamp attestation node (TSP), which will add a timestamp and sign the entry. From that point, the hostname becomes valid and can be used in the network. The timestamp attestation nodes are listed under the "settings" key in the JSON file, and only the dm-network's admin can modify this list.
-
This means there needs to be one or many trusted dm-nodes, which attest that the timestamps are correct. If one of the trusted dm-nodes is compromised, hostnames can be malicously claimed and redirected to attacker controlled nodes.
-
This also means that hostnames are not to be trusted, and instead a Certificate Authority should be used to verify the authenticity of endpoints.
-
This design has been chosen because:
- It enables having completely off-grid nodes, that are only inside the mesh VPN
- A node can start claiming it's hostname offline and just sync it into the VPN network once it's online
- No timing attacks: An attacker cannot pre-fetch timestamps to then use them to override hostnames
Security: Invalid Signatures
- If a hostname or timestamp has an invalid signature, it won’t be shared with other nodes.
- An alert will be triggered for further action.
- Additionally, hosts must go through a verification step to ensure that the IPs and ports are reachable and that the machine behind them holds the correct private keys that signed the entry. Before accepting a new host into the configuration, the node will attempt to contact the host’s IP and port, and a challenge-response protocol will be used to verify that it is the correct machine.
Below is an example data.json:
{
"22excOG1Q7hlNMyRPWz4eZNeTqsH18p0+r0KGPUqVR8=": {
"hosts": {
"7BZSfLVyoTc12xgpvMUSWGTNsjjP4iqv/JSgpYbHQC4=": {
"hostnames": {
"green": {
"hostname": "green"
}
},
"ip": "fdcc:c5da:5295:c853:d499:937c:31a2:1e86",
"last_seen": 1731199277,
"port": 7331,
"signature": "RUZEqQoH1E2TuuB0rcQeaEuyxLTB70xgcj2VvRpvDwRtxvbaXegErJ7ei5obS46x3ApjgVP+3Di7OTXBSxqUCnsiaG9zdG5hbWVzIjogeyJncmVlbiI6IHsiaG9zdG5hbWUiOiAiZ3JlZW4ifX0sICJpcCI6ICJmZGNjOmM1ZGE6NTI5NTpjODUzOmQ0OTk6OTM3YzozMWEyOjFlODYiLCAibGFzdF9zZWVuIjogMTczMTE5OTI3NywgInBvcnQiOiA3MzMxfQ=="
},
"D9mq63wEznl4kHhsoQbq8hpncvGZeWC0vEOekcB8Nko=": {
"hostnames": {
"mors": {
"hostname": "mors"
}
},
"ip": "fdcc:c5da:5295:c853:d499:93e9:c5fc:c8b5",
"last_seen": 1731180076,
"port": 7331,
"signature": "/9o91MnAmSQTnbJCOK29zc2NcoAg8jI3SHbJ1NLiVQfCWafZ9MRqakkT/yLbgOTaepTCy2VFmu2HXalqnnUyC3siaG9zdG5hbWVzIjogeyJtb3JzIjogeyJob3N0bmFtZSI6ICJtb3JzIn19LCAiaXAiOiAiZmRjYzpjNWRhOjUyOTU6Yzg1MzpkNDk5OjkzZTk6YzVmYzpjOGI1IiwgImxhc3Rfc2VlbiI6IDE3MzExODAwNzYsICJwb3J0IjogNzMzMX0="
}
},
"settings": {
"banned_keys": [],
"host_signing_keys": [],
"hostname_overrides": {},
"last_update": 1724161701,
"public": true,
"signature": "K3UIjSbQkjKRM2yBlj8PIoeIZq4PyvImJss6SWremYBzggzibjnx8A5mifh0GF0xHig0J4gVhDmsYqogovRuDA==",
"tld": "nether"
}
}
}
Examples for merging
for settings bigger last_update always wins
a:
{ hosts: { ... },
settings: {
last_update: 2,
tld: "test",
signature: "sig2"
}
}
b:
{ hosts: { ... },
settings: {
last_update: 3,
tld: "test2",
signature: "sig3"
}
}
result:
{ hosts: { ... },
settings: {
last_update: 3,
tld: "test2",
signature: "sig3"
}
}
hosts with bigger last_update win, new hosts will be added
a:
{
hosts: {
pub1: {
ip: "42::1",
port: 7331,
last_update 3,
signature "sig_pub1_3"
}
},
settings: { ... },
}
b:
{
hosts: {
pub1: {
ip: "42::3",
port: 7331,
last_update 1,
signature "sig_pub1_1"
},
pub2: {
ip: "42::2",
port: 7331,
last_update 2,
signature "sig_pub2_2"
}
},
settings: { ... },
}
result:
{
hosts: {
pub1: {
ip: "42::1",
port: 7331,
last_update 3,
signature "sig_pub1_3"
},
pub2: {
ip: "42::2",
port: 7331,
last_update 2,
signature "sig_pub2_2"
}
},
settings: { ... },
}
Time-to-Live (TTL) and Gossip Protocol
In the Data-Mesher, all information should "decay" over time, meaning it automatically expires after a set period (this feature is not yet implemented). The settings field should include a configurable Time-to-Live (TTL), which removes old information, such as host entries, once they exceed the specified TTL.
- This makes attacks possible where Bobs Laptop is offline for TTL time (because Bob is on vacation) and another user claims their hostname.
- This means hostnames are only to be trusted if A) no trusted dm-node has been compromised and B) The hosts are never longer offline then the TTL
To prevent their data from expiring, hosts must regularly send updates to other peers. However, these updates don’t need to reach every peer directly. Since nodes share their entire data set during communication, information can relay through other nodes. As a result, a gossip-style communication system can be used, where information spreads gradually across the network through indirect connections, ensuring that all nodes stay up-to-date without overwhelming network traffic.
Joining Multiple DM-Networks
Here’s an example of how you can configure Data-Mesher in NixOS to join two different DM-Networks. In this setup, the .qubasa.clan domain gets higher priority than the .clan domain.
What does this mean? If both networks have a home hostname (e.g., home.qubasa.clan), the one from .qubasa.clan will take precedence over the one from .clan. The network with the lower priority number wins in the case of conflicts (closer to 0).
Here's the Nix configuration:
services.data-mesher = {
enable = true; # Enable Data-Mesher service
interface = "<mesh_vpn>"; # The network interface Data-Mesher will use
openFirewall = true; # Ensure the firewall allows Data-Mesher traffic
# Define the DM-Networks to join
networks = {
"qubasa.clan" = {
pubkey = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKTi2h4X56CzjeY4L1INl1d5JvYwh7HpaSuUlD33RhnY"; # Public key for the qubasa.clan network
priority = 1; # Higher priority (lower number = higher priority)
bootstrapPeers = [
"http://[fd27:bb88:dbef:737b:3799:9318:aa77:ec12]:7331" # A peer within this network to bootstrap from
];
};
"clan" = {
pubkey = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIJ1UM2Cza+GIRyuB9C3NqY0pSWnGC4DzmQOcWOa4SafV"; # Public key for the clan network
priority = 2; # Lower priority (higher number = lower priority)
bootstrapPeers = [
"http://[fd16:aa77:dbef:737b:3799:9316:aa77:dbef]:7331" # A peer within this network to bootstrap from
];
};
};
};
Key Points:
- Priority System:
qubasa.clanhas a priority of1.0, whileclanhas2.0. So, if there's a conflicting hostname,qubasa.clanwins since it has a smaller priority number. - Bootstrap Peers: Each network defines one or more bootstrap peers—these are known nodes that help your node join the network by sharing the network’s data.
- Public Keys: Each network you join requires its public key to verify the network’s integrity.
- Note: We use floats here because they are infinitely divisible. Which means one can add a higher priority node anywhere without having to change other nodes priorities.
This setup allows you to join and sync with multiple peer-to-peer DM-Networks and ensures the network with the defined priority takes precedence where necessary.
DNS Output (dns.json)
The Data-Mesher generates a file called dns.json, which contains all valid and verified hostnames in a simple, flat JSON format. This file is designed to be easy to consume by other applications or services that need to reference known network hosts.
Here’s an example of a dns.json file:
{"hostname": "mors.nether", "ip": "fdcc:c5da:5295:c853:d499:93e9:c5fc:c8b5"}
{"hostname": "green.nether", "ip": "fdcc:c5da:5295:c853:d499:937c:31a2:1e86"}
Each entry consists of:
- hostname: A valid, peer-reviewed hostname in the network.
- ip: The corresponding IP address (IPv6) of the hostname.
This file provides an up-to-date list of hostnames that are known to be good and usable within the network.
signing timestaps by trusted nodes
It should probably look like this:
{
"xxx": {
"hosts": {
"123": {
"hostnames": {
"123": {
"xyz": {
"signed_at": 128389182,
"signature": "signedbyxyz"
}
}
},
last_seen: 11,
signature "signedby123withtime11"
},
"456": {
"hostnames": {
"456": [
"xyz": {
"signed_at": 1231881298,
"signature": "signedbyxyz"
}
]
}
}
},
"settings": {
"host_signing_keys": [
"xyz"
]
}
}
}
Bootstrapping
Currently, the Data-Mesher starts by using a list of URLs to pull bootstrap data from. These URLs can point to peers in the network or just a web server that makes data.json available.
If a node suddenly finds itself isolated because all other hosts have "decayed" or become unreachable, it should go back to the bootstrap peers to retrieve fresh network information.
It would also be useful to have a feature that allows you to add new peers to a running Data-Mesher instance. For example, you could say, "Hey, connect to this peer and get its data" while the system is still running, without needing to restart or manually reconfigure everything.
Future Ideas
Ideas that are not currently on our roadmap but we want to see in the future.
Host Schema
It would be great if we could make the host schema more flexible, so it isn't tied specifically to hostnames. Hostnames could then just become one possible implementation of this schema. Here's an idea of how the settings could look:
{
"settings": {
"hostSchema": {
"hostnames": {
"schema": $some JSON schema,
"merge": oldest_wins,
}
}
}
}
However, this doesn't cover the requirement of signing hosts (i.e., ensuring hostnames are signed by trusted peers). One possible solution could be to require signatures by trusted third-party hosts for all data in these schemas, since we're already using them for merging as well.
Data Schema
It would also be interesting to expand the current schema (which supports hosts and settings) with a third top-level key, like data. This new data section would have its own schema defined in the settings, along with a merge function. Specific hosts will be authorized to modify this data, meaning nodes can verify if the data has been changed by an allowed writer.
To accomplish this, we'd need a protocol to verify the signatures and the origin of the data (this part hasn’t been specified fully yet, but I’m happy to discuss ideas!).
Here’s a rough idea of how the structure could look:
{
hosts: {
"123": { ... },
"456": { ... }
},
settings: {
dataSchema: {
x: {
schema: $some JSON schema,
merge: newest_wins,
allowed_writers: [
"123"
]
}
}
},
"data":{
x: {
value: "10.42.0.1",
signed_at: 1231212,
signature: "signedby123"
}
}
}
In this example:
- The dataSchema would define the rules for the data field, including how it should be merged (e.g., newest-wins) and which hosts are allowed to write the data (e.g., only host "123").
- The data section includes information such as the value, when it was signed, and the signature to verify who modified it.