Add experimental hybrid post-quantum handshake using Kyber-1024 and Dilithium

[Mateusvff]

Summary

This pull request adds experimental support for a hybrid post-quantum handshake to wireguard-go, combining the existing X25519-based key agreement with ML-KEM (Kyber-1024) and ML-DSA (Dilithium) for authentication. The goal is to provide a prototype implementation of a post-quantum–ready WireGuard handshake while preserving the original Noise_IK pattern and existing behaviour for non-PQ peers.

The implementation is based on the circl cryptographic library from Cloudflare, which provides Go implementations of Kyber and Dilithium.

---

Design overview

The implementation is split into two main parts:

1. Hybrid key agreement (ML-KEM / Kyber-1024)

   • Each device now maintains an ML-KEM static key pair (Kyber-1024).
   • During the initiation message, the initiator performs both X25519 Diffie–Hellman and Kyber encapsulation using the responder’s ML-KEM public key.
   • The ML-KEM ciphertext is carried in an extended field of MessageInitiation and is encrypted within the Noise handshake.
   • The classical X25519 shared secret and the ML-KEM shared secret are combined using a KDF (KDF2) to derive a single combinedSecret, which feeds into the existing Noise key schedule.

2. Hybrid authentication (ML-DSA / Dilithium)

   • Each device also maintains an ML-DSA static key pair (Dilithium5).
   • MessageInitiation is extended with a Signature field that carries a Dilithium signature over the serialized message fields (excluding MAC and signature fields).
   • On the responder side, after resolving the peer via the classical identity, the responder loads the peer’s ML-DSA public key and verifies the signature before proceeding with ML-KEM decapsulation and key derivation.

---

Implementation details

• New sizes and types

  • Added ML-KEM constants: MLKEMPublicKeySize, MLKEMPrivateKeySize, MLKEMCiphertextSize.
  • Added ML-DSA constants: MLDSAPublicKeySize, MLDSAPrivateKeySize, MLDSASignatureSize.
  • These are defined in device/noise-types.go.

• Extended identity and handshake structures

  • staticIdentity now includes:
    • mlkemPrivateKey, mlkemPublicKey
    • mldsaPrivateKey, mldsaPublicKey
  • Handshake now includes:
    • remoteMLKEMStatic (peer’s ML-KEM public key)
    • remoteMLDSAStatic (peer’s ML-DSA public key)

• Handshake protocol changes

  • In device/noise-protocol.go:
    • MessageInitiation was extended with:
      • MLKEM field to carry the Kyber ciphertext.
      • Signature field to carry the Dilithium signature.
    • CreateMessageInitiation:
      • Performs X25519 DH as before.
      • Calls kyber1024.Scheme().Encapsulate(remoteMLKEMStatic) to obtain mlkemSecret and ciphertext.
      • Encrypts and stores ciphertext into msg.MLKEM.
      • Combines the X25519 secret and mlkemSecret via KDF2 into combinedSecret.
      • Serializes the message fields and signs them using the local ML-DSA private key, storing the result in msg.Signature.
    • ConsumeMessageInitiation:
      • Resolves the peer via the classical identity (X25519).
      • Retrieves remoteMLDSAStatic and verifies msg.Signature with the Dilithium verification routine.
      • Decrypts msg.MLKEM and calls Decapsulate with the local ML-KEM private key to recover mlkemSecret.
      • Recomputes combinedSecret using the same KDF and continues with the existing Noise key schedule.

• UAPI and key management

  • device/uapi.go was updated to accept:
    • mlkem_private_key, mlkem_public_key
    • mldsa_private_key, mldsa_public_key
  • A helper module device/quantum-keys.go was added providing:
    • GenerateQuantumKeyPair for Kyber-1024 key pairs.
    • GenerateMLDSAKeyPair for Dilithium5 key pairs.
  • The implementation uses circl’s kem.Scheme API for Kyber (via GenerateKeyPair) and sign.Scheme for Dilithium (via GenerateKey).

---

Compatibility

• Existing behaviour and classical X25519-only handshakes remain unchanged when PQ keys are not configured.
• The hybrid mode is only used when both sides provide ML-KEM and ML-DSA keys via UAPI.
• The Noise_IK pattern and MAC validation logic remain intact; PQ data is carried in additional fields and integrated into the key schedule and authentication steps.

---

Testing

• Unit tests

  • Added tests for:
    • ML-KEM key generation, encapsulation and decapsulation.
    • ML-DSA key generation, signing and verification.
    • KDF combination of classical and PQ secrets.

• Integration tests

  • End-to-end handshake tests between two PQ-enabled peers (initiator and responder).
  • Regression tests with classical-only peers to ensure no change in existing behaviour.

• Manual validation

  • Verified that both peers derive identical session keys in the hybrid mode.
  • Verified that invalid signatures or corrupted PQ ciphertexts abort the handshake.
  • Performed basic performance checks to confirm that handshake latency remains within acceptable bounds.

---