Applied Cryptography, PKI, TLS & Certificate Management
Applied Cryptography, PKI, TLS & Certificate Management
CIPHER Training Module — Deep Dive Reference Sources: OWASP Cheat Sheets, Mozilla Server Side TLS, NIST SP 800-57/131a, RFC 8446, tool documentation
1. Algorithm Recommendations
1.1 Symmetric Encryption
| Algorithm | Key Size | Mode | Use Case | Status |
|---|---|---|---|---|
| AES-256-GCM | 256-bit | AEAD | Data-at-rest, TLS | Recommended |
| AES-128-GCM | 128-bit | AEAD | Data-at-rest, TLS | Acceptable |
| ChaCha20-Poly1305 | 256-bit | AEAD | TLS (mobile/embedded), disk | Recommended (software-only) |
| AES-256-CCM | 256-bit | AEAD | Constrained environments | Acceptable |
| AES-CBC + HMAC | 256-bit | Encrypt-then-MAC | Legacy compatibility | Acceptable with caution |
Selection guidance:
- Always prefer authenticated encryption (AEAD): GCM or ChaCha20-Poly1305
- ChaCha20-Poly1305 outperforms AES-GCM on hardware without AES-NI instructions
- AES-GCM requires unique nonces per encryption — nonce reuse is catastrophic (breaks authenticity and confidentiality)
- GCM performance degrades and leaks with messages exceeding ~64 GiB under a single key
- AES-GCM-SIV (RFC 8452) provides nonce-misuse resistance at slight performance cost — consider for contexts where nonce uniqueness is hard to guarantee
AVOID:
- ECB mode — deterministic, leaks patterns. No legitimate use outside narrow block-cipher building blocks
- AES-CBC without MAC — vulnerable to padding oracle attacks (CVE-2014-3566, Vaudenay 2002)
- RC4 — biases in keystream, prohibited by RFC 7465
- DES/3DES — 64-bit block size, Sweet32 attack (CVE-2016-2183)
- Blowfish — 64-bit block, same Sweet32 class
1.2 Asymmetric Encryption & Key Agreement
| Algorithm | Key Size | Use Case | Status |
|---|---|---|---|
| X25519 | 256-bit | Key agreement (ECDH) | Recommended |
| Ed25519 | 256-bit | Digital signatures | Recommended |
| ECDSA P-256 | 256-bit | Signatures, TLS certs | Acceptable |
| ECDSA P-384 | 384-bit | High-assurance signatures | Acceptable |
| RSA-OAEP | 3072+ bit | Key transport, encryption | Acceptable |
| RSA-PSS | 3072+ bit | Digital signatures | Acceptable |
| X448 / Ed448 | 448-bit | Higher security margin | Acceptable |
Key size equivalences (NIST SP 800-57):
| Symmetric | RSA/DH | ECC | Hash |
|---|---|---|---|
| 128-bit | 3072 | 256 | SHA-256 |
| 192-bit | 7680 | 384 | SHA-384 |
| 256-bit | 15360 | 512 | SHA-512 |
Minimum RSA key size: 2048-bit (legacy), 3072-bit for new deployments (NIST, BSI, ANSSI convergence).
AVOID:
- RSA < 2048 bit — factorable with current resources
- RSA PKCS#1 v1.5 encryption — Bleichenbacher/ROBOT attacks (CVE-2017-13099)
- DSA — deprecated in FIPS 186-5
- ElGamal — no modern advantage over ECDH
- Plain Diffie-Hellman with small/custom groups — logjam attack
1.3 Hashing Algorithms
| Algorithm | Output | Use Case | Status |
|---|---|---|---|
| SHA-256 | 256-bit | General integrity, certificates | Recommended |
| SHA-384 | 384-bit | TLS with P-384 | Acceptable |
| SHA-512 | 512-bit | High-security contexts | Acceptable |
| SHA-3 (Keccak) | 256/512-bit | Diversity from SHA-2 | Acceptable |
| BLAKE2b | 256-512-bit | Performance-critical hashing | Recommended for non-compliance contexts |
| BLAKE3 | 256-bit | Fastest secure hash | Acceptable (newer, less scrutiny) |
AVOID:
- MD5 — collision attacks trivial (2^18 complexity), CVE-2004-2761
- SHA-1 — practical collision (SHAttered, 2017), prohibited in certificates since 2016
- CRC32 — not a cryptographic hash
1.4 Key Derivation Functions (KDFs)
| KDF | Use Case | Parameters |
|---|---|---|
| HKDF (RFC 5869) | Deriving keys from shared secrets | Extract-then-Expand with SHA-256/512 |
| Argon2id | Password-to-key derivation | See password hashing section |
| scrypt | Password-to-key derivation | See password hashing section |
| PBKDF2-HMAC-SHA256 | FIPS-compliant key derivation | 600,000+ iterations |
Critical distinction: HKDF is for high-entropy inputs (DH shared secrets). Password KDFs (Argon2id, scrypt, PBKDF2) are for low-entropy inputs (human passwords). Never use HKDF on passwords. Never use Argon2id on DH outputs.
2. Password Hashing
2.1 Algorithm Selection Hierarchy
- Argon2id — memory-hard, resists GPU/ASIC attacks [CONFIRMED]
- scrypt — memory-hard, established [CONFIRMED]
- bcrypt — CPU-hard only, 72-byte input limit [CONFIRMED]
- PBKDF2-HMAC-SHA256 — only when FIPS 140-2/3 compliance required [CONFIRMED]
2.2 Argon2id Parameters
Equivalent security configurations (choose based on available memory):
| Memory (m) | Iterations (t) | Parallelism (p) | Notes |
|---|---|---|---|
| 47,104 KiB (46 MiB) | 1 | 1 | Maximum memory option |
| 19,456 KiB (19 MiB) | 2 | 1 | Minimum recommended |
| 12,288 KiB (12 MiB) | 3 | 1 | Balanced |
| 9,216 KiB (9 MiB) | 4 | 1 | Lower memory |
| 7,168 KiB (7 MiB) | 5 | 1 | Constrained environments |
Tuning approach:
- Set p=1 (parallelism adds complexity without proportional security gain in most deployments)
- Maximize m (memory) to what the system can sustain under peak auth load
- Increase t (iterations) to fill remaining time budget
- Target: hash computation < 1 second per authentication attempt
2.3 scrypt Parameters
| N (CPU/memory cost) | r (block size) | p (parallelism) | Memory |
|---|---|---|---|
| 2^17 (131072) | 8 | 1 | 128 MiB |
| 2^16 (65536) | 8 | 2 | 64 MiB |
| 2^15 (32768) | 8 | 3 | 32 MiB |
| 2^14 (16384) | 8 | 5 | 16 MiB |
| 2^13 (8192) | 8 | 10 | 8 MiB |
Memory = 128 * N * r bytes.
2.4 bcrypt
- Minimum work factor: 10 (2^10 = 1024 iterations)
- Recommended work factor: 12-14 depending on hardware (benchmark to ~250ms)
- Hard limit: 72-byte input. Passwords longer than 72 bytes are silently truncated
- Pre-hashing workaround: SHA-256 the password first, then bcrypt the hash. Beware null byte issues with some implementations (use base64 encoding of the SHA-256 output)
2.5 PBKDF2
Use only when FIPS 140 compliance is mandatory:
| Variant | Minimum Iterations |
|---|---|
| PBKDF2-HMAC-SHA256 | 600,000 |
| PBKDF2-HMAC-SHA512 | 210,000 |
| PBKDF2-HMAC-SHA1 | 1,300,000 |
2.6 Salting and Peppering
Salt: Unique per credential, minimum 16 bytes from CSPRNG. Modern algorithms (Argon2id, bcrypt) handle salt generation and storage internally.
Pepper: HMAC key or encryption key applied to the hash, stored separately from the database (HSM, config management, environment). Provides defense-in-depth if database is compromised but application layer is not.
Upgrade strategy for legacy hashes: Wrap old hash — argon2id(legacy_md5_hash) — and re-hash on next successful login with the plaintext password.
3. TLS 1.3 Deep Dive
3.1 Protocol Overview
TLS 1.3 (RFC 8446, August 2018) is a major overhaul — not an incremental update. Key changes from TLS 1.2:
- 1-RTT handshake (down from 2-RTT in TLS 1.2)
- 0-RTT resumption for repeat connections (with replay caveats)
- Removed: RSA key exchange, CBC cipher suites, compression, renegotiation, custom DH groups, DSA, RC4, SHA-1 in signatures, export ciphers, static DH/ECDH
- All cipher suites provide forward secrecy — no exceptions
- Encrypted handshake — server certificate and most extensions encrypted after ServerHello
- Simplified cipher suite naming — algorithm specifies only AEAD + hash, key exchange negotiated separately via supported_groups/key_share
3.2 TLS 1.3 Handshake (1-RTT Full)
Client Server
ClientHello
+ supported_versions (0x0304)
+ key_share (X25519 public key)
+ signature_algorithms
+ psk_key_exchange_modes
+ server_name (SNI)
+ supported_groups
-------->
ServerHello
+ supported_versions
+ key_share (X25519 public key)
{EncryptedExtensions}
{CertificateRequest*}
{Certificate}
{CertificateVerify}
{Finished}
<--------
{Certificate*}
{CertificateVerify*}
{Finished}
-------->
[Application Data] <-------> [Application Data]
Legend: {} = encrypted with handshake keys, [] = encrypted with application keys, * = optional
Key differences from TLS 1.2:
- Client sends key_share in ClientHello (speculative key exchange)
- Server responds in single flight with ServerHello + encrypted extensions + cert + finished
- Server certificate is encrypted — passive observers cannot see it
- No separate ChangeCipherSpec message (compatibility mode sends a dummy one)
- Handshake transcript hash binds all messages together cryptographically
3.3 TLS 1.3 Key Schedule
The key schedule derives all keys from a single chain using HKDF:
0
|
v
PSK -------> HKDF-Extract = Early Secret
|
v
Derive-Secret(., "ext binder" | "res binder", "")
= binder_key
|
v
Derive-Secret(., "c e traffic", ClientHello)
= client_early_traffic_secret (0-RTT data key)
|
v
Derive-Secret(., "derived", "")
|
v
(EC)DHE ---> HKDF-Extract = Handshake Secret
|
+-> Derive-Secret(., "c hs traffic", ClientHello...ServerHello)
| = client_handshake_traffic_secret
|
+-> Derive-Secret(., "s hs traffic", ClientHello...ServerHello)
| = server_handshake_traffic_secret
|
v
Derive-Secret(., "derived", "")
|
v
0 ---------> HKDF-Extract = Master Secret
|
+-> Derive-Secret(., "c ap traffic", ClientHello...server Finished)
| = client_application_traffic_secret_0
|
+-> Derive-Secret(., "s ap traffic", ClientHello...server Finished)
| = server_application_traffic_secret_0
|
+-> Derive-Secret(., "res master", ClientHello...client Finished)
= resumption_master_secret
Critical properties:
- Forward secrecy: compromising the server's long-term key does not compromise past sessions because ephemeral (EC)DHE is always used
- Key separation: each direction and phase has distinct keys
- Transcript binding: keys depend on hash of all handshake messages up to that point
3.4 TLS 1.3 Cipher Suites
Only five cipher suites defined (all AEAD):
| Cipher Suite | AEAD | Hash | Status |
|---|---|---|---|
| TLS_AES_128_GCM_SHA256 | AES-128-GCM | SHA-256 | Mandatory |
| TLS_AES_256_GCM_SHA384 | AES-256-GCM | SHA-384 | Recommended |
| TLS_CHACHA20_POLY1305_SHA256 | ChaCha20-Poly1305 | SHA-256 | Recommended |
| TLS_AES_128_CCM_SHA256 | AES-128-CCM | SHA-256 | IoT/constrained |
| TLS_AES_128_CCM_8_SHA256 | AES-128-CCM (8-byte tag) | SHA-256 | IoT only, reduced security |
Key exchange is negotiated separately via supported_groups extension:
- X25519 (most common, recommended)
- secp256r1 (P-256)
- secp384r1 (P-384)
- x448
- ffdhe2048, ffdhe3072 (finite-field, rare in practice)
3.5 0-RTT Resumption (Early Data)
0-RTT allows clients to send application data in the first flight using a pre-shared key from a previous session:
Client Server
ClientHello
+ early_data
+ key_share
+ psk_identity
(Application Data) -------->
ServerHello
+ pre_shared_key
{EncryptedExtensions
+ early_data}
{Finished}
<--------
{EndOfEarlyData}
{Finished}
-------->
[Application Data] <-------> [Application Data]
0-RTT security properties and risks:
| Property | Status |
|---|---|
| Confidentiality | Yes (encrypted with PSK-derived key) |
| Forward secrecy | NO — uses PSK only, no ephemeral DH for early data |
| Replay protection | NO — server cannot guarantee at-most-once delivery |
| Source authentication | Yes (PSK authenticates client implicitly) |
Replay attack mitigations:
- Single-use tickets: server tracks used tickets in a database (eliminates replay but requires shared state across server fleet)
- Client hello recording: server remembers ClientHello hashes within a time window
- Application-level idempotency: only allow 0-RTT for safe (idempotent) HTTP methods (GET, HEAD) — never for POST, PUT, DELETE
- Time window restriction: reject 0-RTT data if ticket age exceeds a threshold (e.g., 10 seconds)
Recommendation: Disable 0-RTT unless the performance benefit is critical AND the application layer handles replay safely. Most deployments should not enable it.
Nginx 0-RTT configuration:
# Enable 0-RTT (use with caution)
ssl_early_data on;
# Pass early data indicator to application
proxy_set_header Early-Data $ssl_early_data;
# Application MUST check Early-Data header and reject non-idempotent requests
3.6 Pre-Shared Keys (PSK)
Two PSK modes in TLS 1.3:
- PSK-only (psk_ke): Key derived from PSK alone. No forward secrecy. Suitable for IoT/constrained devices.
- PSK with (EC)DHE (psk_dhe_ke): PSK combined with ephemeral DH. Provides forward secrecy. Recommended.
Session resumption flow:
- After full handshake completes, server sends NewSessionTicket message containing ticket and ticket_nonce
- Client stores ticket + resumption_master_secret
- On reconnection, client derives PSK = HKDF-Expand-Label(resumption_master_secret, "resumption", ticket_nonce, Hash.length)
- Client includes PSK identity in ClientHello pre_shared_key extension
External PSKs: Pre-provisioned shared secrets (not from resumption). Used in:
- IoT device authentication
- Datacenter-to-datacenter links with pre-shared credentials
- Environments where certificate infrastructure is impractical
3.7 Encrypted Client Hello (ECH)
ECH (draft-ietf-tls-esni) encrypts the ClientHello to hide SNI from passive observers:
Client Server
ClientHelloOuter
+ encrypted_client_hello (contains real SNI)
+ server_name: "public.example.com" (cover)
-------->
Decrypts inner ClientHello
Routes to real backend
ECH deployment requirements:
- Server publishes ECH config in DNS HTTPS record
- Client fetches config, encrypts inner ClientHello with server's HPKE public key
- Requires DNS-over-HTTPS/TLS to prevent DNS-level SNI leakage
- Currently supported in Firefox and Chrome (behind flags in some versions)
Privacy impact: Without ECH, passive network observers can see which specific site a user visits even over TLS (via SNI). ECH closes this metadata leak.
4. TLS Configuration
4.1 Protocol Versions
| Protocol | Status | Notes |
|---|---|---|
| TLS 1.3 | Required | Default for all new deployments |
| TLS 1.2 | Acceptable | Only with AEAD cipher suites (GCM, ChaCha20) |
| TLS 1.1 | Prohibited | Deprecated RFC 8996 (2021), PCI DSS non-compliant |
| TLS 1.0 | Prohibited | Deprecated RFC 8996, BEAST/POODLE class |
| SSL 3.0 | Prohibited | POODLE (CVE-2014-3566) |
| SSL 2.0 | Prohibited | Fundamentally broken (DROWN) |
4.2 Cipher Suites — Mozilla Profiles
Modern (TLS 1.3 only)
TLS_AES_128_GCM_SHA256
TLS_AES_256_GCM_SHA384
TLS_CHACHA20_POLY1305_SHA256
- Certificate: ECDSA P-256
- Curves: X25519, prime256v1, secp384r1
- Cipher preference: client chooses
- Certificate lifespan: 90 days
- Oldest compatible: Firefox 63, Chrome 70, Safari 12.1, Android 10
Intermediate (TLS 1.2 + 1.3) — recommended for most deployments
TLS 1.3 ciphers as above, plus TLS 1.2:
ECDHE-ECDSA-AES128-GCM-SHA256
ECDHE-RSA-AES128-GCM-SHA256
ECDHE-ECDSA-AES256-GCM-SHA384
ECDHE-RSA-AES256-GCM-SHA384
ECDHE-ECDSA-CHACHA20-POLY1305
ECDHE-RSA-CHACHA20-POLY1305
DHE-RSA-AES128-GCM-SHA256
DHE-RSA-AES256-GCM-SHA384
DHE-RSA-CHACHA20-POLY1305
- Certificate: ECDSA P-256 preferred, RSA 2048-bit acceptable
- DH parameter: ffdhe2048 (RFC 7919)
- Curves: X25519, prime256v1, secp384r1
- Certificate lifespan: 90-366 days
- Oldest compatible: Firefox 27, Chrome 31, IE 11/Win7, Android 4.4.2
4.3 Server Configuration — Nginx
Nginx Modern (TLS 1.3 Only)
server {
listen 443 ssl;
listen [::]:443 ssl;
server_name example.com;
ssl_certificate /etc/letsencrypt/live/example.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem;
# TLS 1.3 only
ssl_protocols TLSv1.3;
# Not needed for TLS 1.3 (client chooses), but explicit for clarity
ssl_prefer_server_ciphers off;
# Curve preference
ssl_ecdh_curve X25519:secp256r1:secp384r1;
# Session configuration
ssl_session_timeout 1d;
ssl_session_cache shared:SSL:10m; # ~40,000 sessions
ssl_session_tickets off; # Disable for forward secrecy guarantee
# OCSP stapling
ssl_stapling on;
ssl_stapling_verify on;
ssl_trusted_certificate /etc/letsencrypt/live/example.com/chain.pem;
resolver 1.1.1.1 8.8.8.8 valid=300s;
resolver_timeout 5s;
# Security headers
add_header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload" always;
add_header X-Content-Type-Options "nosniff" always;
add_header X-Frame-Options "DENY" always;
}
Nginx Intermediate (TLS 1.2 + 1.3)
server {
listen 443 ssl;
listen [::]:443 ssl;
server_name example.com;
ssl_certificate /etc/letsencrypt/live/example.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem;
# Protocol versions
ssl_protocols TLSv1.2 TLSv1.3;
# TLS 1.2 cipher suites (TLS 1.3 suites configured automatically)
ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384;
ssl_prefer_server_ciphers off;
# Curves and DH
ssl_ecdh_curve X25519:prime256v1:secp384r1;
ssl_dhparam /etc/nginx/ffdhe2048.pem; # Generate: openssl dhparam -out ffdhe2048.pem 2048
# Session configuration
ssl_session_timeout 1d;
ssl_session_cache shared:SSL:10m;
ssl_session_tickets off;
# OCSP stapling
ssl_stapling on;
ssl_stapling_verify on;
ssl_trusted_certificate /etc/letsencrypt/live/example.com/chain.pem;
resolver 1.1.1.1 8.8.8.8 valid=300s;
resolver_timeout 5s;
# HSTS
add_header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload" always;
}
4.4 Server Configuration — Apache
Apache Intermediate
<VirtualHost *:443>
ServerName example.com
SSLEngine on
SSLCertificateFile /etc/letsencrypt/live/example.com/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/example.com/privkey.pem
# Protocol versions
SSLProtocol all -SSLv3 -TLSv1 -TLSv1.1
# Cipher suites
SSLCipherSuite ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
SSLHonorCipherOrder off
# TLS 1.3 cipher suites (Apache 2.4.53+)
SSLCipherSuite TLSv1.3 TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_128_GCM_SHA256
# Curves
SSLOpenSSLConfCmd Groups X25519:prime256v1:secp384r1
# Session
SSLSessionTickets off
# OCSP stapling
SSLUseStapling On
SSLStaplingResponderTimeout 5
SSLStaplingReturnResponderErrors off
# HSTS
Header always set Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
</VirtualHost>
# OCSP stapling cache (must be outside VirtualHost)
SSLStaplingCache shmcb:/var/run/ocsp(128000)
Apache Modern (TLS 1.3 Only)
<VirtualHost *:443>
ServerName example.com
SSLEngine on
SSLCertificateFile /etc/letsencrypt/live/example.com/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/example.com/privkey.pem
SSLProtocol -all +TLSv1.3
SSLSessionTickets off
SSLUseStapling On
Header always set Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
</VirtualHost>
4.5 Server Configuration — HAProxy
HAProxy Intermediate
global
# SSL/TLS settings
ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
ssl-default-bind-options prefer-client-ciphers no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets
ssl-default-bind-curves X25519:P-256:P-384
ssl-default-server-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384
ssl-default-server-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
ssl-default-server-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets
tune.ssl.default-dh-param 2048
frontend https
bind *:443 ssl crt /etc/haproxy/certs/ alpn h2,http/1.1
http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
default_backend servers
backend servers
server srv1 10.0.0.1:8080 check ssl verify required ca-file /etc/haproxy/ca.pem
4.6 HSTS (HTTP Strict Transport Security)
Header format:
Strict-Transport-Security: max-age=63072000; includeSubDomains; preload
Deployment strategy (graduated rollout):
- Testing:
max-age=86400; includeSubDomains(24 hours) - Staging:
max-age=604800; includeSubDomains(1 week) - Production:
max-age=63072000; includeSubDomains; preload(2 years)
Preload requirements:
- Valid TLS certificate
- Redirect HTTP to HTTPS on same host
- All subdomains served over HTTPS
- HSTS header on base domain with
includeSubDomainsandpreload - max-age >= 31536000 (1 year)
- Submit to hstspreload.org
WARNING: Preload is effectively permanent. Removal from the preload list takes months and requires browser update cycles. Verify all subdomains support HTTPS before submitting.
Security properties:
- Prevents SSL stripping (Moxie Marlinspike, 2009)
- Blocks click-through on certificate warnings
- Converts HTTP links to HTTPS before request (307 Internal Redirect)
- First-visit vulnerability exists without preload
5. Certificate Management
5.1 CA Hierarchy
Root CA (offline, air-gapped HSM, 20-30 year validity)
├── Intermediate CA - TLS Server Certificates (5-10 year validity)
│ ├── *.example.com (wildcard, 90 days)
│ └── api.example.com (SAN cert, 90 days)
├── Intermediate CA - Client Certificates (5-10 year validity)
│ ├── user@example.com (1 year)
│ └── service-account-01 (90 days)
├── Intermediate CA - Code Signing (5-10 year validity)
│ └── build-pipeline (1 year)
└── Intermediate CA - Email/S-MIME (5-10 year validity)
└── user@example.com (1-2 years)
Design principles:
- Root CA is offline, stored in HSM, used only to sign intermediate CA certificates
- Intermediate (issuing) CAs are online, purpose-separated
- Short-lived leaf certificates (90 days or less) reduce revocation dependency
- Path length constraints (
pathLenConstraint) limit delegation depth - Name constraints restrict what domains an intermediate can issue for
5.2 Certificate Lifecycle
Generation -> CSR -> Validation -> Issuance -> Deployment -> Monitoring -> Renewal/Revocation -> Destruction
| | | | | | | |
v v v v v v v v
CSPRNG Subject DV/OV/EV CA signs Install + CT logs, ACME auto Secure
on target + SANs + CAA cert chain expiry or CRL/OCSP key
or HSM + key check verify alerting on compromise deletion
Step-by-step with OpenSSL:
# 1. Generate private key (ECDSA P-256)
openssl ecparam -genkey -name prime256v1 -noout -out server.key
# Or RSA 3072-bit
openssl genrsa -out server.key 3072
# 2. Create CSR
openssl req -new -key server.key -out server.csr \
-subj "/CN=api.example.com" \
-addext "subjectAltName=DNS:api.example.com,DNS:api2.example.com"
# 3. Verify CSR contents
openssl req -in server.csr -noout -text -verify
# 4. Self-sign (for testing only)
openssl x509 -req -in server.csr -signkey server.key -out server.crt \
-days 90 -copy_extensions copyall
# 5. Verify certificate
openssl x509 -in server.crt -noout -text
# 6. Verify certificate chain
openssl verify -CAfile ca-chain.pem server.crt
# 7. Check certificate expiry
openssl x509 -in server.crt -noout -enddate
# 8. Check remote server certificate
openssl s_client -connect example.com:443 -servername example.com < /dev/null 2>/dev/null | \
openssl x509 -noout -dates -subject -issuer
5.3 Certificate Revocation — CRL
Certificate Revocation Lists — CA publishes a signed list of revoked serial numbers:
# Download and inspect CRL
curl -sO http://crl.example.com/intermediate.crl
openssl crl -in intermediate.crl -noout -text
# Check if a certificate is on a CRL
openssl verify -crl_check -CRLfile intermediate.crl -CAfile ca-chain.pem server.crt
CRL limitations:
- Size grows linearly with revocations (can be megabytes for large CAs)
- Clients must download full CRL periodically (latency, bandwidth)
- CRL cache lifetime creates a window where revoked certs are still trusted
- Many clients soft-fail (accept connection if CRL is unavailable)
5.4 Certificate Revocation — OCSP
Online Certificate Status Protocol — real-time revocation checking:
# Query OCSP responder directly
openssl ocsp -issuer intermediate.pem -cert server.crt \
-url http://ocsp.example.com -resp_text
# Extract OCSP URI from certificate
openssl x509 -in server.crt -noout -ocsp_uri
OCSP privacy concern: Every TLS connection triggers an OCSP request to the CA, revealing which sites the user visits. This is why OCSP stapling exists.
5.5 OCSP Stapling
Server fetches OCSP response from CA and serves it during TLS handshake, eliminating client-side OCSP lookup:
Nginx:
ssl_stapling on;
ssl_stapling_verify on;
ssl_trusted_certificate /etc/letsencrypt/live/example.com/chain.pem;
resolver 1.1.1.1 8.8.8.8 valid=300s;
resolver_timeout 5s;
Apache:
SSLUseStapling On
SSLStaplingCache shmcb:/var/run/ocsp(128000)
SSLStaplingResponderTimeout 5
SSLStaplingReturnResponderErrors off
Verify OCSP stapling is working:
openssl s_client -connect example.com:443 -servername example.com -status < /dev/null 2>/dev/null | \
grep -A 20 "OCSP Response"
# Expected output includes:
# OCSP Response Status: successful (0x0)
# Cert Status: good
Must-Staple extension (RFC 7633): Certificate includes OID 1.3.6.1.5.5.7.1.24 — browser rejects connection if stapled response is missing. Recommended for high-security deployments but increases operational complexity (if stapling fails, site is unreachable).
5.6 Certificate Transparency (CT) Logs
CT (RFC 6962) provides a public, append-only log of all issued certificates. CAs must submit certificates to CT logs and include Signed Certificate Timestamps (SCTs) in the certificate.
Chrome requires CT compliance — certificates without SCTs are rejected.
Monitoring with crt.sh:
# Search by domain
curl -s "https://crt.sh/?q=example.com&output=json" | jq '.[0:5]'
# Search by wildcard
curl -s "https://crt.sh/?q=%.example.com&output=json" | jq '.[].name_value' | sort -u
# Search by organization
curl -s "https://crt.sh/?q=O=Example+Corp&output=json" | jq '.[0:5]'
Automated CT monitoring tools:
- certspotter (SSLMate):
certspotter -watchlist example.com - ct-exposer: Enumerates subdomains from CT logs
- Cert Alert (Facebook): email notifications for new certificates
- Custom: subscribe to CT log Merkle tree updates via CT log APIs
Offensive use (RECON):
- Enumerate subdomains via CT logs — reveals internal hostnames, staging environments
- Identify shadow IT — certificates issued for unauthorized services
- Track CA changes — detect potential compromise or policy violations
Defensive use (BLUE):
- Monitor for unauthorized certificate issuance
- Alert on certificates from non-approved CAs
- Detect domain impersonation (typosquatting certs)
- Combine with CAA records for defense-in-depth
5.7 CAA Records
DNS CAA (RFC 8659) restricts which CAs may issue certificates for a domain:
example.com. IN CAA 0 issue "letsencrypt.org"
example.com. IN CAA 0 issuewild "letsencrypt.org"
example.com. IN CAA 0 iodef "mailto:security@example.com"
Advanced CAA with account binding:
; Only allow Let's Encrypt with specific account
example.com. IN CAA 0 issue "letsencrypt.org;accounturi=https://acme-v02.api.letsencrypt.org/acme/acct/123456"
; Allow DigiCert for wildcard only
example.com. IN CAA 0 issuewild "digicert.com"
; Prevent all wildcard issuance
example.com. IN CAA 0 issuewild ";"
CAs are required to check CAA records before issuance (CA/Browser Forum Ballot 187). Verify with:
dig CAA example.com +short
5.8 ACME Protocol & Let's Encrypt
ACME (RFC 8555) automates certificate issuance:
Challenge types:
| Challenge | Port | Automation | Wildcard |
|---|---|---|---|
| http-01 | 80 | Easy | No |
| dns-01 | DNS | Medium (needs DNS API) | Yes |
| tls-alpn-01 | 443 | Medium | No |
Certbot usage:
# Obtain certificate (nginx plugin)
certbot --nginx -d example.com -d www.example.com
# Obtain certificate (standalone — needs port 80)
certbot certonly --standalone -d example.com
# Obtain certificate (webroot — existing web server)
certbot certonly --webroot -w /var/www/html -d example.com
# Wildcard via DNS (Cloudflare)
certbot certonly --dns-cloudflare \
--dns-cloudflare-credentials ~/.secrets/cloudflare.ini \
-d "*.example.com" -d example.com
# ECDSA key (recommended)
certbot certonly --key-type ecdsa --elliptic-curve secp256r1 -d example.com
# Dual certificates (ECDSA + RSA for compatibility)
certbot certonly --key-type ecdsa -d example.com --cert-name example.com-ecdsa
certbot certonly --key-type rsa -d example.com --cert-name example.com-rsa
# Renew all certificates
certbot renew --quiet --deploy-hook "systemctl reload nginx"
# Dry run (test renewal without actually renewing)
certbot renew --dry-run
Automation with systemd timer:
# /etc/systemd/system/certbot-renewal.timer
[Unit]
Description=Certbot renewal timer
[Timer]
OnCalendar=*-*-* 00,12:00:00
RandomizedDelaySec=3600
Persistent=true
[Install]
WantedBy=timers.target
# /etc/systemd/system/certbot-renewal.service
[Unit]
Description=Certbot renewal
[Service]
Type=oneshot
ExecStart=/usr/bin/certbot renew --quiet --deploy-hook "systemctl reload nginx"
6. Certificate Pinning
6.1 HPKP History and Deprecation
HTTP Public Key Pinning (HPKP, RFC 7469) allowed sites to declare which public keys should be trusted via HTTP header:
Public-Key-Pins: pin-sha256="base64..."; pin-sha256="base64backup..."; max-age=5184000; includeSubDomains; report-uri="https://report.example.com/hpkp"
Why HPKP was deprecated:
- Bricking risk: Misconfiguration or key loss rendered sites permanently inaccessible for the max-age duration
- Hostile pinning: Attacker with temporary control could pin their own key, creating persistent DoS
- Operational complexity: Key rotation required careful coordination with pin updates
- Low adoption: Most site operators considered the risk too high
- Removed from Chrome 72 (2019), Firefox 72 (2020)
Expect-CT (RFC 9163) partially replaced HPKP for certificate validation, but is itself now largely superseded by browsers requiring CT by default.
6.2 Where Pinning Is Still Relevant
| Context | Recommendation |
|---|---|
| Web browsers | Do NOT pin (HPKP deprecated, CT provides better protection) |
| Mobile apps (iOS/Android) | Pin intermediate CA SPKI hash with backup pins |
| IoT devices | Pin root or intermediate CA SPKI hash |
| Desktop API clients | Pin leaf or intermediate SPKI hash |
| Internal microservices | mTLS preferred over pinning |
6.3 Generating Pin Hashes
# Pin from certificate file
openssl x509 -in cert.pem -pubkey -noout | \
openssl pkey -pubin -outform der | \
openssl dgst -sha256 -binary | base64
# Pin from live server
openssl s_client -connect example.com:443 -servername example.com 2>/dev/null | \
openssl x509 -pubkey -noout | \
openssl pkey -pubin -outform der | \
openssl dgst -sha256 -binary | base64
# Pin from CSR (pin before certificate is issued)
openssl req -in cert.csr -pubkey -noout | \
openssl pkey -pubin -outform der | \
openssl dgst -sha256 -binary | base64
# Pin from private key (derive public key, then hash)
openssl pkey -in server.key -pubout -outform der | \
openssl dgst -sha256 -binary | base64
6.4 Mobile Certificate Pinning
Android — Network Security Config (API 24+)
<!-- res/xml/network_security_config.xml -->
<network-security-config>
<domain-config>
<domain includeSubdomains="true">api.example.com</domain>
<pin-set expiration="2025-12-31">
<!-- Primary: intermediate CA SPKI hash -->
<pin digest="SHA-256">YLh1dUR9y6Kja30RrAn7JKnbQG/uEtLMkBgFF2Fuihg=</pin>
<!-- Backup: different CA intermediate SPKI hash -->
<pin digest="SHA-256">sRHdihwgkaib1P1gN7akajqIzMt0EjVfUmKkQ8ei1Sg=</pin>
</pin-set>
</domain-config>
</network-security-config>
<!-- AndroidManifest.xml -->
<application android:networkSecurityConfig="@xml/network_security_config">
iOS — TrustKit
// AppDelegate.swift
import TrustKit
let trustKitConfig: [String: Any] = [
kTSKSwizzleNetworkDelegates: false,
kTSKPinnedDomains: [
"api.example.com": [
kTSKEnforcePinning: true,
kTSKIncludeSubdomains: true,
kTSKExpirationDate: "2025-12-31",
kTSKPublicKeyHashes: [
"YLh1dUR9y6Kja30RrAn7JKnbQG/uEtLMkBgFF2Fuihg=", // Primary
"sRHdihwgkaib1P1gN7akajqIzMt0EjVfUmKkQ8ei1Sg=", // Backup
],
]
]
]
TrustKit.initSharedInstance(withConfiguration: trustKitConfig)
OkHttp (Android/JVM)
val certificatePinner = CertificatePinner.Builder()
.add("api.example.com",
"sha256/YLh1dUR9y6Kja30RrAn7JKnbQG/uEtLMkBgFF2Fuihg=", // Primary
"sha256/sRHdihwgkaib1P1gN7akajqIzMt0EjVfUmKkQ8ei1Sg=") // Backup
.build()
val client = OkHttpClient.Builder()
.certificatePinner(certificatePinner)
.build()
6.5 Pinning Bypass Techniques (RED Team / Testing)
| Technique | Tool | Notes |
|---|---|---|
| Frida script injection | frida -U -l ssl-unpin.js com.target.app |
Dynamic instrumentation, works on rooted/jailbroken devices |
| objection | objection -g com.target.app explore -s "android sslpinning disable" |
Frida-based, simplified CLI |
| Xposed/LSPosed module | TrustMeAlready, JustTrustMe | System-level hook, survives app restart |
| Magisk module | MagiskTrustUserCerts | Moves user CA certs to system store |
| Network Security Config override | Repackage APK with permissive config | Requires APK decompilation and re-signing |
| Reverse proxy + custom CA | mitmproxy, Burp Suite | Device configured to trust proxy CA |
| Binary patching | Patch pinning check in native library | For apps using native SSL pinning (e.g., libssl) |
Defensive countermeasures:
- Root/jailbreak detection (not bulletproof but raises bar)
- Certificate transparency verification in-app
- Binary integrity checks (tamper detection)
- Multiple pinning implementations (Java + native layer)
- Runtime integrity monitoring (detect Frida/Xposed)
7. Mutual TLS (mTLS)
7.1 Overview
Standard TLS authenticates only the server. mTLS adds client certificate authentication — both parties present and verify certificates.
Client Server
ClientHello -------->
ServerHello
Certificate (server)
CertificateRequest <-- server requests client cert
ServerHelloDone
Certificate (client)
CertificateVerify --------> Verify client cert
ChangeCipherSpec
Finished -------->
ChangeCipherSpec
<-------- Finished
[Application Data] <-------> [Application Data]
7.2 mTLS for Microservices — Implementation Patterns
Pattern 1: Service Mesh (Istio/Linkerd)
Service mesh injects sidecar proxies that handle mTLS transparently:
# Istio PeerAuthentication — enforce mTLS across namespace
apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
name: default
namespace: production
spec:
mtls:
mode: STRICT # STRICT = require mTLS, PERMISSIVE = accept both
---
# Istio AuthorizationPolicy — control which services can communicate
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
name: api-server-policy
namespace: production
spec:
selector:
matchLabels:
app: api-server
rules:
- from:
- source:
principals: ["cluster.local/ns/production/sa/frontend"]
to:
- operation:
methods: ["GET", "POST"]
paths: ["/api/*"]
Istio certificate management:
- istiod acts as CA, issues SPIFFE-format certificates to sidecars
- Certificates auto-rotate (default 24-hour lifetime)
- Identity format:
spiffe://cluster.local/ns/{namespace}/sa/{service-account} - No application code changes required
Pattern 2: Application-Level mTLS (Go)
package main
import (
"crypto/tls"
"crypto/x509"
"fmt"
"log"
"net/http"
"os"
)
func main() {
// Load CA certificate pool
caCert, err := os.ReadFile("/etc/certs/ca.pem")
if err != nil {
log.Fatal(err)
}
caCertPool := x509.NewCertPool()
caCertPool.AppendCertsFromPEM(caCert)
// Load server certificate and key
serverCert, err := tls.LoadX509KeyPair("/etc/certs/server.crt", "/etc/certs/server.key")
if err != nil {
log.Fatal(err)
}
// Configure mTLS
tlsConfig := &tls.Config{
Certificates: []tls.Certificate{serverCert},
ClientAuth: tls.RequireAndVerifyClientCert, // Require client certificate
ClientCAs: caCertPool, // CA pool for verifying client certs
MinVersion: tls.VersionTLS13,
}
server := &http.Server{
Addr: ":8443",
TLSConfig: tlsConfig,
Handler: http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
// Access verified client identity
clientCN := r.TLS.PeerCertificates[0].Subject.CommonName
fmt.Fprintf(w, "Hello, %s\n", clientCN)
}),
}
log.Fatal(server.ListenAndServeTLS("", ""))
}
// Client with mTLS
func createMTLSClient() (*http.Client, error) {
caCert, _ := os.ReadFile("/etc/certs/ca.pem")
caCertPool := x509.NewCertPool()
caCertPool.AppendCertsFromPEM(caCert)
clientCert, err := tls.LoadX509KeyPair("/etc/certs/client.crt", "/etc/certs/client.key")
if err != nil {
return nil, err
}
return &http.Client{
Transport: &http.Transport{
TLSClientConfig: &tls.Config{
Certificates: []tls.Certificate{clientCert},
RootCAs: caCertPool,
MinVersion: tls.VersionTLS13,
},
},
}, nil
}
Pattern 3: Nginx as mTLS Terminator
server {
listen 8443 ssl;
ssl_certificate /etc/nginx/certs/server.crt;
ssl_certificate_key /etc/nginx/certs/server.key;
ssl_protocols TLSv1.3;
# Client certificate verification
ssl_client_certificate /etc/nginx/certs/client-ca.pem;
ssl_verify_client on; # on = required, optional = allow but don't require
ssl_verify_depth 2; # Maximum chain depth for client certs
# Optional: CRL for client cert revocation
ssl_crl /etc/nginx/certs/client-crl.pem;
location / {
# Pass client identity to upstream
proxy_set_header X-Client-DN $ssl_client_s_dn;
proxy_set_header X-Client-CN $ssl_client_s_dn_cn;
proxy_set_header X-Client-Serial $ssl_client_serial;
proxy_set_header X-Client-Verify $ssl_client_verify;
proxy_pass http://backend;
}
}
Pattern 4: SPIFFE/SPIRE
SPIFFE (Secure Production Identity Framework for Everyone) provides workload identity:
SPIRE Server (central CA)
|
+-- SPIRE Agent (per node)
|
+-- Workload A: spiffe://trust-domain/service/api
+-- Workload B: spiffe://trust-domain/service/db
SPIFFE ID format: spiffe://trust-domain/path
# Register workload identity
spire-server entry create \
-spiffeID spiffe://example.com/service/api \
-parentID spiffe://example.com/agent/node1 \
-selector k8s:pod-label:app:api
# Workload fetches X.509 SVID (short-lived certificate)
# Libraries: go-spiffe, java-spiffe, py-spiffe
7.3 mTLS Certificate Generation with OpenSSL
# 1. Create CA key and certificate
openssl ecparam -genkey -name prime256v1 -noout -out ca.key
openssl req -new -x509 -key ca.key -out ca.crt -days 3650 \
-subj "/CN=Internal mTLS CA/O=Example Corp"
# 2. Create server certificate
openssl ecparam -genkey -name prime256v1 -noout -out server.key
openssl req -new -key server.key -out server.csr \
-subj "/CN=api.internal" \
-addext "subjectAltName=DNS:api.internal,DNS:localhost,IP:127.0.0.1"
openssl x509 -req -in server.csr -CA ca.crt -CAkey ca.key -CAcreateserial \
-out server.crt -days 365 -copy_extensions copyall
# 3. Create client certificate
openssl ecparam -genkey -name prime256v1 -noout -out client.key
openssl req -new -key client.key -out client.csr \
-subj "/CN=frontend-service/O=Example Corp"
openssl x509 -req -in client.csr -CA ca.crt -CAkey ca.key -CAcreateserial \
-out client.crt -days 365
# 4. Test mTLS connection
openssl s_client -connect api.internal:8443 \
-cert client.crt -key client.key -CAfile ca.crt
# 5. Test with curl
curl --cert client.crt --key client.key --cacert ca.crt \
https://api.internal:8443/health
8. PKI Architecture
8.1 Root CA Operations
Root CA security requirements:
- Air-gapped machine, no network connectivity
- Hardware Security Module (HSM) for key storage (FIPS 140-2 Level 3+)
- Physical security: safe, access logging, dual-person control
- Ceremony procedures: documented, witnessed, audited
- Used only to sign intermediate CA certificates and CRLs
- Typical validity: 20-30 years
Root CA key ceremony outline:
- Generate key pair inside HSM (never exportable)
- Create self-signed root certificate
- Export root certificate (public key only) for distribution
- Sign intermediate CA CSRs
- Secure HSM in vault
- Distribute root certificate to trust stores
8.2 Intermediate CA Design
Separation by purpose:
| Intermediate CA | Issues | Constraints |
|---|---|---|
| TLS Server CA | Server certificates | Name constraints to organizational domains |
| TLS Client CA | Client authentication certs | Extended key usage: clientAuth only |
| Code Signing CA | Code signing certificates | Extended key usage: codeSigning only |
| Email CA | S/MIME certificates | Extended key usage: emailProtection only |
Path length constraints:
Root CA (pathLenConstraint: 1)
└── Intermediate CA (pathLenConstraint: 0) <-- cannot sign further CAs
└── Leaf certificates only
8.3 Name Constraints
Name constraints (RFC 5280) restrict the namespaces an intermediate CA can issue certificates for:
# Intermediate CA certificate extensions
X509v3 Name Constraints: critical
Permitted:
DNS:.example.com
DNS:.internal.example.com
IP:10.0.0.0/8
email:.example.com
Excluded:
DNS:.evil.example.com
OpenSSL config for name-constrained intermediate:
# openssl.cnf excerpt for intermediate CA
[intermediate_ca_ext]
basicConstraints = critical, CA:TRUE, pathlen:0
keyUsage = critical, keyCertSign, cRLSign
nameConstraints = critical, permitted;DNS:.example.com, permitted;DNS:.internal.corp, excluded;DNS:.test.example.com
8.4 Cross-Signing
Cross-signing allows a new CA to be trusted by clients that only trust an old root:
Old Root CA (in all trust stores)
|
+-- Cross-signs --> New Root CA
|
+-- Intermediate CA
|
+-- Leaf cert
New Root CA (not yet in all trust stores)
|
+-- Intermediate CA
|
+-- Leaf cert (same cert, two trust paths)
Real-world example: Let's Encrypt ISRG Root X1 was cross-signed by IdenTrust DST Root CA X3 until ISRG Root was widely distributed. The cross-sign expired September 2021, causing issues with older Android devices.
8.5 Policy OIDs
Certificate policies identify the assurance level:
| Validation Level | OID | Assurance |
|---|---|---|
| Domain Validation (DV) | 2.23.140.1.2.1 | Domain control verified |
| Organization Validation (OV) | 2.23.140.1.2.2 | Organization identity verified |
| Extended Validation (EV) | CA-specific (e.g., 2.16.840.1.114412.2.1 for DigiCert) | Legal entity verified, prominent display |
Note: EV certificates no longer show green bar or company name in most browsers (Chrome removed this in 2019). The practical security value of EV over DV is debatable.
8.6 Private PKI with step-ca
step-ca provides a private ACME CA for internal infrastructure:
# Install
curl -sLO https://github.com/smallstep/certificates/releases/latest/download/step-ca_linux_amd64.tar.gz
# Initialize CA
step ca init --name="Internal CA" --dns="ca.internal" --address=":8443"
# Start CA
step-ca $(step path)/config/ca.json
# Issue certificate
step ca certificate "api.internal" api.crt api.key
# Renew (daemon mode — auto-renew before expiry)
step ca renew --daemon api.crt api.key
# SSH certificates
step ca ssh certificate user@example.com id_ecdsa --principal user --principal admin
# Revoke
step ca revoke --cert api.crt --key api.key
Integration pattern for automated TLS:
- Deploy step-ca as intermediate CA under existing root
- Use ACME provisioner for automated cert issuance
- Set certificate lifetime to 24 hours with automated renewal
- Short-lived certs eliminate need for CRL/OCSP infrastructure
8.7 HashiCorp Vault PKI Engine
Vault as an internal CA:
# Enable PKI engine
vault secrets enable pki
# Set max TTL
vault secrets tune -max-lease-ttl=87600h pki
# Generate root CA (or import existing)
vault write pki/root/generate/internal \
common_name="Internal Root CA" \
ttl=87600h
# Enable intermediate CA
vault secrets enable -path=pki_int pki
vault write pki_int/intermediate/generate/internal \
common_name="Internal Intermediate CA"
# Sign intermediate with root
vault write pki/root/sign-intermediate \
csr=@pki_int.csr \
ttl=43800h
# Create role for issuing certificates
vault write pki_int/roles/server-cert \
allowed_domains="internal.example.com" \
allow_subdomains=true \
max_ttl=720h \
key_type=ec \
key_bits=256
# Issue certificate
vault write pki_int/issue/server-cert \
common_name="api.internal.example.com" \
ttl=24h
9. Key Management
9.1 Key Management Lifecycle
Generation -> Storage -> Distribution -> Usage -> Rotation -> Destruction
| | | | | |
v v v v v v
CSPRNG HSM/KMS Encrypted Single Automated Secure
FIPS 140 Vault transport purpose versioned zeroize
validated Never (TLS/SSH) only re-wrap memory +
plaintext DEKs media
9.2 Envelope Encryption
┌─────────────────────────────────────────┐
│ Application Layer │
│ ┌─────────┐ ┌──────────────────┐ │
│ │ Plaintext│───>│ Encrypt with DEK │ │
│ └─────────┘ └────────┬─────────┘ │
│ │ │
│ ┌─────────▼─────────┐ │
│ │ Encrypted Data │ │
│ │ + Encrypted DEK │────│──> Storage
│ └───────────────────┘ │
│ ▲ │
│ ┌─────────┴─────────┐ │
│ │ Wrap DEK with KEK │ │
│ └─────────┬─────────┘ │
│ │ │
│ ┌─────────▼─────────┐ │
│ │ Vault / KMS / HSM │ │ KEK never leaves
│ └───────────────────┘ │ secure boundary
└─────────────────────────────────────────┘
- DEK (Data Encryption Key): generated per record/file, encrypts data
- KEK (Key Encryption Key): stored in Vault/KMS/HSM, encrypts DEK
- Encrypted DEK stored alongside ciphertext
- KEK rotation does not require re-encrypting all data — only re-wrap DEKs
9.3 Cloud KMS
AWS KMS
# Create key
aws kms create-key --description "Application encryption key" --key-spec AES_256
# Encrypt data
aws kms encrypt --key-id alias/app-key --plaintext fileb://secret.txt --output text --query CiphertextBlob | base64 -d > secret.enc
# Decrypt data
aws kms decrypt --ciphertext-blob fileb://secret.enc --output text --query Plaintext | base64 -d > secret.txt
# Generate data key (envelope encryption)
aws kms generate-data-key --key-id alias/app-key --key-spec AES_256
# Returns: Plaintext (DEK) + CiphertextBlob (encrypted DEK)
# Use Plaintext to encrypt data, store CiphertextBlob alongside, zeroize Plaintext from memory
# Key rotation (automatic annual rotation)
aws kms enable-key-rotation --key-id alias/app-key
GCP Cloud KMS
# Create keyring and key
gcloud kms keyrings create my-keyring --location=global
gcloud kms keys create app-key --keyring=my-keyring --location=global \
--purpose=encryption --rotation-period=90d --next-rotation-time=$(date -u +%Y-%m-%dT%H:%M:%SZ -d "+90 days")
# Encrypt
gcloud kms encrypt --key=app-key --keyring=my-keyring --location=global \
--plaintext-file=secret.txt --ciphertext-file=secret.enc
# Decrypt
gcloud kms decrypt --key=app-key --keyring=my-keyring --location=global \
--ciphertext-file=secret.enc --plaintext-file=secret.txt
9.4 Hardware Security Modules (HSMs)
| HSM Type | FIPS Level | Use Case | Cost |
|---|---|---|---|
| AWS CloudHSM | 140-2 Level 3 | Cloud-native, PKCS#11 | ~$1.50/hr |
| Azure Dedicated HSM | 140-2 Level 3 | Cloud-native | ~$4.85/hr |
| GCP Cloud HSM | 140-2 Level 3 | Cloud KMS backed by HSM | Per-operation pricing |
| YubiHSM 2 | 140-2 Level 3 | Small deployments, dev | ~$650 one-time |
| Thales Luna | 140-3 Level 3 | Enterprise, on-prem | $$$$ |
| nCipher nShield | 140-3 Level 3 | Enterprise, on-prem | $$$$ |
HSM key properties:
- Key generated inside HSM, never exportable in plaintext
- Cryptographic operations happen inside HSM boundary
- Tamper-evident and tamper-resistant physical enclosure
- Zeroization on tamper detection
9.5 Vault Transit Engine (Encryption as a Service)
Application never sees the encryption key:
# Enable transit
vault secrets enable transit
# Create encryption key
vault write -f transit/keys/app-data type=aes256-gcm96
# Encrypt
vault write transit/encrypt/app-data \
plaintext=$(echo "sensitive data" | base64)
# Decrypt
vault write transit/decrypt/app-data \
ciphertext="vault:v1:..."
# Key rotation (new version, old versions still decrypt)
vault write -f transit/keys/app-data/rotate
# Set minimum decryption version (effectively destroys old key versions)
vault write transit/keys/app-data \
min_decryption_version=3
# Re-wrap ciphertext to latest key version (without revealing plaintext)
vault write transit/rewrap/app-data \
ciphertext="vault:v1:..."
9.6 Key Rotation Triggers
- Scheduled cryptoperiod expiry (NIST SP 800-57 guidelines)
- Suspected or confirmed compromise
- Personnel changes (departures, role changes)
- Algorithm deprecation
- Data volume threshold (~2^32 blocks for AES-GCM with random nonces)
- Regulatory requirement (e.g., PCI DSS requires annual rotation)
9.7 Key Destruction
- Overwrite key material in memory before deallocation
- Cryptographic erasure for storage media (destroy KEK to render encrypted data irrecoverable)
- Document destruction in audit log
- HSM: use zeroize command per FIPS 140 requirements
- Verify destruction: attempt decryption after destruction to confirm failure
10. Post-Quantum Cryptography
10.1 NIST PQC Standards
| Standard | Algorithm | Type | FIPS | Status |
|---|---|---|---|---|
| ML-KEM | CRYSTALS-Kyber | Key Encapsulation | FIPS 203 | Standardized (2024) |
| ML-DSA | CRYSTALS-Dilithium | Digital Signature | FIPS 204 | Standardized (2024) |
| SLH-DSA | SPHINCS+ | Digital Signature (hash-based) | FIPS 205 | Standardized (2024) |
| FN-DSA | FALCON | Digital Signature (lattice) | Draft | Expected 2025 |
10.2 ML-KEM (CRYSTALS-Kyber) — Key Encapsulation
| Parameter Set | Security Level | Public Key | Ciphertext | Shared Secret |
|---|---|---|---|---|
| ML-KEM-512 | ~AES-128 | 800 bytes | 768 bytes | 32 bytes |
| ML-KEM-768 | ~AES-192 | 1,184 bytes | 1,088 bytes | 32 bytes |
| ML-KEM-1024 | ~AES-256 | 1,568 bytes | 1,568 bytes | 32 bytes |
Recommended: ML-KEM-768 for general use, ML-KEM-1024 for high-security contexts.
10.3 ML-DSA (CRYSTALS-Dilithium) — Digital Signatures
| Parameter Set | Security Level | Public Key | Signature |
|---|---|---|---|
| ML-DSA-44 | ~AES-128 | 1,312 bytes | 2,420 bytes |
| ML-DSA-65 | ~AES-192 | 1,952 bytes | 3,293 bytes |
| ML-DSA-87 | ~AES-256 | 2,592 bytes | 4,595 bytes |
Size comparison with classical algorithms:
| Algorithm | Public Key | Signature |
|---|---|---|
| Ed25519 | 32 bytes | 64 bytes |
| RSA-3072 | 384 bytes | 384 bytes |
| ML-DSA-65 | 1,952 bytes | 3,293 bytes |
| SLH-DSA-SHA2-128s | 32 bytes | 7,856 bytes |
PQ signatures are significantly larger — impacts certificate size, TLS handshake, bandwidth.
10.4 SLH-DSA (SPHINCS+) — Stateless Hash-Based Signatures
Conservative choice — security relies only on hash function security (no lattice assumptions):
| Parameter Set | Public Key | Signature | Signing Speed |
|---|---|---|---|
| SLH-DSA-SHA2-128s | 32 bytes | 7,856 bytes | Slow |
| SLH-DSA-SHA2-128f | 32 bytes | 17,088 bytes | Fast |
| SLH-DSA-SHA2-256s | 64 bytes | 29,792 bytes | Slow |
Trade-off: Small public keys but very large signatures. Use when lattice-based assumptions are considered too risky.
10.5 Hybrid Mode — Transition Strategy
Combine classical and PQ algorithms so security holds if either is unbroken:
Hybrid Key Exchange:
shared_secret = HKDF(X25519_shared_secret || ML-KEM-768_shared_secret)
Hybrid Signature:
valid = Ed25519_verify(msg, sig1) AND ML-DSA-65_verify(msg, sig2)
Current hybrid implementations:
- TLS: X25519Kyber768Draft00 (Chrome, Firefox — experimental)
- SSH: sntrup761x25519-sha512@openssh.com (OpenSSH 9.0+)
- age:
-pqflag uses X25519 + ML-KEM-768 - Signal Protocol: PQXDH using X25519 + ML-KEM-768
10.6 Migration Planning
NSA CNSA 2.0 Timeline:
| Capability | Transition | Exclusive Use |
|---|---|---|
| Software/firmware signing | 2025 | 2030 |
| Web browsers/servers (TLS) | 2025 | 2030 |
| Cloud services | 2025 | 2030 |
| Networking equipment (VPN, routers) | 2026 | 2030 |
| Operating systems | 2027 | 2033 |
| Niche equipment | 2028 | 2033 |
Migration steps:
- Inventory: catalog all cryptographic usage (algorithms, key sizes, protocols)
- Risk assessment: identify harvest-now-decrypt-later exposure (long-lived secrets are highest priority)
- Test hybrid modes: deploy X25519+ML-KEM-768 in TLS where supported
- Update libraries: ensure crypto libraries support FIPS 203/204/205
- Certificate migration: plan for PQ certificates (larger sizes impact handshake performance)
- Monitor standards: NIST may revise recommendations as cryptanalysis progresses
Harvest-now-decrypt-later (HNDL) threat: Nation-state adversaries may be recording encrypted traffic today to decrypt when quantum computers become available. Data with long confidentiality requirements (state secrets, medical records, financial data) should use hybrid PQ encryption NOW. [CONFIRMED threat model — NSA, GCHQ, BSI all acknowledge this risk]
11. TLS Attacks — Deep Dive
11.1 Attack Reference Table
| Attack | CVE/Ref | Affected | Root Cause | Mitigation |
|---|---|---|---|---|
| BEAST | CVE-2011-3389 | TLS 1.0 CBC | Predictable IV in CBC | TLS 1.1+ (or 1/n-1 split) |
| CRIME | CVE-2012-4929 | TLS compression | Compression oracle | Disable TLS compression |
| BREACH | CVE-2013-3587 | HTTP compression | Compression oracle on HTTP body | Disable HTTP compression on secrets, SameSite cookies, CSRF tokens |
| Lucky13 | CVE-2013-0169 | TLS CBC | Timing side-channel in MAC verification | Use AEAD ciphers, constant-time implementations |
| POODLE | CVE-2014-3566 | SSL 3.0 CBC | Padding oracle | Disable SSL 3.0 |
| Heartbleed | CVE-2014-0160 | OpenSSL 1.0.1-1.0.1f | Buffer over-read in heartbeat extension | Patch OpenSSL, rotate ALL keys and certs |
| FREAK | CVE-2015-0204 | RSA export ciphers | Downgrade to 512-bit RSA | Disable export cipher suites |
| Logjam | CVE-2015-4000 | DHE export + small groups | Downgrade to 512-bit DH, precomputation on common 1024-bit groups | DH >= 2048-bit, prefer ECDHE |
| DROWN | CVE-2016-0800 | SSL 2.0 | Cross-protocol attack using SSLv2 | Disable SSL 2.0 on ALL servers sharing a private key |
| Sweet32 | CVE-2016-2183 | 3DES, Blowfish | Birthday attack on 64-bit block ciphers | Disable 64-bit block ciphers |
| ROBOT | CVE-2017-13099 | RSA key exchange | Bleichenbacher oracle in RSA PKCS#1 v1.5 | Disable RSA key exchange, use ECDHE |
| Raccoon | CVE-2020-1968 | DH key exchange | Timing side-channel in DH | DH >= 2048-bit, prefer TLS 1.3 |
| ALPACA | CVE-2021-3618 | Cross-protocol (TLS) | Application layer confusion between TLS services | Separate certificates per service, SNI enforcement |
11.2 Heartbleed — Deep Dive
CVE-2014-0160 — affected OpenSSL 1.0.1 through 1.0.1f (April 2014).
Mechanism: TLS Heartbeat extension (RFC 6520) allows keep-alive messages. The client sends a heartbeat request with a payload and declares its length. Vulnerable OpenSSL trusted the declared length without bounds checking, returning up to 64KB of adjacent server memory.
What leaked:
- Server private keys (confirmed by CloudFlare challenge)
- Session cookies and tokens
- User credentials in transit
- Other users' request data from the same process
Detection:
# Test for Heartbleed
nmap -p 443 --script ssl-heartbleed example.com
# With testssl.sh
./testssl.sh --heartbleed example.com
# OpenSSL direct test
openssl s_client -connect example.com:443 -tlsextdebug 2>&1 | grep -i heartbeat
Response checklist:
- Patch OpenSSL immediately
- Regenerate ALL private keys (assume compromised)
- Revoke and reissue ALL certificates
- Invalidate ALL session tokens
- Force password resets for affected services
- Check for unauthorized access using leaked credentials
11.3 ROBOT Attack
Return Of Bleichenbacher's Oracle Threat (CVE-2017-13099).
RSA PKCS#1 v1.5 encryption used in TLS RSA key exchange is vulnerable to adaptive chosen-ciphertext attacks. The attacker sends modified ciphertexts and observes different error responses to gradually decrypt the pre-master secret.
Mitigation: Disable RSA key exchange entirely. Use ECDHE or DHE cipher suites only.
# Test for ROBOT
./testssl.sh --robot example.com
# Check if RSA key exchange is enabled
nmap -p 443 --script ssl-enum-ciphers example.com | grep -i "RSA" | grep -v "ECDHE\|DHE"
11.4 Renegotiation Attacks
CVE-2009-3555 — TLS renegotiation allows injecting plaintext before a legitimate client's traffic.
Mitigation: Require secure renegotiation (RFC 5746). TLS 1.3 removes renegotiation entirely — uses KeyUpdate message for rekeying.
# Nginx — disable client-initiated renegotiation (default in modern versions)
# No explicit config needed; nginx does not support client renegotiation
11.5 Downgrade Attacks
TLS_FALLBACK_SCSV (RFC 7507) prevents protocol downgrade:
When a client retries a connection with a lower TLS version, it includes the SCSV cipher suite value. If the server supports a higher version, it rejects the connection.
TLS 1.3 downgrade protection: The server includes a specific sentinel value in the ServerHello random field when negotiating TLS 1.2 or below. TLS 1.3-capable clients detect this and abort.
12. TLS/SSL Testing and Auditing
12.1 testssl.sh
Comprehensive TLS/SSL scanner — no dependencies beyond bash and OpenSSL.
# Install
git clone --depth 1 https://github.com/drwetter/testssl.sh.git
# Basic scan
./testssl.sh example.com
# Specific checks
./testssl.sh --protocols example.com # Protocol support
./testssl.sh --ciphers example.com # Cipher suites
./testssl.sh --vulnerabilities example.com # Known vulnerabilities
./testssl.sh --headers example.com # Security headers
./testssl.sh --server-defaults example.com # Certificate details
./testssl.sh --fs example.com # Forward secrecy ciphers
./testssl.sh --rc4 example.com # RC4 cipher usage
# Full scan with JSON output
./testssl.sh --jsonfile results.json example.com
# CSV output for spreadsheet analysis
./testssl.sh --csvfile results.csv example.com
# Batch scan
./testssl.sh --file hosts.txt --parallel 10
# STARTTLS protocols
./testssl.sh --starttls smtp mail.example.com:25
./testssl.sh --starttls imap mail.example.com:143
./testssl.sh --starttls ftp ftp.example.com:21
./testssl.sh --starttls xmpp chat.example.com:5222
# Specific vulnerability checks
./testssl.sh --heartbleed example.com
./testssl.sh --robot example.com
./testssl.sh --poodle example.com
./testssl.sh --beast example.com
./testssl.sh --drown example.com
# Check specific IP (bypass DNS)
./testssl.sh --ip 1.2.3.4 example.com
# Use specific OpenSSL binary
./testssl.sh --openssl /opt/openssl-3.0/bin/openssl example.com
# Quiet mode for CI/CD (exit code indicates severity)
./testssl.sh --quiet --severity HIGH example.com
echo $? # 0=OK, non-zero=issues found
12.2 SSLyze
Python-based TLS scanner with CI/CD integration:
# Install
pip install sslyze
# Basic scan
python -m sslyze example.com
# Mozilla compliance check
python -m sslyze --mozilla_config=intermediate example.com
python -m sslyze --mozilla_config=modern example.com
# JSON output
python -m sslyze --json_out=results.json example.com
# STARTTLS
python -m sslyze --starttls smtp mail.example.com:25
# Multiple targets
python -m sslyze example.com:443 api.example.com:443 mail.example.com:465
Python API for automation:
from sslyze import Scanner, ServerScanRequest, ServerNetworkLocation
from sslyze.plugins.scan_commands import ScanCommand
server = ServerNetworkLocation("example.com", 443)
request = ServerScanRequest(server, {
ScanCommand.CERTIFICATE_INFO,
ScanCommand.SSL_2_0_CIPHER_SUITES,
ScanCommand.TLS_1_3_CIPHER_SUITES,
ScanCommand.HEARTBLEED,
ScanCommand.ROBOT,
ScanCommand.TLS_COMPRESSION,
ScanCommand.HTTP_HEADERS,
})
scanner = Scanner()
scanner.queue_scans([request])
for result in scanner.get_results():
cert_info = result.scan_result.certificate_info
heartbleed = result.scan_result.heartbleed
robot = result.scan_result.robot
if heartbleed.result.is_vulnerable_to_heartbleed:
print("CRITICAL: Vulnerable to Heartbleed!")
if robot.result.robot_result.name != "NOT_VULNERABLE_NO_ORACLE":
print("CRITICAL: Vulnerable to ROBOT!")
12.3 Nmap SSL Scripts
# Enumerate cipher suites
nmap -p 443 --script ssl-enum-ciphers example.com
# Check for known vulnerabilities
nmap -p 443 --script ssl-heartbleed example.com
nmap -p 443 --script ssl-poodle example.com
nmap -p 443 --script ssl-dh-params example.com
nmap -p 443 --script ssl-ccs-injection example.com
# Certificate details
nmap -p 443 --script ssl-cert example.com
# Combined scan
nmap -p 443 --script "ssl-*" example.com
# Check multiple ports
nmap -p 443,8443,465,993,995 --script ssl-enum-ciphers example.com
12.4 OpenSSL s_client Commands
# Basic connection test
openssl s_client -connect example.com:443 -servername example.com < /dev/null
# Force specific TLS version
openssl s_client -connect example.com:443 -tls1_2
openssl s_client -connect example.com:443 -tls1_3
# Show full certificate chain
openssl s_client -connect example.com:443 -servername example.com -showcerts < /dev/null
# Check OCSP stapling
openssl s_client -connect example.com:443 -servername example.com -status < /dev/null 2>/dev/null | grep -A 20 "OCSP"
# Test specific cipher suite
openssl s_client -connect example.com:443 -cipher ECDHE-RSA-AES256-GCM-SHA384
openssl s_client -connect example.com:443 -ciphersuites TLS_AES_256_GCM_SHA384 # TLS 1.3
# STARTTLS
openssl s_client -connect mail.example.com:25 -starttls smtp
openssl s_client -connect mail.example.com:143 -starttls imap
openssl s_client -connect mail.example.com:587 -starttls smtp
# Verify certificate against CA bundle
openssl s_client -connect example.com:443 -CAfile /etc/ssl/certs/ca-certificates.crt -verify_return_error
# Check certificate dates
echo | openssl s_client -connect example.com:443 -servername example.com 2>/dev/null | \
openssl x509 -noout -dates
# Extract certificate in PEM format
echo | openssl s_client -connect example.com:443 -servername example.com 2>/dev/null | \
openssl x509 -outform PEM > example.com.pem
# Check if server supports session resumption
openssl s_client -connect example.com:443 -reconnect -no_ticket 2>/dev/null | grep -i "reuse"
# Test with client certificate (mTLS)
openssl s_client -connect api.internal:8443 \
-cert client.crt -key client.key -CAfile ca.pem
# Check supported curves/groups
openssl s_client -connect example.com:443 -curves X25519 < /dev/null
12.5 Online Testing Tools
| Tool | URL | Use |
|---|---|---|
| SSL Labs | ssllabs.com/ssltest | Comprehensive public server analysis (A+ grading) |
| Hardenize | hardenize.com | TLS + email + DNS security |
| Mozilla Observatory | observatory.mozilla.org | Headers + TLS + third-party |
| ImmuniWeb | immuniweb.com/ssl | PCI DSS compliance check |
| CryptCheck | tls.imirhil.fr | French alternative to SSL Labs |
| testssl.sh online | testssl.sh (self-hosted) | Private scanning |
12.6 CI/CD Integration Script
#!/bin/bash
# tls-check.sh — CI/CD TLS compliance checker
# Exit codes: 0 = pass, 1 = fail
TARGET="${1:?Usage: tls-check.sh hostname:port}"
FAILURES=0
echo "=== TLS Compliance Check: ${TARGET} ==="
# Check protocol support
echo "[*] Checking protocols..."
if openssl s_client -connect "${TARGET}" -tls1 < /dev/null 2>&1 | grep -q "CONNECTED"; then
echo " FAIL: TLS 1.0 is enabled"
FAILURES=$((FAILURES + 1))
fi
if openssl s_client -connect "${TARGET}" -tls1_1 < /dev/null 2>&1 | grep -q "CONNECTED"; then
echo " FAIL: TLS 1.1 is enabled"
FAILURES=$((FAILURES + 1))
fi
if ! openssl s_client -connect "${TARGET}" -tls1_3 < /dev/null 2>&1 | grep -q "CONNECTED"; then
echo " FAIL: TLS 1.3 is NOT supported"
FAILURES=$((FAILURES + 1))
fi
# Check certificate expiry
echo "[*] Checking certificate expiry..."
EXPIRY=$(echo | openssl s_client -connect "${TARGET}" -servername "${TARGET%%:*}" 2>/dev/null | \
openssl x509 -noout -enddate 2>/dev/null | cut -d= -f2)
if [ -n "$EXPIRY" ]; then
EXPIRY_EPOCH=$(date -d "$EXPIRY" +%s 2>/dev/null)
NOW_EPOCH=$(date +%s)
DAYS_LEFT=$(( (EXPIRY_EPOCH - NOW_EPOCH) / 86400 ))
if [ "$DAYS_LEFT" -lt 30 ]; then
echo " WARN: Certificate expires in ${DAYS_LEFT} days"
FAILURES=$((FAILURES + 1))
else
echo " OK: Certificate expires in ${DAYS_LEFT} days"
fi
fi
# Check HSTS
echo "[*] Checking HSTS..."
if ! curl -sI "https://${TARGET%%:*}" 2>/dev/null | grep -qi "strict-transport-security"; then
echo " FAIL: HSTS header missing"
FAILURES=$((FAILURES + 1))
fi
echo ""
if [ "$FAILURES" -gt 0 ]; then
echo "RESULT: ${FAILURES} issue(s) found"
exit 1
else
echo "RESULT: All checks passed"
exit 0
fi
13. Code Signing and Software Integrity
13.1 GPG/PGP Signing
# Generate signing key
gpg --full-generate-key # Select Ed25519 or RSA 4096
# Sign a file (detached signature)
gpg --armor --detach-sign --output file.sig file.tar.gz
# Verify signature
gpg --verify file.sig file.tar.gz
# Sign a git commit
git commit -S -m "Signed commit"
# Sign a git tag
git tag -s v1.0.0 -m "Release v1.0.0"
# Verify git signature
git log --show-signature -1
git tag -v v1.0.0
# Export public key for distribution
gpg --armor --export user@example.com > pubkey.asc
13.2 Sigstore — Keyless Signing
Sigstore eliminates key management for code signing by using ephemeral keys tied to OIDC identity:
Developer authenticates via OIDC (GitHub, Google, etc.)
|
v
Fulcio CA issues short-lived signing certificate (10 minutes)
|
v
Developer signs artifact with ephemeral key
|
v
Signature + certificate logged in Rekor transparency log
|
v
Ephemeral private key destroyed
cosign (container signing):
# Install
go install github.com/sigstore/cosign/v2/cmd/cosign@latest
# Keyless signing (OIDC-based)
cosign sign ghcr.io/example/app:v1.0.0
# Verify
cosign verify ghcr.io/example/app:v1.0.0 \
--certificate-identity=user@example.com \
--certificate-oidc-issuer=https://accounts.google.com
# Key-based signing (traditional)
cosign generate-key-pair
cosign sign --key cosign.key ghcr.io/example/app:v1.0.0
cosign verify --key cosign.pub ghcr.io/example/app:v1.0.0
# Sign a blob (non-container)
cosign sign-blob --bundle artifact.bundle artifact.tar.gz
cosign verify-blob --bundle artifact.bundle artifact.tar.gz \
--certificate-identity=user@example.com \
--certificate-oidc-issuer=https://accounts.google.com
13.3 Notary v2 (OCI Artifacts)
# Sign OCI image
notation sign ghcr.io/example/app:v1.0.0
# Verify
notation verify ghcr.io/example/app:v1.0.0
# List signatures
notation list ghcr.io/example/app:v1.0.0
13.4 SLSA Framework (Supply Chain Levels for Software Artifacts)
| Level | Requirements |
|---|---|
| SLSA 1 | Build process documented, provenance generated |
| SLSA 2 | Hosted build service, authenticated provenance |
| SLSA 3 | Hardened build platform, non-falsifiable provenance |
| SLSA 4 | Hermetic builds, two-person review |
Provenance verification:
# Verify SLSA provenance with slsa-verifier
slsa-verifier verify-artifact artifact.tar.gz \
--provenance-path provenance.intoto.jsonl \
--source-uri github.com/example/repo \
--source-tag v1.0.0
14. Encryption at Rest
14.1 Full Disk Encryption (FDE)
LUKS (Linux Unified Key Setup)
# Create encrypted partition
cryptsetup luksFormat /dev/sda2
# Open (decrypt) partition
cryptsetup luksOpen /dev/sda2 encrypted_root
# Create filesystem
mkfs.ext4 /dev/mapper/encrypted_root
# Mount
mount /dev/mapper/encrypted_root /mnt
# Add additional key slot (backup passphrase)
cryptsetup luksAddKey /dev/sda2
# Remove key slot
cryptsetup luksRemoveKey /dev/sda2
# Check LUKS header info
cryptsetup luksDump /dev/sda2
# Backup LUKS header (CRITICAL for recovery)
cryptsetup luksHeaderBackup /dev/sda2 --header-backup-file luks-header.bak
# Use keyfile instead of passphrase
dd if=/dev/urandom of=/root/keyfile bs=4096 count=1
chmod 600 /root/keyfile
cryptsetup luksAddKey /dev/sda2 /root/keyfile
LUKS2 with Argon2id (default in modern distros):
cryptsetup luksFormat --type luks2 --pbkdf argon2id \
--pbkdf-memory 1048576 --pbkdf-parallel 4 /dev/sda2
BitLocker (Windows)
# Enable BitLocker with TPM
Enable-BitLocker -MountPoint "C:" -EncryptionMethod XtsAes256 -TpmProtector
# Enable with TPM + PIN
Enable-BitLocker -MountPoint "C:" -EncryptionMethod XtsAes256 -TpmAndPinProtector -Pin (Read-Host -AsSecureString)
# Check status
Get-BitLockerVolume
# Backup recovery key to AD
Backup-BitLockerKeyProtector -MountPoint "C:" -KeyProtectorId $KeyProtectorId
FileVault (macOS)
# Enable FileVault
sudo fdesetup enable
# Check status
fdesetup status
# List recovery key
sudo fdesetup showrecoverykey
# Verify encryption
diskutil apfs list | grep -i "FileVault"
14.2 File-Based Encryption (FBE)
fscrypt (Linux, ext4/F2FS):
# Setup fscrypt on filesystem
sudo tune2fs -O encrypt /dev/sda2
fscrypt setup
fscrypt setup /mnt/data
# Encrypt a directory
fscrypt encrypt /mnt/data/secrets
# Lock (remove keys from kernel)
fscrypt lock /mnt/data/secrets
# Unlock
fscrypt unlock /mnt/data/secrets
Use case: Multi-user systems where different users' data should be encrypted with different keys (e.g., Android uses FBE so each user profile is independently encrypted).
14.3 Database Transparent Data Encryption (TDE)
PostgreSQL (pgcrypto + application-level)
-- Application-level encryption with pgcrypto
CREATE EXTENSION IF NOT EXISTS pgcrypto;
-- Encrypt sensitive column
INSERT INTO users (email, ssn_encrypted)
VALUES (
'user@example.com',
pgp_sym_encrypt('123-45-6789', 'encryption-key-from-vault')
);
-- Decrypt
SELECT email, pgp_sym_decrypt(ssn_encrypted::bytea, 'encryption-key-from-vault') as ssn
FROM users WHERE email = 'user@example.com';
MySQL/MariaDB TDE
-- Enable TDE (InnoDB tablespace encryption)
-- my.cnf:
-- [mysqld]
-- early-plugin-load=keyring_file.so
-- keyring_file_data=/var/lib/mysql-keyring/keyring
ALTER TABLE sensitive_data ENCRYPTION='Y';
-- Verify
SELECT TABLE_SCHEMA, TABLE_NAME, CREATE_OPTIONS
FROM INFORMATION_SCHEMA.TABLES
WHERE CREATE_OPTIONS LIKE '%ENCRYPTION%';
MongoDB Encrypted Storage Engine
# mongod.conf
security:
enableEncryption: true
encryptionKeyFile: /etc/mongodb/encryption-key
# Or use KMIP for external key management
# kmip:
# serverName: kmip.example.com
# port: 5696
# clientCertificateFile: /etc/mongodb/client.pem
14.4 Application-Level Encryption
Best practice for sensitive fields (PII, financial data, health records):
# Python example using cryptography library
from cryptography.fernet import Fernet
from cryptography.hazmat.primitives import hashes
from cryptography.hazmat.primitives.kdf.pbkdf2 import PBKDF2HMAC
import base64
import os
# In practice, retrieve KEK from Vault/KMS
def get_kek_from_vault() -> bytes:
"""Retrieve Key Encryption Key from secrets manager."""
# vault_client.read('transit/keys/app-data')
raise NotImplementedError("Integrate with your secrets manager")
def generate_dek() -> tuple[bytes, bytes]:
"""Generate a Data Encryption Key and encrypt it with KEK."""
dek = Fernet.generate_key()
# In production: encrypt DEK with KEK via Vault Transit
encrypted_dek = dek # placeholder — use vault transit/encrypt
return dek, encrypted_dek
def encrypt_field(plaintext: str, dek: bytes) -> bytes:
"""Encrypt a single field value."""
f = Fernet(dek)
return f.encrypt(plaintext.encode('utf-8'))
def decrypt_field(ciphertext: bytes, dek: bytes) -> str:
"""Decrypt a single field value."""
f = Fernet(dek)
return f.decrypt(ciphertext).decode('utf-8')
15. FIPS 140-2/3 Compliance
15.1 Overview
| Aspect | FIPS 140-2 | FIPS 140-3 |
|---|---|---|
| Standard | Published 2001 | Published 2019, mandatory for new submissions since 2021 |
| Levels | 1-4 | 1-4 (enhanced requirements) |
| Testing | CMVP (NIST + CSE) | CMVP with ISO/IEC 19790 alignment |
| Physical security | Level 2+ | Enhanced at all levels |
15.2 FIPS 140 Security Levels
| Level | Physical | Key Management | Use Case |
|---|---|---|---|
| 1 | No physical security | Software implementation | Software crypto modules |
| 2 | Tamper-evident seals | Role-based authentication | Cloud HSMs, server modules |
| 3 | Tamper-resistant | Identity-based authentication, key zeroization | Payment processing, PKI |
| 4 | Environmental failure protection | Complete physical security envelope | Military, classified systems |
15.3 FIPS-Approved Algorithms
| Category | Approved | NOT Approved |
|---|---|---|
| Symmetric | AES (128/192/256), 3DES (legacy only) | ChaCha20, Blowfish, Twofish |
| Hash | SHA-2 family, SHA-3 family | MD5, BLAKE2/3 |
| MAC | HMAC, CMAC, GMAC | Poly1305 |
| Signatures | RSA (2048+), ECDSA, EdDSA, ML-DSA, SLH-DSA | — |
| Key Agreement | DH (2048+), ECDH, ML-KEM | X25519 (under review) |
| KDF | PBKDF2, HKDF, SP 800-108 KDF | Argon2, scrypt, bcrypt |
| RNG | SP 800-90A (CTR_DRBG, Hash_DRBG, HMAC_DRBG) | Dual_EC_DRBG (withdrawn) |
Critical note: ChaCha20-Poly1305 is NOT FIPS-approved. FIPS environments must use AES-GCM. This affects TLS cipher suite selection.
15.4 FIPS Mode in Practice
OpenSSL FIPS provider (3.0+):
# Check if FIPS provider is available
openssl list -providers
# Enable FIPS mode in openssl.cnf
# [openssl_init]
# providers = provider_sect
# [provider_sect]
# fips = fips_sect
# base = base_sect
# [fips_sect]
# activate = 1
# Test FIPS mode
openssl list -providers | grep fips
Go FIPS mode:
// Use go-crypto-fips or boringcrypto build tag
// Build with: CGO_ENABLED=1 GOEXPERIMENT=boringcrypto go build
Java FIPS (Bouncy Castle FIPS):
// Add BC-FIPS provider
Security.addProvider(new BouncyCastleFipsProvider());
// Use FIPS-approved algorithms only
Cipher cipher = Cipher.getInstance("AES/GCM/NoPadding", "BCFIPS");
Linux FIPS mode:
# Enable FIPS mode (RHEL/CentOS)
fips-mode-setup --enable
reboot
# Verify
fips-mode-setup --check
# or
cat /proc/sys/crypto/fips_enabled # 1 = enabled
# Ubuntu
sudo ua enable fips
15.5 Common FIPS Compliance Pitfalls
| Pitfall | Impact | Fix |
|---|---|---|
| Using ChaCha20-Poly1305 | Non-compliant | Use AES-GCM |
| Using Argon2id for password hashing | Non-compliant | Use PBKDF2-HMAC-SHA256 (600K+ iterations) |
| Using Ed25519/X25519 | Uncertain (not yet validated in all modules) | Use ECDSA P-256/P-384 and ECDH P-256/P-384 |
| Non-FIPS OpenSSL build | Entire crypto stack non-compliant | Build with FIPS provider enabled |
| Using /dev/urandom directly | May not meet SP 800-90A | Use FIPS-validated DRBG |
| Self-tests disabled | Module validation invalid | Ensure power-on self-tests run |
16. Secure Random Number Generation
16.1 CSPRNG by Language
| Language | Secure API | AVOID |
|---|---|---|
| Python | secrets module, os.urandom() |
random module |
| Go | crypto/rand |
math/rand (even math/rand/v2 is not crypto-safe) |
| Node.js | crypto.randomBytes(), crypto.randomInt(), crypto.randomUUID() |
Math.random() |
| Java | java.security.SecureRandom |
java.util.Random |
| Rust | rand::rngs::OsRng, getrandom crate |
— |
| C/C++ | getrandom(2) (Linux 3.17+), BCryptGenRandom (Windows), arc4random_buf (BSD) |
rand(), srand(), random() |
| Ruby | SecureRandom |
rand() |
| PHP | random_bytes(), random_int() |
rand(), mt_rand() |
| .NET | RandomNumberGenerator |
System.Random |
16.2 OS-Level Entropy Sources
Linux:
# Check available entropy (traditional — less meaningful with modern kernels)
cat /proc/sys/kernel/random/entropy_avail
# getrandom(2) — preferred syscall (blocks until sufficient entropy at boot)
# /dev/urandom — always available, safe after boot initialization
# /dev/random — blocks on older kernels, same as urandom on Linux 5.6+
# Check hardware RNG
cat /sys/devices/virtual/misc/hw_random/rng_available
# Use rng-tools to feed hardware entropy
sudo rngd -r /dev/hwrng
Entropy sources in modern Linux (5.x+):
- CPU timing jitter (jitterentropy)
- Hardware RNG (RDRAND/RDSEED on Intel/AMD, RNDR on ARM)
- Interrupt timing
- Disk I/O timing
- Input device timing
16.3 Common RNG Mistakes
| Mistake | Impact | Example |
|---|---|---|
| Using Math.random() for tokens | Predictable output, account takeover | Session tokens generated with PRNG |
| Seeding with time | Predictable seed, key recovery | srand(time(NULL)) in C |
| Reusing nonces/IVs | Catastrophic for GCM, stream ciphers | Counter reset after restart |
| Insufficient entropy at boot | Predictable keys on first boot | VM cloning, embedded devices |
| Hardcoded seed | All outputs identical across instances | Random(42) in production |
| Modulo bias | Non-uniform distribution | random() % 10 skews distribution |
16.4 Generating Secure Tokens
# Python — secure token generation
import secrets
import uuid
# URL-safe token (recommended for session IDs, API keys)
token = secrets.token_urlsafe(32) # 32 bytes = 256 bits of entropy
# Hex token
token = secrets.token_hex(32)
# UUID v4 (122 bits of randomness)
uid = uuid.uuid4()
# Secure random integer in range
code = secrets.randbelow(1000000) # 6-digit verification code
# Secure choice from sequence
password_char = secrets.choice("abcdefghijklmnopqrstuvwxyz0123456789")
// Go — secure token generation
package main
import (
"crypto/rand"
"encoding/base64"
"encoding/hex"
"fmt"
"math/big"
)
func generateToken(nbytes int) (string, error) {
b := make([]byte, nbytes)
_, err := rand.Read(b)
if err != nil {
return "", err
}
return base64.URLEncoding.EncodeToString(b), nil
}
func generateHexToken(nbytes int) (string, error) {
b := make([]byte, nbytes)
_, err := rand.Read(b)
if err != nil {
return "", err
}
return hex.EncodeToString(b), nil
}
func secureRandomInt(max int64) (int64, error) {
n, err := rand.Int(rand.Reader, big.NewInt(max))
if err != nil {
return 0, err
}
return n.Int64(), nil
}
17. File Encryption Tools
17.1 age / rage
Modern, simple file encryption. No configuration options — secure by default.
# Key generation
age-keygen -o key.txt
# Output: public key age1... and private key AGE-SECRET-KEY-1...
# Post-quantum hybrid key
age-keygen -pq -o key.txt
# Encrypt to recipient
age -r age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8p -o secret.age secret.txt
# Encrypt to SSH key
age -R ~/.ssh/id_ed25519.pub -o secret.age secret.txt
# Passphrase-based
age -p -o secret.age secret.txt
# Armor mode (PEM-encoded output)
age -a -r age1... secret.txt > secret.age.pem
# Multiple recipients
age -R recipients.txt -o secret.age secret.txt
# Decrypt
age -d -i key.txt secret.age > secret.txt
Cryptographic internals:
- X25519 for key agreement
- ChaCha20-Poly1305 for symmetric encryption
- HKDF-SHA-256 for key derivation
- Post-quantum: ML-KEM-768 + X25519 hybrid
17.2 Tool Selection Matrix
| Scenario | Tool |
|---|---|
| Encrypt files for known recipients | age/rage |
| Encrypt backups with passphrase | age -p |
| Application-level encryption | Vault Transit |
| Disk/volume encryption | LUKS (Linux), BitLocker (Windows), FileVault (macOS) |
| Database field encryption | Vault Transit or application-layer with libsodium |
| Container image signing | cosign (Sigstore) |
| Git commit signing | GPG or SSH signing |
18. Secrets Management with HashiCorp Vault
18.1 Core Concepts
- Seal/Unseal: Vault starts sealed. Unsealing requires threshold of key shares (Shamir's Secret Sharing). Auto-unseal available via cloud KMS
- Secrets Engines: Pluggable backends — KV, PKI, Transit, database, AWS, SSH
- Auth Methods: Token, LDAP, OIDC, Kubernetes, AppRole, TLS certificates
- Policies: HCL-based ACLs controlling path access
- Audit Devices: Log every request/response for compliance
18.2 Dynamic Secrets
# Database credentials (PostgreSQL example)
vault write database/config/mydb \
plugin_name=postgresql-database-plugin \
connection_url="postgresql://{{username}}:{{password}}@db:5432/mydb" \
allowed_roles="readonly"
vault write database/roles/readonly \
db_name=mydb \
creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; GRANT SELECT ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
default_ttl=1h \
max_ttl=24h
# Get dynamic credentials
vault read database/creds/readonly
18.3 AppRole Authentication (Machine-to-Machine)
# Enable AppRole
vault auth enable approle
# Create role
vault write auth/approle/role/my-app \
secret_id_ttl=10m \
token_ttl=20m \
token_max_ttl=30m \
policies="my-app-policy"
# Get RoleID (deploy with application)
vault read auth/approle/role/my-app/role-id
# Generate SecretID (deliver securely, short-lived)
vault write -f auth/approle/role/my-app/secret-id
# Application authenticates
vault write auth/approle/login \
role_id="role-id-value" \
secret_id="secret-id-value"
19. Common Crypto Pitfalls
19.1 Implementation Errors
| Pitfall | Impact | Fix |
|---|---|---|
| Nonce reuse in AES-GCM | Breaks authenticity + leaks plaintext XOR | Use randomized nonces (96-bit) or counter-based with strict state management |
| ECB mode | Deterministic — identical blocks produce identical ciphertext | Use AEAD modes (GCM, ChaCha20-Poly1305) |
| CBC without MAC | Padding oracle attacks | Use Encrypt-then-MAC or switch to AEAD |
| Encrypt-and-MAC / MAC-then-Encrypt | Various attacks depending on construction | Always Encrypt-then-MAC if not using AEAD |
| Rolling your own crypto | Virtually guaranteed to be broken | Use libsodium, OpenSSL, BoringSSL, ring (Rust) |
| Hardcoded keys/IVs | Key extraction via reverse engineering | Key management system (Vault, KMS) |
| Predictable IVs | BEAST-class attacks, chosen plaintext | CSPRNG for IV generation |
| String comparison for MACs | Timing side-channel leaks validity byte-by-byte | Constant-time comparison (hmac.compare_digest, crypto.timingSafeEqual) |
| Using encryption for integrity | Encryption alone does not detect tampering | Use authenticated encryption or separate MAC |
| Base64 as encryption | It is encoding, not encryption | Use actual encryption algorithms |
| Compression before encryption | CRIME/BREACH attacks leak plaintext | Disable compression or use length-hiding padding |
19.2 Architecture Errors
| Pitfall | Impact | Fix |
|---|---|---|
| Keys in source control | Permanent exposure — git history persists | Pre-commit hooks (gitleaks, truffleHog), secrets manager |
| Keys in environment variables | Visible in /proc, crash dumps, logging | Secrets manager with short-lived leases |
| Symmetric key for multiple purposes | Compromise of one use compromises all | Separate keys per purpose (encryption, signing, wrapping) |
| No key rotation plan | Unbounded exposure window | Automated rotation with Vault or KMS |
| Self-signed certs in production | No revocation, no chain of trust, user habituation | ACME automation (Let's Encrypt, step-ca) |
| Wildcard certs everywhere | Compromise of one server exposes all subdomains | SAN certs for specific services, segment by risk |
| Long-lived certificates | Extended window for compromised key use | 90-day max, automated renewal |
| No CT monitoring | Unauthorized certificates go undetected | Monitor crt.sh, deploy CAA records |
| Trusting default trust stores | Compromised or rogue CAs accepted | Audit trust store, pin for high-value connections |
19.3 Protocol Errors
| Pitfall | Impact | Fix |
|---|---|---|
| Allowing TLS fallback | Downgrade attacks (POODLE, DROWN) | Disable legacy protocols, TLS_FALLBACK_SCSV |
| RSA key exchange (no PFS) | Past traffic decryptable if server key compromised | ECDHE-only cipher suites |
| Mixed content | Active MitM on HTTPS pages | CSP: upgrade-insecure-requests, audit all resources |
| Missing HSTS | SSL stripping on first visit | HSTS with preload |
| Custom DH parameters < 2048-bit | Logjam attack | Use named groups (ffdhe2048+) or ECDHE only |
| Ignoring certificate validation | MitM trivial (verify=False, InsecureSkipVerify) |
Never disable in production, use proper CA trust |
| Session tickets without rotation | Long-term key compromise decrypts past sessions | Rotate session ticket keys, or disable tickets |
| Missing SNI | Wrong certificate served, connection failures | Always set SNI in client connections |
20. Quick Reference
20.1 OpenSSL One-Liners
# Generate ECDSA P-256 key
openssl ecparam -genkey -name prime256v1 -noout -out key.pem
# Generate Ed25519 key
openssl genpkey -algorithm Ed25519 -out key.pem
# Generate RSA 3072 key
openssl genrsa -out key.pem 3072
# Self-signed certificate (testing)
openssl req -x509 -newkey ec -pkeyopt ec_paramgen_curve:prime256v1 \
-keyout key.pem -out cert.pem -days 365 -nodes -subj "/CN=localhost"
# View certificate details
openssl x509 -in cert.pem -noout -text
# View CSR details
openssl req -in cert.csr -noout -text -verify
# Check key matches certificate
openssl x509 -noout -modulus -in cert.pem | openssl md5
openssl rsa -noout -modulus -in key.pem | openssl md5
# Convert PEM to DER
openssl x509 -in cert.pem -outform der -out cert.der
# Convert DER to PEM
openssl x509 -in cert.der -inform der -outform pem -out cert.pem
# Convert PKCS#12 to PEM
openssl pkcs12 -in cert.pfx -out cert.pem -nodes
# Create PKCS#12 from PEM
openssl pkcs12 -export -out cert.pfx -inkey key.pem -in cert.pem -certfile ca-chain.pem
# Encrypt file with AES-256-GCM (OpenSSL 3.x)
openssl enc -aes-256-gcm -salt -pbkdf2 -iter 600000 -in plain.txt -out encrypted.bin
# Generate random bytes
openssl rand -hex 32 # 32 bytes as hex
openssl rand -base64 32 # 32 bytes as base64
# Hash a file
openssl dgst -sha256 file.txt
# HMAC
openssl dgst -sha256 -hmac "secret-key" file.txt
# Benchmark crypto performance
openssl speed aes-256-gcm chacha20-poly1305 sha256 sha512
20.2 Compliance Mapping
| Requirement | PCI DSS 4.0 | HIPAA | SOC 2 | NIST 800-53 |
|---|---|---|---|---|
| TLS 1.2+ required | 4.2.1 | 164.312(e)(1) | CC6.1 | SC-8 |
| Strong cipher suites | 4.2.1 | 164.312(e)(2) | CC6.1 | SC-13 |
| Certificate management | 4.2.1 | 164.312(e)(1) | CC6.1 | SC-17 |
| Key rotation | 3.6.1 | 164.312(a)(2)(iv) | CC6.1 | SC-12 |
| Encryption at rest | 3.5.1 | 164.312(a)(2)(iv) | CC6.1 | SC-28 |
| Key management | 3.6 | 164.312(a)(2)(iv) | CC6.1 | SC-12 |
| Audit logging | 10.x | 164.312(b) | CC7.2 | AU-2 |
References
- OWASP Cryptographic Storage Cheat Sheet
- OWASP Key Management Cheat Sheet
- OWASP Transport Layer Security Cheat Sheet
- OWASP Password Storage Cheat Sheet
- OWASP Pinning Cheat Sheet
- Mozilla Server Side TLS Guidelines (wiki.mozilla.org/Security/Server_Side_TLS)
- NIST SP 800-57 (Key Management), SP 800-131a (Transitions), SP 800-175B (Guideline for Using Crypto)
- NIST FIPS 140-3, FIPS 203/204/205 (Post-Quantum Standards)
- RFC 8446 (TLS 1.3), RFC 7919 (FFDHE), RFC 8996 (TLS 1.0/1.1 Deprecation)
- RFC 6797 (HSTS), RFC 8659 (CAA), RFC 6962 (Certificate Transparency)
- RFC 8555 (ACME), RFC 7469 (HPKP, deprecated), RFC 7633 (Must-Staple)
- RFC 5280 (X.509 PKI), RFC 6960 (OCSP), RFC 5869 (HKDF)
- NSA CNSA 2.0 Suite (September 2022)
- Sigstore documentation (docs.sigstore.dev)
- SLSA framework (slsa.dev)
- Tool repos: testssl.sh, SSLyze, Certbot, step-ca, HashiCorp Vault, age, rage, cosign