Goal

Understand entropy and randomness as the foundation of cryptographic security, learn binary encoding (Base64, hex), and combine everything into practical encrypted workflows with best practices.

Prerequisites: Week 2b (Symmetric Encryption)

This is Part 3 of 3 - Covers entropy, encoding, workflows, and best practices.


1. Entropy and Randomness: The Foundation of Security

What Is Entropy?

Entropy is a measure of unpredictability (randomness).

In cryptography:

  • High entropy = Truly random, unpredictable
  • Low entropy = Patterns, predictable

Why it matters: Encryption keys must be random, and a predictable key means broken encryption, so random number generators (RNGs) are security-critical.

Sources of Randomness in Linux

Linux provides two key sources:

1. /dev/random (Historically Blocking)

# Historically blocked when the kernel's entropy estimate ran low.
# Since kernel 5.6 it blocks only until the CSPRNG is first seeded at boot,
# then behaves like /dev/urandom.
head -c 32 /dev/random | base64

2. /dev/urandom (Non-Blocking, Cryptographically Secure)

# Never blocks, always returns data
# Used for: Most cryptographic operations (recommended)
head -c 32 /dev/urandom | base64

Modern Linux (kernel 5.6+): After boot-time seeding both behave the same and are equally secure; /dev/urandom is fine for all uses.

How Linux Generates Randomness

Entropy sources:

  • Hardware events: Keyboard timings, mouse movements, disk I/O
  • CPU jitter: Timing variations in instruction execution
  • Hardware RNG: Intel RDRAND, AMD RdRand (if available)
  • Network packet timings

Process:

Hardware Events → Entropy Pool → CSPRNG → /dev/urandom
(keyboard, disk)   (mixed)     (ChaCha20)  (secure random bytes)

CSPRNG = Cryptographically Secure Pseudo-Random Number Generator

Generating Random Keys

# Generate 256-bit (32-byte) encryption key
head -c 32 /dev/urandom > encryption.key

# Generate random password (base64 encoded for readability)
head -c 32 /dev/urandom | base64
# Example output (yours will differ; base64 alphabet is only A-Za-z0-9+/=):
# AFV3/ZP7rvuUSjBLfL2ylrJqY3tmvv9p1XF6qTabXpE=

# Generate hex-encoded random data
xxd -l 32 -p /dev/urandom
# Output: 7d8f4c3a1b2e5f6d9c8a7b6e5f4d3c2b1a9e8d7c6b5a4f3e2d1c0b9a8e7f6d5c

The Danger of Weak RNGs

Historical failure: Debian OpenSSL bug (2006-2008)

  • Bug: a Debian packaging patch (Sept 2006 - May 2008) removed entropy mixing from OpenSSL, leaving the process ID as essentially the only seed.
  • Result: at most 32,767 possible keys per key type and size, instead of the full key-size entropy.
  • SSH/SSL keys generated on Debian, Ubuntu and other derivatives with the affected openssl package during this period were weak. Keys from other systems were not affected.
  • Attackers could precompute every possible key and check for a match quickly.

Lesson: Don’t patch out or bypass the seeding of a cryptographic RNG. Use the OS CSPRNG (/dev/urandom) and keep your crypto libraries unmodified.


2. Binary Encoding: Base64, Hex, and Why They Matter

The Problem: Binary Data in Text Systems

Encryption produces binary data (random bytes 0-255), but many systems expect text:

  • Email (originally 7-bit ASCII)
  • JSON configuration files
  • Copy/paste in terminals

Solution: Encode binary as text using Base64 or Hex.

Base64 Encoding

Concept: Represent binary data using 64 printable ASCII characters.

Character set: A-Z, a-z, 0-9, +, / (64 total)

Efficiency: 3 bytes binary → 4 bytes base64 (33% overhead)

# Encode binary to base64
echo "Hello, cypherpunk!" | base64
# Output: SGVsbG8sIGN5cGhlcnB1bmshCg==

# Decode base64 to binary
echo "SGVsbG8sIGN5cGhlcnB1bmshCg==" | base64 -d
# Output: Hello, cypherpunk!

Hexadecimal Encoding

Concept: Represent each byte as two hex digits (0-9, A-F).

Efficiency: 1 byte binary → 2 bytes hex (100% overhead)

When to use:

  • Debugging (more readable than base64)
  • Cryptographic hashes (SHA-256 output always shown in hex)
  • Low-level binary analysis
# Encode to hex
echo "Hello" | xxd -p
# Output: 48656c6c6f0a

# Decode from hex
echo "48656c6c6f0a" | xxd -r -p
# Output: Hello

Why Base64 Has “=” Padding

Base64 encodes 3 bytes at a time:

  • 3 bytes (24 bits) → 4 base64 characters
  • If input not divisible by 3, add padding
2 bytes: "He" → SGU= (one = for padding)
1 byte:  "H"  → SA== (two = for padding)

The “=” is filler, not part of data.


3. Putting It All Together: Encrypted Workflow

This section combines everything into one encrypted workflow.

Scenario: Encrypting Sensitive Documents

# 1. Create directory structure
mkdir -p ~/secure/{documents,keys,logs}

# 2. Generate strong encryption key (256-bit, base64-encoded as text)
openssl rand -base64 32 > ~/secure/keys/master.key
chmod 600 ~/secure/keys/master.key  # Only you can read

# 3. Create sensitive document
cat > ~/secure/documents/secrets.txt <<EOF
Bank account: 1234567890
SSH passphrase: MySecretPhrase123
API keys: sk_live_abc123xyz
EOF

# 4. Hash the original (catches accidental corruption)
sha256sum ~/secure/documents/secrets.txt > ~/secure/logs/secrets.sha256

# 5. Encrypt with AES-256-CBC + PBKDF2
#    (openssl's `enc` can't do GCM/AEAD, so this is confidentiality only.
#     The SHA-256 hash from step 4, checked in step 8, catches accidental
#     corruption, not tampering. For authenticated encryption use `age`, see Week 2b)
openssl enc -aes-256-cbc -pbkdf2 -iter 600000 -salt \
  -in ~/secure/documents/secrets.txt \
  -out ~/secure/documents/secrets.enc \
  -pass file:"$HOME/secure/keys/master.key"

# 6. Remove the plaintext original (shred is best-effort only: it does not reliably
#    erase on SSDs, journaling or copy-on-write filesystems such as btrfs/ZFS.
#    Week 4's encrypted volumes are the more dependable protection)
shred -u ~/secure/documents/secrets.txt

# 7. Later: Decrypt when needed
openssl enc -d -aes-256-cbc -pbkdf2 -iter 600000 \
  -in ~/secure/documents/secrets.enc \
  -out ~/secure/documents/secrets-decrypted.txt \
  -pass file:"$HOME/secure/keys/master.key"

# 8. Verify integrity (hash should match)
sha256sum ~/secure/documents/secrets-decrypted.txt
cat ~/secure/logs/secrets.sha256

What We Just Did

  1. Generated random key - High-entropy 256-bit key from openssl rand, encoded as text
  2. Protected key file - Set strict permissions (chmod 600)
  3. Hashed original - SHA-256 fingerprint to catch accidental corruption
  4. Encrypted with AES-256-CBC + PBKDF2 - Confidentiality only (no authentication tag; for authenticated encryption use age or gpg --symmetric, see Week 2b)
  5. Deleted plaintext - Removed unencrypted copy
  6. Decrypted securely - Retrieved data with key file
  7. Verified integrity - Checked the hash matches (catches accidental corruption; an attacker who can edit the files could replace the hash too, so this is not tamper-proofing)

4. Cryptographic Best Practices

Algorithm Selection Guide

For Hashing:

  • SHA-256 - General purpose, widely supported
  • BLAKE2b - Faster alternative, modern applications
  • SHA-1 - Only if legacy system requires it
  • MD5 - Never use (broken)

For Symmetric Encryption:

  • AES-256-GCM - Default choice, authenticated
  • ChaCha20-Poly1305 - Mobile/embedded devices, Tor
  • AES-256-CBC - Legacy systems only
  • AES-ECB - Never use (leaks patterns)

For Key Derivation:

  • Argon2id - Best choice (if available)
  • scrypt - Good alternative
  • PBKDF2 - Minimum acceptable (PBKDF2-HMAC-SHA256 600k+ iterations, OWASP guidance)
  • Plain SHA-256 - Never use for passwords

Common Mistakes to Avoid

1. Using ECB mode

# WRONG
openssl enc -aes-256-ecb -in file.txt -out file.enc

2. Not using a proper key-derivation function

# WRONG
openssl enc -aes-256-cbc -in file.txt -out file.enc  # Missing -pbkdf2 -iter: falls back to legacy single-iteration key derivation (openssl warns "deprecated key derivation used")

3. Weak passwords with strong encryption

# WRONG (the cipher is fine, the passphrase is guessable)
openssl enc -aes-256-cbc -pbkdf2 -iter 600000 -salt -in file.txt -out file.enc -pass pass:password123
# An attacker brute-forces the password and never touches AES

4. Storing keys with encrypted data

# WRONG
encryption.key  # Same directory as encrypted files
secrets.enc     # Attacker gets both!

5. Not verifying integrity

# WRONG (no way to detect tampering)
# Attacker could modify ciphertext

Security Checklist

Before deploying encryption:

  • Using modern algorithm (AES-256-GCM or ChaCha20-Poly1305)
  • Strong key (256-bit random key, encoded as text)
  • Key derivation if password-based (PBKDF2-HMAC-SHA256 600k+ iterations, OWASP guidance; prefer Argon2id or scrypt)
  • Authenticated encryption (GCM or Poly1305)
  • Corruption check only (SHA-256 hash; does not detect tampering)
  • Secure key storage (separate from encrypted data, chmod 600)
  • Tested decryption (verify you can recover data)

Week 2 Checklist

  • Understand difference between hashing and encryption
  • Explain why MD5 is broken (collision attacks)
  • Hash files with SHA-256 and verify integrity
  • Encrypt and decrypt a file with authenticated encryption (age -p or gpg --symmetric, Week 2b) and confirm that a flipped ciphertext byte makes decryption fail
  • Generate 256-bit random keys and encode them as text
  • Understand encryption modes (ECB broken, CBC legacy, GCM modern)
  • Know when to use AES vs ChaCha20
  • Use key derivation for password-based encryption
  • Encode binary data with base64 and hex
  • Demonstrate avalanche effect with tiny input changes
  • Create encrypted workflow (hash → encrypt → verify)

Journal & Git Commit

echo "Week 2: Mastered hash functions (SHA-256), symmetric encryption (authenticated, via age/gpg), key derivation (PBKDF2/Argon2), and entropy sources. Built encrypted workflow." >> notes/week02_journal.md

git add .
git commit -S -m "Week 2 - Cryptographic fundamentals, hashing, encryption"

Up Next: Week 3

Now that you understand symmetric encryption (same key for encrypt/decrypt), Week 3 introduces asymmetric encryption:

  • Public/private keypairs - Math that makes key exchange possible
  • GPG (GnuPG) - The cypherpunk’s Swiss Army knife
  • Digital signatures - Prove you wrote a message
  • Web of trust - Decentralized identity verification

Week 2 covered how encryption works; Week 3 covers how to use it without sharing your secret key.


Additional Resources

Further Reading:

Tools Covered This Week:

  • openssl - Swiss Army knife of cryptography
  • sha256sum - SHA-2 family hashing
  • xxd / hexdump - Binary data visualization
  • base64 - Binary-to-text encoding
  • /dev/urandom - Cryptographically secure RNG

Key Takeaways

  • Entropy measures randomness - high entropy keys are unpredictable
  • /dev/urandom is the recommended source for cryptographic randomness
  • Base64 encodes binary for text systems (33% overhead)
  • Hex encoding is more readable for debugging (100% overhead)
  • Never implement your own RNG - use system-provided sources
  • Complete workflow: Generate key → Hash original → Encrypt → Delete plaintext → Verify