Homelab v2: The Proxmox Architecture

This document outlines the plan and architecture for the second iteration of the homelab. The primary goal of this evolution is to introduce a bare-metal hypervisor (Proxmox VE) to enable a deeper learning experience in virtualization, resource management, and other core concepts of platform and infrastructure engineering.

This plan supersedes the original "Homelab v1" architecture.

Core Principles of Homelab v2

Virtualization First: The majority of services will run within virtual machines (VMs) or lightweight containers (LXC) on a dedicated hypervisor. This allows for better resource management, service isolation, and rapid experimentation.
Decoupled Critical Infrastructure: Core network services (like DNS) will be kept physically separate from the virtualization host to ensure network stability during host maintenance or failure.
Dedicated Control Plane: Management of the lab will be performed from a dedicated "SRE Workstation," simulating a professional environment where infrastructure is managed remotely.
Separation of Compute and Storage: The virtualization host will handle computation, while a separate, dedicated machine will handle storage and backups.

The Final Architecture

The homelab consists of four distinct physical machines, with all primary services virtualized on the Proxmox host.

                                       (The Cloud)
                                    ┌────────────────┐
                                    │  HiveMQ Cloud  │
                                    └──────┬───┬─────┘
                                           │   │ (MQTT Subscribe)
                                (Internet) │   │
                                           │   │
                  ┌────────────────────────┘   └──────────────────────────┐
                  │                                                       │
           ┌──────▼──────┐                                           ┌────▼────┐
           │   ROUTER    │                                           │ ESP32 #2│
           │ 192.168.68.1│                                           │ (MQTT)  │
           └──────┬──────┘                                           └─────────┘
                  │
           ┌──────▼──────┐
           │  8-PORT     │
           │  SWITCH     │
           └──────┬──────┘
                  │
┌─────────────────┼──────────────────────────────────────────────────────────┐
│                 │                      Your Homelab                        │
├─────────────────┼──────────────────────────────────────────────────────────┤
│                 │                                                          │
│     ┌───────────▼───────────┐                            ┌─────────────────▼───────────────┐
│     │   PROXMOX HOST        │                            │   NETWORK CORE (Raspberry Pi)     │
│     │   (Thinkcentre)       ├─ Manages ───────────►───┤   192.168.68.58                   │
│     │   192.168.68.11       │                            │   - Pi-hole (DNS)                 │
│     └───────────┬───────────┘                            │   - Tailscale VPN                 │
│                 │                                        └─────────────────────────────────┘
│                 │
│     ┌───────────▼───────────┐                            ┌─────────────────────────────────┐
│     │   SRE CONTROL PLANE   │                            │   STORAGE (MacBook Pro) [Paused]  │
│     │   (Mac Mini)          │                            │   192.168.68.12                   │
│     │   192.168.68.10       │                            │   - NAS / Backups                 │
│     └───────────────────────┘                            │   - See: nas-planning.md          │
│                                                          └─────────────────────────────────┘
│
└────────────────────────────────────────────────────────────────────────────────────────────┘

Proxmox Host Service Layout

The Thinkcentre runs Proxmox VE, hosting the following containers:

lxc-docker (192.168.68.30): A specialized container with nesting enabled to securely run the Docker daemon. It hosts the entire telemetry backend.
- TimescaleDB Container: Stores all incoming time-series data.
- Python Bridge Service: Subscribes to HiveMQ and inserts data into TimescaleDB.
lxc-monitoring (192.168.68.22): A dedicated container for observability.
- Grafana: Provides the visualization front-end for the telemetry dashboard.

Application Architecture

The primary application running on this homelab is the TSI-Telemetry project. For a detailed breakdown of its end-to-end data flow, please see the dedicated document:

TSI-Telemetry Application Architecture

Phased Migration Plan

The transition from v1 to v2 will be a careful, step-by-step process:

Prepare the Control Plane: The first step is to configure the Mac Mini with a proper SSH configuration to allow seamless, key-based access to all other nodes in the lab.
Full Backup: Before any destructive changes, a complete backup of all configurations and data from the Thinkcentre, Pi, and Mac Mini will be made to the NAS.
Install Proxmox: The Thinkcentre will be wiped and Proxmox VE will be installed as its bare-metal operating system.
Incremental Migration: We will create the new LXC containers and VMs one at a time. We will start with a low-risk service (like setting up the lxc-infra container with a test Nginx page) to verify the Proxmox network configuration. Services will be migrated over, and data restored, one by one.
Verify and Decommission: After a new virtualized service is confirmed to be running correctly, the old service on its original machine will be decommissioned. This ensures a smooth transition with no loss of service.

Installation Notes and Troubleshooting

Creating a Bootable Proxmox USB on macOS

Using the dd command to create a bootable USB:

# 1. Download Proxmox VE ISO from https://www.proxmox.com/en/downloads

# 2. Identify USB drive
diskutil list

# 3. Unmount the USB (replace diskX with your disk number)
diskutil unmountDisk /dev/diskX

# 4. Write ISO to USB (use rdiskX for better performance)
sudo dd if=~/Downloads/proxmox-ve_*.iso of=/dev/rdiskX bs=1m

# 5. Eject when complete
diskutil eject /dev/diskX

Booting from USB: Manual UEFI Boot Entry

On some systems (like the Thinkcentre), the BIOS boot menu may not display properly or the USB drive may not appear in the boot options. In this case, you can create a manual UEFI boot entry from the existing OS before wiping the machine.

From Arch Linux (or any Linux system):

Install efibootmgr:

sudo pacman -S efibootmgr  # On Arch
# sudo apt install efibootmgr  # On Debian/Ubuntu

Identify the USB device:
```
lsblk
sudo fdisk -l
```
Look for your USB drive (usually /dev/sdb, /dev/sdc, etc.) matching your USB size.

Mount the USB EFI partition:

sudo mkdir -p /mnt/usb
sudo mount /dev/sdX1 /mnt/usb  # Usually partition 1

# Locate the EFI bootloader
find /mnt/usb -name "*.efi"

The bootloader is typically at /EFI/BOOT/bootx64.efi.

Create a UEFI boot entry:

sudo efibootmgr --create --disk /dev/sdX --part 1 \
  --label "Proxmox USB" --loader '\EFI\BOOT\bootx64.efi'

Set as next boot:

# List entries to find the new entry number
efibootmgr

# Set as next boot (replace XXXX with the entry number)
sudo efibootmgr --bootnext XXXX

# Unmount and reboot
sudo umount /mnt/usb
sudo reboot

Display Issues During Boot: If the screen appears blank during USB boot or BIOS access, this is typically a display output issue:

The boot screen may output to a different video port (try DisplayPort, VGA, or other HDMI ports)
Wait 1-2 minutes - the display may appear once the Proxmox installer fully loads
The issue usually resolves once the graphical installer starts

Proxmox Installation Configuration

During the Proxmox installation, use these specific values:

Hostname: proxmox.local or proxmox
IP Address: 192.168.68.11
Netmask: 255.255.255.0 (or /24)
Gateway: 192.168.68.1
DNS Server: 192.168.68.58 (Raspberry Pi)
Management Interface: Usually eno1 or eth0

After installation, access the web interface at: https://192.168.68.11:8006

Post-Installation: SSH Key Setup from Mac Mini

Once Proxmox is installed, configure SSH key-based authentication from your Mac Mini (SRE Control Plane):

Remove old host key (if you previously had SSH access to this IP):
```
ssh-keygen -R 192.168.68.11
```
This is necessary because the machine was wiped and reinstalled, generating a new SSH host key.
Copy your SSH key to Proxmox:
```
ssh-copy-id root@192.168.68.11
```
- Accept the new host fingerprint (type yes)
- Enter the root password you set during installation
- Your SSH key will be copied to the server
Test passwordless SSH access:
```
ssh root@192.168.68.11
```
You should now be able to login without entering a password.

This completes Phase 1 of the migration plan: "Prepare the Control Plane" with proper SSH configuration for managing all lab nodes.