Résumé · Storage, Infrastructure & Reliability Engineering
Twenty-six years keeping systems running where failure is expensive: live mutual-fund trading, a federal client's 40-petabyte object-storage estate, and the observability that proves it is all still working.
The work that best shows how I operate. Open one for the problem, what I did, and the outcome.
Problem. A multi-year defect in an object-storage platform meant freed capacity was never actually reclaimed. Fleet utilization crept past 97%, toward write-stop, across eight sites.
What I did. Found and proved the defect, drove it to resolution with the vendor, and built the remediation toolchain myself in Python: an unattended fleet mode with drive rotation and stall detection, a terminal dashboard, and a FastAPI service feeding the ops dashboard.
Outcome. 6,952 drives cycled across 32 nodes at 8 sites; utilization back below the alerting threshold. I co-authored the root-cause analysis with the vendor.
Problem. The monitoring platform's own time-series database had grown to 340 GB on local disk, and history was the first thing at risk.
What I did. Moved long-term storage onto the object storage the platform was monitoring, using Thanos: sidecar, store, query and compactor as Podman Quadlets with TLS throughout.
Outcome. Local disk went from 340 GB to two days, with three years of downsampled history kept. Nine operational teams used the platform: 14 custom dashboards, about 460 panels, 120+ endpoints. I presented its architecture to executive leadership.
Problem. Commvault HyperScale X backups to object storage were failing, and both vendors had stalled, each pointing at the other.
What I did. Traced it at the S3 layer: object metadata reads (HEAD) succeeded while reads of the data (GET) were refused. The cause was a gap in the access-control configuration between the two platforms.
Outcome. Root cause found and fixed, and backups resumed. It's the kind of problem I like most: the answer was in the protocol, not in either vendor's documentation.
What I built. Each site's load-balancer cluster in front of the object storage: two physical nodes per site, each with a 4×25 GbE bond and a 2×100 GbE bond, and each bond fronted by its own DNS-published VIP.
Outcome. 300 Gb of bonded capacity per node at every site. I was also the go-to escalation SME on Ivanti vADC and F5 LTM/GTM for Hitachi delivery teams on four continents.
Starting point. A failing NT4 environment with hub networking and no firewall, and me as the only infrastructure engineer.
What I built. Every layer, rebuilt: network, servers, security, databases and backup. Then the colo, platform and DR environments behind live production financial applications: mutual-fund trading (TradeManager), PositionManager, RevenueManager, CashManager, a SEC Rule 22c-2 compliance system and the DTCC's MFPS2 prospectus database.
Outcome. 10M+ transactions a day on Oracle RAC and Data Guard, SQL Server mirroring and MySQL replication, with a written and tested BC/DR plan. From 2007 I hired and led the team that ran it: 5 operations engineers and 2 DBAs.
What I did. In my public lab, I destroyed all three Kubernetes VMs, their disks and their DNS records, then rebuilt everything using only the repository, the CI pipeline and offsite S3 backups.
Outcome. RTO 23 min 10 s against a 60-minute target. RPO was well inside the 24-hour target, with no data lost. The drill turned up two real findings, regenerated SSH host keys and a broken health-probe secret, and both were fixed in code the same day. See the repo →
Professional services: a resident subject-matter expert placed with enterprise and federal clients.
S3 semantics at petabyte scale: Hitachi HCP, HCP for Cloud Scale, VSP One Object (S-Series); Ceph; erasure coding, replication topology, failure domains, capacity planning, data migration.
Backup and recovery architecture; S3 targets for Cohesity and Commvault HyperScale X; BC/DR planning; VSP block, Isilon/PowerScale, Nasuni; NFS, SMB, iSCSI, Fibre Channel.
Prometheus, PromQL, Alertmanager, Thanos, Grafana, Loki, Grafana Alloy, OpenTelemetry, Jaeger, Splunk Enterprise, Filebeat / Logstash / OpenSearch, InfluxDB, Telegraf (SNMP), custom Python exporters, SLO burn-rate alerting.
Terraform, Ansible / AWX, Semaphore, GitHub Actions (plan-on-PR, gated apply, drift detection), Checkov and tflint policy-as-code, Python (FastAPI), Go, Bash, REST APIs.
Anthropic Claude API, Claude Code, MCP, agentic workflows; retrieval-augmented generation on PostgreSQL + pgvector.
Linux (RHEL, Rocky, Debian, Fedora), systemd, LVM/XFS; Windows Server NT4 → 2025; Kubernetes (k3s, RKE2), Podman / Quadlets, Docker; VMware, Hyper-V, Proxmox VE, KVM; bare metal via iLO/iDRAC.
L2/L3 data-center networking, spine-leaf, MLAG, LACP, tagged VLANs; Cisco Nexus/ACI, Arista, Juniper; F5 LTM/GTM, Ivanti vADC; DNS (Technitium, Infoblox); OPNsense, Suricata IDS.
CyberArk EPV and PAM; Authentik and Keycloak SSO (OIDC, SAML, forward-auth); Active Directory / LDAP; zero-trust segmentation on the ACI fabric; PKI/TLS (Venafi TLS Protect, private CA).
Oracle 10g–12c (OCP, RAC, Data Guard), SQL Server, MySQL, PostgreSQL; AWS, Azure.
ServiceNow change control and asset tracking, CAB presentations, runbooks, design documents and root-cause analyses; Jira, Confluence, draw.io.
I learn a new layer by building it, breaking it on purpose, and writing down what I found. Most of it is public.
sre-lab: infrastructure built as a product
Terraform with PostgreSQL state. GitHub Actions on a hardened self-hosted runner, with plan and policy checks on every PR, a change-risk gate, gated apply of the reviewed plan, automatic rollback and nightly drift detection. Ansible builds a k3s cluster; SLOs use burn-rate alerts. Every decision and failure is written up as an ADR.
github.com/NoelLucasIO/sre-lab
Project Pulse
Continuous per-node metrics for data-integrity verification runs that last days to weeks: SSH collection every five minutes and rates derived from persisted state, in a systemd chain that survives reboots and fails closed. It surfaced an outlier node and made completion ETAs possible.
python · systemd · prometheus
Observability lab
Two log pipelines on one host (Grafana Alloy → Loki, and Filebeat → Logstash → OpenSearch), plus Jaeger traces and OpenTelemetry instrumentation of a RAG service, with one-click log-to-trace correlation in Grafana.
loki · opensearch · jaeger · otel