← noellucas.io GitHub

Résumé · Storage, Infrastructure & Reliability Engineering

Noel Lucas

Twenty-six years keeping systems running where failure is expensive: live mutual-fund trading, a federal client's 40-petabyte object-storage estate, and the observability that proves it is all still working.

By the numbers

40+ PBobject storage owned for a federal client, across 8 sites
10M+financial transactions a day on platforms I built and ran
6,952drives remediated across 32 nodes to fix a multi-year platform defect
50M+objects a day ingested on a Walmart deployment
200+management endpoints behind SSO I built
23 minto rebuild a destroyed Kubernetes cluster from code and offsite backups (60-min target)

Case files

The work that best shows how I operate. Open one for the problem, what I did, and the outcome.

01 The fleet that couldn't reclaim space Object storage · Python · Vendor escalation

Problem. A multi-year defect in an object-storage platform meant freed capacity was never actually reclaimed. Fleet utilization crept past 97%, toward write-stop, across eight sites.

What I did. Found and proved the defect, drove it to resolution with the vendor, and built the remediation toolchain myself in Python: an unattended fleet mode with drive rotation and stall detection, a terminal dashboard, and a FastAPI service feeding the ops dashboard.

Outcome. 6,952 drives cycled across 32 nodes at 8 sites; utilization back below the alerting threshold. I co-authored the root-cause analysis with the vendor.

02 340 GB of metrics, two days on disk, three years of history Prometheus · Thanos · Grafana

Problem. The monitoring platform's own time-series database had grown to 340 GB on local disk, and history was the first thing at risk.

What I did. Moved long-term storage onto the object storage the platform was monitoring, using Thanos: sidecar, store, query and compactor as Podman Quadlets with TLS throughout.

Outcome. Local disk went from 340 GB to two days, with three years of downsampled history kept. Nine operational teams used the platform: 14 custom dashboards, about 460 panels, 120+ endpoints. I presented its architecture to executive leadership.

03 HEAD 200, GET 403: the backup failure two vendors couldn't solve Commvault · S3 · Root cause

Problem. Commvault HyperScale X backups to object storage were failing, and both vendors had stalled, each pointing at the other.

What I did. Traced it at the S3 layer: object metadata reads (HEAD) succeeded while reads of the data (GET) were refused. The cause was a gap in the access-control configuration between the two platforms.

Outcome. Root cause found and fixed, and backups resumed. It's the kind of problem I like most: the answer was in the protocol, not in either vendor's documentation.

04 300 Gb per node in front of 40 PB Ivanti vADC · Load balancing · DNS

What I built. Each site's load-balancer cluster in front of the object storage: two physical nodes per site, each with a 4×25 GbE bond and a 2×100 GbE bond, and each bond fronted by its own DNS-published VIP.

vADC node A bond0 · 4×25 GbE bond1 · 2×100 GbE vADC node B bond0 · 4×25 GbE bond1 · 2×100 GbE VIP · DNSVIP · DNS

Outcome. 300 Gb of bonded capacity per node at every site. I was also the go-to escalation SME on Ivanti vADC and F5 LTM/GTM for Hitachi delivery teams on four continents.

05 Live trading platforms, from a failing NT4 network Financial services · DR · Leadership

Starting point. A failing NT4 environment with hub networking and no firewall, and me as the only infrastructure engineer.

What I built. Every layer, rebuilt: network, servers, security, databases and backup. Then the colo, platform and DR environments behind live production financial applications: mutual-fund trading (TradeManager), PositionManager, RevenueManager, CashManager, a SEC Rule 22c-2 compliance system and the DTCC's MFPS2 prospectus database.

Outcome. 10M+ transactions a day on Oracle RAC and Data Guard, SQL Server mirroring and MySQL replication, with a written and tested BC/DR plan. From 2007 I hired and led the team that ran it: 5 operations engineers and 2 DBAs.

06 Destroy the cluster, rebuild it in 23 minutes Terraform · GitHub Actions · DR drill

What I did. In my public lab, I destroyed all three Kubernetes VMs, their disks and their DNS records, then rebuilt everything using only the repository, the CI pipeline and offsite S3 backups.

Outcome. RTO 23 min 10 s against a 60-minute target. RPO was well inside the 24-hour target, with no data lost. The drill turned up two real findings, regenerated SSH host keys and a broken health-probe secret, and both were fixed in code the same day. See the repo →

Career

Feb 2015 – Present · Remote

Master Technical Consultant — Hitachi Vantara

Professional services: a resident subject-matter expert placed with enterprise and federal clients.

A Federal Client — Resident Storage & Observability SME 2022–2026
  • Architected and operated 40+ PB of S3 object storage (HCP, HCP for Cloud Scale, S-Series) across 8 data center and colocation sites. It was the primary backup target, backing Cohesity, Commvault, Splunk, internal ransomware-recovery tooling and the Nasuni filer cache.
  • Installed and configured Splunk Enterprise with object storage as its storage tier. Used CyberArk EPV and PAM for privileged access.
  • Built the open-source observability platform (Prometheus, Thanos, Grafana, Alertmanager, Blackbox Exporter, PostgreSQL, Podman Quadlets) used by nine teams: Operational Recovery, Network, InfoSec, Incident Response, IAM, IPAM, PKI, DCIM and storage.
  • Wrote Python exporters for namespace capacity and for Venafi TLS Protect (75,000+ certificates). A custom alert caught an expiring Active Directory domain-controller certificate before it caused an org-wide authentication outage.
  • Built an identity and automation platform: Authentik syncing 8,800+ users and 7,000+ AD groups, Semaphore with OIDC SSO, and NGINX Proxy Manager forward-auth, together gating 200+ management endpoints.
  • Ran every change through ServiceNow change control, including freeze windows. Authored Cisco ACI fabric contract changes for my services and presented their justification to the Change Approval Board.
  • Planned a four-site colocation buildout: cabling specifications, a 10-zone DNS bundle and automated provisioning.
Earlier engagements 2015–2022
  • Resident SME and lead implementer of HCP, HCP for Cloud Scale, HCP Anywhere, HCP Gateway and Hitachi Data Ingestor for Walmart (50M+ objects/day), PNC Bank, TD Bank, Bell Canada, Standard Chartered Bank, FIS Global, Disney, Marvel, Starz and NASA. That covered design, bare-metal deployment, configuration and customer handover.
  • Escalation SME and mentor for Hitachi delivery teams across North America, EMEA, APAC and LATAM on HCP, Cloud Scale, S-Series, Ivanti vADC, F5 LTM/GTM and replication failover testing. I trained engineers and wrote their runbooks.

Jan 2000 – Feb 2015 · Columbus, GA

IT Director → Senior Database Administrator — Delta Data

  • Rebuilt the company's infrastructure from a failing NT4 network as its sole infrastructure engineer. From 2007 I built and led the team: 5 operations engineers and 2 DBAs, with the final hiring decision on each.
  • Ran the colo, platform and DR environments for live mutual-fund trading, position, revenue and cash-management applications, a SEC Rule 22c-2 compliance app and the DTCC MFPS2 database: 10M+ transactions per day.
  • Acted as liaison between development and operations, guiding the product from client-server toward distributed, web-native design.
  • Ran production on AWS and Azure alongside Hyper-V and VMware. Wrote the Business Continuity and Disaster Recovery Plans, and led the data-center migration.
  • Ran Oracle 10g–12c (RAC, Data Guard), SQL Server (mirroring) and MySQL (replication) across dev, QA and production.

Skills

Object storage

S3 semantics at petabyte scale: Hitachi HCP, HCP for Cloud Scale, VSP One Object (S-Series); Ceph; erasure coding, replication topology, failure domains, capacity planning, data migration.

Data protection

Backup and recovery architecture; S3 targets for Cohesity and Commvault HyperScale X; BC/DR planning; VSP block, Isilon/PowerScale, Nasuni; NFS, SMB, iSCSI, Fibre Channel.

Observability

Prometheus, PromQL, Alertmanager, Thanos, Grafana, Loki, Grafana Alloy, OpenTelemetry, Jaeger, Splunk Enterprise, Filebeat / Logstash / OpenSearch, InfluxDB, Telegraf (SNMP), custom Python exporters, SLO burn-rate alerting.

Automation & IaC

Terraform, Ansible / AWX, Semaphore, GitHub Actions (plan-on-PR, gated apply, drift detection), Checkov and tflint policy-as-code, Python (FastAPI), Go, Bash, REST APIs.

AI-assisted engineering

Anthropic Claude API, Claude Code, MCP, agentic workflows; retrieval-augmented generation on PostgreSQL + pgvector.

Platforms

Linux (RHEL, Rocky, Debian, Fedora), systemd, LVM/XFS; Windows Server NT4 → 2025; Kubernetes (k3s, RKE2), Podman / Quadlets, Docker; VMware, Hyper-V, Proxmox VE, KVM; bare metal via iLO/iDRAC.

Networking

L2/L3 data-center networking, spine-leaf, MLAG, LACP, tagged VLANs; Cisco Nexus/ACI, Arista, Juniper; F5 LTM/GTM, Ivanti vADC; DNS (Technitium, Infoblox); OPNsense, Suricata IDS.

Security & identity

CyberArk EPV and PAM; Authentik and Keycloak SSO (OIDC, SAML, forward-auth); Active Directory / LDAP; zero-trust segmentation on the ACI fabric; PKI/TLS (Venafi TLS Protect, private CA).

Data & cloud

Oracle 10g–12c (OCP, RAC, Data Guard), SQL Server, MySQL, PostgreSQL; AWS, Azure.

Process

ServiceNow change control and asset tracking, CAB presentations, runbooks, design documents and root-cause analyses; Jira, Confluence, draw.io.

The lab

I learn a new layer by building it, breaking it on purpose, and writing down what I found. Most of it is public.

  1. sre-lab: infrastructure built as a product

    Terraform with PostgreSQL state. GitHub Actions on a hardened self-hosted runner, with plan and policy checks on every PR, a change-risk gate, gated apply of the reviewed plan, automatic rollback and nightly drift detection. Ansible builds a k3s cluster; SLOs use burn-rate alerts. Every decision and failure is written up as an ADR.

    github.com/NoelLucasIO/sre-lab

  2. Project Pulse

    Continuous per-node metrics for data-integrity verification runs that last days to weeks: SSH collection every five minutes and rates derived from persisted state, in a systemd chain that survives reboots and fails closed. It surfaced an outlier node and made completion ETAs possible.

    python · systemd · prometheus

  3. Observability lab

    Two log pipelines on one host (Grafana Alloy → Loki, and Filebeat → Logstash → OpenSearch), plus Jaeger traces and OpenTelemetry instrumentation of a RAG service, with one-click log-to-trace correlation in Grafana.

    loki · opensearch · jaeger · otel

Credentials

EducationB.S., Information Technology — Western Governors University, 2010
OracleOCP 10g · 11g · 12c
MicrosoftMCSE (NT4 · 2000 · 2003) · MCITP Enterprise Administrator (2008)
CompTIALinux+ · Network+ · Project+