travis@deluca-home:~$
Senior Product & Field Solutions Engineer (Everything Engineer) — Milestone Computer Technology, Inc.
Travis DeLuca is a Senior Product & Field Solutions Engineer at Milestone Technologies, working on virtualization platforms: KVM, HPE Morpheus and VM Essentials (HVM), VMware and Proxmox. He is an HPE CloudOps Maestro Expert and publishes open-source Morpheus plugins and infrastructure tooling under the GitHub organization builtbyfood — including Argus (an iLO and Redfish tab for HPE hosts in Morpheus), Perch (network topology), Pinax (infrastructure inventory and reporting), VM Folders, and vme-iscsi-setup.
travisdeluca.com -- a terminal that happens to be a resume. ls list the current directory cat <file> read a file cd projects change directory cd notes the part most people skip help everything else Tab completes. Up/Down walks history. Ctrl-L clears. There are a few things in here that are not on the map.
TRAVIS(1) User Commands TRAVIS(1)
NAME
travis - senior product & field solutions engineer
SYNOPSIS
travis [--kvm] [--morpheus] [--vmware] [--proxmox] [--field] [-v]
DESCRIPTION
Builds, breaks, and re-builds virtualization platforms for a
living. Currently at Milestone Computer Technology, Inc. doing product and
field solutions engineering across the HPE Morpheus and VM
Essentials ecosystem. Recognized as an HPE CloudOps Maestro
Expert.
Job title says Senior Product & Field Solutions Engineer. What it
means in practice is Everything Engineer: the platform, the
storage under it, the network under that, the plugin that fills
the gap, the migration off the old thing, the runbook, and the
call at 2am when all six of those interact.
Came up through Proxmox and VMware. Currently resident in HPE's
stack, which is where most of the interesting migration work is
right now.
Ten years before that, in order: secure communications support at
a bank, five years of VMware and storage engineering at a hosted
virtualization provider, then systems and senior systems
engineering for county government -- 3PAR, Synergy, Nimble,
Brocade, vSphere, vSAN, NSX, Cohesity, Splunk -- and finally
running the datacenter for a county court. The uptime was real
and so were the consequences.
Writes the tooling that should have shipped with the product.
Publishes it publicly under github.com/builtbyfood, usually after
hitting the gap in the field first and getting annoyed about it.
Runs a Helldivers 2 themed homelab that is substantially harder
to operate than most production estates, on purpose: it exists to
reproduce customer failure modes before customers find them.
OPTIONS
--kvm libvirt, OVMF/UEFI, HVM clustering, shared block
storage, multipath iSCSI, OVS/LACP, passthrough
--morpheus Groovy plugin SDK, tasks, workflows, reports,
custom tabs, option sources, provisioning
--vmware vSphere/ESXi, and the migration path off it
--proxmox where a lot of this started, and a migration
source that keeps coming up
--field the part where you fix it while someone watches
SEE ALSO
resume.txt, projects/, notes/, certs.txt, links.txt, contact.txt
TRAVIS(1) TRAVIS(1)
TRAVIS DELUCA
Senior Product & Field Solutions Engineer (Everything Engineer)
----------------------------------------------------------------
Datacenter infrastructure, enterprise storage, virtualization,
and everything in between.
EXPERIENCE
Senior Product & Field Solutions Engineer
Milestone Computer Technology, Inc. Oct 2025 - present
- Field and product engineering across HPE Morpheus and VM
Essentials deployments: design, migration, remediation, and the
calls that begin with 'it worked yesterday.'
- Build and publish plugins and tooling that close the gap between
what the platform ships and what a real deployment needs --
several now in use outside my own environments.
- Lead VMware and Proxmox to HVM migration work end to end:
assessment, conversion tooling, cutover, and the boot failures
nobody warned you about.
- Author operational runbooks and automation for KVM clusters:
iSCSI and multipath, shared cluster storage, OVS networking,
UEFI/OVMF, device passthrough.
Senior Systems Administrator
Maricopa County Clerk of the Superior Court Mar 2025 - Oct 2025
- Ran the datacenter infrastructure.
- VMware vCenter 6.7/7/8, vSAN 7/8, SDDC and NSX.
- Cohesity data protection.
- Dell ECS 2 and 3, PowerStore, VxRail.
- AWS.
Maricopa County Aug 2021 - Apr 2025
Senior Systems Engineer Jul 2022 - Apr 2025
- HPE 3PAR, HPE Synergy, Nimble.
- Brocade fibre channel switching.
- VMware vCenter 6.7/7/8, vSAN 7/8, SDDC and NSX.
- Cohesity data protection, Splunk.
Systems Engineer Aug 2021 - Jul 2022
Systems Engineer
Infinitely Virtual Mar 2016 - Jul 2021
- Five years of VMware and enterprise storage engineering at a
hosted virtualization provider. Multi-tenant, and somebody else's
production the entire time, which is a particular kind of school.
Transmission Control Support Specialist
Wells Fargo Jul 2015 - Mar 2016
- Support for secure communications.
RECOGNITION
HPE CloudOps Maestro Expert
Full certification list: cat certs.txt
STACK
Virtualization KVM/libvirt, HPE Morpheus + VM Essentials (HVM),
VMware vSphere/ESXi, vCenter 6.7/7/8, vSAN 7/8, SDDC,
NSX, Proxmox, OVMF/UEFI, qemu-img, qemu-nbd, virtio
Clustering HVM cluster layouts, shared cluster filesystems,
HA and failover behavior, live migration
Storage HPE 3PAR, Nimble, Dell PowerStore, Dell ECS, VxRail,
iSCSI, multipath, NVMe-FC, fibre channel, NFSv3/v4,
local block
Data protection Cohesity, snapshot and restore design
Networking Open vSwitch, LACP bonding, VLAN design, jumbo frames,
Brocade FC switching, zone-based firewalling
Observability Splunk, log pipeline design, custom instrumentation
Cloud AWS
Code Groovy + Gradle, Bash, Python, Redfish and REST
integration, YAML-shaped suffering
Platforms Ubuntu, RHEL/Rocky, Fedora, Debian, Windows Server
SELECTED PUBLIC WORK github.com/builtbyfood
kvm-monitor CPU Ready and vCPU scheduling metrics for KVM
kvm-echo per-VM packet capture, encrypted before it lands
argus iLO and Redfish inside Morpheus
gfs2-rescue adopt GFS2 storage from a dead cluster, no reformat
migration-assist finishes VMware to HVM migrations
hvm-updater appliance and fleet upgrades from inside Morpheus
vm-xml-editor libvirt domain XML in the UI, plus OVA/GNS3 convert
vm-folders folder organization for the VM list
vme-netconfig build, repair and audit host networking
vme-iscsi-setup iSCSI and multipath bring-up, MIT licensed
ls projects/plugins -- in-product Morpheus plugins
ls projects/tools -- scripts, workflows and guides
ls notes -- how the work actually goes
CERTIFICATIONS
----------------------------------------------------------------
HPE CloudOps Maestro Expert
Hewlett Packard Enterprise
HPE ASE - Edge-to-cloud Architect exp Oct 2028
HPE ATP - Hybrid Cloud exp Oct 2028
HPE Product Certified - Morpheus VM Essentials exp Mar 2029
HPE Sales Certified - Hybrid Cloud Solutions [2026] Apr 2026
VMware / Broadcom
VCP - Data Center Virtualization 2021
VMware Software-Defined Networking - Trained Professional (2023)
VMware vSphere 6.5 Foundations (2020)
Verify: credly.com/users/travis-deluca
$ cat /proc/travis/capabilities
virtualization [##########] KVM, HVM/VME, Morpheus, ESXi, Proxmox
clustering [#########-] HVM clusters, GFS2, HA, live migration
storage [#########-] iSCSI multipath, FC, NFS, local block
networking [########--] OVS, LACP, VLANs, jumbo frames
automation [#########-] Bash, Python, Groovy, Gradle, CI glue
plugin dev [##########] Morpheus SDK: tabs, reports, controllers,
option sources, dashboards, providers
hardware [#########-] Redfish/iLO, passthrough, GPU, NVMe
observability [########--] Splunk, SNMP, metric pipelines, SQLite
desktop / rust [#######---] Tauri, Rust, cross-platform packaging
incident work [##########] calm voice, wrong hour, right fix
documentation [#########-] yes, actually. See notes/.
golf [##--------] hopeful
Ratings are self-reported and therefore inadmissible.
argus -- iLO and Redfish inside Morpheus
----------------------------------------------------------------
stack Groovy, Redfish, Morpheus plugin SDK
target Morpheus HVM (VM Essentials / Enterprise 9.0)
status published -- builtbyfood/morpheus-argus
THE GAP
Morpheus is very good at what Morpheus is for: provisioning,
inventory, orchestration, day-2 operations on the operating
system. On bare-metal HPE there is an entire layer underneath the
OS it has never been exposed to, and that layer is iLO.
iLO is where you go when the OS will not boot. When the boot order
is wrong. When BIOS settings are wrong. When you need to flash the
indicator LED so the tech on the floor knows which 1U to pull.
When a drive has predicted its own failure and the OS-side agent
has not noticed yet.
The usual workaround is OneView, or a browser bookmark per iLO
address plus a shared vault for the BMC credentials.
WHAT IT ADDS
An iLO tab on every HPE ProLiant host detail page:
- live Redfish status: power, health, BIOS, CPU, memory,
indicator LED, boot progress, hostname
- power and cooling: current draw, PSUs, hottest CPU temp and
ambient in C or F, fans, cooling zones
- power trend sparkline where the hardware tracks it
- network adapters with per-port link, speed, MAC, and
at-a-glance LED dots
- drives, DIMMs, RAID volumes, firmware inventory, active iLO
sessions, recent IML events
- one-click launch into the iLO HTML5 console, pre-authenticated
- a credentials fallback in pure HTML and CSS, no JavaScript, so
it survives restrictive browser policies
- read-only mode for operators who should see status but not
open a console
Credentials come from Infrastructure > Trust > Credentials. No
copy-paste out of a vault, no leaving Morpheus to look at the box.
A DESIGN DECISION WORTH DEFENDING
The plugin declares no custom permissions and registers no
controller routes. It inherits the permission model that is
already there.
A plugin that invents its own access control is a plugin that has
quietly become a second, worse security boundary that nobody
audits. Inheriting is less code and a much better answer.
gfs2-rescue -- adopting shared storage from a dead cluster
----------------------------------------------------------------
stack Groovy plugin, cluster tab
status published -- builtbyfood/gfs2-rescue
target Morpheus HVM 9.1
WHAT
Lets an HVM cluster take over GFS2 shared storage created by a
different cluster and attach it as an HPE Clustered Datastore --
without reformatting it, and with proof that nothing on it was lost.
For moving storage between clusters (1.3 to 2.0, for instance),
bringing it back after a cluster rebuild or a lost appliance, taking
over a replicated LUN for disaster recovery, or attaching a snapshot
alongside its original to restore VMs from it.
SEVEN STEPS, TWO OF WHICH WRITE
re-UUID (snapshots only), collect, check, repair, adopt, attach,
verify.
Collect takes a superblock backup and a manifest of every entry with
hashes and qcow2 backing chains. Check runs fsck.gfs2 read-only and
grades the result into tiers, repairing only the two it should.
Verify compares the mounted datastore against that manifest, entry
by entry and hash by hash.
Every step is validated server-side against a fresh scan of the
whole cluster, not against whatever the page last saw.
WHAT IT REFUSES TO DO
It works on GFS2 and nothing else -- devices are confirmed by
probing their superblock, and only HPE Clustered datastores are ever
matched or created. It never formats. It never writes to a volume
that is mounted, or whose lockspace is live anywhere in the cluster.
It ships switched off until an administrator turns it on.
A tool that adopts storage is one bad assumption away from eating a
LUN. The refusals are the feature.
kvm-echo -- per-VM packet capture, encrypted before it lands
----------------------------------------------------------------
stack Groovy, Morpheus 9.0+, plugin API 1.3.1
status published -- builtbyfood/kvm-echo
license Apache 2.0
THE PROBLEM
Capturing packets for a VM on an HVM host means SSHing in, working
out which vnet tap belongs to which guest, running tcpdump by hand,
then getting the pcap off the box -- usually leaving a cleartext
capture sitting on the hypervisor, and usually forgetting to delete
it.
That last part is the actual problem. A packet capture is one of the
most sensitive artifacts a platform can produce: credentials,
session tokens, internal traffic, all in the clear. The convenient
workflow and the safe workflow point in opposite directions.
WHAT IT DOES
Puts the whole thing in the Morpheus UI, on the host or the VM you
are already looking at. tcpdump output pipes straight into
AES-256-GCM before anything is written down, so ciphertext is the
only form that ever exists on the hypervisor, and it lands on tmpfs
rather than a block device. The appliance decrypts on download and
hands back a standard pcap that opens in Wireshark.
48 BPF presets grouped by traffic type, or write your own. Ad hoc
from a tab, or scheduled as an Automation Task. Every export is
byte-verified against a host-computed SHA-256 before it is decrypted.
CONSTRAINTS, NOT PREFERENCES
These are written down as hard rules, and the framing is the point:
anything that violates one is a defect, not a tradeoff.
- cleartext pcap never touches disk
- keys never appear in argv, environment, command strings or any
persistent filesystem. A fresh key per capture, delivered out of
band, opened as a file descriptor, the file unlinked immediately
- BPF filters are allowlist-parsed on the host before the command
is built; the argument vector is assembled as an array, never by
string interpolation
- ciphertext is integrity-protected against chunk reorder and
stream truncation, not merely tampering, by binding the chunk
index and a finality flag into the AEAD associated data
Filters apply on the host before encryption, so a scoped capture
contains less than an unfiltered one ever did. That is a
confidentiality property, not a convenience.
Keys expire. Once a capture's key lapses its ciphertext is
unrecoverable, deliberately.
morpheus-kvm-monitor -- CPU Ready for KVM
----------------------------------------------------------------
stack Groovy plugin + JSX dashboard widget
status published -- builtbyfood/morpheus-kvm-monitor
license Apache 2.0
WHAT
Morpheus has no equivalent of VMware's CPU Ready. This builds one.
Per-VM vCPU scheduling delay collected from virsh domstats, stored
locally, surfaced in three places: a native dashboard widget, a host
detail tab, and an Operations report with CSV export.
THE METRIC
Ready % comes from vcpu.N.delay -- nanoseconds a vCPU spent runnable
but waiting for a physical CPU -- computed as deltas between
collection cycles. Over 5% is contention; over 10% is affecting the
guest. Alongside it: Used %, Steal %, per-VM disk and network
throughput, and per-host load, CPU breakdown and MHz ratio.
A STORAGE DECISION WORTH DEFENDING
Metrics land in SQLite rather than in Morpheus's OpenSearch backend.
That looks like the lazy option right up until somebody migrates
their search backend and every historical metric living there goes
with it. A self-contained database outlives the platform decision it
would otherwise have depended on.
Every metric in the UI carries an info bubble explaining what it is
and which number should worry you. A metric nobody can interpret is a
number, not a signal.
It is the most-starred thing I have published, which says something
about how many people have been trying to answer "is this host
oversubscribed" on KVM with no tool to answer it.
migration-assist -- finishing VMware to HVM migrations
----------------------------------------------------------------
stack Groovy plugin, PowerShell guest script
status published -- builtbyfood/migration-assist, 0.3.0
WHY
The built-in migration keeps each NIC's MAC address, which only goes
so far for static-IP VMs. Four things it does not handle, each of
which breaks a cutover:
- Windows binds IP settings to the adapter's device instance, not
its MAC. The new virtio NIC is a new device; the old vmxnet3
becomes a hidden ghost adapter still holding the static IP.
- Morpheus's VirtIO injection installs boot and network drivers
only. The full guest tools suite is left for later.
- VMware Tools stays installed, and its uninstaller often fails
with MSI 1603 once the VM is no longer on VMware.
- The source VM's IP addresses vanish from Morpheus the moment the
plan powers it off, so they must be captured while the plan is
still pending.
WHAT IT DOES
Before: snapshots every source NIC, checks the plan's network
mapping, DHCP and subnet fit, that the VM is managed, that the agent
is healthy and WinRM is reachable, and scans the appliance log for
rejected guest logins. Installs the agent itself over WinRM when
Morpheus's own push fails.
During: shows seven migration phases per VM, flags phases that have
stopped updating, and surfaces vCenter's real Export OVF percentage
-- Morpheus's own transfer meter reads 0% the whole way, which is
not a reassuring thing to watch at hour two.
After: scans for VMware, Broadcom and Omnissa software, then runs
the guest script -- full VirtIO set, Tools removed, NIC settings
restored by MAC, reboot, final check.
Completed end to end on a 9.0.2 appliance with Windows Server 2022.
Linux guests and HA appliances are not covered yet, and the README
says so rather than leaving it to be found out.
hvm-updater -- upgrading the appliance from inside the appliance
----------------------------------------------------------------
stack Groovy plugin
status published -- builtbyfood/hvm-updater
WHAT
Tracks releases in My HPE Software Center, downloads and SHA-512
verifies them onto the appliance, runs the appliance upgrade, then
upgrades the Morpheus agent across HVM hosts and managed VMs. One
page.
First real run on a test appliance: 9.0.1 to 9.0.2 in 41 minutes end
to end, including a backup of the appliance VM, with the runner
journal readable throughout.
THE INTERESTING PROBLEM
You are asking a web application to upgrade itself, and step three
of the upgrade stops the web application.
So the upgrade does not run inside it. It runs as a transient
systemd unit, writing state and a log at every step. The page polls
those files; while the UI is down the browser keeps retrying; when
it returns, the page shows where the run got to. For watching from
outside entirely there are optional sinks: a webhook, a status file
mirrored to NFS or CIFS, Morpheus notifications at the bookends, and
journalctl -f.
CREDENTIALS, THE LONG WAY ROUND
Several architectures came and went here for one reason: on this
build the credential store will not hand a plugin a decrypted secret
through any in-process path. Not accountCredential, not the REST
endpoint, which masks by design. Tasks running in user context read
them without trouble; plugins get a redacted view of the same store.
The answer was to stop trying. HPE sign-in is interactive and
store-free -- a box on the page, credentials sent as headers on a
same-origin request, only the resulting token held in memory and
cleared on sign-out. Privileged appliance work runs through
executeCommandOnServer against the appliance's own ComputeServer
record.
Downloads run on the appliance, curling the pre-signed URL straight
to disk and verifying it there, so nothing multi-gigabyte ever
passes through the plugin JVM.
Four rewrites to arrive somewhere simpler than where it started. That
is usually what the ones worth keeping look like.
vm-xml-editor -- the libvirt domain XML, in the UI
----------------------------------------------------------------
stack Groovy, Morpheus 9.0.1+, plugin API 1.3.3
status published -- builtbyfood/vm-xml-editor
license Apache 2.0
WHAT
Morpheus exposes the VM settings most people need most of the time.
Underneath each KVM guest sits a libvirt domain XML holding a great
deal more: disk cache and I/O modes, IOThreads, PCI and mediated
device passthrough, USB host devices, firmware and Secure Boot,
per-device boot order, CPU model. Reaching any of it means SSH and
virsh edit.
Nine panels -- Boot, Disks, CD-ROM, USB, PCI, Network, CPU/Mem, Raw
XML, Snapshots. Nothing is written until you review a diff and
confirm, and a backup is taken before every apply. Server-side
validation refuses firmware, boot-order and disk-driver combinations
libvirt would reject, rather than letting you discover them at boot.
THE CONVERTER
If a vendor ships an OVA, a vSphere template or a GNS3 appliance and
no KVM image, you are not stuck: .ova, .ovf, .vmx, .vmdk and .gns3a
all convert, disks and NICs included. With virt-v2v present the
guest itself is adapted; without it the conversion is disk-format
only, which matters most for Windows. That limitation is in the
README rather than waiting to be discovered.
TWO THINGS IT DELIBERATELY DOES NOT DO
It does not edit a running VM's live configuration. Changes go to
the persistent definition and take effect at the next power cycle.
Live migration moves the running config, so power cycle first if you
want pending edits to travel with the VM.
It does not define a domain without being asked. Conversion produces
a disk and an XML; applying them is always a separate, explicit
step.
vm-folders -- folder organization for the VM list
----------------------------------------------------------------
stack Groovy, single jar
status published -- builtbyfood/morpheus-vm-folders
A flat list of virtual machines stops being useful somewhere around
the fiftieth one. This adds the organizational layer:
- create, rename, delete, and nest folders
- move multiple VMs at once
- start, stop, and open a console in one click
- search and sort to get to a specific machine quickly
- backup, restore, and JSON-based data management
- light and dark themes
- log viewing and filtering
Installs as one jar, no appliance restart. Also runs standalone as
plain HTML or in Docker, for people who want the organization
without the plugin.
It is the least technically interesting thing I have built and
probably the most used, which is a lesson I try to keep in view.
It has been written up by community members in three languages,
none of which I speak.
Community contribution, not an official HPE feature.
perch -- network topology for Morpheus
----------------------------------------------------------------
stack Groovy, Gradle, Morpheus plugin SDK
status internal -- not published
release 0.3.73, 1495 tests passing
WHAT
Renders live topology graphs inside Morpheus from relationships the
platform already knows about but never draws. It has outgrown its
own name: it covers the whole environment now -- storage paths and
hardware device connections as well as networking.
Seven registered surfaces: topology report, instance tab, server
tab, cluster tab, network tab, controller, and an option source.
Host telemetry collection and device/GPU passthrough mapping are
the current build front.
A THING WORTH KNOWING
Plugin.registerProvider is a Map put with no capability check --
confirmed against plugin-api 1.4.1. A provider registers cleanly
on an appliance that has no menu to reach it. So the plugin logs
7/7 registered and then logs, immediately underneath, that
registered does not mean reachable. If a surface is missing from
the UI, the licensed tier does not offer it, and the plugin cannot
detect that from inside initialize().
Making a tool honest about the limits of its own instrumentation
is most of what separates one that gets trusted from one that gets
ignored.
DEFECT CLASS E
A documentation pass turned up a prerequisite -- a sudoers drop-in
for host telemetry -- that had never actually been installed on
any appliance. Telemetry worked anyway, because the agent
installer had been granting a far broader NOPASSWD grant the whole
time.
The instruction was not wrong when it was written. It stopped
being true and nothing announced it. I started calling that a
rationale that outlives its own truth, and now I go looking for it
on purpose.
pinax -- infrastructure inventory and reporting for VME / HVM
----------------------------------------------------------------
stack Groovy, multi-module Gradle, Morpheus plugin API
status internal -- not published
release 0.4.0, 197 tests passing
WHAT
RVTools for a platform that shipped without reporting. Pinax walks
an HVM or VM Essentials estate and produces the inventory nobody
can currently get out of the product: hosts, guests, storage,
networks, and the relationships between them, at ALL / TENANT /
CLOUD / CLUSTER scope.
NAME
The Pinakes -- Callimachus's catalog of the Library of Alexandria.
First known attempt to inventory something nobody had inventoried.
ARCHITECTURE
Three modules with one hard invariant: pinax-core has zero
dependency on morpheus-plugin-api. That is enforced twice -- a
source grep and a resolved-classpath assertion in the build -- so
the collection logic can never quietly grow a coupling to the
plugin runtime.
pinax-core collection, modeling, output. Plugin-agnostic.
pinax-cli run it against an appliance without installing.
pinax-plugin the in-product surfaces.
THE INTERESTING PROBLEM
VM Essentials does not expose a Reports section at all -- its
Operations nav is Dashboard, Wiki, Activity. So ReportProvider is
effectively an Enterprise-only surface, and three of the four
scopes had to be delivered through a controller + SPA +
GlobalUIComponentProvider pattern instead, with a ClusterTabProvider
covering the fourth. Same plugin, two entirely different delivery
mechanisms depending on the tier the customer bought.
HOW THE API GETS PINNED DOWN
Vendored the OpenAPI spec and built a conformance test that
resolves $ref chains, rather than comparing against a hand-copied
list that rots in a week. That caught a real one: tenant-scoped
server queries take tenantId, not accountId. Other corrections
came from reading the shipped jars directly --
- DataQuery has no withUser(); it takes a UserIdentity ctor arg
- ComputeServer has no vmHypervisor property -- it lives on
computeServerType.vmHypervisor
- OptionType in API 1.3.1 has no optionList, which makes a
plugin-declared select without an optionSource unconditionally
broken rather than merely misconfigured
Reading the bytecode is faster than reading the docs, and it is
right more often.
vm-diagnostics -- VM logs and stuck task control
----------------------------------------------------------------
stack Groovy (report provider + server tab provider)
status internal -- not published
Surfaces per-VM log collection and stuck-task control where the
problem actually appears, instead of on an appliance shell.
IT PAID FOR ITSELF IMMEDIATELY
A VM was shutting down at random overnight. The chain: libvirtd
restarts before NFS storage is reachable, so the storage pool
fails to autostart, so the VM cannot start, so the orchestrator
enters a retry storm -- forty-plus attempts at five-second
intervals for ten minutes -- until the NFS comes back and
everything pretends it is fine.
Fix: add After=network-online.target remote-fs.target to the
libvirtd unit. Second-order fix: hardcoded USB passthrough paths
break on device re-enumeration after a libvirtd restart, so use
startupPolicy='optional'.
Nothing about that is visible from a dashboard. It is visible from
a log timeline, which is why the plugin exists.
vme-iscsi-setup -- iSCSI and multipath bring-up for VME hosts
----------------------------------------------------------------
stack Bash, MIT licensed
status published -- builtbyfood/morpheus-vme-iscsi-setup, v1.2.0
note largely superseded for VME cluster builds, which now handle
this themselves. Still the fastest way to do it on plain
Linux and KVM.
WHAT
Four-phase, idempotent iSCSI initiator and multipath configuration
for HPE VM Essentials hosts staging shared cluster storage:
1 configuration config file, env vars, or interactive prompts,
always ending in a summary and a y/N gate
2 pre-flight read-only: packages, services, NIC link, IPv4,
MTU, jumbo ping with the DF bit set, TCP 3260
reachability from every NIC to every portal
3 apply initiator name, multipath.conf, ifaces,
discovery, node-record binding, login, CHAP,
autostart
4 verify sessions, multipath -ll, lsblk, and a WWID
summary you can paste straight into the
datastore step in Morpheus
Distro-aware across the RHEL and Debian families. Optional one-way
CHAP with three credential sources in strict precedence order.
Multi-host mode over SSH.
THE BUG THAT MATTERED
iface-bound sendtargets discovery returns every portal regardless
of which one you queried. The original loop was Cartesian -- NIC
times portal -- so every host with more than one NIC and more than
one portal came up with a full mesh of sessions instead of the
intended one-session-per-fabric topology. Which is to say: every
real host.
Fixed with positional pair derivation, a NIC_PORTAL_PAIRING
override for anyone whose cabling disagrees, and a reconciliation
pass gated behind a multipath safety check.
WHY IT EXISTS
Doing this by hand across three hosts, twice, in front of a
production cutover window is an excellent way to typo yourself
into a five-hour incident. Several later fixes came from other
people running it in the field and telling me what broke.
vme-netconfig -- build, repair and audit host networking
----------------------------------------------------------------
stack Bash
status published -- builtbyfood/vme-netconfig, v2.13.0
target HVM 9.0.1+ hosts on Ubuntu 22.04 / 24.04
WHY
Network setup is the single biggest failure point in a VME install,
and it has to be right before the appliance exists -- so there is no
UI to help, and a mistake means starting over.
THREE MOMENTS IN A HOST'S LIFE
--wizard after Ubuntu, before hpe-vm. Discovers every NIC
with driver, speed, PCI address, NUMA node,
permanent MAC, carrier and LLDP peer; walks you
through role assignment; renders netplan and applies
it behind a dead-man rollback. Writes a reusable
profile.
--apply clone that profile to the rest of the cluster,
non-interactive.
--reconfigure for a host already prepped by hpe-vm whose uplinks
are wrong.
--doctor read-only audit. --doctor --fix repairs.
--rollback undo the last change.
The wizard refuses to run if netplan already declares OVS bridges --
that host is prepped and wants --reconfigure instead. Knowing which
of its own modes applies is a thing the tool should work out, not a
thing the operator should have to get right under pressure.
THE PROPERTY THAT MATTERS
Everything it changes is backed up first, verified, and restorable
from an iLO console with no network at all. A network tool that can
strand you is a network tool you will not run on a host that
matters.
Supports converged and decoupled topologies. Still correct for 9.0.x
and below; newer cluster builds handle more of this themselves.
migration toolkit -- VMware and Proxmox to HVM
----------------------------------------------------------------
stack Groovy plugin + Bash/Python task suite
status published in parts -- see builtbyfood
A guided migration wizard living inside Morpheus, plus the task
library underneath it:
- Proxmox to HVM migration path (7 tasks, vma-extractor based)
- VM storage migration toolkit (5 tasks)
- vmdk to qcow2 conversion
- UEFI/OVMF enablement, 5 variants for 5 flavors of broken
- BIOS boot menu toggling
THREE THINGS THE SDK DOES NOT TELL YOU
Script tags inside tab provider templates are stripped by the UI's
innerHTML injection. JavaScript has to arrive through a
GlobalUIComponentProvider with a CSP nonce, then wait on a
MutationObserver for the tab content to actually exist.
Permission.build() throws MissingPropertyException at runtime
because Groovy resolves class names dynamically -- @CompileStatic
on getRoutes() fixes it. An empty permission list denies access
rather than granting it, which is the safe default and also not
what anyone expects.
Handlebars left anywhere in an extracted JS file gets processed
against an empty model and silently destroys the script. Read the
data from a DOM data-island instead.
Every one of those cost hours. They are written down now so they
cost somebody else zero.
unmanaged VM disk migration -- moving discovered VMs between pools
----------------------------------------------------------------
stack Python, five Morpheus Operational Workflow tasks
status published -- builtbyfood/morpheus-unmanaged-vm-disk-migrate
license MIT
WHY
Morpheus cannot migrate disk files between storage pools for VMs it
manages as unmanaged or discovered instances on HVM. The built-in
Move only applies to fully provisioned, Morpheus-native instances.
For anything built outside Morpheus and later discovered there is no
UI path at all.
HOW
Five sequential tasks -- discovery, pre-flight, copy, switch,
cleanup -- each independent and idempotent, communicating through a
JSON state file. You run them one at a time and read the output
before continuing, which puts a natural checkpoint in front of the
two destructive ones.
rsync runs on the KVM host itself, so no disk data crosses the
network between appliance and host. Copy speed is the storage, not
the fabric.
THINGS THAT ONLY SHOW UP IN PRACTICE
qemu-img info -U uses force-share, so a running VM's disk can be
inspected without taking a write lock -- discovery needs no
downtime.
Autostart is disabled before the shutdown, not after. Power a VM off
with autostart enabled and libvirt will bring it back, sometimes
immediately, in the middle of your copy.
Progress polling captures the rsync PID directly rather than running
pgrep, because pgrep matches every other rsync on the host and two
concurrent migrations should not be able to confuse each other.
All virsh calls carry -c qemu:///system explicitly. Non-interactive
SSH sessions do not inherit LIBVIRT_DEFAULT_URI, so omitting it
fails with an error that looks like a permissions problem.
The original domain XML is base64-encoded into the state file, so
the rollback material sits alongside the migration record even after
the host-side backup is gone.
intel igpu passthrough -- GVT-g for Immich on HVM
----------------------------------------------------------------
stack libvirt, mdev, VAAPI
status published -- builtbyfood/morpheus-vme-intel-gpu-passthrough
WHAT
HPE VM Essentials has no native GPU passthrough feature in its
management UI. This documents enabling Intel GVT-g mediated device
passthrough at the KVM level, to give a dedicated Immich VM
hardware-accelerated transcoding through VAAPI.
Done as part of a full Proxmox to VME migration, to reproduce GPU
transcoding that already worked on Proxmox rather than wait for a
native solution.
THE PART THAT MAKES IT USEFUL
Full PCIe passthrough via VFIO should work on a Kaby Lake iGPU with
VT-d enabled. On this board it cannot, because the firmware DMAR
table explicitly disables IOMMU for graphics:
DMAR: Disabling IOMMU for graphics on this chipset
That is an OEM firmware decision, not a silicon limitation, and no
amount of software configuration changes it. GVT-g is the correct
path here because a mediated device needs no IOMMU at all -- and it
slices the GPU across multiple VMs, which full passthrough does not.
Saying which of those two it is matters. One sends a reader to a
BIOS setting; the other sends them to a different approach.
Proof in the repo is intel-gpu-top on the host showing the HD 630
video engine at 88% while Immich transcodes.
retro dos lab -- MS-DOS 6.22 on a modern hypervisor
----------------------------------------------------------------
stack Python Morpheus tasks, libvirt XML rewriting
status published -- builtbyfood/morpheus-vme-msdos-6.22
and builtbyfood/morpheus-vme-msdos-game-loader
WHAT
Automated, in-place installation of MS-DOS 6.22 on a KVM VM managed
by Morpheus and HPE VM Essentials. Then a second task that loads
Wolfenstein, Duke and Doom into it.
HOW, AND WHY IT IS NOT A DISK IMAGE
Rather than shipping a prebuilt image or nesting a hypervisor, the
task transforms an ordinary Morpheus-provisioned VM's libvirt XML
in place, from modern virtualization back to something DOS can boot:
machine pc-q35-8.2 -> pc (SeaBIOS)
arch x86_64 -> i686
firmware UEFI / OVMF -> SeaBIOS
disk bus virtio -> IDE
video virtio-vga -> VGA
RAM as provisioned -> 64 MB
controllers q35 PCIe tree -> removed, auto-generated
watchdog itco (q35 only) -> removed
boot per-device -> floppy, then HD
Then it boots from floppy images, swaps them between installer
phases, and cleans up after itself -- floppy removed, HD-only boot
restored, the instance tagged in Morpheus.
It is a toy, and it is also the clearest demonstration I have of what
the XML layer actually controls. Every line in that table is a thing
that can break a real migration when it is wrong.
This site has an iddqd command. It did not get there by accident.
ha-silo -- an HPE iLO fleet in Home Assistant
----------------------------------------------------------------
stack Python, Redfish, HACS integration
status published -- builtbyfood/ha-silo
license MIT
WHY
iLO 6 dropped the old RIBCL interface. Integrations built on
python-hpilo still connect, then read nothing at all -- which is a
worse failure than refusing to connect, because everything looks
fine until you notice every sensor is empty.
This one speaks Redfish end to end, so it works on Gen10 and Gen11.
DESIGN
Session-token auth: log in once per cycle, reuse the X-Auth-Token
for every read, then log out. Per-request Basic auth exhausts the
iLO's session pool and produces NoValidSession errors that look like
a network problem.
One batched coordinator poll feeds every entity -- a handful of
requests rather than dozens. The management processor on a server is
a small computer with a small budget, and a monitoring integration
that treats it like an API gateway will take it down.
Health, power state, CPU and memory, per-sensor temperatures, fan
speeds, iLO and BIOS firmware versions, firmware integrity and the
full component inventory. UI config flow, read-only iLO account, no
YAML.
The same Redfish work that went into argus, pointed at the house
instead of the datacenter.
hpe-vm rpm port -- VME host console on RHEL-family distros
----------------------------------------------------------------
stack RPM spec, Bash, Python
status internal -- proof of concept, booted successfully
The VME host console ships as a .deb and hard-gates on Ubuntu.
This port takes it to Rocky Linux, which means a cluster host no
longer has to be an Ubuntu host.
WHAT THE TEARDOWN FOUND
The binary is a GraalVM native image with bundled JVM libs, linked
against the standard ELF interpreter -- distro-agnostic in
practice. Confirmed with ldd and a strings pass looking for
baked-in netplan references before writing a single line.
The scripts sorted into three buckets: pure virsh XML (copy
verbatim), one netplan apply call (trivial swap), and the OVS
builder, which reads, mutates, and writes netplan YAML for its
entire networking layer, then hard-exits if os-release is not
Ubuntu. That one got re-targeted at nmstatectl, which is also
YAML-declarative and maps almost one-to-one.
ALSO PATCHED
libvirt domain XML for the RHEL machine type, video and sound
models, threaded disk IO; OVS from the NFV repo; and a netplan
shim for the calls baked into the binary itself.
Result: a manager VM that boots on a distro it was never supposed
to run on. The assessment document is more useful than the package.
esx-odus -- a hypervisor quest
----------------------------------------------------------------
stack single-file HTML5, 320x180 canvas, Web Audio
status published -- play it at /esx-odus.html
version 0.1.5
WHAT
A 16-bit comedy adventure about leading 463 VMs out of the land of
vSpheer and into HVM. You sit through the sales deck -- all 47
slides, for the 112th time -- survive discovery, SSH into hosts
vCenter had been lying about, catch a boot menu during POST, install
on the correct disk, open a cease and desist, babysit a migration
while Dale asks if it is done yet, and finally fight the
Subscriptionator. Devourer of budgets. Priced per core.
THE PART THAT MAKES IT A WORK ARTIFACT
The terminals take real commands. PowerCLI, esxcli, vim-cmd,
ipmitool, dlm_tool. If you have done this job your muscle memory
carries you; if you have not, the hints teach you, at half a
billable hour each, because nothing is free.
Sanity is health. Run out and you are reassigned to Sales.
Coffee powers your big moves. Billable hours are the score.
Commands that need root need sudo. Every time.
There is a golden rule of GFS2, and you will be tested on it.
The optional dungeon is a 3am page: a cluster has lost a node and
you have to recover it without making things worse. Which is to say
it is the kind of night notes/gotchas.md got written after, rendered
as a boss fight.
Choices made early come back later. Prepare properly in discovery
and the migration goes smoother, exactly as in life.
UNDER THE HOOD
One self-contained HTML file. Canvas pixel art scaled up SNES-style,
chiptune through the Web Audio API, no frameworks and no build step.
The three pixel fonts are inlined as subset WOFF2 -- Latin-1 plus the
ten symbols the game actually uses, 23 KB for all four faces -- so it
makes no outbound requests at all. A game you can save to a thumb
drive and still play on a machine with no network.
Progress and engineer name saved in the browser, chapter select
unlocking as you go.
The same constraints as the site you are reading, which was less a
decision than a habit.
vSpheer, Broadkomm, the Subscriptionator, Chad, Dale and Gary are
fictional. Any resemblance to real hypervisors, licensing models, or
people named Gary is entirely coincidental. HPE, Morpheus, ProLiant
and VM Essentials are trademarks of Hewlett Packard Enterprise; this
is a parody and is not affiliated with or endorsed by any vendor.
Type `play` to start it.
in progress -- built, building, or queued
----------------------------------------------------------------
Not everything has a repo behind it yet. These are real and running
somewhere; they are listed so this page is a description of the work
rather than a description of the public half of it.
CHARON in development
One tool in place of PuTTY, Notepad++ and MobaXterm. Tauri 2, Rust
and Svelte. Portable on Windows, deb and rpm on Linux, CI building
all three plus macOS. Profile-based connection manager, vault,
ssh_config / PuTTY / MobaXterm importers, host-key TOFU, local PTY
and serial, jump hosts, native .ppk reading, OSC 133 command marks
so the console can colour-code where each command ran and you can
pinpoint output to copy. SFTP pane that survives sudo -i, a
VS Code-like editor, CLI flag autocomplete, and an MCP server so
agents can do dev work through it.
HVM-DESK in development
VMware Workstation Pro, for KVM. A desktop hypervisor GUI with
local QEMU -- KVM acceleration on Linux, WHPX on Windows, the same
qcow2, virtio and guest agent either way -- plus Send to Morpheus
and Pull from Morpheus for handing a VM to the cluster and back.
OVA, OVF, VMX and VMDK import carrying hardware one to one with MAC
preservation, export to OVA, and a working Morpheus console. First
full laptop-to-cluster hand-off succeeded in September.
PHRYKTORIA in development
Named for the Greek fire-signal relay. A Morpheus plugin that
relays HVM quorum, fencing and cluster storage state out over SNMP,
so an RMM or SolarWinds can alarm on a fenced host without needing
to understand Morpheus. It monitors and signals; it does not act.
HVM-BACKUP in development
File-level backup and restore for HVM VMs, with a client that runs
on any OS, and a Backups view on instance and server pages that
lists what third-party products hold as well as its own. The
governing requirement: no lock-in. Backups must stay restorable
from the format itself if the tool disappears.
HVM WIKI queued
The built-in Morpheus wiki, made Confluence-shaped and locally
hosted -- real markdown, document import, wikis that are pleasant
to write.
PERCH, PINAX, VM DIAGNOSTICS, HPE-VM RPM PORT internal
Built and in use, not published. See their own pages.
fedora on a 2017 macbook pro (14,1)
----------------------------------------------------------------
stack Shell, DKMS, dracut, udev, V4L2
status published, verified July 2026
license CC BY 4.0
WHAT
A complete guide to running Fedora 44 on a MacBookPro14,1 -- the
2017 13" model without a Touch Bar. Audio, camera, WiFi, suspend and
resume, and the trackpad, all working, on kernel 7.1.3 and GNOME 50
under Wayland. Verified including an eight-hour overnight lid-close
suspend, because a suspend fix you have not slept on is not a
suspend fix.
SUSPEND
Two overlapping problems that present as one. S3 deep sleep wedges
the machine outright -- an Apple ACPI and PCIe root port interaction
that no driver works around alone. Then PCIe port power management
fights the Thunderbolt controller on the way back, hanging resume
for a minute and a half while it refuses to leave D3cold.
mem_sleep_default=s2idle pcie_port_pm=off
Both are required. Either one alone leaves you holding exactly one
of the two failures, which is why the internet is full of
half-answers for this machine.
AUDIO
The mainline driver loads, creates a sink, accepts audio and plays
nothing. The tell is one line of boot log: speaker_outs=0. The codec
is detected; the MacBook pin routing to the amplifier is not. A DKMS
driver carrying the pin quirks fixes it, built with a CFLAGS
override for a newer GCC than the source expects.
The good part is the side effect. If Bluetooth audio was choppy
before, it stops being choppy afterwards -- the malformed HDA bus
state destabilizes the whole audio graph, and Bluetooth rides on
that same graph. Two symptoms, one cause, and nobody had connected
them.
CAMERA
No mainline driver exists. The working one is reverse-engineered and
needs two proprietary blobs: base firmware pulled from Apple's own
CDN, and sensor calibration extracted from Apple's Windows Boot Camp
driver. Calibration data for a Mac camera, shipped inside a .sys
file. That is not a typo and it is my favourite fact in the guide.
The driver then applies V4L2 controls at stream-open rather than
per-frame, so tuning has to be persisted in a udev rule before any
application opens the device. There is an optional loopback and
filter pipeline for people who want more correction than that.
Documented honestly: even fully configured the image is dimmer than
macOS, because the reverse-engineered ISP does not implement the
full auto-exposure logic. That is a limitation, not something you
did wrong, and saying so saves the reader an evening.
WHAT THE REPO DOES NOT SHIP
Any of the Apple content. No firmware, no calibration files, no
installer -- instructions and an extraction script, so every user
obtains it from Apple themselves. The community convention is that
tools are fine to share and blobs are not, and that line is worth
holding even when nobody is checking.
Fedora sections are verified on my own machine. Debian, Ubuntu, KDE,
XFCE, OpenBSD and FreeBSD sections are marked, explicitly, as
best-effort translations I have not tested. Labelling that difference
is the whole distance between a guide and a rumour.
plymouth-morph-theme -- a boot animation with a narrative
----------------------------------------------------------------
stack Plymouth script plugin, Shell, ImageMagick
status published
license CC BY 4.0
WHAT
A scripted Plymouth theme that fades one logo through another to a
third during boot. Ships as apple silhouette to Tux to Fedora, but
nothing in the code is tied to those images -- hand it any three and
it plays them.
WHY
Power on a Mac and the firmware draws an Apple logo. Under macOS
that flows into the macOS boot. Under Linux it flows into a black
screen and then a login manager. It seemed worth having the boot
say, in order: this was an Apple machine, now it is a Linux machine,
specifically a Fedora machine.
THE VERSION I DID NOT BUILD
The obvious approach is replacing the firmware logo itself. On real
Apple hardware that means SPI flashing, a genuine brick risk, and a
firmware layer that does not animate anyway. Plymouth is the correct
layer: it owns the screen from immediately after firmware handoff
until the display manager appears. Choosing the layer is most of the
work, and it happened before a line of code was written.
THREE APPROACHES, ONE CHOSEN
Pre-rendered morph frames played back like a flip-book. Simple, but
linear pixel blending looks double-exposed at the midpoint.
Sprite crossfade at Plymouth's native refresh. Smooth, but two
overlapping shapes bleed through each other in the middle.
Fade through black. Each logo fades out completely, a beat of pure
black, then the next fades in alone. Chosen -- every logo appears by
itself, so nothing can look muddy.
THE PART THAT MADE IT INTERESTING
Apple hardware hands Plymouth a small early framebuffer, then swaps
it when the real graphics driver initializes. The display size
changes mid-animation and anything positioned in fixed pixels is
suddenly wrong. Handled through the display-size-changed callback,
rescaling to a fraction of screen height instead.
The detail I am fondest of: the tick counter deliberately never
resets across that swap, or across switch-root. The animation keeps
its place in the story rather than starting over. On hardware
without the transition the callback simply never fires, so the code
is a no-op there.
Ships no images at all. The inputs directory is gitignored, and the
README explains where to source each logo and which one you must not
commit to a public repository. A theme that earns somebody a DMCA
notice is not a good theme.
homelab -- 'super earth'
----------------------------------------------------------------
Named after Helldivers 2. Every host is named after something that
shoots at you. This was a choice and I am living with it.
PLATFORM
HPE Morpheus Enterprise Community Edition on 9.1, currently mid-way
through moving VMs and hardware from an HVM 1.3 cluster onto a new
HVM 2.0 one. Corosync still carries membership; Pacemaker is gone
and the appliance is the cluster manager. HA, placement and the
shared filesystem belong to the platform now, and the failure modes
moved with them.
That migration is why gfs2-rescue exists. Taking a GFS2 LUN from a
cluster that no longer owns it, without reformatting, turned out to
be a thing the platform could not do and several people needed.
COMPUTE
6 nodes. Live migration, HA restart on host loss, and distributed
placement all exercised regularly, mostly by me breaking things
on purpose.
STORAGE
Deliberately heterogeneous, because customers are:
HPFS shared cluster filesystem on multipath block
NFS v3 and v4 exports
local block on individual hosts
Running all four side by side is the fastest way to learn which
behaviors belong to the platform and which belong to the storage.
NETWORK
LACP bond for management, separate storage and compute VLANs,
jumbo frames on storage, zone-based firewalling at the edge.
ALSO RESIDENT
An ESXi environment -- an entire second faction, because migration
paths need somewhere to migrate from -- and Proxmox for the same
reason on the other side.
SERVICES
Home Assistant, running a genuinely unreasonable amount of the
house: lighting and presence automation, irrigation scheduled
against local weather and seasonal watering guidance, pool pump
scheduling, appliance integrations, arrival and door automations,
school and meal-tracking integrations built against APIs that
were never meant to be used this way
Immich, self-hosted photos with Intel GVT-g passthrough for
hardware transcoding, on a deliberately pinned 6.8 kernel --
GVT-g was removed at 6.10, so the kernel is held and documented
as held
Media, backup targets, monitoring, DNS, reverse proxy, and the
usual self-hosted sprawl that accumulates when the platform
underneath it is already there
Custom Home Assistant integrations and cards, published where
they are useful to anyone else
PURPOSE
Reproduce customer failure modes before customers do. It works.
Most of notes/gotchas.md was learned here first, at a cost of
sleep rather than a support case.
garage -- the other kind of reverse engineering
----------------------------------------------------------------
Not professional work. Same instincts, different substrate, and the
reason I can say read the bytecode with a straight face.
The car is a W211 E55 AMG. It has been apart more than most
production systems I have supported.
ECU BINARY ANALYSIS -- M113 / M113K
Bosch Motronic ME2.8.1 on an Infineon C167 running the ETAS ERCSU
runtime. Pulled apart factory .ori images and a modified flash to
work out how the thing is actually organized:
- one binary carries datasets for three chassis at once, selected
by coding block -- SL55, E55/CLS55 and S55/CL55 all live in the
same file at different offsets
- the header carries the Bosch hardware and software IDs plus the
calibration engineer's name and date, which is a delightful
thing to find in a binary from 2002
- entropy is measurably higher on the supercharged file, which
tracks: it carries blower control, bypass scheduling, boost-
corrected fueling and ignition, and a much denser torque limiter
Then the real question. The car runs a fixed pulley with the
supercharger's electromagnetic clutch physically removed, so what
did the tune have to change to stop the ECU looking for hardware
that is not there?
Answer by shape rather than by documentation: look for flat regions.
Maps that should vary and do not. A zeroed enable table, RPM
hysteresis thresholds pinned below any speed a running engine
reaches -- so the condition is permanently true -- per-gear tables
with flattened tails, neutralized correction blocks.
The tune turned out to be built on a different software version than
the reference image, which makes a direct diff unreliable. So I said
so, marked the findings as strong candidates rather than confirmed,
and wrote down exactly what would prove them. Candidate is not
verified, and the difference belongs in the writeup.
TWO CAN BUSES, ONE ADAPTER
The 2004 car speaks KWP2000. Mercedes did not monitor air/fuel ratio
until 2008, so I added a wideband gauge -- which speaks CAN, at a
different bit width and a different bus speed.
That meant logging software could read the chassis PIDs or the
wideband, never both. Either-or is a useless dataset: AFR is only
interesting next to timing, intake temp and boost at the same
instant.
The fix was to stop treating the protocol as a property of the
session and make it a property of the reading. One custom PID with
its own adapter init string: switch protocol, set the header, set
the flow-control header, mode and data, set the receive filter, take
the sample, then hand the bus back. The gauge became a sensor in the
same log as everything else.
Same answer as most integration problems. Two systems that will not
talk usually both speak fine -- what is missing is the shim that
knows how to switch contexts and switch back cleanly.
MECHANICAL
Supercharger upgrade from the factory blower to a twin-screw, and a
chiller that runs the intercooler circuit through the car's air
conditioning to pull intake temperatures down. Both are the kind of
job where the install is half the work and making everything else
tolerate the change is the other half.
Also: diagnosed a split-temperature fault where one side of the
cabin blew warmer than the other, tracked it to root cause, and
published the repair as a DIY writeup rather than keeping it.
And a large-screen head unit retrofit, which on a car of this
vintage is mostly an integration problem: keep the factory buses,
controls and amplifier working while replacing the thing in the
middle of them.
THE OTHER KIND OF CLUSTER WORK
Instrument cluster repair across W211, W219, R230 and C209. These
fail in known, repeatable ways, and they are repairable rather than
replaceable if you are willing to open them.
I have now spent real hours fixing clusters in both senses of the
word, and the diagnostic method is identical: know the normal state,
find the one component behaving differently, do not replace the
whole assembly because one part of it is loud.
TELEMETRY -- BMW 750i
OBD-II data into Home Assistant as a live dashboard: RPM, coolant
and oil temperature, intake air temp, boost, measured air/fuel,
timing advance, adapter voltage; acceleration and braking times;
instant and long-term economy, fuel level and odometer off the ECU.
Which is to say I built a monitoring stack for a car, because the
data was there and nothing was collecting it.
THE TRUCK -- 2016 F-350, 6.7 POWER STROKE
The BMW dashboard was read-only. This one has a database behind it.
A Home Assistant dashboard with Overview, Diagnostics, Maintenance,
Parts and Garage tabs, backed by a YAML package of interval helpers
and template sensors -- oil, fuel filters, rear diff, brake check --
each one a countdown rather than a date somebody is supposed to
remember.
Live data arrives off a WiCAN Pro adapter through Torque Pro, using
custom PIDs for the things a generic profile does not know about:
odometer, diesel particulate soot load, TPMS on its own header.
Two local add-ons sit underneath it:
parts intake photograph a box, barcode or OCR it, and the part
lands on a to-do list. The alternative is typing
part numbers off a shelf, which means it does not
happen.
garage a SQLite vehicle-history database with its own web
UI, Carfax import, and records for mods, components
and inspections. It pushes service baselines back
into Home Assistant, so the maintenance timers count
from what actually happened rather than from the day
I set them up.
The truck's full Carfax history went in, and every interval is
baselined off it. Intervals follow Ford's published 6.7L schedule,
taking the severe-duty variants wherever heat makes them the honest
choice.
WHERE THE SPEC IS NOT THE TRUTH
The truck runs below placard tire pressure on purpose -- it spends
most of its life unloaded and rides better soft. So the TPMS alerts
have to be built against my targets rather than Ford's, or the
dashboard cries wolf permanently and gets ignored within a week.
Corner labels are also wrong after every rotation until a relearn.
That is the same lesson as the vendor prerequisite that had quietly
stopped being true. A documented value is a default, not a fact
about this particular installation, and monitoring that alerts on
the spec instead of the system in front of you is monitoring nobody
keeps.
PUBLISHED
Rewritten vehicle-agnostic and out as builtbyfood/ha-garage: a
generator driven by a per-vehicle YAML file, generic Garage and
Parts Intake add-ons, the PID definitions, and the setup notes -- so
it works for anyone with an OBD dongle and a vehicle they would
rather not forget to service. My own environment details are dummy
values in it, which is the part people skip and then regret.
Which is to say I keep building monitoring stacks for cars, because
the data is there and nothing is collecting it. That impulse is the
entire career, honestly.
primemyai -- free AI prompt generator
----------------------------------------------------------------
primemyai.com
WHAT
Pick a template, fill in a structured brief, get a formatted
prompt to hand to Claude, ChatGPT, Gemini, Grok, Perplexity, or
Copilot. No signup, no ads, and nothing leaves the browser.
35 TEMPLATES, SIX GROUPS
write words for someone else -- writing, email, marketing copy,
pitch decks, creative, transform, documentation, sales
plan organize an action -- software, business, career move,
event, meeting prep, architecture decision, sprint,
hiring
think analyze or decide -- research, decision, brainstorm,
post-mortem, data analysis, competitive analysis, root
cause analysis
do work on existing things -- debug, code review, design
brief, threat model, incident response, API design,
database schema, test plan
grow develop yourself -- learning, interview or hard
conversation, reflection
other a custom brief for anything that does not fit
WHY
The difference between a bad answer and a good one is almost
never the model. It is whether the question arrived with its
context attached. That is the same problem as a well-written
ticket, a proper incident report, or a runbook that actually
works -- so the fix is the same: give the structure, ask for the
specifics, produce something complete.
Half of these templates exist because I wanted them for
infrastructure work. Threat model, incident response, root cause
analysis, and architecture decision are the day job wearing a
different hat.
gotchas -- hard-won, saves you an evening ---------------------------------------------------------------- QEMU / KVM Always pass --fork to qemu-nbd or the device disconnects the instant the command returns. Clean stale nbd connections before every run. A running VM's XML dump is full of runtime artifacts -- alias, index, seclabel, resource, partition -- strip them before you define it anywhere else. Post-migration boot failures are almost always one of three things: boot disk ordering (a placeholder disk landing as vda), USB hostdev expressed by address instead of vendor/product, or a libvirt boot order conflict. Remove a VM from orchestration BEFORE virsh undefine. Backwards, and you get an orphan record that outlives the thing it described. Use --nvram on undefine for UEFI guests or you leave the varstore behind. OVS A fresh install can come up with the compute bridge broken -- the bond added as type internal. Delete the port, re-add without the type, set fail_mode standalone. CLUSTER MEMBERSHIP Current Morpheus HVM layouts still run Corosync for membership -- what went away is Pacemaker. The Morpheus appliance is the cluster manager now, so resource control lives in the platform rather than in pcs on the node. A two-node cluster needs a witness to hold quorum, for the reason two-node clusters have always needed one: without a third vote a partition is a coin flip and both halves are confident they won. Give it a witness or give it a third node. There is no clever way around arithmetic. Net effect: less to misconfigure, and less to inspect directly. Know which half of the stack you are standing on before you need to. CLUSTER STORAGE HPFS is the shared filesystem on current layouts. The rule that survived the move from GFS2 intact: never change storage paths underneath a live cluster filesystem. Drain the node first. I learned that the expensive way, and nothing about a newer filesystem makes it less true. When every node fails identically and inside the same millisecond, stop investigating the nodes and go look at the array. MULTIPATH Config writers that append instead of replace will happily leave you two multipaths blocks in one file, plus stale conf.d entries from datastores that no longer exist. PERMISSIONS Correct ownership and mode on a disk file means nothing if the hypervisor user cannot traverse the parent directory. Check the whole path, not just the file. Every time. PLUGIN DEVELOPMENT Inputs come from the custom options map, not environment variables -- sudo strips those. Checkbox values arrive as the strings on and off. The credential API masks passwords, so use key auth. Enumerate real servers, not the hypervisor nodes. And when the version shown in the UI disagrees with the filename you uploaded, the manifest is what is being read, not the filename. ---------------------------------------------------------------- Written down because a fix nobody recorded is a fix you get to find again, later, under worse conditions.
how i work ---------------------------------------------------------------- BEFORE YOU TOUCH IT Reproduce it before you explain it. A theory that has not been reproduced is a guess wearing a lab coat. Run the read-only pass first. Every tool I write has a preflight phase that changes nothing and tells you what it found -- packages, links, MTU, reachability, existing state -- and then asks. Half the failures never happen because the preflight refused to proceed. Make it reversible before you run it. Sentinel values, a cleanup trap on exit, a backup of anything being overwritten. If you cannot undo it, you are not testing, you are committing. Read the license. On one plugin it turned out an SDK could not be redistributed at all, which changed the architecture: ship without the assets, probe for them at startup, degrade gracefully. Legal reading is engineering work, and finding out late is expensive. WHILE YOU BUILD IT Read the bytecode. Documentation describes intent; the shipped jar describes behavior. Disassembly settles arguments that docs prolong. Enforce the invariant in the build. If a rule matters -- a module boundary, an API contract -- make the build fail when it breaks. A rule that only lives in someone's head has already been broken and nobody has noticed yet. Inherit the security model, do not invent one. A plugin that adds its own access control has quietly become a second, worse boundary that nobody audits. Write the security properties down as constraints, not preferences. kvm-echo's README states four: cleartext never on disk, keys never in argv or environment, filters allowlist-parsed before the command is built, ciphertext protected against reorder and truncation. Phrased that way, anything violating one is a defect rather than a tradeoff, and nobody has to relitigate it at review time. Stop and re-read the approach before writing more code. The most useful thing I do in a long build is put the keyboard down and ask whether the shape is still right. It usually is. When it isn't, that question just saved a week. Automate the thing you did twice by hand. Not the first time -- that is procrastination with syntax highlighting. The second time. BEFORE YOU SHIP IT Scrub before you publish. Grep the repo for your own name, hostnames, addresses, tokens, and anything that looks base64. Every public thing I release goes through that pass, and it has caught something more than once. Ship the tool with its own limits printed on it. A tool honest about what it cannot see gets trusted with what it can. Single jar, single script, no restart. Tools with installation instructions do not get installed. AFTERWARD Write it down while it still hurts. Documentation written after the pain fades is missing the part that mattered. Then go back and check whether the reason is still true.
OUTBOUND ---------------------------------------------------------------- github github.com/builtbyfood The workshop, and currently 22 repositories. Morpheus and VM Essentials plugins written in Groovy against the plugin SDK, plus the Bash and Python tooling that grew out of field work: storage bring-up and adoption, migration kits, network build and repair, inventory, metrics, packet capture, appliance upgrades. Everything published there started as something I needed on a real deployment and could not find. Most of it installs as a single jar or a single script, because tools with installation instructions do not get installed. primemyai primemyai.com A free AI prompt generator. Pick a template, fill in a structured brief, get a formatted prompt for Claude, ChatGPT, Gemini, Grok, Perplexity, or Copilot. 35 templates across six groups -- write, plan, think, do, grow, and one open-ended custom brief. No signup, no ads, and the data never leaves your browser. Same instinct as everything else here: the useful part was always the structured brief, so build the thing that produces it. esx-odus travisdeluca.com/esx-odus.html A 16-bit comedy adventure about migrating 463 VMs off vSpheer and onto HVM, with real commands in the terminals and a final boss priced per core. Made for laughs by somebody who has sat through a lot of sales decks. No install, no account, works on a phone. More: cat projects/esx-odus.md linkedin linkedin.com/in/travisdeluca Where new plugin releases get announced first, and where the HPE and Morpheus community argues about storage. credly credly.com/users/travis-deluca Verifiable badges. See also: cat certs.txt Tip: run `open github` to launch one of these from here.
CONTACT
----------------------------------------------------------------
mail me@travisdeluca.com
github github.com/builtbyfood
linkedin linkedin.com/in/travisdeluca
GOOD REASONS TO GET IN TOUCH
Virtualization platform work, VMware or Proxmox to HVM migration,
Morpheus and VM Essentials plugin development, cluster storage
design, or why your shared datastore is behaving strangely.
ENGAGING PROFESSIONALLY
Consulting and delivery work goes through Milestone Computer Technology, Inc.,
where I do product and field solutions engineering. Reach out and
we can talk about the right shape for it.
milestonetech.com
ICQ: I genuinely wish I still knew my number.