Security and open source at the hardware/software interface. Embedded sec @ IOActive. Lead dev of ngscopeclient/libscopehal. GHz probe designer. Open source networking hardware. "So others may live" Toots searchable on tootfinder.
Public Key
npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Profile Code
nprofile1qqsvxk504cz6sw9cvdc8sqamrl0uljanntw0q8hp48799m6ynxxwfsqpz3mhxue69uhhyetvv9ujuerpd46hxtnfduqs6amnwvaz7tmwdaejumr0ds75w78g
Show more details
Published at
2026-05-19T13:23:57+02:00 Event JSON
{
"id": "795cc3b7d630645804b6f3cd8c3928222bdda61031f391e35a35a768b34aadd8" ,
"pubkey": "c35a8fae05a838b863707803bb1fdfcfcbb39adcf01ee1a9fc52ef44998ce4c0" ,
"created_at": 1779189837 ,
"kind": 0 ,
"tags": [
[
"proxy",
"https://ioc.exchange/users/azonenberg",
"activitypub"
],
[
"client",
"Mostr",
"31990:6be38f8c63df7dbf84db7ec4a6e6fbbd8d19dca3b980efad18585c46f04b26f9:mostr",
"wss://relay.ditto.pub"
]
],
"content": "{\"name\":\"Andrew Zonenberg\",\"about\":\"Security and open source at the hardware/software interface. Embedded sec @ IOActive. Lead dev of ngscopeclient/libscopehal. GHz probe designer. Open source networking hardware. \\\"So others may live\\\"\\n\\nToots searchable on tootfinder.\",\"picture\":\"https://files.ioc.exchange/iocexchange/accounts/avatars/109/320/697/263/826/535/original/452045644aa6671b.jpg\",\"banner\":\"https://files.ioc.exchange/iocexchange/accounts/headers/109/320/697/263/826/535/original/e9bac12e369b65ab.jpg\",\"nip05\":\"[email protected] \",\"fields\":[[\"ngscopeclient\",\"https://www.ngscopeclient.org/\"],[\"Blog\",\"https://serd.es\"],[\"Location\",\"Seattle area\"],[\"GitHub\",\"https://github.com/azonenberg\"]]}" ,
"sig": "7177ecf9dca71cdd8d496f5c9802282de40ee8a68e79edc6601a8f60cc89fff6b445c914e3b3a6a2fa2a2413c194f74a047276f699e91e7aace77c6e862bda17"
}
Last Notes npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg If you're cleaning a computer monitor and multiple kimwipes come off visibly discolored, and you can feel changes in surface texture from surface contamination, that's a bad sign right? This clean was clearly waaay overdue lol npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg @nprofile…czzc @nprofile…t9xu how much juice does a shipping container size store? Not that i have enough solar to make it worth it even if i did, but can you fit like 30 MWh in one? npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg @nprofile…czzc @nprofile…t9xu ah interesting, but the heating probably makes it impractical for small scale applications where you have a high surface to volume ratio causing lots of losses there. Like, for me to store a year of power generated over the summer I'd need to be storing a few tens of kW during the afternoon over months, and trying to hold a shed-sized tank of stuff at 500C for that long seems like it would use a significant fraction of the input power. If it's a facility the size of an apartment building, you have much less surface area per unit volume so it'll be easier to insulate with less losses. npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg I was hoping I'd be sub $700 for April but didn't quite generate enough juice to do it. May started out strong with four days of >60 kWh solar generation (which counted as part of the April billing cycle) before some clouds rolled in, so we'll see... May around here tends to be the transition between the monsoon season and cloudless summer weather. Hoping to hit below 3 MWh imported in May, so let's see how that goes. Here's some graphs from my analysis spreadsheet showing my demand vs utility rates over the last few years. https://files.ioc.exchange/iocexchange/media_attachments/files/116/529/636/620/185/035/original/c9af5918369a39b5.png https://files.ioc.exchange/iocexchange/media_attachments/files/116/529/639/468/136/893/original/324b3adc555abe65.png npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg Thinking more about energy upgrades in the lab: i have a 2U that just acts as a host for a bunch of FPGA devkits to run jtag/uart hosts etc. I can shut it down if I can manage to connect the relevant USB devices to the VM server (and if they work in passthrough). The problem is that the cable is going to have to go from the bottom of the rack at right up and over into the tray and down to the rack at left before reaching the hub. What are the odds of this working well with normal (not active) usb cables? How reliable are active cables, does the added latency sometimes break devices? https://files.ioc.exchange/iocexchange/media_attachments/files/116/522/555/217/096/777/original/c3cd6c5a22b681f5.jpg npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg @nprofile…02n5 something something taylor swift private jet npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg For comparison... my 16 GHz LeCroy oscilloscope puts out 40 Gsps * 4 channels * 8 bits of raw ADC samples, not counting the flatness corrections done in gateware/firmware. That's 160 GB/s or 1.28 Tbps of raw samples. That would even fit in NVLink 2.0 much less the current gen4/5 stuff. Imagine four channels of 16 GHz bandwidth waveform data straight into a (very large) GPU nonstop... We'd have to do a hell of a lot of optimization to ngscopeclient to keep up and probably add multi-GPU support but it would be so much fun lol. npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg Whenever me and my wife are driving somewhere together and are backing out of a driveway, making a turn at a sketchy intersection, etc whoever is in the passenger seat always acts as a second set of eyes calling out traffic or "clear right" etc as appropriate. From what I've seen this is common practice in military and fire department circles but pretty rare in the civilian world for some reason. Not sure why, better situational awareness definitely makes things safer. npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg Silly idea: Monocrystalline rock candy. I'm thinking you start with a normal polycrystalline rock candy block, break off one of the larger crystals as a seed, maybe polish or something to reduce defects on the broken surface, then lower into a supersaturated sucrose solution. Then let it grow for a while and hopefully extract a relatively clear, defect-free <100> sucrose crystal. npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg @nprofile…a65n it claimed submersible and i certainly got it pretty wet over the years. Pretty sure it's been submerged at least once. npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg @nprofile…a65n air quality + IP rated might be harder since you'll need the sensor exposed npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg @nprofile…a65n doesn't exist afaik npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg I have a pair of end-of-life phones (CAT S61 and S62) with FLIR Lepton thermal cores in them. Anybody know if there is a good way to disassemble them to salvage the thermal modules without destroying them? I don't care about the rest of the phone, it's gonna end up in ewaste. npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg I just tried to ctrl-O to save a file in a GUI text editor. I think I've been spending too much time in nano lately npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg @nprofile…8f6p i tried adding Trixie-backports-debug to my sources.list but apt isn't finding it... Do you have an example config line? Is it on a special mirror not like deb.debian.org? npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg @nprofile…3tsk lol moss is native to this area the grass is probably what doesn't belong npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg Hey @nprofile…8f6p am I missing something or does the trixie-backports version of mesa-vulkan-drivers not come with a -dbgsym package? I'm looking for symbols for libvulkan_radeon.so npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg Weird things that happen when you have a 100GbE pipe to your desk. 8 Gbps of sustained network traffic and the network monitor is like "yeah you're not using much bandwidth" Also I think there's a 32-bit overflow or something in xfce4-netload-plugin because the rate shows 0.00 Mbps when I get above some threshold (not sure what it is exactly but it's in the 15-40 Gbps range) https://files.ioc.exchange/iocexchange/media_attachments/files/116/497/247/334/128/502/original/8f0c619c80aec6bf.png npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg @nprofile…k7h5 you could probably write one without too much trouble but all of the ones I've seen, despite being freely downloadable, have restrictive termss attached npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg @nprofile…c63w yeah this is a CI runner so it's something I can blow away whenever I want. the issue is that I want to clone the snapshot, run the build, and shut it down and it seems like if a new kernel came out since the snapshot I need to install that and reboot before anything else npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg @nprofile…c63w yeah but like if I'm 24 hours out of date and install a package i wouldn't expect my gpu to stop working until i reboot npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg @nprofile…u57d How do you get years of uptime with arch? I have a VM snapshot that was pacman -Syu'd yesterday night and doing a pacman -Syu of a random library today broke Vulkan until I rebooted it. npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg In debian land i'm used to installing packages whenever I need them, patching every month or so, and rebooting ~annually when I have hardware maintenance to do npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg Do normal arch users just reboot constantly? I'm trying to imagine the UX of installing a random package and having vulkan suddenly stop working until you reboot because your kernel driver and userspace are out of sync or something like that npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg Using Arch as a CI runner has been quite the interesting journey due to their packaging workflow and the fact that installing any new package effectively brings in a full system upgrade with it. An upgrade that can include some combination of kernel, nvidia kernel module, and userspace Vulkan stack. Which can result in Vulkan context creation failing with errorIncompatibleDriver until you reboot and everything is in sync again. So now the "spawn VM" script has special cased logic for Arch that patches, shuts down, and restarts the VM before actually launching the build on it https://github.com/ngscopeclient/scopehal-ci-scripts/blob/main/vm/spawn-vm npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg Well this was not expected. The 100G optics I ordered for my desk came in, and swapping the pipe to my desk from 40 to 100G gave a huge improvement in Ceph performance. But I don't understand *why*. Baseline config from earlier in the year: client on 40G, cluster nodes on dual 10G, 1558 MB/s on linear reads Moving cluster nodes to 40G, client on 40G: 1864 MB/s Moving client to 100G, cluster nodes on 40G: 3787 MB/s. The confusing thing is, even 3787 MB/s is only about 30 Gbps, so after protocol overhead I would expect it to fit comfortably in 40G. Why can I get this performance with the client on 100G, but not on 40? https://files.ioc.exchange/iocexchange/media_attachments/files/116/485/423/725/240/394/original/9cc10ec6167dd1c0.png npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg @nprofile…scs2 I'm gonna be buying a r&s nrp18t soon but not for work, for my own lab mostly as a cal standard for other equipment npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg Random thought inspired vaguely by my fighting with msys2: has anyone ever built an inverse WINE? Are there any major technical blockers to doing so? By which I mean, a custom ELF loader and libc6.so that runs unmodified Linux binaries on Windows, live translating Linux syscalls and ioctls to Windows API calls npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg @nprofile…4424 let me put it this way if you are the kind of shop that runs cadence virtuoso on linux, you can afford a RHEL license :p npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg @nprofile…4424 yes RHEL is what the big eda folks run for linux, fedora is close enough we expect it to work on RHEL despite not officially testing npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg @nprofile…4424 We do have some, yes. Not sure how many. Arch has caused me a lot more pain than Fedora to date, mostly due to random package updates causing breakage but also the recent llvmpipe saga causing the GitHub release status to show as failing for the past few months (I might just disable that test suite on GH for now). npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg So now we have builders for * arch + nvidia * debian 13 + nvidia * debian 12 + nvidia * ubuntu 26.04 + llvmpipe * ubuntu 24.04 + nvidia * win11 + nvidia Fedora is next on the list as soon as I fix an issue around updating the msys2 runtime on the Windows build... npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg OK, the Arch runner is now set up and working after my rather annoying tangle with llvmpipe. The entire six-job build ran for 17 minutes. I could probably optimize it further: right now the Windows and Ubuntu 24.04 builds are among the slower ones and they share the same GPU forcing them to run sequentially. If I shuffled around which VM ran which job, I could probably fine tune a bit. Max concurrent jobs was four, three using nvidia cards plus the Ubuntu 26.04 using llvmpipe. I was watching YouTube videos in a SSH+VNC session to another VM on the server the whole time and it didn't stutter or slow down noticeably, suggesting that I wasn't overloading the server. I think my earlier problems with performance of other VMs during builds were due to the default xcp-ng scheduler granularity being "CPU" not "core"? After changing that things seemed to be a lot better. Was it maybe allocating the entire 32-core CPU to a single VM at a time or something in that mode?? npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg @nprofile…a65n well i dont have easy access to anything else mesa supported on this vm (or any arch instance i have access to) and nvidia doesnt use mesa. npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg @nprofile…a65n I was able to initialize vulkan and run a couple of shaders without any problems; 13 of my 17 unit tests crashed with weird pointer or memory corruption looking faults in code that seemed to be llvmpipe's JIT binaries from my SPIR-V. When I put an nvidia card in the same VM, it runs without issue npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg @nprofile…a65n The crash I was getting was segfaults deep inside LLVM-generated code with no symbols, and a thread named "llvmpipe". I don't have a full vulkaninfo dump from the VM at that snapshot handy but here's a subset of the info https://dashboard.ngscopeclient.org/tests/920 npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg @nprofile…a65n well i dont even have a display server *installed*, this is a headless VM, so it shouldn't be confused by that npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg @nprofile…a65n there is no x or wayland server running so i dont think there is any mesa? or is mesa involved in the vulkan stack still even when nothing is being drawn? npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg @nprofile…a65n I do not have any AMD cards in the VM server at the moment only nvidia. So my options are currently llvmpipe or a GTX 1630. Down the road when I shuffle some stuff around and put a 100G NIC in the box, I am considering trying to squeeze an AMD in to enable testing on a larger cross section of platforms npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg Is anybody using llvmpipe on Arch Linux with Vulkan? Have you tested recently? I'm seeing all kinds of horrible segfaults of trivially simple shaders in ngscopeclient + llvmpipe with Arch on both my own infrastructure and GitHub Actions. When I put a nvidia card on my Arch VM and use their Vulkan driver instead, everything works fine. npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg Alder Lake isn't far off but i need to find the chip first and they're new enough to still be a bit pricey npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg Using the solar company's 9 MWh/yr rough estimate for generation since I don't have actual numbers for the year yet, and my current power price rather than what it was when I quoted, and the actual rate of price increase, extrapolating... 2026: $1944 savings 2027: $2439 2028: $3062 ... 2033: $9532, total since install $39.2K That's roughly 8 years to breakeven, not 18. Of course, who knows if power prices will keep climbing at this rate (and how much I can cut my consumption between now and then with some of the other upgrades I have planned). I'm also not looking forward to what the same math extrapolates would be a nearly $60K/year power bill in 2033 if I can't cut my usage dramatically! npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg @nprofile…huwm my vision was a laptop with a tablet class mainboard and nice big hot swappable batteries, fully IP67 including keyboard and display, in a cnc'd aluminum orthogrid chassis with hex bolts and gaskets to close panels npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg @nprofile…huwm mmmm anodized aluminum with waterproof gaskets. Looks like the chassis of my dream laptop if you scaled it up a bit and cut the thickness down to like 4mm npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg @nprofile…4424 I have a TBT to 10G SFP+ dongle on my work laptop and keep meaning to put an intel 10G controller asic in a custom framework expansion bay board so i can have SFP+ to the other laptop natively with no dongles npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg @nprofile…4424 10gbaseT is way worse I don't use it because it's trash npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg @nprofile…4424 lol oh when i have my switch prototype on a metered lab supply i can very clearly see the few hundred mW from a 1000baseT link toggling npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg @nprofile…4424 to give you a better idea... The barely perceptible jump around 21:00 is, I think, me turning on the switch, Compare this to the much sharper and more well defined jump around 01:00 where I was playing a game on my desktop causing increased CPU/GPU load. Assuming worst case 75W 24/7 that's 1.8 kWh a day, or about $11 a month at my current power prices. But that's a worst-case number and I expect the actual number will be quite a bit less. Put a different way, this time of year on a sunny day *one* of my solar panels puts out enough power during daylight hours to run this switch for the entire day, including overnight. https://files.ioc.exchange/iocexchange/media_attachments/files/116/442/129/582/012/248/original/059d392f6d57ac1c.png npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg @nprofile…4424 I mean I have a $800ish power bill it's hard to see a load this small npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg I wish "show int status" output on the FS switch showed the vlan number that would be helpful to have at a glance. https://files.ioc.exchange/iocexchange/media_attachments/files/116/441/539/407/015/835/original/6bca3c967441b9d7.png npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg I might have to do some tuning on the Ceph cluster to push performance up a bit more, though. Right now with all 3 cluster nodes and the client on 40G, I get 991 MB/s (7.92 Gbps) of writes, 1864 MB/s (14.9 Gbps) of sequential reads, and 1841 MB/s (14.7 Gbps) of random reads. This is certainly "usably fast" but not saturating the 40G like I wanted/ npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg There are of course a lot more ports wired up and just not actively lit, I have OM4 all over the house going to wall ports around the lab and office that I can hook things to when experimenting with an FPGA board or something. But this is a nice near-term upgrade. npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg All wired up, now using 5x 40G + 1x 100G ports for the moment. Office workstation will get bumped from 40 to 100 at some point so then I'll have two free 40G ports, one of which will eventually get hooked to the VM server. Plus three free 40G ports on the 10G switch. Also I picked a hostname, it's argon2 https://files.ioc.exchange/iocexchange/media_attachments/files/116/441/480/998/810/783/original/5a71fe4ac4c7006d.jpg npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg On this switch, trunks default to *no* vlans allowed. You have to do "switchport trunk allowed vlan all" before it will pass any traffic. npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg Getting ready to deploy the new switch and realized a major problem: it doesn't have a hostname yet. My general theme for switch hostnames is hash algorithms although I've expanded a bit due to exhausting the common ones so I'm using MACs etc as namesakes too. My current fleet is keccak, blake, skein, poly1305, and ethash. All of the SHA/MD algorithms are previously used. https://files.ioc.exchange/iocexchange/media_attachments/files/116/440/553/417/582/155/original/783c4571149902dc.jpg npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg @nprofile…s32f not at all, each of those windows has dozens to hundreds of them. there's hierarchy :p npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg 740.3 kWh month to date. more than we've got during all of March. Solar power season is definitely starting to hit, we're on track to be generating something around 1.2 MWh this month. Of course this is nowhere near enough to equal my projected consumption of ~4.8 MWh but it'll certainly put a dent in my power bill. npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg @nprofile…u57d I'm pretty sure the last unsolicited call I got from work was when the low-oxygen alarm in the mechanical room went off at like 5:30am when one lab tech was the only guy in the building. Luckily it turned out to be the sensor hitting end-of-life and bricking itself rather than going out of calibration (because apparently this sensor had slipped through the cracks and not got the regular maintenance it was supposed to), not an actual N2/Ar leak. npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg @nprofile…u57d And if it's a major emergency, just go shotgun approach and call/text/email simultaneously and see which gets their attention first. (That's what my SAR unit uses for their dispatch system, you get mission callouts by SMS, email, and TTS robocall in hopes that one will catch your attention. In a literal life-or-death emergency this is an entirely justifiable level of spam) npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg @nprofile…u57d and because "be ready to answer the phone and talk to a prospect at any time during normal business hours" is pretty much the job description of a sales rep so it's not like I'm annoying them npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg @nprofile…u57d If I call a friend or coworker outside of a prearranged meeting/discussion, it's because I have a matter that requires *immediate attention* for the most part. Businesses, or people acting on behalf of a business like sales reps, are a different story. I do sometimes call rather than emailing because I'm usually looking to have a discussion that will lead to them sending me a quote or something, and the call will get it out of the way quicker than a bunch of back and forth asynchronous messaging (and most businesses aren't reachable by SMS or signal) npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg And the second one is now POSTing after the BIOS reflash too. Just gotta put the GPU and rest of the RAM back in that I took out while troubleshooting. Seriously though, *how*??? npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg The machine hanging during POST is on BIOS 1.1 which is even older (from 2021). I had never had a reason to reflash it and while I do usually run fwupd from time to time, maybe I forgot on this machine or supermicro didnt push updates? Here's hoping that's the problem. npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg @nprofile…3tsk like i either had two simultaneous cpu failures, two simultaneous psu failures, or two simultaneous mobo failures, that manifest in different ways, triggered by the ceph kernel module crash, or ????? npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg @nprofile…3tsk i seriously need an exorcist every workstation in the lab is down and i have no idea why npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg Pulled all but one dimm out of each. No change in behavior npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg oh. And the machine that is bootlooping originally came up after reboot enough that i could ping and start an ssh connection but not make it all the way to a shell. No idea what was going on there, had no visibility into the failed state npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg Already pulled the GPUs from both, both because debugging supermicro stuff is easier using the VGA port, and to reduce PSU load in case that was part of the problem. npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg The two media machines, also running Trixie with the same filesystem mounted, are unaffected. The two failed machines both have Supermicro X12SPI-TF mainboards. One has a recently acquired secondhand Xeon 8362, the other a 5320 that had previously lived in my VM server for the last few years and had no problems. RAM on both is recently installed, but hand-me-downs from other machines with no errors during POST or operation. Everything is ECC. npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg Poking around a bit more at the PCIe configuration on my storage cluster nodes and between the UEFI config and the manual I think I fully understand the topology now. (do consumer mainboards ever publish this kind of info? I'm gonna guess no... but I havent bought one in ~10 years) The CPU itself has 48 PCIe lanes, which was the most available when I deployed these (icelake-SP bumped this up to 64, and sapphire rapids to 80). They're divided into three x16 root complexes. Group 1 goes into a mux (Br3) which either connects to slot 1 as x16, or bifurcates to x4 to each of the four M.2 ports. Group 2 goes into a Broadcom PCIe switch which splits off to four slots 5:2. These slots are paired, and each pair (2/3 and 4/5) can be either dual x8s or a single x16, but all four share the single x16 to the root. Group 3 goes to a mux (Br2) which either connects to slot 7 as x16, or slots 6/7 as dual x8. It can also bifurcate to quad x4+x4+x4+x4 in slot 7 or dual x4+x4 in each of 6/7. https://files.ioc.exchange/iocexchange/media_attachments/files/116/429/518/204/191/700/original/1a697eb27094959d.png npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg @nprofile…2u6u No recording, only slides npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg @nprofile…thd8 as usual the long answer is "there's no closer form solution because it's stackup and geometry dependent so just throw it in a field solver and when the plots with/without the in plane metal look close enough that's your answer" npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg @nprofile…thd8 3x the distance to the ref plane is a reasonable first order estimate but e.g. half oz vs 2oz copper will have a big difference in edgewise coupling strength npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg This is very much a WIP, we don't have a full break of the target (it's been kind of a back-burnered project around other stuff) but I thought the obfuscation was worth talking about just because there's so little detail on this sort of tradecraft out in the public sphere npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg Slides from my HARRIS 2026 talk, "Anti-RE Countermeasures in a Real Secure Element", are up at http://harris-frab.mpi-sp.org/media/harris26/question_uploads/harris2026-zonenberg_sM1gyme.pdf npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg Has anyone done a test implementation of https://www.ietf.org/archive/id/draft-meow-mrrp-00.html yet? Is there any open example code? npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg @nprofile…s32f i have 192gb on this box so if something leaks a few tens of gigs i might not notice until I'm about to do a memory hungry task and check and go "wait why is kicad using more ram than vivado" npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg @nprofile…s32f it's definitely a leak it's not that much if i close and reopen lol npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg @nprofile…5a85 My solar system is grid tied and no use without utility power, although if I get a battery to go with it that will change and I'll be able to run off grid in a pinch supplementing with generator. My current plan is... for brief blips in power the UPS handles things for the computers, nothing else cares. For longer duration (30 minutes to maybe one day) outages I fire up the generator to keep everything running so I can work, not lose open documents, and remain fully productive. The only things that degrade in this state are the heat pump, water heater, and oven - all of which use too much power to run on generator. If I anticipate the outage being multiple days in length (i.e. it's the result of a large scale disaster not a random storm or tree on a power line), we're no longer in "business as usual" mode. In that case priority #1 is to shut down all of the tech as quickly as possible to keep the batteries charged for later. Then go dark but set up the generator. A few times a day, fire it up to cool down the fridge and run the microwave or toaster oven or something to eat as much perishable food as we can (charging phones etc at the same time), then shut it off and keep the fridge closed to stretch our limited fuel supply as long as possible. When we run out of generator fuel or fresh food, switch to shelf stable stuff. npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg @nprofile…z3nt @nprofile…a638 Sure, by all means be prepared to escape and/or resist if a band of violent looters shows up. But you also have to consider the more practical issues - some people might have medications that need refrigeration. Perishable food is going to spoil unless you can provide backup power to refrigerators and freezers. But if you can't preserve that food, you're not going to be able to last nearly as long on shelf stable pantry goods. How much water do you have? Is it drinkable? I'm downhill of a water tower so even if treatment plants and pumps fail, we would probably still have water pressure at least for a while, but if pipes are damaged it might have to be filtered or boiled before it's safe to drink. Who has stockpiles of bottled water, and how much? Who's injured? How bad? Who has first aid or medical training, and supplies? Can people be treated for minor cuts and scrapes and sent home unsupervised? Can someone with a broken limb be splinted and left with a friend until you can get them to proper care, potentially days out? Is there something really critical that demands immediate higher care, and is there anyone nearby capable of providing it? Can you get them there on foot? Sure, tourniquets and quik-clot are nice to have. But can you manage someone with a broken arm for a week until a boat shows up to take them over to the one undamaged hospital in the area? npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg @nprofile…z3nt @nprofile…a638 But the part they miss is that 99.9% of the time, going solo isn't the way to do it. The most likely scenario is that a town or neighborhood will have most or all paths in and out cut off, and some or all utilities out of service for an extended period of time. So what you need to do is address any imminent life threats (put out fires, evacuate unstable structures, rescue people trapped under debris) to the extent possible, then get together with neighbors (ideally not meeting them for the first time) and make a plan to deal with whatever is happening right where you are, because you're all you've got. The three fire stations in my city of 13000 are not sending anyone to my tiny street when the big one hits regardless of how bad the damage is or how many injuries there are. There's too many of us and not enough of them. npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg @nprofile…z3nt @nprofile…a638 And like, the preppers aren't fundamentally wrong. Natural disasters are real. Where I live especially (pacific northwest, so earthquake country split up into literal or virtual islands by bodies of water connected by bridges, ferries, etc) it's common knowledge among savvy people at all levels that when the big one hits, we're going to get cut off. The island I used to live on is connected to the outside world by one ferry and one bridge that dates to the 1940s. The county I live in now is connected to the rest of the state by... three ferries, one of which is on that island, and I think two or three bridges providing the main ways in and out. Evacuating over a quarter million people through those routes post/mid disaster is never going to be on the menu, and they're likely to all be damaged or obstructed enough that getting help in will be impractical for a while too. So you really *will* be on your own to an extent. npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg @nprofile…z3nt @nprofile…a638 It wasn't until COVID hit that I realized the brainwashing and cognitive dissonance of the American conservative/prepper community. I expected them to realize their moment had come, retreat into their bunkers, eat expired MREs when not patrolling the area wearing body armor and asbestos-filled Soviet gas masks, and threatening to shoot anyone who got close enough to infect them. Like, the apocalypse you've been waiting for is finally here. This is everything you've trained for. And you... pretend it's not real. npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg it was just a kicad project manager window i didnt even have a schematic or layout open lol. although it had been open for a month or so and i definitely had schs and layouts opened in the past npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg ... why is kicad using 40GB of RAM? npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg Predictable network interface names strike again. Removed a 10G NIC from a box and the 100G NIC that was enp27xxxx is now enp26xxx because PCIe enumeration changed npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg Getting marketing spam from airlines is the weirdest thing ever. Like are there people who get an email from Delta and just go "oh I think I'll book a flight to Paris for next week"?? If I fly somewhere it's because I have a reason to be there, usually because I am speaking at a conference or am visiting a customer. Nothing the airline spams me with will magically conjure up a presentation or client. npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg @nprofile…cy5p @nprofile…k6a7 For me the question was not whether to read the encyclopedia. It was whether to read... * The World Book from the late 80s * The massive Britannica set in my parents bedroom * The... I forget who made it, but it was a natural sciences focused one, brown binding, B&W photos, smaller volumes only about 6" tall * The kids encyclopedia with lots of pretty drawings * The British science encyclopedia * The computer/technology focused one npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg @nprofile…kuux i mean i recognize a ton of PMK probes and accessories by sight so... npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg Firestopping on cable tray penetrations between my lab and house (cc: @nprofile…vnlz ) https://files.ioc.exchange/iocexchange/media_attachments/files/116/389/978/735/148/892/original/f863a5833703c49f.jpg https://files.ioc.exchange/iocexchange/media_attachments/files/116/389/979/177/788/957/original/bff2553b16e50a50.jpg https://files.ioc.exchange/iocexchange/media_attachments/files/116/389/979/632/629/598/original/8ad0dae4ed417e90.jpg https://files.ioc.exchange/iocexchange/media_attachments/files/116/389/980/031/748/442/original/6f241b72195eb11e.jpg npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg @nprofile…8f6p @nprofile…ahke my thought was something like, for example linux-image-amd64: unchanged linux-headers-amd64: provides linux-headers linux-image-arm64: unchanged linux-headers-arm64: provides linux-headers nvidia-driver-dkms: depends on linux-headers then if you try to install nvidia-driver-dkms you would have to install a linux-headers-* package for it to work, but you'd have to explicitly pick one of them (I think). This won't save you from installing arm headers on an amd64 machine or something but at least "complain if headers are missing" would be better than the current situation alternatively maybe have update-initramfs complain at run time if DKMS is installed and there is a module that wants to build, but there's no headers/compiler for it to do so? npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg @nprofile…8f6p @nprofile…ahke well i was simplifying, i assumed in reality it would be a "kernel headers" metapackage provided by any of several different header packages depending on which image you had. npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg @nprofile…pelx scientists don't seem immune either I just rejected a slop paper as a peer reviewer. It's bad. npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg @nprofile…f38z so which is it really npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg References to specific paragraphs and clauses of IEEE 802.3 is a perfectly normal thing to find in a GLSL shader, right? Just normal shader developers doing normal shader things. https://github.com/ngscopeclient/scopehal/blob/master/scopeprotocols/shaders/Ethernet100BaseT1_Decoder.glsl?ts=4#L100 npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg if the app says "I'm OK to kill and restart for PM" that's fine. but it should be a capability you advertise, that the OS doesn't try to use if you aren't able to handle it npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg (is this something graphene or any of the other more user-respecting android forks / mobile platforms fix? When I launch an executable I want it to stay running until I tell it to stop, or it segfaults due to a bug, or I run out of ram or something. But barring exceptional circumstances it should run forever) npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg Cranky again about how Android randomly reserves the right to kill applications for power management or its own inscrutable reasons, even if you have power management settings for the app set to "unrestricted". So every time I open firefox it's a 50/50 shot whether my incognito tabs from my last browsing session (my default browsing mode to minimize leaving residue in history etc) are there or not. On a desktop OS, apps randomly being terminated for no reason would be a sev1 issue. On mobile, it's just expected. npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg It's... Bigger than i thought it would be. About time I had one of these. Quite the upgrade from my old clicky thing that doesn't go low enough for small screws on PCBs etc https://files.ioc.exchange/iocexchange/media_attachments/files/116/371/683/033/886/488/original/43d08f911e8e7f39.jpg https://files.ioc.exchange/iocexchange/media_attachments/files/116/371/683/377/578/556/original/3355666c95cd05ee.jpg npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg @nprofile…m7ah @nprofile…cs94 @nprofile…gvek wait, so if the certificate expires *existing signed binaries* will no longer run? Does this mean any signed bootloader has an inherent shelf life and will need to be re-signed every so many years even if no changes are being made to it?