Security and open source at the hardware/software interface. Embedded sec @ IOActive. Lead dev of ngscopeclient/libscopehal. GHz probe designer. Open source networking hardware. "So others may live" Toots searchable on tootfinder.
Public Key
npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Profile Code
nprofile1qqsvxk504cz6sw9cvdc8sqamrl0uljanntw0q8hp48799m6ynxxwfsqpz4mhxue69uhhyetvv9ujumt0wd68ytnsw43qn5gcpf
Show more details
Published at
2026-05-22T10:20:29Z Event JSON
{
"id": "dea2c82d09b43bfaa225d44fe4e5ec89490cae8090327ab918386fca570e5884" ,
"pubkey": "c35a8fae05a838b863707803bb1fdfcfcbb39adcf01ee1a9fc52ef44998ce4c0" ,
"created_at": 1779445229 ,
"kind": 0 ,
"tags": [
[
"proxy",
"https://ioc.exchange/users/azonenberg",
"activitypub"
],
[
"client",
"Mostr",
"31990:6be38f8c63df7dbf84db7ec4a6e6fbbd8d19dca3b980efad18585c46f04b26f9:mostr",
"wss://relay.ditto.pub"
]
],
"content": "{\"name\":\"Andrew Zonenberg\",\"about\":\"Security and open source at the hardware/software interface. Embedded sec @ IOActive. Lead dev of ngscopeclient/libscopehal. GHz probe designer. Open source networking hardware. \\\"So others may live\\\"\\n\\nToots searchable on tootfinder.\",\"picture\":\"https://files.ioc.exchange/iocexchange/accounts/avatars/109/320/697/263/826/535/original/452045644aa6671b.jpg\",\"banner\":\"https://files.ioc.exchange/iocexchange/accounts/headers/109/320/697/263/826/535/original/e9bac12e369b65ab.jpg\",\"nip05\":\"[email protected] \",\"fields\":[[\"ngscopeclient\",\"https://www.ngscopeclient.org/\"],[\"Blog\",\"https://serd.es\"],[\"Location\",\"Seattle area\"],[\"GitHub\",\"https://github.com/azonenberg\"]]}" ,
"sig": "deb203397ad2edeb7f9c1be15c898fdcee532fd736f942734f5ed0717b456133709b785a51e00a952b2d62a2dd8508bdbe71cf349f5f466f29ed4a1cecb1fbc6"
}
Last Notes npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg @nprofile…3tsk my monthly total consumption is usually in the 4.5 range npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg (I'm kinda surprised they at least were able to figure out which patients they had scheduled, and contact info for them in order to cancel, offline) npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg Set up this OEM approved cooling solution. Temps dropped by 10C or so but still getting 570 Mbps in / 138 out on iperf3. On a 40Gbase-SR4 pipe. Something is definitely wrong. https://files.ioc.exchange/iocexchange/media_attachments/files/116/574/732/766/052/930/original/17782daed6223d3c.jpg https://files.ioc.exchange/iocexchange/media_attachments/files/116/574/733/143/439/207/original/f92f6eb906d9824e.jpg npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg @nprofile…thd8 that will be nice but take a fair bit of work so not any time soon npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg Thinking more about energy upgrades in the lab: i have a 2U that just acts as a host for a bunch of FPGA devkits to run jtag/uart hosts etc. I can shut it down if I can manage to connect the relevant USB devices to the VM server (and if they work in passthrough). The problem is that the cable is going to have to go from the bottom of the rack at right up and over into the tray and down to the rack at left before reaching the hub. What are the odds of this working well with normal (not active) usb cables? How reliable are active cables, does the added latency sometimes break devices? https://files.ioc.exchange/iocexchange/media_attachments/files/116/522/555/217/096/777/original/c3cd6c5a22b681f5.jpg npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg @nprofile…02n5 something something taylor swift private jet npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg Whenever me and my wife are driving somewhere together and are backing out of a driveway, making a turn at a sketchy intersection, etc whoever is in the passenger seat always acts as a second set of eyes calling out traffic or "clear right" etc as appropriate. From what I've seen this is common practice in military and fire department circles but pretty rare in the civilian world for some reason. Not sure why, better situational awareness definitely makes things safer. npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg Silly idea: Monocrystalline rock candy. I'm thinking you start with a normal polycrystalline rock candy block, break off one of the larger crystals as a seed, maybe polish or something to reduce defects on the broken surface, then lower into a supersaturated sucrose solution. Then let it grow for a while and hopefully extract a relatively clear, defect-free <100> sucrose crystal. npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg @nprofile…a65n it claimed submersible and i certainly got it pretty wet over the years. Pretty sure it's been submerged at least once. npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg @nprofile…a65n air quality + IP rated might be harder since you'll need the sensor exposed npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg I have a pair of end-of-life phones (CAT S61 and S62) with FLIR Lepton thermal cores in them. Anybody know if there is a good way to disassemble them to salvage the thermal modules without destroying them? I don't care about the rest of the phone, it's gonna end up in ewaste. npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg I just tried to ctrl-O to save a file in a GUI text editor. I think I've been spending too much time in nano lately npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg @nprofile…8f6p i tried adding Trixie-backports-debug to my sources.list but apt isn't finding it... Do you have an example config line? Is it on a special mirror not like deb.debian.org? npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg @nprofile…3tsk lol moss is native to this area the grass is probably what doesn't belong npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg Having a bit of fun with my 100G LAN, new GPU, and a pair of ThunderScopes. Here's not one but *two* 1 Gsps live waveform streams being FFT'd simultaneously in real time. https://files.ioc.exchange/iocexchange/media_attachments/files/116/499/861/404/440/171/original/373d7d2f9bf5bc82.png https://files.ioc.exchange/iocexchange/media_attachments/files/116/499/862/680/763/256/original/483932b71acf3a52.png npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg Weird things that happen when you have a 100GbE pipe to your desk. 8 Gbps of sustained network traffic and the network monitor is like "yeah you're not using much bandwidth" Also I think there's a 32-bit overflow or something in xfce4-netload-plugin because the rate shows 0.00 Mbps when I get above some threshold (not sure what it is exactly but it's in the 15-40 Gbps range) https://files.ioc.exchange/iocexchange/media_attachments/files/116/497/247/334/128/502/original/8f0c619c80aec6bf.png npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg @nprofile…k7h5 you could probably write one without too much trouble but all of the ones I've seen, despite being freely downloadable, have restrictive termss attached npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg @nprofile…c63w yeah this is a CI runner so it's something I can blow away whenever I want. the issue is that I want to clone the snapshot, run the build, and shut it down and it seems like if a new kernel came out since the snapshot I need to install that and reboot before anything else npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg @nprofile…c63w yeah but like if I'm 24 hours out of date and install a package i wouldn't expect my gpu to stop working until i reboot npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg @nprofile…u57d How do you get years of uptime with arch? I have a VM snapshot that was pacman -Syu'd yesterday night and doing a pacman -Syu of a random library today broke Vulkan until I rebooted it. npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg Do normal arch users just reboot constantly? I'm trying to imagine the UX of installing a random package and having vulkan suddenly stop working until you reboot because your kernel driver and userspace are out of sync or something like that npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg Using Arch as a CI runner has been quite the interesting journey due to their packaging workflow and the fact that installing any new package effectively brings in a full system upgrade with it. An upgrade that can include some combination of kernel, nvidia kernel module, and userspace Vulkan stack. Which can result in Vulkan context creation failing with errorIncompatibleDriver until you reboot and everything is in sync again. So now the "spawn VM" script has special cased logic for Arch that patches, shuts down, and restarts the VM before actually launching the build on it https://github.com/ngscopeclient/scopehal-ci-scripts/blob/main/vm/spawn-vm npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg Well this was not expected. The 100G optics I ordered for my desk came in, and swapping the pipe to my desk from 40 to 100G gave a huge improvement in Ceph performance. But I don't understand *why*. Baseline config from earlier in the year: client on 40G, cluster nodes on dual 10G, 1558 MB/s on linear reads Moving cluster nodes to 40G, client on 40G: 1864 MB/s Moving client to 100G, cluster nodes on 40G: 3787 MB/s. The confusing thing is, even 3787 MB/s is only about 30 Gbps, so after protocol overhead I would expect it to fit comfortably in 40G. Why can I get this performance with the client on 100G, but not on 40? https://files.ioc.exchange/iocexchange/media_attachments/files/116/485/423/725/240/394/original/9cc10ec6167dd1c0.png npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg @nprofile…scs2 silver chloride plus sodium chloride and potassium bitartrate. I think (don't fully understand the theory of operation) the NaCl improves the extremely poor solubility of silver chloride slightly while the tartrate is ... Doing something to improve deposition and ensure a shiny smooth deposition (apparently after deposition you're supposed to wipe it down with just aqueous potassium bitartrate which somehow cleans the surface) npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg @nprofile…scs2 I'm gonna be buying a r&s nrp18t soon but not for work, for my own lab mostly as a cal standard for other equipment npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg Testing the silver chloride based plating powder on some PCB samples that I milled. It plates bare copper as well as, interestingly, ENIG coated surfaces but doesn't seem to deposit on non-metals. I milled a cavity into layer 2 on this scrap board (ignore the solder paste granules this board has been used for stencil testing too and I didn't bother fully cleaning lol. I used a combination of a lint-free swab and wooden toothpick to rub the plating composition over the surface. Plating quality deep in the cavity near the edges was defintiely sub-par, you can see the middle area is brighter looking compared to the edges because I was able to rub it more completely. But if I was trying to do rework or wire bonding to this it would probably be good. Gonna take it into work tomorrow and test if it is in fact wire bondable (and how well it stands up to oxidation etc when heated) https://files.ioc.exchange/iocexchange/media_attachments/files/116/475/022/391/964/946/original/5e120af59fb33043.jpeg https://files.ioc.exchange/iocexchange/media_attachments/files/116/475/022/392/910/293/original/ca9363165f0955b2.jpeg npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg @nprofile…4424 let me put it this way if you are the kind of shop that runs cadence virtuoso on linux, you can afford a RHEL license :p npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg @nprofile…4424 yes RHEL is what the big eda folks run for linux, fedora is close enough we expect it to work on RHEL despite not officially testing npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg @nprofile…4424 We do have some, yes. Not sure how many. Arch has caused me a lot more pain than Fedora to date, mostly due to random package updates causing breakage but also the recent llvmpipe saga causing the GitHub release status to show as failing for the past few months (I might just disable that test suite on GH for now). npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg So now we have builders for * arch + nvidia * debian 13 + nvidia * debian 12 + nvidia * ubuntu 26.04 + llvmpipe * ubuntu 24.04 + nvidia * win11 + nvidia Fedora is next on the list as soon as I fix an issue around updating the msys2 runtime on the Windows build... npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg I ran a few quick calculations on power consumption: measuring at the UPS, the baseline consumption of the lab a minute or so before kicking off a CI job was 3828W. During testing it hovered around 3900-3950, with a peak of 4007. Using worst-case numbers assuming (4007-3828) = 179W extra power consumption during the build compared to idle, sustained over 17 minutes, a build would use about 0.05 kWh of power, which as of my last bill would cost me about one cent. Since the actual build is not running at that peak power the entire time, the actual energy requirement is substantially less. npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg @nprofile…a65n well i dont have easy access to anything else mesa supported on this vm (or any arch instance i have access to) and nvidia doesnt use mesa. npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg @nprofile…a65n I was able to initialize vulkan and run a couple of shaders without any problems; 13 of my 17 unit tests crashed with weird pointer or memory corruption looking faults in code that seemed to be llvmpipe's JIT binaries from my SPIR-V. When I put an nvidia card in the same VM, it runs without issue npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg @nprofile…a65n The crash I was getting was segfaults deep inside LLVM-generated code with no symbols, and a thread named "llvmpipe". I don't have a full vulkaninfo dump from the VM at that snapshot handy but here's a subset of the info https://dashboard.ngscopeclient.org/tests/920 npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg @nprofile…a65n well i dont even have a display server *installed*, this is a headless VM, so it shouldn't be confused by that npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg @nprofile…a65n there is no x or wayland server running so i dont think there is any mesa? or is mesa involved in the vulkan stack still even when nothing is being drawn? npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg @nprofile…a65n I do not have any AMD cards in the VM server at the moment only nvidia. So my options are currently llvmpipe or a GTX 1630. Down the road when I shuffle some stuff around and put a 100G NIC in the box, I am considering trying to squeeze an AMD in to enable testing on a larger cross section of platforms npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg Is anybody using llvmpipe on Arch Linux with Vulkan? Have you tested recently? I'm seeing all kinds of horrible segfaults of trivially simple shaders in ngscopeclient + llvmpipe with Arch on both my own infrastructure and GitHub Actions. When I put a nvidia card on my Arch VM and use their Vulkan driver instead, everything works fine. npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg Tossed a PCIe passthrough GTX 1630 in the builder VM and fixed a few other things and now it's churning away again. We'll let this build do its thing and hopefully it'll pass tests, if not I need to troubleshoot further. Gotta love those plots of 100% CPU usage lol. No sense having all this big iron if it's not going to be put to use. https://files.ioc.exchange/iocexchange/media_attachments/files/116/465/321/026/731/045/original/b69197e80d61f965.png npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg Using the solar company's 9 MWh/yr rough estimate for generation since I don't have actual numbers for the year yet, and my current power price rather than what it was when I quoted, and the actual rate of price increase, extrapolating... 2026: $1944 savings 2027: $2439 2028: $3062 ... 2033: $9532, total since install $39.2K That's roughly 8 years to breakeven, not 18. Of course, who knows if power prices will keep climbing at this rate (and how much I can cut my consumption between now and then with some of the other upgrades I have planned). I'm also not looking forward to what the same math extrapolates would be a nearly $60K/year power bill in 2033 if I can't cut my usage dramatically! npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg Looking at the trend, I don't see it getting better soon if ever. I generated ~0.7 MWh last month which means that my actual consumption sans solar was more like 4.8 MWh. Back in December 2025 (pre solar) I paid $920 for about that much power, but the rate went up 14.8% between December and April so had I used that much power in April it would have cost me $1056. In April 2027 if I get another 25.5% price jump like I did between April 2025 and 2026, the same consumption would be $1326. https://files.ioc.exchange/iocexchange/media_attachments/files/116/450/564/670/453/269/original/283cb17b4a623659.png npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg @nprofile…huwm my vision was a laptop with a tablet class mainboard and nice big hot swappable batteries, fully IP67 including keyboard and display, in a cnc'd aluminum orthogrid chassis with hex bolts and gaskets to close panels npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg @nprofile…huwm mmmm anodized aluminum with waterproof gaskets. Looks like the chassis of my dream laptop if you scaled it up a bit and cut the thickness down to like 4mm npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg @nprofile…4424 I have a TBT to 10G SFP+ dongle on my work laptop and keep meaning to put an intel 10G controller asic in a custom framework expansion bay board so i can have SFP+ to the other laptop natively with no dongles npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg @nprofile…4424 10gbaseT is way worse I don't use it because it's trash npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg @nprofile…4424 lol oh when i have my switch prototype on a metered lab supply i can very clearly see the few hundred mW from a 1000baseT link toggling npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg @nprofile…4424 to give you a better idea... The barely perceptible jump around 21:00 is, I think, me turning on the switch, Compare this to the much sharper and more well defined jump around 01:00 where I was playing a game on my desktop causing increased CPU/GPU load. Assuming worst case 75W 24/7 that's 1.8 kWh a day, or about $11 a month at my current power prices. But that's a worst-case number and I expect the actual number will be quite a bit less. Put a different way, this time of year on a sunny day *one* of my solar panels puts out enough power during daylight hours to run this switch for the entire day, including overnight. https://files.ioc.exchange/iocexchange/media_attachments/files/116/442/129/582/012/248/original/059d392f6d57ac1c.png npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg @nprofile…4424 (that should go down now that the sun is shining more) npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg @nprofile…4424 I mean I have a $800ish power bill it's hard to see a load this small npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg @nprofile…4424 75w is datasheet max. ups is metered but there's 4 kW of load tens of watts are lost in the noise npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg I wish "show int status" output on the FS switch showed the vlan number that would be helpful to have at a glance. https://files.ioc.exchange/iocexchange/media_attachments/files/116/441/539/407/015/835/original/6bca3c967441b9d7.png npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg I might have to do some tuning on the Ceph cluster to push performance up a bit more, though. Right now with all 3 cluster nodes and the client on 40G, I get 991 MB/s (7.92 Gbps) of writes, 1864 MB/s (14.9 Gbps) of sequential reads, and 1841 MB/s (14.7 Gbps) of random reads. This is certainly "usably fast" but not saturating the 40G like I wanted/ npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg All wired up, now using 5x 40G + 1x 100G ports for the moment. Office workstation will get bumped from 40 to 100 at some point so then I'll have two free 40G ports, one of which will eventually get hooked to the VM server. Plus three free 40G ports on the 10G switch. Also I picked a hostname, it's argon2 https://files.ioc.exchange/iocexchange/media_attachments/files/116/441/480/998/810/783/original/5a71fe4ac4c7006d.jpg npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg Getting ready to deploy the new switch and realized a major problem: it doesn't have a hostname yet. My general theme for switch hostnames is hash algorithms although I've expanded a bit due to exhausting the common ones so I'm using MACs etc as namesakes too. My current fleet is keccak, blake, skein, poly1305, and ethash. All of the SHA/MD algorithms are previously used. https://files.ioc.exchange/iocexchange/media_attachments/files/116/440/553/417/582/155/original/783c4571149902dc.jpg npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg @nprofile…s32f not at all, each of those windows has dozens to hundreds of them. there's hierarchy :p npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg Do I have a problem? https://files.ioc.exchange/iocexchange/media_attachments/files/116/435/566/289/755/804/original/26891bbf1a1eec55.png npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg Spider Georg (who lives in a drain not a cave and definitely does not eat 10,000 other spiders a day) came out of the drain for a bit and I got a pic. Pretty sure he's a hobo spider or giant house spider but not 100%. He can't get out of the tub because the walls are too slippery (I've seen him try) so whatever he is, he's stuck in there lol. https://files.ioc.exchange/iocexchange/media_attachments/files/116/435/521/250/535/642/original/7fc70a691494e8f0.jpg npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg 740.3 kWh month to date. more than we've got during all of March. Solar power season is definitely starting to hit, we're on track to be generating something around 1.2 MWh this month. Of course this is nowhere near enough to equal my projected consumption of ~4.8 MWh but it'll certainly put a dent in my power bill. npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg @nprofile…u57d I'm pretty sure the last unsolicited call I got from work was when the low-oxygen alarm in the mechanical room went off at like 5:30am when one lab tech was the only guy in the building. Luckily it turned out to be the sensor hitting end-of-life and bricking itself rather than going out of calibration (because apparently this sensor had slipped through the cracks and not got the regular maintenance it was supposed to), not an actual N2/Ar leak. npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg @nprofile…u57d And if it's a major emergency, just go shotgun approach and call/text/email simultaneously and see which gets their attention first. (That's what my SAR unit uses for their dispatch system, you get mission callouts by SMS, email, and TTS robocall in hopes that one will catch your attention. In a literal life-or-death emergency this is an entirely justifiable level of spam) npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg @nprofile…u57d and because "be ready to answer the phone and talk to a prospect at any time during normal business hours" is pretty much the job description of a sales rep so it's not like I'm annoying them npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg Ok all reassembled and patched and took care of some other housekeeping plus reopened most of the wip stuff i wanted to keep active. Now it's almost 5am and I'm about where I wanted to be at 2200 last night (pcie all shuffled, 100g nic installed in last cluster node, desktops functional) npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg And the second one is now POSTing after the BIOS reflash too. Just gotta put the GPU and rest of the RAM back in that I took out while troubleshooting. Seriously though, *how*??? npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg The machine hanging during POST is on BIOS 1.1 which is even older (from 2021). I had never had a reason to reflash it and while I do usually run fwupd from time to time, maybe I forgot on this machine or supermicro didnt push updates? Here's hoping that's the problem. npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg @nprofile…3tsk like i either had two simultaneous cpu failures, two simultaneous psu failures, or two simultaneous mobo failures, that manifest in different ways, triggered by the ceph kernel module crash, or ????? npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg @nprofile…3tsk i seriously need an exorcist every workstation in the lab is down and i have no idea why npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg Pulled all but one dimm out of each. No change in behavior npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg oh. And the machine that is bootlooping originally came up after reboot enough that i could ping and start an ssh connection but not make it all the way to a shell. No idea what was going on there, had no visibility into the failed state npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg Already pulled the GPUs from both, both because debugging supermicro stuff is easier using the VGA port, and to reduce PSU load in case that was part of the problem. npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg Poking around a bit more at the PCIe configuration on my storage cluster nodes and between the UEFI config and the manual I think I fully understand the topology now. (do consumer mainboards ever publish this kind of info? I'm gonna guess no... but I havent bought one in ~10 years) The CPU itself has 48 PCIe lanes, which was the most available when I deployed these (icelake-SP bumped this up to 64, and sapphire rapids to 80). They're divided into three x16 root complexes. Group 1 goes into a mux (Br3) which either connects to slot 1 as x16, or bifurcates to x4 to each of the four M.2 ports. Group 2 goes into a Broadcom PCIe switch which splits off to four slots 5:2. These slots are paired, and each pair (2/3 and 4/5) can be either dual x8s or a single x16, but all four share the single x16 to the root. Group 3 goes to a mux (Br2) which either connects to slot 7 as x16, or slots 6/7 as dual x8. It can also bifurcate to quad x4+x4+x4+x4 in slot 7 or dual x4+x4 in each of 6/7. https://files.ioc.exchange/iocexchange/media_attachments/files/116/429/518/204/191/700/original/1a697eb27094959d.png npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg @nprofile…2u6u No recording, only slides npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg @nprofile…thd8 as usual the long answer is "there's no closer form solution because it's stackup and geometry dependent so just throw it in a field solver and when the plots with/without the in plane metal look close enough that's your answer" npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg @nprofile…thd8 3x the distance to the ref plane is a reasonable first order estimate but e.g. half oz vs 2oz copper will have a big difference in edgewise coupling strength npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg @nprofile…thd8 there's no hard cutoffs since fields never decay fully to zero at any distance. The practical answer is, the point at which the coupling in-plane becomes negligible compared to the coupling to the reference plane. This is dependent on copper thickness, soldermask presence and Dk compared to substrate Dk, etc. And exactly how "negligible" you want to let the coupling get before you can ignore it npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg This is very much a WIP, we don't have a full break of the target (it's been kind of a back-burnered project around other stuff) but I thought the obfuscation was worth talking about just because there's so little detail on this sort of tradecraft out in the public sphere npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg Slides from my HARRIS 2026 talk, "Anti-RE Countermeasures in a Real Secure Element", are up at http://harris-frab.mpi-sp.org/media/harris26/question_uploads/harris2026-zonenberg_sM1gyme.pdf npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg Has anyone done a test implementation of https://www.ietf.org/archive/id/draft-meow-mrrp-00.html yet? Is there any open example code? npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg @nprofile…s32f i have 192gb on this box so if something leaks a few tens of gigs i might not notice until I'm about to do a memory hungry task and check and go "wait why is kicad using more ram than vivado" npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg @nprofile…s32f it's definitely a leak it's not that much if i close and reopen lol npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg @nprofile…z3nt @nprofile…a638 Sure, by all means be prepared to escape and/or resist if a band of violent looters shows up. But you also have to consider the more practical issues - some people might have medications that need refrigeration. Perishable food is going to spoil unless you can provide backup power to refrigerators and freezers. But if you can't preserve that food, you're not going to be able to last nearly as long on shelf stable pantry goods. How much water do you have? Is it drinkable? I'm downhill of a water tower so even if treatment plants and pumps fail, we would probably still have water pressure at least for a while, but if pipes are damaged it might have to be filtered or boiled before it's safe to drink. Who has stockpiles of bottled water, and how much? Who's injured? How bad? Who has first aid or medical training, and supplies? Can people be treated for minor cuts and scrapes and sent home unsupervised? Can someone with a broken limb be splinted and left with a friend until you can get them to proper care, potentially days out? Is there something really critical that demands immediate higher care, and is there anyone nearby capable of providing it? Can you get them there on foot? Sure, tourniquets and quik-clot are nice to have. But can you manage someone with a broken arm for a week until a boat shows up to take them over to the one undamaged hospital in the area? npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg @nprofile…z3nt @nprofile…a638 But the part they miss is that 99.9% of the time, going solo isn't the way to do it. The most likely scenario is that a town or neighborhood will have most or all paths in and out cut off, and some or all utilities out of service for an extended period of time. So what you need to do is address any imminent life threats (put out fires, evacuate unstable structures, rescue people trapped under debris) to the extent possible, then get together with neighbors (ideally not meeting them for the first time) and make a plan to deal with whatever is happening right where you are, because you're all you've got. The three fire stations in my city of 13000 are not sending anyone to my tiny street when the big one hits regardless of how bad the damage is or how many injuries there are. There's too many of us and not enough of them. npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg @nprofile…z3nt @nprofile…a638 And like, the preppers aren't fundamentally wrong. Natural disasters are real. Where I live especially (pacific northwest, so earthquake country split up into literal or virtual islands by bodies of water connected by bridges, ferries, etc) it's common knowledge among savvy people at all levels that when the big one hits, we're going to get cut off. The island I used to live on is connected to the outside world by one ferry and one bridge that dates to the 1940s. The county I live in now is connected to the rest of the state by... three ferries, one of which is on that island, and I think two or three bridges providing the main ways in and out. Evacuating over a quarter million people through those routes post/mid disaster is never going to be on the menu, and they're likely to all be damaged or obstructed enough that getting help in will be impractical for a while too. So you really *will* be on your own to an extent. npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg @nprofile…z3nt @nprofile…a638 It wasn't until COVID hit that I realized the brainwashing and cognitive dissonance of the American conservative/prepper community. I expected them to realize their moment had come, retreat into their bunkers, eat expired MREs when not patrolling the area wearing body armor and asbestos-filled Soviet gas masks, and threatening to shoot anyone who got close enough to infect them. Like, the apocalypse you've been waiting for is finally here. This is everything you've trained for. And you... pretend it's not real. npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg it was just a kicad project manager window i didnt even have a schematic or layout open lol. although it had been open for a month or so and i definitely had schs and layouts opened in the past npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg Predictable network interface names strike again. Removed a 10G NIC from a box and the 100G NIC that was enp27xxxx is now enp26xxx because PCIe enumeration changed npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg @nprofile…cy5p @nprofile…k6a7 For me the question was not whether to read the encyclopedia. It was whether to read... * The World Book from the late 80s * The massive Britannica set in my parents bedroom * The... I forget who made it, but it was a natural sciences focused one, brown binding, B&W photos, smaller volumes only about 6" tall * The kids encyclopedia with lots of pretty drawings * The British science encyclopedia * The computer/technology focused one npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg @nprofile…kuux i mean i recognize a ton of PMK probes and accessories by sight so... npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg Firestopping on cable tray penetrations between my lab and house (cc: @nprofile…vnlz ) https://files.ioc.exchange/iocexchange/media_attachments/files/116/389/978/735/148/892/original/f863a5833703c49f.jpg https://files.ioc.exchange/iocexchange/media_attachments/files/116/389/979/177/788/957/original/bff2553b16e50a50.jpg https://files.ioc.exchange/iocexchange/media_attachments/files/116/389/979/632/629/598/original/8ad0dae4ed417e90.jpg https://files.ioc.exchange/iocexchange/media_attachments/files/116/389/980/031/748/442/original/6f241b72195eb11e.jpg npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg @nprofile…8f6p @nprofile…ahke my thought was something like, for example linux-image-amd64: unchanged linux-headers-amd64: provides linux-headers linux-image-arm64: unchanged linux-headers-arm64: provides linux-headers nvidia-driver-dkms: depends on linux-headers then if you try to install nvidia-driver-dkms you would have to install a linux-headers-* package for it to work, but you'd have to explicitly pick one of them (I think). This won't save you from installing arm headers on an amd64 machine or something but at least "complain if headers are missing" would be better than the current situation alternatively maybe have update-initramfs complain at run time if DKMS is installed and there is a module that wants to build, but there's no headers/compiler for it to do so? npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg @nprofile…8f6p @nprofile…ahke well i was simplifying, i assumed in reality it would be a "kernel headers" metapackage provided by any of several different header packages depending on which image you had. npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg Hey @nprofile…8f6p any idea why nvidia-kernel-dkms (on trixie at least) does not depend on linux-headers-$arch? The end result of installing it without kernel headers is that the driver doesn't work, because it can't compile without the headers. I can't think of any situation in which this is a desirable state npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg @nprofile…pelx scientists don't seem immune either I just rejected a slop paper as a peer reviewer. It's bad. npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg @nprofile…f38z so which is it really npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg References to specific paragraphs and clauses of IEEE 802.3 is a perfectly normal thing to find in a GLSL shader, right? Just normal shader developers doing normal shader things. https://github.com/ngscopeclient/scopehal/blob/master/scopeprotocols/shaders/Ethernet100BaseT1_Decoder.glsl?ts=4#L100 npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg if the app says "I'm OK to kill and restart for PM" that's fine. but it should be a capability you advertise, that the OS doesn't try to use if you aren't able to handle it npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg (is this something graphene or any of the other more user-respecting android forks / mobile platforms fix? When I launch an executable I want it to stay running until I tell it to stop, or it segfaults due to a bug, or I run out of ram or something. But barring exceptional circumstances it should run forever) npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg Cranky again about how Android randomly reserves the right to kill applications for power management or its own inscrutable reasons, even if you have power management settings for the app set to "unrestricted". So every time I open firefox it's a 50/50 shot whether my incognito tabs from my last browsing session (my default browsing mode to minimize leaving residue in history etc) are there or not. On a desktop OS, apps randomly being terminated for no reason would be a sev1 issue. On mobile, it's just expected. npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg It's... Bigger than i thought it would be. About time I had one of these. Quite the upgrade from my old clicky thing that doesn't go low enough for small screws on PCBs etc https://files.ioc.exchange/iocexchange/media_attachments/files/116/371/683/033/886/488/original/43d08f911e8e7f39.jpg https://files.ioc.exchange/iocexchange/media_attachments/files/116/371/683/377/578/556/original/3355666c95cd05ee.jpg npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg @nprofile…m7ah @nprofile…cs94 @nprofile…gvek wait, so if the certificate expires *existing signed binaries* will no longer run? Does this mean any signed bootloader has an inherent shelf life and will need to be re-signed every so many years even if no changes are being made to it? npub1cddglts94qutscms0qpmk87lel9m8xku7q0wr20u2th5fxvvunqqxz9vpd Andrew Zonenberg As a test, to verify the hypothesis that the eye is stealing GPU time from the demux, I deleted the eye pattern from the filter graph. Refresh rate went from about 9.8 WFM/s to 11.5, not *quite* real time (that would be 12.5) but pretty close and certainly a big improvement. But I don't want to delete it, and I have no easy way to delay it or de-prioritize it: while Vulkan queues have priority associated with them, the assignment of worker threads (and thus queues) to filter blocks is essentially random (whichever one catches the condition variable first) and thus not usable as a precedence mechanism. Messing with the scheduler also seems like a bad idea here, because any changes to tune for this particular workload unless very well tested might cause problems for other filter graphs. So I don't want any hacky special casing.