"A start job is running for dev-disk-by" and other horrors

My desktop system at home is currently running OpenSUSE Tumbleweed.  It used to run Debian until I got caught in an upgrade version-locked disaster.  Then it ran OpenSolaris until Oracle made that an unlikely proposition.

I've been reasonably happy with OpenSUSE until I made the mistake of trying to do a "zypper dup" recently, and the reboot showed me this:

[***   ] A start job is running for dev-disk-by\x2duuid-1a0dc1c5\x2d26cc\x2d45ff\x2da7b1\x2d1f827c971ff9.device (15s / no limit)

As long as one might care to sit there and watch, it never completed whatever task it was trying to perform.

I was able to boot up with a rescue CD (thank goodness I downloaded that first), and was able to mount the disks with no trouble.  But no amount of fooling around would make it boot.  It was not a happy evening.

After quite a bit of fooling about, I discovered that there were two serious problems that I had to fix manually, and I'm writing this for those who might have run into similar problems:

1. dracut is missing bits

Crucial kernel drivers go missing when dracut builds a new initrd image, and other bits get included whether you like it or not.  I have a mix of file systems in use, and here are the new configuration bits I had to add to /etc/dracut.conf.d:

add_dracutmodules+="btrfs"
add_drivers+="btrfs zlib_deflate xor raid6_pq"
add_drivers+="md-mod raid1 raid456"
omit_drivers+="nouveau"

That it would exclude the RAID and btrfs drivers by default was very surprising.

2. udevd is broken by default

The default configuration of udevd simply doesn't work right.  It limits itself to an absurdly tiny number of processes, and ends up failing to run trivial scripts needed by the Linux "MD" disk subsystem.  That's a big part of my boot problem.  The solution is to create a file named /etc/systemd/system/systemd-udevd.service with this inside:

[Unit]
Description=udev Kernel Device Manager
Documentation=man:systemd-udevd.service(8) man:udev(7)
DefaultDependencies=no
Wants=systemd-udevd-control.socket systemd-udevd-kernel.socket
After=systemd-udevd-control.socket systemd-udevd-kernel.socket systemd-sysusers.service
Before=sysinit.target
ConditionPathIsReadWrite=/sys

[Service]
Type=notify
OOMScoreAdjust=-1000
Sockets=systemd-udevd-control.socket systemd-udevd-kernel.socket
Restart=always
RestartSec=0
ExecStart=/usr/lib/systemd/systemd-udevd
MountFlags=slave
KillMode=mixed
WatchdogSec=3min
TasksMax=infinity

The important part is that "TasksMax=infinity" line.  That's what fixes the system so that it will actually boot again.

Missing Windows users? The badly-named "net" command is your friend.

I have a lousy old laptop that I use for IMC Club presentations and the like.  I wish I could afford something better, but it mostly works almost well enough to keep me from bothering to look around.

Except every once in a while, it falls apart.  Windows is, unfortunately, really terrible in that way.  The latest problem it had is so strange and so obscure that I felt I had to write about it just in case someone else runs into it.

First symptom: all users accounts are gone from the login screen.  In fact, the login screen itself is skipped, and it goes straight to asking for the Admin password on boot.

On getting in, the "user accounts" tool shows nothing but Admin and Guest.  All user accounts are just plain gone.  Attempting to add the user accounts back results in an error message saying that the user "already has permission to access this computer."  Well, that's unhelpful.

The parental controls section shows the accounts.  The files are still there under C:\Users.  Everything seems in place, but nobody can log in.  Regedit shows nothing interesting.  Googling around for all sorts of related phrases shows that quite a few people have experienced this, but nobody has solved it.

I just solved it.  Typing "net user Jim", I can see output that ends like this:

    Local Group Memberships
    Global Group memberships   *None
    The command completed successfully.

I tried adding a dummy account, and it showed up with "Local Group Memberships" set to *Users.  That's the key.  For some reason, all of the accounts had been kicked out of the "Users" group, and that's why they were gone from the login screen.  Adding them back in looks like this:

    net localgroup Users Jim /add
    net localgroup Users Madeline /add

After doing that, the system was back to normal.  Ah, Windows.  Thanks for wasting so many hours of my life.

Unfortunate Result

I had high hopes for AMC's new "Halt and Catch Fire."  It's great to see the pie man back on TV.  And the subject is one that I'm very interested in, as I've been involved with computers since well before the introduction of the PC.

From the start, though, the show went off the rails.  They tried to explain the title with a "definition" that was complete gibberish.  That phrase doesn't refer to an actual computer instruction or program; it's an old joke.  And, sadly, it seems that the folks working on that show didn't get the joke.

Back then, writing in assembly was more common than it is today.  Just for fun, many people made lists of fake instructions and passed them around by photocopies and pinning them to bulletin boards in office hallways.  Typical entries were "Rewind and Stretch Tape," "Jump To Random Location," and "Execute Operator."  No machine had those instructions.  They were jokes, as was "Halt and Catch Fire."  But they missed that entirely.

That was just the start.  They missed the boat in so many ways that it's hard to count them.  I don't want to just pick nits -- the pattern "on, on, off, on" or 1101 is a hexadecimal D, not a B -- but the basic plot points are way off the mark.  The BIOS wasn't in any way the "secret sauce."  It just does basic hardware initialization, a little bit of self-test, and then loads bootstrap code from a floppy drive or (back then) a cassette recorder.  Heck, IBM published the source code in the technical reference manual!  And if you wanted to extract the ROM contents, even back then, it wouldn't be as crazily hard as it was portrayed for dramatic purposes.  You can just read the memory.

So, it's already headed in a bad direction, and it's hard to see why that's the case.  The real story has all sorts of interesting twists and turns -- just google "Gary Kildall" for part of it -- and it's a shame to see the opportunity squandered.  I'll keep watching for a few episodes, hoping that they'll turn it around, but it's not looking good so far.  Instead of portraying life in a start-up, though, it seems to be mimicking the failure of one.

Next Adventure

I'm now working on buying a plane. It's a bit more complicated -- and taxing -- than I'd expected.

The first one I looked at seemed almost perfect: right model and year, no significant damage history, good price, low time, and hangared within an hour's drive. I went out to see it, and it seemed well-kept. He had a couple of sports cars stored in the hangar with the plane, and he obviously took good care of his machines. The owner just didn't get to fly much anymore and wanted to sell. (I'm going to avoid specifically identifying the plane or owner here.)

He let me look over the books, and everything seemed to be in order. The I&A logs looked right, the engine tests looked great, and the known sorts of problems for that plane were all absent.

After reading detailed postings on a few other sites (particularly www.aopa.org and www.cardinalflyers.com), I decided that it'd be worthwhile to invest in an appraisal before going further. I contacted the NAAA, and was given a pointer to Ken Dantzig. He was able to make some time to go out and pore over the log books and the plane for several hours. (Appraiser's aren't mechanics and don't open things up; they spend most of the time with the records and the observable details.)

I was expecting this to be a slightly expensive formality. It wasn't. He found several issues that I just wouldn't have noticed on my own because I wasn't looking in the right places. After a few days of anxiously waiting, I got the written report. It surprised me -- the value was much below what either the seller or I expected, and the details backed this up. In particular, the engine and prop had essentially no value left. It would probably continue to fly for another ten years like this ... but who knows? The risk of either an expensive overhaul or just plain failure was too high for me.

The seller wasn't happy to hear this. He obviously disagreed with the appraiser and, as long as it keeps passing annual inspection with good compression, I can sympathize with his view. And he's right that I'm not going to find another comparable aircraft out there. But I'm going to keep on trying.

I learned a lot in the process, and I think I'll be in a better position to view offers more critically in the future. If nothing else, when you see "XXX SMOH," you should ask, "so what year was that last major overhaul?" If it's 5-10 years, not bad. If it's more like 30, that might be a problem.

To anyone else who might be in the position of buying (or even selling) an aircraft: appraisals count. It looks expensive at first, particularly when you realize that if everything's fine, you're still going to have to get the plane to a mechanic and pay him for a day's labor in pre-purchase inspection. But I'm certain that he's saved me from a very likely (and very expensive) overhaul in the near future. If nothing else, it's an extra set of eyes on the problem, and as an engineer, I know the value of that.

A family member asked for a copy of the eulogy I gave at my father's memorial service. I didn't have one, because I spoke off the cuff with just a few handwritten notes. The following is the text, as best I recall it.

My father asked that I speak to you today.

I could tell you stories. I could tell you stories like the time Charlie Law and I towed a broken-down '64 Rambler into my father's driveway. And instead of doing the right thing -- having it towed away -- he helped us try to fix it.

I could tell you that he was my answer man. Any time I had a question about anything -- anything in the world -- I would ask him first.

I could tell you how much he loved music, and wanted his children and grandchildren to have it in their lives. It was important to him.

I could tell you how in the last few months, he was learning a new language, because that's who he was; he was always learning new things.

But instead I'll tell you this:

He worked with his mind, but he respected most those who worked hard with their hands.

He was irreligious, but he loved this church and its people.

He was fiercely rational, but he enjoyed a good conspiracy.

That was my father. He defied conventions. He did and said things in his own way. That's how I'll remember him.

To Schenectady with Ben

Ben and I flew out to Schenectady, NY, today (KSCH). It was about 1:30 flying time, and the first time I've had to buy fuel. I likely could have gotten away without buying any, but I'd end up with less than an hour's worth by the time I got back, and that's too little. That's not a rule I want to break.

The original plan was to stop in at the Empire State AeroSciences Museum that's on the field, but when we got there, we stopped at Richmor, which turns out to be at the wrong end of the airport. We could have taxied over, but Ben wasn't too enthused about walking around the museum anyway, and I just wanted to stretch and get some great vending machine food. ;-}

It's a nice little airport with three runways, one ILS approach, and a lot of Air National Guard activity. When we were landing, there was a C130 landing on the cross runway, so we had to cross "without delay" to avoid them. After we shut down, we watched them practicing in the area, and performing some impressive aerial work -- at least 70 degree bank angle just a couple of hundred feet off of the ground.

We spent some time relaxing in the lobby next to the rental car counter, and then went back out to the plane. The take-off was unremarkable. Switching over to Albany Approach was a bit confused. I expected to go back the direction I had come in, but that's not what they wanted me to do, and they had me head out to the East. Compounding the issue, 13081 seems to have a failing DG; it just wanders all over the place, and I was constantly resetting it to the compass. I ended up flying by compass, VOR, and GPS. The DG was a bit of a distraction. (No, it wasn't a vacuum problem; that was fine.)

Eventually, after a mild scolding, I was left to navigate back home. There was one really good lift-you-out-of-your-seat thump as we were nearing Mt. Greylock, but the rest was fairly smooth. Visibility was great; from about Turner's Falls at 5500, it was easy to see Boston and the ocean beyond.

Coming back into Lawrence, it was very busy. There were two directly ahead of us, and a helicopter and yet another airplane behind. We came into 32 a few knots fast. The landing was smooth, but uncomfortable as I knew I was too fast. We got off promptly at Delta and taxied back to the ramp to shut down. It was so busy that they actually had separate people running tower and ground -- at Lawrence, that's usually just one person talking on both frequencies.

That's another 3.1 for the log book. I think I need about another 10 or so before I can start with real instrument work.

To Hyannis With Ben

I'd planned earlier last week to go to Hyannis on Sunday morning, but as the weekend approached, it looked like Saturday would be the better choice, and I was able to change the reservation.

It turned out well. Ben came along and took pictures along the way, so when I get a chance to unload that camera, there'll be even more to post.

We went by way of Bedford and Norwood, under and around the Class B at 2700. I picked that route after realizing that going direct (if we could get it) would put us out over the water for a bit, and I didn't really want that.

My usual N61976 wasn't available, so I used N13081. Remembering the different identification when talking was probably the hardest part of the trip.

Where 976 has an ADF, 081 has a VFR-only GPS unit. What a difference that makes! It's like cheating, because I can just look at it and get an accurate position. I still tuned the VORs and identified them and watched the needles, but there was a lot less guessing involved.

The bad part of 081 is the mechanical radios. Having no flip-flop tuning makes dealing with ATC much harder, especially as Boston tends not to use the published frequencies all the time; they give you different ones depending on workload. So, it's not uncommon to have both a new squawk code *and* a new frequency to dial in.

It was hazy and warm, but quite smooth both there and back. We stopped for a few minutes in the run-up area at HYA to check in back home, and then took off again. Maybe next time I'll plan for a little more time so that we can actually stop somewhere for that $250 hamburger.

I haven't decided where to go next, but probably somewhere different. Maybe New York.