GNU make reports "process_begin: CreateProcess(NULL, pwd, ...) failed." on Cygwin

I very rarely build anything on Windows.  My day job doesn't call for it much, and I certainly wouldn't want to do that for "fun."  So, when I do, I often run into problems that (a) I don't understand and (b) nobody else seems to have seen before.  This is one of those sorts of problems.

My build on a new machine failed almost immediately after typing "make" with this error:

process_begin: CreateProcess(NULL, pwd, ...) failed.

I have no idea what that means.  The "pwd" command, of course, works just fine for me at the command line, and "whence" tells me it comes from /usr/bin.  Everything looks fine there.

After a lot of debugging and comparisons with others who had working configurations, I found this: near the front of my PATH, I had an entry like this:

/cygdrive/c/this/does/../not/exist

That's not the actual path, of course, but it gives the idea.  The bug I encountered is this: if you have a path that includes "/.." and if the previous directory in that path doesn't physically exist on that machine, then the path search function stops right there.  It doesn't look at the rest of the path entries at all.  So, if you have something like this before "/usr/bin", you're sunk.  That works fine on all UNIX and Unix-like systems, and it even works fine for the Cygwin shells.  But, for some reason, it doesn't work within GNU make's code that deals with path searches.

Changing that path so that it didn't have "/.." in it fixed the problem.

Not Hertz

This happened many years ago, but I still think about it at times, and a recent exchange on Twitter made me remember that I really should have written it up a long time ago.  Better late than never, I suppose.

In 2009, my father was being treated for cancer, and his health was up and down.  I had been planning for some time to take a day of vacation from work on Friday, November 6th, 2009, and stay the weekend.  I had a flight booked on Jet Blue and a car reserved with Hertz.

On Monday, November 2nd, I got word that my father had taken a turn for the worse.  The radiation treatment had caused swelling in his throat, and he was rushed to the hospital to insert a trach tube.  After talking to my brother in Pittsburgh, I decided to change my plans.  I would fly out as soon as possible so I could be with my brother, father, and step-mother.

I called Jet Blue first.  No problem.  It was a $50 fee to change the flight, but I could have the first plane out in the morning on Tuesday.  That was pretty easy, and I was encouraged.

Then I called Hertz reservations through the 800 number.  I'd booked with them because I had a good bit of experience with them.  They were one of the preferred providers when I was at Sun, and I traveled a lot for work when I was there.  The customer service folks couldn't change my previous reservation, but they did offer an alternative: they could book a second car from the 3rd to the 6th, so I had a car the whole time, and they told me that the reservations counter in Pittsburgh would be able to help when I got there.  Nothing they could do by phone; it had to be done in person.  That was less than ideal, but what do I know about their reservations system?

The next morning, I flew to Pittsburgh, and got to the reservations counter.  They told me, no, they couldn't change the two contiguous reservations into one.  They had no idea why the 800 people told me that.  They told me I should call the 800 number again and ask for help.

So, I stepped out of line and into the waiting area, and called the 800 number again from my cell phone.  No dice.  They couldn't do anything for me and told me to talk to the people at the desk again.  So, I got back in line and waited again at the desk.  When I got up there, I asked if there was a manager I could talk to.  The answer was just "no."  They helpfully said that I should call the desk on Friday and ask to have it fixed before having to drive in.  They assured me there was no way I'd have to return just to swap cars; that would be silly.

I took the keys and went off to visit my father.  We had to make some really tough (and possibly wrong) decisions over the next few days.  It was a difficult time, but I'm very glad I made the trip.

On Friday, I called the Hertz desk in Pittsburgh.  I was told that, no, they could not help me.  I could not extend the current reservation.  I could not combine reservations.  The only thing I could do would be to return the car as agreed on Friday and then take another one out.  So, that night, I drove the 20 miles / 30 minutes to the airport, dropped off one ugly champagne colored Elantra, picked up a nearly identical champagne colored Elantra -- the only difference was that the XM radio worked in one and not in the other -- then drove back to my father's place.

That Sunday, November 8th, I dropped off the second car and returned to Boston.  That's the last time I've ever done business with Hertz, and the last time I ever will.

This sort of thing, to me, reeks of a corporate culture problem.  None of the customer service representatives had the remit to fix things.  In a company that is serious about customer service, the employees are given the power to "make things right" -- even if this means breaking company policies.  The service I got indicates the reverse.  Nobody I dealt with had the power to fix anything.

The problem is not necessarily being treated like dirt by every single customer service representative I dealt with.  The problem is having no prospect of things ever getting better, because it wasn't just one or two people having a bad day or not knowing how to make changes.  In fact, the representatives were pleasant to deal with, but completely unhelpful.  A systemic problem like that is something I can't put up with.  So when I need ground transport, it's anyone but them.

"A start job is running for dev-disk-by" and other horrors

My desktop system at home is currently running OpenSUSE Tumbleweed.  It used to run Debian until I got caught in an upgrade version-locked disaster.  Then it ran OpenSolaris until Oracle made that an unlikely proposition.

I've been reasonably happy with OpenSUSE until I made the mistake of trying to do a "zypper dup" recently, and the reboot showed me this:

[***   ] A start job is running for dev-disk-by\x2duuid-1a0dc1c5\x2d26cc\x2d45ff\x2da7b1\x2d1f827c971ff9.device (15s / no limit)

As long as one might care to sit there and watch, it never completed whatever task it was trying to perform.

I was able to boot up with a rescue CD (thank goodness I downloaded that first), and was able to mount the disks with no trouble.  But no amount of fooling around would make it boot.  It was not a happy evening.

After quite a bit of fooling about, I discovered that there were two serious problems that I had to fix manually, and I'm writing this for those who might have run into similar problems:

1. dracut is missing bits

Crucial kernel drivers go missing when dracut builds a new initrd image, and other bits get included whether you like it or not.  I have a mix of file systems in use, and here are the new configuration bits I had to add to /etc/dracut.conf.d:

add_dracutmodules+="btrfs"
add_drivers+="btrfs zlib_deflate xor raid6_pq"
add_drivers+="md-mod raid1 raid456"
omit_drivers+="nouveau"

That it would exclude the RAID and btrfs drivers by default was very surprising.

2. udevd is broken by default

The default configuration of udevd simply doesn't work right.  It limits itself to an absurdly tiny number of processes, and ends up failing to run trivial scripts needed by the Linux "MD" disk subsystem.  That's a big part of my boot problem.  The solution is to create a file named /etc/systemd/system/systemd-udevd.service with this inside:

[Unit]
Description=udev Kernel Device Manager
Documentation=man:systemd-udevd.service(8) man:udev(7)
DefaultDependencies=no
Wants=systemd-udevd-control.socket systemd-udevd-kernel.socket
After=systemd-udevd-control.socket systemd-udevd-kernel.socket systemd-sysusers.service
Before=sysinit.target
ConditionPathIsReadWrite=/sys

[Service]
Type=notify
OOMScoreAdjust=-1000
Sockets=systemd-udevd-control.socket systemd-udevd-kernel.socket
Restart=always
RestartSec=0
ExecStart=/usr/lib/systemd/systemd-udevd
MountFlags=slave
KillMode=mixed
WatchdogSec=3min
TasksMax=infinity

The important part is that "TasksMax=infinity" line.  That's what fixes the system so that it will actually boot again.

Missing Windows users? The badly-named "net" command is your friend.

I have a lousy old laptop that I use for IMC Club presentations and the like.  I wish I could afford something better, but it mostly works almost well enough to keep me from bothering to look around.

Except every once in a while, it falls apart.  Windows is, unfortunately, really terrible in that way.  The latest problem it had is so strange and so obscure that I felt I had to write about it just in case someone else runs into it.

First symptom: all users accounts are gone from the login screen.  In fact, the login screen itself is skipped, and it goes straight to asking for the Admin password on boot.

On getting in, the "user accounts" tool shows nothing but Admin and Guest.  All user accounts are just plain gone.  Attempting to add the user accounts back results in an error message saying that the user "already has permission to access this computer."  Well, that's unhelpful.

The parental controls section shows the accounts.  The files are still there under C:\Users.  Everything seems in place, but nobody can log in.  Regedit shows nothing interesting.  Googling around for all sorts of related phrases shows that quite a few people have experienced this, but nobody has solved it.

I just solved it.  Typing "net user Jim", I can see output that ends like this:

    Local Group Memberships
    Global Group memberships   *None
    The command completed successfully.

I tried adding a dummy account, and it showed up with "Local Group Memberships" set to *Users.  That's the key.  For some reason, all of the accounts had been kicked out of the "Users" group, and that's why they were gone from the login screen.  Adding them back in looks like this:

    net localgroup Users Jim /add
    net localgroup Users Madeline /add

After doing that, the system was back to normal.  Ah, Windows.  Thanks for wasting so many hours of my life.

Unfortunate Result

I had high hopes for AMC's new "Halt and Catch Fire."  It's great to see the pie man back on TV.  And the subject is one that I'm very interested in, as I've been involved with computers since well before the introduction of the PC.

From the start, though, the show went off the rails.  They tried to explain the title with a "definition" that was complete gibberish.  That phrase doesn't refer to an actual computer instruction or program; it's an old joke.  And, sadly, it seems that the folks working on that show didn't get the joke.

Back then, writing in assembly was more common than it is today.  Just for fun, many people made lists of fake instructions and passed them around by photocopies and pinning them to bulletin boards in office hallways.  Typical entries were "Rewind and Stretch Tape," "Jump To Random Location," and "Execute Operator."  No machine had those instructions.  They were jokes, as was "Halt and Catch Fire."  But they missed that entirely.

That was just the start.  They missed the boat in so many ways that it's hard to count them.  I don't want to just pick nits -- the pattern "on, on, off, on" or 1101 is a hexadecimal D, not a B -- but the basic plot points are way off the mark.  The BIOS wasn't in any way the "secret sauce."  It just does basic hardware initialization, a little bit of self-test, and then loads bootstrap code from a floppy drive or (back then) a cassette recorder.  Heck, IBM published the source code in the technical reference manual!  And if you wanted to extract the ROM contents, even back then, it wouldn't be as crazily hard as it was portrayed for dramatic purposes.  You can just read the memory.

So, it's already headed in a bad direction, and it's hard to see why that's the case.  The real story has all sorts of interesting twists and turns -- just google "Gary Kildall" for part of it -- and it's a shame to see the opportunity squandered.  I'll keep watching for a few episodes, hoping that they'll turn it around, but it's not looking good so far.  Instead of portraying life in a start-up, though, it seems to be mimicking the failure of one.

Next Adventure

I'm now working on buying a plane. It's a bit more complicated -- and taxing -- than I'd expected.

The first one I looked at seemed almost perfect: right model and year, no significant damage history, good price, low time, and hangared within an hour's drive. I went out to see it, and it seemed well-kept. He had a couple of sports cars stored in the hangar with the plane, and he obviously took good care of his machines. The owner just didn't get to fly much anymore and wanted to sell. (I'm going to avoid specifically identifying the plane or owner here.)

He let me look over the books, and everything seemed to be in order. The I&A logs looked right, the engine tests looked great, and the known sorts of problems for that plane were all absent.

After reading detailed postings on a few other sites (particularly www.aopa.org and www.cardinalflyers.com), I decided that it'd be worthwhile to invest in an appraisal before going further. I contacted the NAAA, and was given a pointer to Ken Dantzig. He was able to make some time to go out and pore over the log books and the plane for several hours. (Appraiser's aren't mechanics and don't open things up; they spend most of the time with the records and the observable details.)

I was expecting this to be a slightly expensive formality. It wasn't. He found several issues that I just wouldn't have noticed on my own because I wasn't looking in the right places. After a few days of anxiously waiting, I got the written report. It surprised me -- the value was much below what either the seller or I expected, and the details backed this up. In particular, the engine and prop had essentially no value left. It would probably continue to fly for another ten years like this ... but who knows? The risk of either an expensive overhaul or just plain failure was too high for me.

The seller wasn't happy to hear this. He obviously disagreed with the appraiser and, as long as it keeps passing annual inspection with good compression, I can sympathize with his view. And he's right that I'm not going to find another comparable aircraft out there. But I'm going to keep on trying.

I learned a lot in the process, and I think I'll be in a better position to view offers more critically in the future. If nothing else, when you see "XXX SMOH," you should ask, "so what year was that last major overhaul?" If it's 5-10 years, not bad. If it's more like 30, that might be a problem.

To anyone else who might be in the position of buying (or even selling) an aircraft: appraisals count. It looks expensive at first, particularly when you realize that if everything's fine, you're still going to have to get the plane to a mechanic and pay him for a day's labor in pre-purchase inspection. But I'm certain that he's saved me from a very likely (and very expensive) overhaul in the near future. If nothing else, it's an extra set of eyes on the problem, and as an engineer, I know the value of that.

A family member asked for a copy of the eulogy I gave at my father's memorial service. I didn't have one, because I spoke off the cuff with just a few handwritten notes. The following is the text, as best I recall it.

My father asked that I speak to you today.

I could tell you stories. I could tell you stories like the time Charlie Law and I towed a broken-down '64 Rambler into my father's driveway. And instead of doing the right thing -- having it towed away -- he helped us try to fix it.

I could tell you that he was my answer man. Any time I had a question about anything -- anything in the world -- I would ask him first.

I could tell you how much he loved music, and wanted his children and grandchildren to have it in their lives. It was important to him.

I could tell you how in the last few months, he was learning a new language, because that's who he was; he was always learning new things.

But instead I'll tell you this:

He worked with his mind, but he respected most those who worked hard with their hands.

He was irreligious, but he loved this church and its people.

He was fiercely rational, but he enjoyed a good conspiracy.

That was my father. He defied conventions. He did and said things in his own way. That's how I'll remember him.