Jeppesen

I know there are companies that people complain about a lot, such as Comcast. There's good cause for that, but trust me when I say that the only reason they complain so much about them is because they haven't had to deal with Jeppesen.

Jeppesen has a monopoly on the delivery of navigation data for aircraft, and they certainly behave like it. The contempt they have for their customers is astounding.

Every year, I have to deal with renewing my subscription. Every year, something is wrong with the renewal and requires manual intervention to fix. Last year, the web site was down for a week or more, and their "support" person swore up and down that the problem was with Edge (I wasn't using Edge and wasn't using Windows), and that the problem was with my computer or firewall -- even though I figured out which one of their content providers was failing and told them exactly what the problem was. The problem was fairly obvious from the error message and anyone with experience in web deployment would have recognized it. The "support" person was argumentative through the whole call. All I could do was to hang up and wait -- without a legal database -- until they eventually figured out the problem on their own and fixed it. (Without notifying anyone, of course.)

This year, as usual, they've redesigned the web site. Again. And it doesn't work. Clicking on the "change credit card" button to update an expired card just hangs. It took multiple tries and multiple different browsers to get through that seemingly simple operation. They also have multiple expired cards on record, but you can't remove them unless you somehow have a copy of your expired account number and CVV code from six years ago. Yes, they require validation of broken data in order to remove broken data. It's hard to see how that makes sense, and I wonder what their credit card processor thinks of that design choice.

Plus, I discovered that they duplicated my subscription at some point last year. I now have two database subscriptions for my single IFD540 in my single airplane. And unbeknownst to me, they charged my credit card back in February for that duplicate entry. No notice, nothing. Just charged me an extra $184 in February although my account normally renews in November. 

The big improvement is that they've removed the phone number from the web site, so now you can't even call and complain to someone who would have little if any interest in solving your problem.

And all of that is without mentioning the Jeppesen Database Manager, the required application you must use in order to download the database updates every 28 days that you (somehow) have purchased. No, you can't just download the files from a web site, you have to use the application, which runs only on Windows and MacOS. On certain versions of those OSes. It's bad. There's no information about the status of your account in the app. The only indication you ever get about needing to update the app is that it will hang with a blank screen on start-up. Well, hang for longer than it usually hangs, because it sometimes takes minutes to start up at all, and may need to be forcefully terminated and started again or reinstalled. I have to keep an old computer running just to run JDM, and I doubt I'm the only person in that situation.

I really like the user interface on my Avidyne IFD540. It's much easier to fly with than any Garmin device - by far. It's a nice design and works well in real world situations, such as with complex re-routes that are common here in the Northeastern US. 

But the big downside to Avidyne compared to Garmin is that the required navigation data (needed to remain legal for IFR flight) is sole-sourced from Jeppesen. There's no alternative. Garmin offers their own navdata source as an alternative, but not Avidyne. You can't even use Seattle Avionics charts with Avidyne. And Avidyne doesn't publish their data format so nobody else can even try to provide the data without investing in reverse engineering, and almost certainly wouldn't be a legal solution based on the ICA.

If I had any choice in the matter, I'd choose someone else for navdata. And when talking to other pilots about Avidyne, I'm forced to say both that the UI is good, but the Jeppesen lock-in is terrible. It's entirely possible they lose potential customers that way, and I wish it were otherwise.

Subtle C++ alignment and size issues

Short version: sizeof in C++ can be misleading and cause problems, particularly inside assignment overloads, and public/private has surprising effects on alignment.

Object alignment requirements can cause some subtle and hard-to-diagnose bugs in C++. The demonstrations here show two issues that have caused problems that I ended up debugging, and I hope they help others find similar issues buried in their code. (Issues seen with gcc-10, gcc-14, clang-14, clang-17, and others. Note that all examples here are distilled down from much more complex code, so many questions about alternative implementations are out of scope.)

First, some background.

Most C and C++ programmers are aware that struct and class members are placed on "natural" alignment boundaries. This placement is dependent on CPU architecture and a number of other factors (including "packed" options supported by some, but not all, compilers and architectures), but the most common rule is that by default each member is placed at an offset is that is a multiple of the size of the object. That is, a 'char' is on any byte boundary, a two-byte 'short' is on an even byte boundary, a four-byte 'int' is on a multiple of 4 bytes, and so on. If the offset after the preceding element doesn't provide the right alignment, then the compiler automatically inserts hidden padding to make it right. For example:

struct {
  /* must be at offset 0
     in the struct */
  char first;
  /* offset 2; 1 one byte
     pad inserted before */
  short second;
};

The overall size of that struct above may be either 3 or 4 (or perhaps even more), depending on the requirements of the platform. This is why it's somewhat common practice to put the large objects first in a structure, followed by smaller ones, or to group small objects together in clusters. Both strategies minimize this wasteful padding. Note that this padding is between successive members in a structure.

In general, these size-based rules apply to fundamental types, and do not directly apply to compounds (nested structures). Instead, a compound has composite size and alignment based on its contents. That is, a struct has a required alignment that is equal to the strictest (largest) requirement of any member inside, and not its overall size.

A subtle corollary of the above rules is that sizeof() on a struct (or class) must return a value that is rounded up based on the alignment of the strictest member inside, effectively producing trailing hidden padding. That's because sizeof() must always give the proper stride of objects within an array. For example:

struct {
  int val1;
  short val2;
};

This struct should have size 6 (assuming a 4-byte int and 2-byte short), with no padding. But the natural alignment of that 'int' is on a 4-byte boundary, so the sizeof() will be 8, making sure that adjacent array entries are all on natural alignment boundaries when multiple objects are allocated. Note that this shows trailing padding, after the last member.

The first problem, shown in align-surprise1.cpp (see attachments at the end of this post), is that the C++ compiler will naturally pack together member variables using an alignment that is less strict than the alignment required for the base class. This requires some explanation.

The definition of SimplePOD includes a naive optimization: we know that all of the members are just plain old data, and the class itself is not virtual (thus has no vft pointer to worry about), so why not take advantage of that, and use memset/memcpy rather than individually assigning each object?

The subtle error here is that the size being copied includes that alignment padding at the end, and the compiler is free to insert other, unrelated members inside that padding. It can't place them between members (as best I can tell), but the trailing padding is fair game.

In this case, this means that the data copied by the SimplePOD assignment operator includes an extra 4 bytes (actual length is 20 but sizeof is 24). This means that the private members in both TestClass and TestClass2 are overwritten by the assignment of the base class. This test just shows the effect of that issue, which is effectively memory corruption. In the case where I originally encountered this problem, the contents of a smart pointer was copied, resulting in a reference count mismatch and a crash.

You can see the problem demonstrated by running "make" and then executing "./surprise1-fail". The fixed version is built as "./surprise1-pass".

The fix (enabled by FIX_BUG) computes the actual size of SimplePOD and uses that for the copy. Note that commenting out the assignment operator overload in SimplePOD also fixes the problem, as the compiler will internally compute the correct amount to copy. C++ itself just provides no convenient means for the user to compute this value, which is important if an overload is needed by the overall class design.

The second problem is shown by align-surprise2.cpp and is even more shocking. In some cases, changing from "public" to private or protected will cause the member alignment and the overall object size both to change. This is demonstrated by the output of "./surprise2-public" and "./surprise2-private". The only difference is whether the members are public or private.

The surprise2-public output looks like this:

baz 12 foo 8
a 0
b 4
c 8
d 9

This shows that the size of baz is 12, and that the alignment of 'c' starts on the next int boundary. But surprise2-private (and surprise2-protected) show this:

baz 8 foo 8
a 0
b 4
c 5
d 6

The size of baz is down to 8 and the alignment of 'c' has changed to start on a char boundary. Again, the only change is the visibility of the members.

This has several implications. One is that if (say) you are debugging a problem and change some members from private to public just to simplify some temporary debug code, you might also be inadvertently changing the actual offset of those members within the object. If the entire project isn't recompiled, you could have mysterious behavior or even crashes as a result. Another is that seemingly innocuous improvements in C++ code (for example, making public members private and providing accessors instead) could easily change the size of the object and affect cache alignment, drastically altering performance by creating new opportunities for false sharing. Still another is that if you cast pointers back and forth between different classes, you may find that the actual offsets of members in those classes depend on the visibility specified, and thus memory corruption may occur.

It's a jungle out there!

https://www.workingcode.com/align-surprise.tar.gz
https://www.workingcode.com/align-surprise.zip 

Funny C++ Initialization

If you have a class (call it "myclass") that has Plain Old Data (POD) members and does not have a defined constructor (i.e., uses the compiler-supplied default constructor), then "new myclass;" and "new myclass();" will do different things.  The parenthesis-free version will leave the POD uninitialized, but the with-parenthesis version will initialize the POD elements to zero.

If you define a constructor in the class, then the difference disappears.  Both forms of "new" result in leaving the POD uninitialized (as you may have originally expected).

Here's a test case that demonstrates the difference:

funny-init.cpp

Compiling (with g++) and running this little program produces:

Default constructor, no parenthesis:  42
Default constructor, parenthesis:     0
Explicit constructor, no parenthesis: 42
Explicit constructor, parenthesis:    42

As an additional bit of hilarity is that the default-constructor variant compiles to an actual object constructor that does nothing, but when you invoke "new" with parenthesis, the site of the "new" invocation is littered with extra instructions just to write zeros over the POD elements in the class.

One surprising place this difference shows up is with "struct" and placement new.  If you use placement new with a "struct" and you add the parenthesis, then the underlying storage is wiped clean.  Here's a test case for that:

placement-wipeout.cpp

And the corresponding output:

Default constructor, no parenthesis:  42
Default constructor, parenthesis:     0

It's hard to see how this is a helpful state of affairs, but forewarned is forearmed.

RTLD_DEEPBIND has deep surprises

Linux RTLD_DEEPBIND has at least one very deep problem: if you load a dynamic object with it, the symbols called from within that one object will ignore LD_PRELOAD and go directly to the dependency, but symbols within the dependency itself are still resolved via LD_PRELOAD.

A real-world example of this failure is with a dynamically loaded object that invokes malloc(3), strdup(3), and free(3).  Suppose we have an application using an LD_PRELOAD that interposes on malloc and free.  The call to malloc, strdup, and free from within the loaded object will go straight to libc, bypassing the preload as expected.  But the implementation of strdup inside libc invokes malloc on its own:

https://sourceware.org/git/?p=glibc.git;a=blob;f=string/strdup.c;hb=HEAD

That invocation of malloc will go to the LD_PRELOAD library, not libc's local definition.  As a result, the pointer that the dynamic object gets back is from the LD_PRELOAD library's implementation of malloc.  If the dynamic object tries to free that pointer, it will go straight to the libc definition of free().  Unless the preload "just" an innocuous wrapper on the libc functions, and doesn't replace them outright, this will fail in spectacular ways.

Here's a demo of the sort of hilarity this causes:

https://www.workingcode.com/deep-disaster.tar

This program produces the following output with "make test":

LD_PRELOAD=./preload.so ./main
Doing normal test: in preload: Inside the normal library
Doing bound test: in libbound: Inside the normal library
Doing inside test: from inside: in preload: Inside the normal library
Doing bound inside test: in libbound: from inside: in preload: Inside the normal library

The first two test results are as expected.  The main program goes through the preload to get to the common library, and the deeply-bound library does not.  The third result is also fine, and represents the main program invoking a function inside the common library that invokes another library function, which redirects through the preload.  The fourth result is the problem.  The deeply-bound library invokes a function in the common library that in turn invokes another library function.  In this case, it (somewhat surprisingly) goes through the preload, even though the user probably expected that RTLD_DEEPBIND would avoid the use of the preload.

At best, it does so "sometimes."

Note that this means that almost any non-trivial use of RTLD_DEEPBIND is incompatible with (at least) the usual LD_PRELOAD=libtcmalloc.so type of wrapper.  Anything you load with RTLD_DEEPBIND is hopelessly compromised if it invokes libc functions that internally use malloc or free.  Or it means that any such wrapper must carefully wrap all exposed libc interfaces (such as strdup and fopen) that can allocate memory, and supply its own implementation -- a feat that may be impossible.

GNU make reports "process_begin: CreateProcess(NULL, pwd, ...) failed." on Cygwin

I very rarely build anything on Windows.  My day job doesn't call for it much, and I certainly wouldn't want to do that for "fun."  So, when I do, I often run into problems that (a) I don't understand and (b) nobody else seems to have seen before.  This is one of those sorts of problems.

My build on a new machine failed almost immediately after typing "make" with this error:

process_begin: CreateProcess(NULL, pwd, ...) failed.

I have no idea what that means.  The "pwd" command, of course, works just fine for me at the command line, and "whence" tells me it comes from /usr/bin.  Everything looks fine there.

After a lot of debugging and comparisons with others who had working configurations, I found this: near the front of my PATH, I had an entry like this:

/cygdrive/c/this/does/../not/exist

That's not the actual path, of course, but it gives the idea.  The bug I encountered is this: if you have a path that includes "/.." and if the previous directory in that path doesn't physically exist on that machine, then the path search function stops right there.  It doesn't look at the rest of the path entries at all.  So, if you have something like this before "/usr/bin", you're sunk.  That works fine on all UNIX and Unix-like systems, and it even works fine for the Cygwin shells.  But, for some reason, it doesn't work within GNU make's code that deals with path searches.

Changing that path so that it didn't have "/.." in it fixed the problem.

Not Hertz

This happened many years ago, but I still think about it at times, and a recent exchange on Twitter made me remember that I really should have written it up a long time ago.  Better late than never, I suppose.

In 2009, my father was being treated for cancer, and his health was up and down.  I had been planning for some time to take a day of vacation from work on Friday, November 6th, 2009, and stay the weekend.  I had a flight booked on Jet Blue and a car reserved with Hertz.

On Monday, November 2nd, I got word that my father had taken a turn for the worse.  The radiation treatment had caused swelling in his throat, and he was rushed to the hospital to insert a trach tube.  After talking to my brother in Pittsburgh, I decided to change my plans.  I would fly out as soon as possible so I could be with my brother, father, and step-mother.

I called Jet Blue first.  No problem.  It was a $50 fee to change the flight, but I could have the first plane out in the morning on Tuesday.  That was pretty easy, and I was encouraged.

Then I called Hertz reservations through the 800 number.  I'd booked with them because I had a good bit of experience with them.  They were one of the preferred providers when I was at Sun, and I traveled a lot for work when I was there.  The customer service folks couldn't change my previous reservation, but they did offer an alternative: they could book a second car from the 3rd to the 6th, so I had a car the whole time, and they told me that the reservations counter in Pittsburgh would be able to help when I got there.  Nothing they could do by phone; it had to be done in person.  That was less than ideal, but what do I know about their reservations system?

The next morning, I flew to Pittsburgh, and got to the reservations counter.  They told me, no, they couldn't change the two contiguous reservations into one.  They had no idea why the 800 people told me that.  They told me I should call the 800 number again and ask for help.

So, I stepped out of line and into the waiting area, and called the 800 number again from my cell phone.  No dice.  They couldn't do anything for me and told me to talk to the people at the desk again.  So, I got back in line and waited again at the desk.  When I got up there, I asked if there was a manager I could talk to.  The answer was just "no."  They helpfully said that I should call the desk on Friday and ask to have it fixed before having to drive in.  They assured me there was no way I'd have to return just to swap cars; that would be silly.

I took the keys and went off to visit my father.  We had to make some really tough (and possibly wrong) decisions over the next few days.  It was a difficult time, but I'm very glad I made the trip.

On Friday, I called the Hertz desk in Pittsburgh.  I was told that, no, they could not help me.  I could not extend the current reservation.  I could not combine reservations.  The only thing I could do would be to return the car as agreed on Friday and then take another one out.  So, that night, I drove the 20 miles / 30 minutes to the airport, dropped off one ugly champagne colored Elantra, picked up a nearly identical champagne colored Elantra -- the only difference was that the XM radio worked in one and not in the other -- then drove back to my father's place.

That Sunday, November 8th, I dropped off the second car and returned to Boston.  That's the last time I've ever done business with Hertz, and the last time I ever will.

This sort of thing, to me, reeks of a corporate culture problem.  None of the customer service representatives had the remit to fix things.  In a company that is serious about customer service, the employees are given the power to "make things right" -- even if this means breaking company policies.  The service I got indicates the reverse.  Nobody I dealt with had the power to fix anything.

The problem is not necessarily being treated like dirt by every single customer service representative I dealt with.  The problem is having no prospect of things ever getting better, because it wasn't just one or two people having a bad day or not knowing how to make changes.  In fact, the representatives were pleasant to deal with, but completely unhelpful.  A systemic problem like that is something I can't put up with.  So when I need ground transport, it's anyone but them.

"A start job is running for dev-disk-by" and other horrors

My desktop system at home is currently running OpenSUSE Tumbleweed.  It used to run Debian until I got caught in an upgrade version-locked disaster.  Then it ran OpenSolaris until Oracle made that an unlikely proposition.

I've been reasonably happy with OpenSUSE until I made the mistake of trying to do a "zypper dup" recently, and the reboot showed me this:

[***   ] A start job is running for dev-disk-by\x2duuid-1a0dc1c5\x2d26cc\x2d45ff\x2da7b1\x2d1f827c971ff9.device (15s / no limit)

As long as one might care to sit there and watch, it never completed whatever task it was trying to perform.

I was able to boot up with a rescue CD (thank goodness I downloaded that first), and was able to mount the disks with no trouble.  But no amount of fooling around would make it boot.  It was not a happy evening.

After quite a bit of fooling about, I discovered that there were two serious problems that I had to fix manually, and I'm writing this for those who might have run into similar problems:

1. dracut is missing bits

Crucial kernel drivers go missing when dracut builds a new initrd image, and other bits get included whether you like it or not.  I have a mix of file systems in use, and here are the new configuration bits I had to add to /etc/dracut.conf.d:

add_dracutmodules+="btrfs"
add_drivers+="btrfs zlib_deflate xor raid6_pq"
add_drivers+="md-mod raid1 raid456"
omit_drivers+="nouveau"

That it would exclude the RAID and btrfs drivers by default was very surprising.

2. udevd is broken by default

The default configuration of udevd simply doesn't work right.  It limits itself to an absurdly tiny number of processes, and ends up failing to run trivial scripts needed by the Linux "MD" disk subsystem.  That's a big part of my boot problem.  The solution is to create a file named /etc/systemd/system/systemd-udevd.service with this inside:

[Unit]
Description=udev Kernel Device Manager
Documentation=man:systemd-udevd.service(8) man:udev(7)
DefaultDependencies=no
Wants=systemd-udevd-control.socket systemd-udevd-kernel.socket
After=systemd-udevd-control.socket systemd-udevd-kernel.socket systemd-sysusers.service
Before=sysinit.target
ConditionPathIsReadWrite=/sys

[Service]
Type=notify
OOMScoreAdjust=-1000
Sockets=systemd-udevd-control.socket systemd-udevd-kernel.socket
Restart=always
RestartSec=0
ExecStart=/usr/lib/systemd/systemd-udevd
MountFlags=slave
KillMode=mixed
WatchdogSec=3min
TasksMax=infinity

The important part is that "TasksMax=infinity" line.  That's what fixes the system so that it will actually boot again.