Archive for the 'Uncategorized' Category

Visual C++: Letting C++ programmers get away with new murder without warning

Consider Visual C++.

This always correct code emits a completely useless warning:



This never ever correct code doesn’t emit any warning at all:

Visual C++ is a totally awesome piece of technology.  So why doesn’t it work the way I want it to?

Two nearly perfect keyboards: Das Ultimate and WASD CODE

For many years I’ve missed the second best typing experience I’ve ever enjoyed:  The Northgate Omnikey Ultra, pictured here in all its glory:


Oh what a wonderful tactile feel!  My fingers just flew over the full size keys, hitting every key correctly!  And what a wonderful clacky sound it made!  Unfortunately, they had old style PC/AT keyboard connectors, which were inconvenient, and were quite large, and weighed about 50 pounds each, and I stupidly got rid of all 4 of mine some time ago.

Anyway, I now have that same experience again:  I’m the proud owner of both a DAS Model S Ultimate, and a WASD CODE keyboard.  They are equally wonderful in the tactile feel department, and have interesting aesthetic differences.

The DAS Model S Ultimate is gorgeous: it has a beautiful case, seemingly black lacquered, with nice geeky blank keycaps.  I got the “blue” clicky switches, and boy is it loud!  Feels and sounds great, but you can’t use it in an apartment or your neighbors will call the police.  In fact, my wife told me it went or I went.  I retrofitted it with WASD red O-rings and that made it much quieter but didn’t hurt the feel. Has other features too (see the link above) including 2 USB ports on the side—useful for plugging in your Logitech “unifying receiver” for your M570 trackball, and a USB stick as well.  Excellent solid construction: You wouldn’t damage it if you ran over it with a tank.

The CODE keyboard feels great, looks very good, and has wonderful backlighting: the keycaps are black with translucent white letters that glow.  The backlight has adjustable levels.  When I bought it I thought the backlighting was a gimmick but I use it every night … and I really like the look of walking into my darkened office and having the keyboard glowing at me.  I got the clicky (but not really loud) green switches (and they come with sound dampening O-rings installed) so I get a terrific tactile feel but it isn’t terribly loud.  I got the 87-key version without the numeric keypad—which I don’t use—and so it is more compact than the DAS Ultimate, and that is convenient.  Plenty of other features too like Dvorak and Colemak alternate layouts built in, and the very nice ability to totally disable caps lock (turn it into another ctrl key).  Excellent solid construction like the DAS keyboard.

I am in typing bliss!  Wouldn’t trade these in for anything, in fact, I’m looking to get another one or two to take on the job.

But no product is perfect, so here are some issues.

First a minor flaw in the CODE keyboard: A sort of nice feature is that instead of a permanently attached cable hanging off it they use a detachable standard Micro USB cable which plugs in under the keyboard.   There are cable routing channels that you can use to route the cable out the back left/center/right or out the left or right sides of the case.  But if you use the back center channel than any tension on the cable will pull it right out.  So use one of the other four.

And now, an extremely serious flaw that affects both the otherwise wonderful DAS keyboard and the CODE keyboard:  In a major design fail, the function keys F1-F12 are at the top row of the keyboard instead of the left side of the keyboard where God put them.  See them on the left of that picture of the Northgate keyboard above?  Well, admittedly, those guys went overboard, duplicating the function keys on the top as well as the left.  Truly, you only need the ones at the left.  Because then you can easily and accurately touch type the function keys, and do it without horrible stretching.  The original IBM PC keyboards were like that and I’ve never understood why IBM and everyone else abandoned that.  DAS, WASD, are you listening?  Put the function keys on the left and you will have the second best keyboard ever created!

(After all this praise: second best?  And that’s the second time I’ve said it.  Well, yes, the ultimate typing experience is to be found only on IBM Selectrics, specifically, the Selectric Model II.  Those were just beautiful. We’ll never see the likes of them again…)


Bad cables can masquerade as other errors

This is just a reminder:  A bad cable can masquerade as other errors.

In this case I was building a new server system – new motherboard, new drives, etc. keeping only the splendid case.  I had one particular new 4TB HDD that would drop out of a Windows Spaces storage pool (“Retired”, as if it had done some hard work and was now taking a well-earned rest)—sometimes within 10 minutes after booting, sometimes it would take an hour.

The event log before one of these drop outs would show a few bad commands, followed by a “bad block” error.

A S.M.A.R.T. diagnostic utility showed excessive errors in a couple of categories, none of them bad blocks.  One such category was “command timeouts” which was a clue, but I didn’t know how to interpret it.

Anyway, it was a brand new disk.  And I buy high-capacity HDDs all the time (I have nearly 50) yet I’ve never had any die of infant mortality.  So I tried moving the disk to a different port on the same brand-new motherboard controller (bad socket?), moving it to an add-in card controller (of the same type, Marvell) (bad motherboard chip?), moving it to a different port on a different motherboard controller (Intel chipset this time, bad driver?).  Failed each time!  So I ordered a new drive (same-day delivery!) and tried it…and it failed too in the same way!

Wracked my brain…and finally…swapped cables with an adjacent hard drive in the same enclosure.  Now the other drive failed!

So it was the cable.  Replaced the cable and all works fine.  I’ve copied 8Tb of data onto the new Storage Spaces array with no issues now.

I’ve never had a bad (internal case) cable before either.  And these were new cables.

Update 2014-09-09:  And bad network cables can make your PC connect at 100Mbps instead of 1Gbps!  This just happened to a colleague at work.

When is a newline required to separate command line arguments? When it is in your eclipse.ini!

This cost me two full days—very embarrassing in front of my new colleagues.  I set up a new Eclipse installation (I am an Eclipse newbie) following their detailed instructions, and tried to load their very large project.  Eclipse kept crashing, after 10 to 30 seconds, with OutOfMemory PermGen errors. The solution, as any one of 100 blog posts will tell you, is to add “-XX:MaxPermSize=512m” to your eclipse.ini. Which I did, to no effect. I tried different values for MaxPermSize, multiple different versions of Eclipse, made sure I was running the proper Sun JVM, etc. etc. etc.  Finally hooked up jconsole and discovered that I was getting these OutOfMemory PermGen errors when PermGen was nowhere near filling my expanded space limit.

Now, note:  ps showed that the vm was started with the right command line parameter, -XX:MaxPermSize=512m. Eclipse’s own error dialog showed it after a crash. jconsole showed it. But it was being ignored!?!

The solution was this: I had, in the interest of making eclipse.ini easier to read, reformatted it so instead of saying, e.g.,


it was instead:

--launcher.defaultAction openFile
-vmargs -Dosgi.requiredJavaVersion=1.5 ⏎
     -XX:MaxPermSize=512m ⏎
     -Xms40m -Xmx512m

And that was the problem!  Putting the parameters to the command line arguments on the same line caused them not to be recognized!

So: don’t do that!

Does enumerating files with FSCTL_ENUM_USN_DATA ever miss any files?

Short answer: No and yes.

I’ve been playing with enumerating all the files on a volume using the FSCTL_ENUM_USN_DATA IOCTL, which reads the MFT.

It is supposed to be very fast, especially compared with other methods such as FindFirst/FindNext.  (There’s a great forum posting on it here, “Reading MFT, which also contains links to posted source code here, and several posts by the same author here, all of which together make a great starting point.)

However, a question is quickly raised when you’re trying this out:  Does the enumerating files through the MFT this way miss any?  If you try a different traversal technique, e.g., using the .NET Framework APIs Directory.GetDirectories() and Directory.GetFiles() you’ll get a different number of results.

In fact, on my system (running Windows Server 2008 R2) I found 1419 directories and files with the MFT enumeration that weren’t listed with the .NET API traversal, and an astonishing (to me) 19199 extra directories and files with the .NET API traversal than with the MFT traversal.  What’s going on with all those “missed” files?  Did I have a bug in my MFT enumeration code?

No.  The answer is: Hard links.  (And symbolic links.)

When scanning the MFT with FSCTL_ENUM_USN_DATA you see each directory and file once and only once, no matter how many directory entries point to it.  For example, on my system, traversing C: with the .NET APIs returns 6 files named “write.exe”, but the MFT enumeration has only 2.

In fact, by using the command “fsutil hardlink list c:Windowswrite.exe” I see that that single file has four names:

(The other two instances of “write.exe” are a single separate file that has two links to it.)

I had no idea that in a standard installation there were so many hard links used. In fact, it even seems that some application installers create multiple hard links to the same file (e.g., MiKTeK).

And that explains nearly all of the files “missed” by the MFT enumeration.

Depending on the reason that you’re enumerating directories and files on a volume, this may or may not be an issue for you.  Actually, likely, it is an issue for you and you may need to resolve it by traversing the directory using FindFirst/FindNext or the .NET APIs and reconciling the two collections.  (Given a filename you can use the Find{First/Next}FileName functions to get all the names of a given file, i.e., all of the names of the hard links to the file. But it may be expensive to use this on every file just to find the ones that have multiple links.  Reconciling with the other kind of traversal may be the better bet – I have yet to measure this.)

On the other hand MFT enumeration does find files that the .NET traversal does not. There are a large number of files under WindowsSystem32 that are returned on the MFT enumeration but not the .NET traversal.  I’m not sure why; it doesn’t appear to be security related.  The .NET enumeration won’t descend past reparse points like “Documents and Settings”, but MFT enumeration won’t descend into directory mount points (FindFirst/FindNext will go through mount points, and I haven’t tried .NET enumeration on that yet.) The MFT enumeration also returns the directory “System Volume Information” and some files under it.  And it also returns directories and files related to the Transactional Resource Manager, namely the directory “$RmMetadata” and its contents.

(The latter is the cause of some minor coding confusion:  The MFT metadata files, e.g., $Bitmap and $Quota, are not returned by the MFT enumeration—and that includes the directory $Extend, which is the parent of $RmMetadata.  So when you’re assembling path names from the entries returned in your MFT enumeration you’ll have to account for the fact that $RmMetadata’s parent isn’t going to be in your collection of directories.)

A plea that the class Point should not be an example for data abstraction

I was just reading a modern “what is clean code” book and once again found the same example being used to motivate the discussion of “data abstraction” as in many previous “clean code”/”O-O design”/”coding standard” books:  The classic Point class.

Yes, the Point is currently represented in Cartesian coordinates but someday you might want to switch it to polar coordinates, and thus you should use data abstraction to expose getters and setters instead of fields.

The problem is that there is no circumstance whatsoever under which you would ever want to change a Point representation from Cartesian to Polar, or vice versa, where you would want that change to happen in isolation without changing the rest of the code.

I’ll explain why—but first I’ll just say that I understand that Point is used as an example because it is simple to understand and doesn’t take a lot of space on a page in a book.  But there are other examples one could use that also satisfy those needs that, in addition, make sense as a motivation for using data abstraction.  For example, a URL class, where you might initially have a representation that separated the components of a URL into protocol (enum)/site (string)/port (int)/path (string)/query (string[]) parameters and then switch to represent it as simple string, or vice versa.  Easily understood as an example, a simple class to put in your book, and you might actually want to make that switch someday.

But back to the Point.  The reasons you would never want to switch a Point representation from Cartesian to Polar, or vice versa, in your application without changing a line of your code is because the two representations have different semantics and performance.  They are just not interchangable.

Continue reading ‘A plea that the class Point should not be an example for data abstraction’

Is the PE Attribute Certificate Table octoword-aligned or octobyte-aligned?

Looking at the Microsoft Portable Executable and Commmon Object File Specification rev 8.2 (Sept 21, 2010)which is the definition of the PE file formatI’m confused about the alignment of the entries in the Attribute Certificate Table.

  • Page 58, section 4.7: “The attribute certificate table is composed of a set of contiguous, octaword-aligned attribute certificate entries.” 
  • Page 59, first paragraph: “Subsequent entries are accessed by advancing that entry’s dwLength bytes, rounded up to an 8-byte multiple, …”
  • Page 59, algorithm step 2: “Round the value from step 1 up to the nearest 8-byte multiple …”
  • Page 59, algorithm step 3: “… and round up to the nearest 8-byte multiple …”
  • Page 60, last paragraph before the bullets: “If the bCertificate does not end on an octaword boundary, the attribute certificate table is padded with zeros, from the end of the bCertificate to the octaword boundary.”

So the documentation is confused.  But, it also clearly says, on page 59, “If the sum of the rounded dwLength values does not equal the Size value, then either the attribute certificate table or the Size field is corrupted.”  And on my sample, signed, executable, the Size field is a multiple of 8 but not 16, and WinVerifyTrust() says that the executable is authentic (and of course the loader will load and execute it).

So on the basis of this experimental evidence (one sample) I think we can conclude that the Attribute Certificate Table is octobyte aligned, not octoword aligned.

An example of documenting a broken feature as normal behavior

Here’s an example of documenting a broken feature as normal behavior – furthermore, hiding it in the documentation as a “remark”.

The issue is the SQL Server “TOP” feature to return the “top” rows of a query result.  Consider the MSDN documentation here.  The “TOP” keyword in a SELECT statement is used to return only the first N rows of a query, for some N (or alternatively, N%).

Naturally—and I really mean, of course, there’s no other reasonable way for it to work—if the SELECT query includes an ORDER BY clause then it is respected and the sort occurs before the “top N” rows are selected.

Now for some reason they let you put the TOP keyword in an INSERT, UPDATE, or DELETE statement after the main keyword.  But in these cases, the ORDER BY clause in the subordinate SELECT is ignored!

E.g., given
INSERT TOP (2) INTO Table2 (ColumnB)
SELECT ColumnA FROM Table1

an arbitrary two rows are inserted, not the first two rows returned after the sort! That’s REMARKable! But documented!

(There’s a workaround they show you: Put the TOP keyword after the SELECT keyword, not after the INSERT keyword. But why allow the other syntax to be at all if it leads to such obviously broken behavior?  Alternatively: Why not fix it if you’re going to continue allowing that syntax?)