No one won NY-23

(Note: when this diary was started, I was working off of a quote from an election official stating unambiguously that a virus was causing problems in some of the voting machines and that it was not ‘fixed’ in voting machines in use on election night. That statement was later said to be incorrect by other officials; and — possibly limited by resources or simply looking ahead to 2010 — the Hoffman campaign ultimately decided against disputing the contest. Without placing hands on the hardware and software configuration utilities, it is currently impossible to do more than speculate whether the official version of events is correct or not. The original version of this diary explores the basic history of NY-23, and describes how such a virus — which is theoretically possible — could have been implanted into the system, how it could have affected results; and ultimately, how to recover the true results of an election. I did correct one ‘error’ below, which could potentially be a problem for future systems.)

After the local Republican party shut down the primary process and selected the terrible candidate and well-known liberal — and sometime Workers Family Party candidate — Dede Scozzafava to run against Owens; conservative Republicans in that district were left in the unlikely position of having to choose either the Republican or the less liberal candidate (a Democrat) in the race.

Instead, as some of conservative’s front line activists, we RedStaters were quick to embrace third party candidate Doug Hoffman: the one-time GOP primary nominee hopeful, who was locked out of the process without a debate.

Heading into the special elections a couple of weeks ago, Doug Hoffman went from a “divisive” thorn in the side of the Republican establishment, to a veritable champion of the conservative base of the Republican party. Once Scozzafava’s liberal (and not moderate at all) positions became widely known, her popularity (locally and nationally) tanked; and it soon became clear that she could not be elected.

As the vote neared, Scozzafava — no longer able to win — seemed satisfied to play spoiler when Hoffman emerged as the front-runner; at least until she realized that she was splitting union votes away from her favored candidate (the Democrat), and dropped out. Interestingly, it seemed to matter little: Doug Hoffman, with broad national support and a surprising amount of local appeal, was steam-rolling his way to victory in both 2-way and 3-way contests. In what appeared at the time to be a major strategic blunder, Scozzafava’s Republican betrayal (the national party injected hundreds of thousands of dollars into her campaign coffers in an absurd attempt to help her win) did little but validate long-held skepticism of such faux Republicanism (to paraphrase Rush Limbaugh, “she screwed every RINO in the country”).

But then, a funny thing happened; when early returns began flowing in, Hoffman was severely under-performing in some of his best districts. We (RedStaters) watched with interest and dismay as his In-Trade numbers plummeted. It wasn’t long before he conceded defeat. As could have been predicted; Owens, just a few hours after being sworn in, completely backtracked on several of his campaign pledges and voted soon after for the abominable Obama Care bill in the house that he had promised to oppose.

Every election has anomalies; and NY-23 was no different. Hoffman’s terrible performance in his best districts? A technical glitch. More surprisingly, he got no votes at all in several districts. Election officials quickly “corrected” the problems; and, not surprisingly, Owens still won.

It is time to issue a federal subpoena to find out exactly how officials “corrected” the vote.

Why, you may ask?

Because, as we learned yesterday, at least some of the certified electronic voting machines in the NY-23 congressional special election were discovered to be running malicious code (a virus) in the week prior to voting. We now hear word that election officials, According to the report, election officials, after cleaning only the identified machines, took no further action to re-certify other voting machines in the district.

Although the technical nature of such a virus is interesting; it is difficult to adequately explain in article format. For the technical minded, the programming technique (called return-oriented programming) utilized is described here; while the application of this technique is described here (h/t Scope).

For the layman, I’ll simply list a few facts regarding the virus:

1. TheSuch a virus could only have been implanted after the voting machine was installed into the polling centers and the machines turned on at least once.

2. TheSuch a virus was could only be (edit: originally working from the election official’s statement, now disclaimed) implanted by an operative with direct, physical access to the voting machines.

3. TheSuch a virus is eliminated only when power is completely removed, and not necessarily when the machine is “turned off”.

4. TheSuch a virus works not by “injecting” new code into the system, but by confusing the system by piecing together valid code fragments to form a new code (for example, a vote for Hoffman could become a vote for either no one or a vote for Owens; but the attacker is quite limited in their ability to, for example, generate a random number). Therefore, given the exact virus code run on each affected machine, it would be quite easy to reconstruct the true vote. (CORRECTION 11/27/09: given enough time, and a linux based system — such as the one discussed here — a full ‘return to libc’ virus could be implemented. That would give lots of freedom to do all kinds of nasty stuff, including random numbers which would be totally unrecoverable)

While the story certainly begins well (after workers discovered a problem, they were able to eliminate the infection with the help of the manufacturer and the machines were recertified), election officials did something inexplicable: rather than reset and recertify all of the voting machines, they took no further action.

Any curiosity at all into the nature of the virus — for example, by simply asking the manufacturer — would have quickly revealed the facts that I have enumerated above. The perpetrator(s) had physical access to the machines (that is, possibly poll workers or administrators), and an express intent of manipulating the vote. Therefore, without being able to identify the source of the virus, the only reasonable course of action of election officials would have been to immediately perform the power cycling procedure on all voting machines to eliminate the virus and provide outside physical security for all machines to prevent reinfection. As far as we know right now, they did neither.

Which brings me back to my point about that innocuous little technical glitch. The workers reporting vote totals in those districts (in which Hoffman received no votes) apparently didn’t find this information worthy of propagating to the Hoffman campaign prior to his concession; which must, in retrospect, present at least some cause for concern.

Without the code (as point #3 indicates, once power is removed from the machines, all traces of the offending code is permanently destroyed), it is now impossible to reconstruct the true vote total. Not only do we not know which, if any, machines were infected; we can not know what the parameters of the virus were (for example, the percentage of Hoffman votes which were either discarded or became Owens votes could have varied from polling place to polling place), although we can begin to piece this information together by comparing the number of paper signatures at the polling places with the number of votes.

However, the most important thing to know now is how election officials reconstructed Hoffman’s vote total in the districts in which he received no votes, as well as where his vote was initially under-reported.

If the lack of Hoffman votes in those district could have been caused by the virus, then the election official’s reaction and actions taken would indicate whether they were complicit in election fraud. The most likely way that they might have reconstructed the Hoffman vote would be to take the total number of voters and subtract off the Owen’s vote (and some percentage of misvotes), because the actual vote would have either been lost (or even more damning, calculable only with an intimate knowledge of the workings of the virus).

It is vitally important to verify how they came up with the Hoffman vote total — as this could not only implicate those potentially guilty of the fraud — but also whether Hoffman votes became Owens votes or whether Hoffman votes were being dropped and help forensically reconstruct the true results of the election.

As they like to say, it’s not the crime, but the cover-up.

If the Hoffman votes were simply dropped, it might still be possible to estimate the true Hoffman vote from the total number of voters (based on the number of signatures on the paper sign-in forms). Unfortunately, without using properly certified machines or having a detailed record of which machines were infected and (precisely) what they were infected with, it is impossible to say for sure who won the election (whether the tainted total shows Owens by 1 vote or 1 million).

In case you’re interested in learning more about how the system works, here are the voting machine training videos.

Update 1

This was in the original article, but it’s worth pointing out specifically:

In St. Lawrence County, machines in Louisville, Waddington, Claire, and Rossie “broke” early in the voting process on election day. Republican Commissioner Deborah Pahler said that the machines kept “freezing up… like Windows does all the time,” and that they experienced several paper jams as well. The voted ballots that could not be scanned were placed in an Emergency Lock Box and re-scanned later at the St. Lawrence County Board of Elections. Election officials in St. Lawrence County were given no advance knowledge of a potential virus in the system.

Because of the nature of return-oriented programming, freeze-ups are practically expected. Again, without going into too many details, it effectively works by “confusing” the system, so that a function should begin and end at well-defined points in memory; but the machine is placed into a stable “state error” and the attacker constructs a pseudo-function out of many non-sequential chunks of code.

It is quite likely that the machines in those counties were simply running a slightly different version of software than the virus was actually intended to work on, so that those stable “state errors” become slightly less… stable.

Update 2

Also in the original article, a note about the paper ballots also in storage:

The problem is that the slot is readily accessible to the voter (or poll worker) to stuff manually. 10 voted ballots could be deposited in the slot for every one voter… and if the electronic count was compromised, the “paper backup” would be useless.

So, these paper ballots are probably not a “true” vote count, but if ballot stuffing can be estimated from the numbers on the check-in rolls, it would certainly be an excellent thing to look at.

Update 3 (11/21/09 8:09PM)

The response from a former election watchdog (Bo Lipari) is being used by the media to run with the claim that the virus story is a hoax.

Lipari claims that the voting machines could not have had a virus because they run Linux:

There was no virus in the NY-23 machines. How do I know? Well, in the first place, the Dominion ImageCast scanners in question run the Linux operating system, which is nearly immune to viruses due to its inherent ability to lock out programs that lack explicit permission to run, unlike the highly vulnerable Windows operating system.

(Italics mine). The insanity of it all is that this person is obviously ignorant about return-oriented programming — he should become familiar. Linux’s security is based on a concept called W-xor-X, which (though cryptic) simply means that the operating system will not allow execution of code which the operating system has the authority to write to. It eliminates the possibility of user injected code — but that’s not what we’re talking about here. The executed code is not writable (as I’ve said, it’s pieced together fragments of good, non-writable code), so W-xor-X is useless.

The piece doesn’t end there; the “hoax” is further debunked with the following cannard:

Because New York votes on paper, everybody’s vote was counted. When the scanner stopped working, the ballots were removed and counted, so no votes were lost.

(Again, italics mine). I’ve already addressed this here, but I’ll restate it anyway: the ballots could be stuffed at a rate of 10 to 1. While this is unlikely in any fair election, the possibility cannot be eliminated given that the perpetrators were likely poll workers.

Much more importantly, it is incorrect to assume that only the misbehaving machines had the virus.

Given this possibility, a full recount must be performed (despite the possibility of ballot stuffing). Recounting only the malfunctioning machines is NOT sufficient!

Update 4 (11/22/09 6:45AM)

Yet more in the media/blogosphere running with the “no virus” meme based on Lipari’s statement. here, here, here, and elsewhere.

Every “there is no virus” story has either cited Lipari or provides a bald assertion.

By the way, here is the direct link to Lipari’s NY-23 blog entry; and, here is where he discusses the possibility of fooling scanning voting machines (like the one we’re talking about here), for example, in order for ACORN/WFP corrupt poll workers honest citizens to do some ballot stuffing while they’re at the polls.

Update 5 (11/22/09 7:AM)

Required reading: what they told the Hoffman campaign.

Just a few tidbits:

There was no virus in the voting machines on Election Day in the 23rd District or anywhere else. The article is full of inaccurate information and unfortunately quoted a single word from a commissioner who mischaracterized the issue in question.

Question: how was the code “updated” to solve this problem? Did it involve unplugging and replugging in the machine, or not?

And then on the USB ports, which the attacker used to upload the virus:

With regard to the use of USB ports, there is a single USB port on the ImageCast scanner. Pursuant to state Election Law 7-202(t) the port does not permit any “functionality potentially capable of externally transmitting or receiving data via the internet, via radio waves or via other wireless means.” The port is sealed, is not accessible and has no capability for any exchange of information. The scanners do not operate like personal computers. Any device, such as a flash drive, placed in the port will not be recognized.

Essentially, they promise that the USB port cannot be used to support a device able to transmit voter data. Wonderful. Terrific. I hope it’s true (it probably isn’t). Not to beat a dead horse, here, but what exactly could the USB port be used for? Booting the machine into diagnostic code, and/or updating source code. That sounds a lot like the type of port you might need to upload a virus.

Lest anybody be confused, the attacker is unlikely to need to call tech support when his/her USB thumb drive doesn’t appear as an icon on the screen with a cheerful bell sound.

Update 6 (11/22/09 7:20AM)

Here is the compact, more or less “official” explanation of the problem:

The problem was determined to be in the ImageCast source code, which may not be modified without retesting and certification prior to use. But it was possible to modify the ballot configuration file to contain less ancillary text, freeing up a bit more memory and preventing the crash. The Board of Elections approved the configuration file change, the 10 counties which had races susceptible to the bug were identified, and the changes made to the files in plenty of time for the election. Everything worked the way it should have, except for one thing – the process of identifying machines which needed the fix missed some of them, so the modified configuration file was not installed, and, as expected, these machines hung.

It’s a tidy explanation: judge for yourself. Keep in mind, however, that the virus code would likely occupy the portion of memory most easily accessed by people with boots on the ground (in other words, right after where the candidates in the race are defined). The easiest way to test this assertion is to pull a machine out of storage (if power has been removed, it would now be wiped of any presumptive virus) and see if it’s still “broken”.

If it is, and this exact glitch is the only indication of a virus, then you’ve got yourself a reasonable explanation from the state. If not (eg,. it works fine), then it’s indirect proof of the presence of the virus.

Update 7 (11/22/09 9:45PM)

Another nice blog regarding the electronic voting machines. He points out that the configuration tool used by the election officials (both to configure the races and cause the supposed memory problem) is actually not a certified tool, but underestimates (in my humble opinion) the possibility of a virus. On the one hand, using uncertified tools would seemingly make weird glitches more expected and help to validate the official story; on the other (if true, of course), what in the heck were they thinking using non-certified configuration tools?

How would they know that they weren’t unwittingly uploading a virus? Sloppy! But, if the virus hypothesis is in any way substantiated, it would certainly be a huge coup if the configuration utility was the delivery mechanism, as that code surely still exists somewhere.

Technical aside: I haven’t been able to find much in the way of specifications on the Sequoia/Dominion ImageCast, if anyone knows how much system RAM it contains, the size of (or better, a copy of) the default firmware image, etc. please post let me know. I am just having a hard time believing that — from an engineering standpoint — there is so little available memory that an extra few dozen characters of data could crash it (without a pretty egregious coding error).

Update 8 (11/23/09 6:20AM)

At this point, the only new spin out there is that the Gouverneur Times (the folks who broke the story when that election official used the ‘v’ word) is poorly edited, new, and hence, not a real news source (I say, have a sense of humor about it: just because they can’t spell “Governor” doesn’t mean that they were wrong about what the election official said or what meaning that statement portends).

We await to see what the Hoffman campaign does today. I’m thinking the only logical choice will be for them to let it go and gear up for 2010, since practically everyone (the local and national Republican party, the local election board, the media…) is against them at this point; but it would certainly do my heart some good if they were able to use some legal leverage to pull one of those machines out of storage — along with the configuration utilities — and let somebody qualified take a few pokes at it.

Update 9 (11/23/09 5:15PM)

An interesting post from Brad Friedman, whom I had previously linked as one of the virus “hoax” band-wagoners (original post here). The new article was much better (gratuitous mention of -16k votes for the Algore not withstanding); and while he didn’t dump on Lipari’s assessment, the updated piece no longer cites it; and he now comes to the conclusion that a manual recount is warranted (so; sorry, Brad — it looks like you’re actually interested in the same thing that we are: proof of a fair election).

Update 10 (11/24/09 8:45AM)

This from the Watertown Daily Times:

John McCaslin, the radio co-host, asked Mr. Hoffman if he suspected that “somebody tampered with the software to change the votes that were actually cast to give a different result than the voters intended.”

“No, we’re not making that allegation,” Mr. Hoffman said. “I think we have to congratulate the elections commissioners for doing the best job that they possibly could.”

Hoffman didn’t announce whether he was challenging the result yesterday; which means he is now simply weighing whether to force a recount based on what amounts to a mere technicality (the supposedly uncertified configuration software). He won’t get a re-vote based on that.

See you again in 2010, Doug.