How to do an invalid benchmarking test

A post on a mail list pointed me to a what was supposed to be a good benchmark test posted on a blog. The test is Ubuntu 8.04 LTS vs. Windows XP SP3: Application Performance Benchmark. I read down through the report and some of the results seemed a little odd. Then I get to the end where the equipment and software details are laid out and see this:

  • HDD (Windows XP): Western Digital, WD1600JD, Capacity:160GB, Cache: 8 MB, SATA150, 7200rpm.
  • HDD (Ubuntu 8.04): Maxtor DiamondMax 21, STM3160215A, Capacity:160GB, Cache: 2MB, ATA100, 7200rpm.

Talk about poor test design, XP and Ubuntu are running from two different disk drives from two different manufacturers. On top of that they are on totally different interface busses and have different size drive caches. It boggles my mind how someone can spend their time doing a benchmark and totally invalidate the results by giving the two OS’s different hardware to work with, duh. I guess this next bit from the details shouldn’t have surprised me.

  • I also disabled RAM swapping on both Windows XP and Ubuntu.
  • OK, you change the OS suppliers recommended default setting to a non-recommended setting and you think you’ll get a fair test, double duh. If you want to do a fair test of two competing OS’s you absolutely must use the same hardware for both OS’s and use the OS’s recommended performance settings. Anything else is ridiculous and completely invalidates the results.

    Mike Norman from Marietta GA in big trouble

    By now you’ve probably heard about the racist bar owner who sold T-shirts with Curious George on them and the words “Obama 08”. As John Lynch of Stranger Fruit pointed out yesterday, “And I’m sure Houghton Mifflin’s lawyers will want to have a word with someone …”.

    Well today Houghton Mifflin Harcourt has stated their displeasure, here’s a couple of quotes from an AP article:

    The publisher of the popular children book’s series “Curious George” is considering legal action against a Georgia bar owner for selling T-shirts that link Democratic presidential candidate Barack Obama to the inquisitive monkey.

    “Houghton Mifflin Harcourt did not nor would we ever authorize or approve this use of the Curious George character, which we find offensive and utterly out of keeping with the values Curious George represents,” said Richard Blake, the company’s spokesman. “We are monitoring the situation and weighing all of our options.”

    Mike had better hope that they offer him a cheap settlement and then take it without hesitation. This is because violating copyright and trademark laws can carry very steep penalties, steep enough to put him in bankruptcy. Last night I had noticed Mike Norman was trying to make himself look better by saying:

    “It wasn’t meant to be racist,” Norman said. “It was just funny to me because they look so much alike – the ears, hairline.” A friend gave him the shirts, Norman said, and he donated the profits to the Muscular Dystrophy Association.

    I wondered if the Muscular Dystrophy Association had seen this story so, I sent them an email last night with a link to the story. Today I got a nice response from the MDA saying they have declined to take his money. They also said that their lawyers have sent him a cease-and-desist order so he doesn’t use their name again.

    Of course some people still don’t get it as evidenced by this comment on a blog post at the AJC:

    everybody relax, curious george is a celebrated and respected storybook character and obama should be honored to be compared to him.

    This sounds like the type of person who thinks there’s nothing offensive with the confederate flag, duh.

    Some of the articles on this subject:

    Georgia bar’s T-shirt links Obama, Curious George | ajc.com

    Marietta Daily Journal – Mulligan’s shirt drawing protests

    Curious George publisher may sue over T-shirt | ajc.com

    T-shirt draws protests | ajc.com

    Free research resources

    I’ve found a couple of great technical research sites and both offer full content PDF downloads for free. The first is from Penn State CiteSeerx alpha here’s the opening description from their about page.

    CiteSeerx is a scientific literature digital library and search engine that focuses primarily on the literature in computer and information science. CiteSeerx aims to improve the dissemination of scientific literature and to provide improvements in functionality, usability, availability, cost, comprehensiveness, efficiency, and timeliness in the access of scientific and scholarly knowledge.

    Tip of the hat to f5r5e5d for sending me there to get a great old PSRR op-amp paper.

    The other research site is Google Patents here’s a quote from their about page:

    Google Patent Search covers the entire collection of issued patents and millions of patent application made available by the USPTO—from patents issued in the 1790s through those most recently issued in the past few months. We don’t currently include international patents, but we look forward to expanding our coverage in the future.

    IMO, Google’s system to be the easiest to use complete patent search around.

    BBC threatens Dr. Who fans

    From the EFF comes this story of extreme copyright holder stupidity, Knitwit BBC Goes After Dr Who Fans.

    Like Dr Who’s Ood, fans are happy to serve their favorite franchises when treated well. But if the BBC starts treating them like this, they can all too easily rise up and attack the very brand value the BBC is overzealously seeking to protect.

    The BBC is following in the footsteps of the RIAA by threatening Dr. Who fans with legal action over knitting patterns. The BBC legal department must not be fans of the show, why else would they work to eliminate the free publicity from the Dr. Who fans.

    May 12th update

    Digging a little deeper into this story I found that there is a bit more that either makes the BBC look worse or better depending on which version is true. Over at the mazzmatazz site the knitting section has an entry for April 10th titled “I am furious”.

    I have added creative commons licenses to all patterns now, as they are NOT to be used commercially, and the patterns are NOT to be resold.

    If I continue to discover that they are being sold, they WILL be removed.

    Then on May 5th is an entry about the BBC take down. Over at the TechnoLlama blog the post about this topic has two comments that add substantially to the story. Both comments point out that someone besides the creator of the knitting patterns had started selling knitted characters on eBay. What happened next is different between the two comments. The first comment says that the BBC got eBay to take the stuff down and then told the fan to take the free patterns down. If that’s true then I’d give the Beeb a little slack, they saw a serious infringement and then kept going a bit too far in stopping it.

    The second comment states that it was the fan who had eBay take down the characters. Then the BBC went after the fan a few days later. If this is what happened then IMO, the Beeb should have thanked the fan for the help and just let the knitting patterns slide for personal use.

    If you want to see pictures of all the characters this very talented fan made check them out at the site.

    LOL Florida

    Everyone have a good laugh out loud at Land O’ Lakes Florida because a substitute teacher at the Charles S. Rushe Middle School was fired for wizardry. This is pure insanity most likely inspired by religious fundamentalism (IMO, religious fundamentalism is a mental health problem).

    Substitute Teacher Says Wizardry Accusation Cost Him Job

    The trick requires a toothpick and transparent tape. A sleight-of-hand maneuver causes the toothpick to disappear then reappear. At least, so it seems. In reality, the toothpick hides behind the performer’s thumb, held in place by the tape.

    “The whole thing lasted 45 seconds,” Piculas said.

    He said the students liked the trick. He showed them how to do it so they could perform it at home.
    One student in the Rushe Middle class apparently took the trick the wrong way, Piculas said. He said he was told the student became so traumatized that the student’s father complained.

    Traumatized by a magic trick! This child and its parents are in need of urgent psychological treatment perhaps a long stay at a mental health facility for their extreme fantasy prone personalities. I’d guess they fall for the magic tricks some fundamentalist/evangelical preachers use to convince people of their faith healing and other scams to gain followers and make money.

    LOL Florida citizens repeat after me, there is no real magic, they are just tricks for entertainment or scams to get your worship and/or money.

    See also:
    Magic trick costs teacher job
    Wizardry In The Classroom
    Back to Blogging: “Wizard” fired from teaching gig in Florida.
    Why Harry Potter will never teach in Florida
    Hexpelled: No Wizarding Allowed
    I can’t believe in Florida anymore

    Tip of the wizard hat to PZ Myers for pointing out the story and to Christopher Petroni at Allusions of Grandeur for pointing out that the abbreviation of Land O’ Lakes is LOL.

    Updates to this article are here and here.

    Card Wiper

    When I looked around for a program to test Compact Flash memory cards I found a good free one. The program is from a company I wrote about before, DataRescue makers of PhotoRescue. You can get a copy at the bottom of the PhotoRescue Expert download page. This is a very dangerous program, not only can it permanently delete CF card data, it sure looks like you could wipe your hard drive if you set it wrong. Please be very careful with CardWiper and heed their warning on the page.

    Card Wiper is a dangerous program: remember, the data you wipe is gone forever. Use it at your own risk! If you have ANY question about CardWiper, you probably should not be using it!

    Now that you’ve been warned, I can say I highly recommend this program for its test function. I had to set the “Media access mode” to “Physical drive” in the options dialog before I could select my card reader. Once that was done it was easy to run a good write/read test of my suspect CF card.

    Expelled reactions

    Randy Olson, creator of the excellent documentary film Flock of Dodos, has weighed in with a post and a lively debate in the comments, Meet Ben Stein, the New Spokesman for the Field of Evolution. I was particularly impressed with JuliaL’s comment in that thread.

    Extremely effective as propaganda.

    I went to see the movie this weekend because I can’t expect to have any credibility in discussing it with my church friends if I haven’t seen it.

    I’m comfortable with the context of witty debate, and, viewed as part of that context, Myers and Dawkins were clever and surprisingly mild-mannered.

    I’m also familiar with the context of conservative Christian conversations about religion, and in that context, Myers and Dawkins look cold, callous, arrogant, and hateful.

    Myers takes something that the audience treasures as the most valuable element of their lives, the source of comfort in pain and tragedy, the source of the ethical and social systems they live by, and compares it to a trivial hobby (accompanied by ad by an old clip of an ugly, foolish-looking woman knitting). The very off-handedness of his comment reads as a powerful insult. The method by which one is dealing with the death of a child, the way in which one makes life-altering decisions, all a trivial hobby? That doesn’t come across so much as a put-down of religion as a dismissing of the pain, hopes, and lives of the people themselves.

    As a Christian who is entirely comfortable with the scientific Theory of Evolution, I hope that very few of my fellow church members see this movie because I think its effect may be hard to counteract. I’m sorry to see today so much scienceblog discussion focused on money – for the fundamentalists, this is an investment in missions, not a source of income – and so much less focused on what a powerful tool this movie is for turning a religious person who has known and cared little about the evolution/ID struggle in public schools into a determined advocate for ID.

    Sadly I think she’s correct about the reactions to the film by conservative/fundamentalist Christians, I recommend you read the whole comment

    Michael Shermer received a letter demonstrating some of the harm Expelled is doing by spreading lies. Richard Dawkins wrote a wonderful reply, ‘Open Letter to a victim of Ben Stein’s lying propaganda’

    A young Christian wrote an interesting post, Expelled, or Why I Won’t Be Seeing It.

    Blake Stacey has put together a list of people actually harmed in the evolution/creationism wars in his post, Creation, Power and Violence.

    Scientific American has many articles about Expelled, here are two to get you started. Ben Stein Launches a Science-free Attack on Darwin and Six Things in Expelled That Ben Stein Doesn’t Want You to Know.

    Peter Manseau wrote a review for Science & Spirit magazine, Is I.D. Ready for Its Close-up?

    Blue Collar Scientist has quotes and links for Some Expelled Reviews.

    Skeptico comments on Ben Stein’s appearance on the Craig Ferguson show, What’s Ben Stein Smoking? Seriously, has Ben lost all his brain, expecting evolution to explain things in other unrelated fields of science, WTF!

    An odd definition of what is pure science

    Someone posted a few good links pertaining to the time_t year 2038 problem and an engineer posted the following statement. (PIC refers to Microchip’s PIC family of micros)

    THIS is a perfect example of how pure science affects the PIC world.

    WTF, he thinks the creation of a computer programming data structure is pure science! This engineer has absolutely no idea what science is, sigh. What he should have said is, this is a perfect example of how large computer programming affects the PIC world. Conflating large computer programming with pure science is ridiculous.

    Processing my digital photos part 2

    In part one I started describing my work flow for handling digital photos and gave my reasons for developing and using it. This part starts off by inserting a new step in my previously described standard work flow. Before making the image files read-only I now add information to the EXIF data contained in the files. Adding information at this point in the work flow ensures that title, location, etc., info will stay with the photo through all edited versions and copies.

    While a quick look might make you think EXIF is a nice consistent standard, my research quickly made me realize this is not the case. EXIF is so flexible that it is more appropriately thought of as an un-standard like PCB Gerber files (RS-274). As with RS-274, EXIF is so flexible that it is not practical for any one program to be able to handle all the possible variations.

    The best solution I found for handling the majority of variations in this image metadata is ExifTool by Phil Harvey. This tool set is a Perl library and command line program that can manipulate nearly any piece of EXIF data. While a command line tool is very handy a GUI shell is often desirable and one is available for ExifTool at the HBx Hobbypage. The program, ExifTool GUI, gives you a file manager type interface that makes it even easier to edit EXIF image metadata.

    I start off updating the EXIF data by batch adding information like artist and copyright using the command line ExifTool. The GUI tool gives me a shortcut for using the command line tool, when you right click the image’s directory you can select the “Open Command Window Here” item (AFAIK, this capability comes from Windows it isn’t a custom bit exclusive to the ExifTool GUI app). Selecting this menu item gives you a command prompt already located in the image directory ready to accept the command with parameters.

    As a time saving shortcut I keep a little text file that has examples and previously used parameter sets. I construct a new command line or copy a previously used one from the text file and paste it into the command window. Here’s an example of a command line I’ve used to mass update image metadata.

    "C:Program FilesEXIFtoolexiftool.exe" -Artist="Paul Hutchinson" -Copyright="Paul Hutchinson" -City="Disney World" -Province-State="FL" -Country-PrimaryLocationName="USA" *.jpg

    Pressing enter updates all the images in the directory with this new/changed metadata in one quick command. The next step is to update the unique information like the image description using the ExifTool GUI program itself. If some images use the same data (e.g. exposure/composition variations deserving of the same title) then I use standard multiple selection techniques before activating the data editing function.

    Once I have all the EXIF data updated, it’s now time to set the read-only attribute of the original image files so that they don’t get accidentally overwritten. The fast way I use is to press Control-A to select all the files in the ExifTool GUI and then press Alt-Enter to open the the standard Windows multiple file properties dialog box. When the multiple file properties dialog opens the “Read-only” checkbox is already in focus so, all I need do is hit Spacebar to mark the checkbox and then hit Enter to change all the selected files to read-only. This is easier to do than to describe in writing, after you’ve done it a few times it will become a fast four keystroke/combination keystroke operation (Ctrl-A, Alt-Enter, Spacebar, Enter). If you don’t or can’t get into the groove of using this quick keyboard operation, then give up on ever being efficient with computers and go ahead and click your way through the process using that killer of UI efficiency the mouse ;-). As an aside why can’t every computer user just stop clicking for Copy/Cut/Paste operations and just use the so much more efficient Ctrl-C/Ctrl-X/Ctrl-V keyboard combinations instead :-).

    The next step is to convert the image files to a loss less format in preparation for editing (Note, ExifTool does not alter the image data so, even though technically it’s re-saving a compressed format, there is no data loss). For this conversion step I use PaintShop Pro’s batch conversion feature and its loss-less PSPimage file format. First I select all the originals in the PaintShop file browser (organizer in new versions) then select the “File-Batch Process…” menu item. This opens the batch process dialog with all the files listed in the “Files to process” list box. I set the “Save Mode:” to “NewType” and in the “Save Options” I select PSPimage in the “Type” drop down list and set the “Folder” to the root of particular images sub-directory structure (e.g. D:My Pictures2008-04-19). Finally click the “Start” button in the dialog and watch as the files are converted and copied to the new location.

    With the files now in a safe format I can edit away to my hearts content knowing that if I screw things up royally I still have my originals to start over with. That’s all for part two in the series, part three will cover how I take the edited pictures the rest of the way to web/CD albums for others to enjoy.