MySQL repair or replacement on OSX Server (Yosemite)


Another scratchpad post.  This one is a reminder of what I did to repair MySQL on OSX Server after the upgrade from Mavericks to Yosemite kinda broke things.

I was working to solve two problems: intermittent “unable to connect to database” errors on all our WordPress sites, and the dreaded “unable to update PID” errors when starting and stopping MySQL.

  • I think the “unable to connect” errors are caused by intruders trying to brute-force break the passwords on my (roughly 35) web sites.  This problem can possibly be cured just by doing the “tuning” steps at the end of the cookbook.
  • “Unable to update PID…” type problems are more symptomatic of a broken MySQL implementation and probably require the whole process.

None of these were terrible (a system-restart every few days kept things more or less in check) so I limped along for a while after upgrading from Mavericks to Yosemite, but it finally drove me crazy and I decided to upgrade MySQL and rebuild all the databases.

The cookbook

I tried several approaches and finally landed on one which I can reliably repeat in the future (as long as the good folks at Mac Mini Vault continue to provide their magnificent script).


I used backups from all sorts of places.  I thought I was being a little over the top, but things went wrong and I was really glad to have all these safety nets.  Here were the backups I had available:

  • Time Machine backups of the /usr/local/mysql/ directory
  • Pre-upgrade copies of the /usr/local/mysql/data/ directory (and their Time Machine backups)
  • Historical (nightly) MYSQLDUMPs of all the databases (and their Time Machine backups).  Use this command to write each database to a text file.  I have a script that does all 35 of my databases at once, every night.
    sudo mysqldump -u root -p[root_password] [database_name] > dumpfilename.sql

clear out the previous installation of MySQL:

This was by far the hardest part to get right.  MySQL doesn’t have an “uninstall” script and reacts badly when little odds and ends are left over from previous installations.  My OSX Server has been running MySQL since OSX Lion and there was a fair amount of cruft left behind that was causing some of the trouble.  Here’s my current list of things to move or remove (although not all of them will exist on any given machine):

  • move the old data directory (/usr/local/mysql/data/) to someplace else (yet another backup)
  • rename the old base MySQL directory (this is the directory with the version-number that the mysql alias points to – I renamed rather than deleted as a backup)
  • remove the /usr/local/mysql alias (it’s going to get recreated during the install, pointing at the new/correct base directory)
  • move out of /Library/StartupItems/
  • move My* out of /Library/PreferencePanes/
  • move My* out of your account’s /Library/PreferencePanes/  (I had two mismatched ones of these, one lurking in /Users/admin/Library/PreferencePanes that was really old)
  • edit /etc/hostconfig and remove the line MYSQLCOM=-YES- (If it’s still there — this was left over from the days when MySQL shipped as part of OSX Server)
  • remove entries for receipts for mysql in /Library/Receipts/Install-history.plist (I edited the plist with a text editor to do this)
  • remove receipts for mysql in /private/var/db/receipts/
  • remove mysql-related scripts from /Librarly/LaunchDaemons/
  • remove any aliases for mysql.sock from /var/mysql/, /etc/ and /private/var/mysql/ (I’ve had good luck leaving the directories in place and just deleting the aliases – ymmv)
  • If the Mac Mini Vault (MMV) script has been run before, here are a couple more things:
    • remove the MYSQL_PASSWORD item from the desktop
    • remove MySQL-install.plist from your /Downloads/ directory

install MySQL using the MMV script:

NOTE: the folks who maintain the script only support it for a clean install of MySQL on a freshly-loaded version of OSX.  So this cookbook is OUTSIDE the bounds of what they support — please don’t complain to them if things break.  Instead thank them for sharing this script publicly, and consider buying some of their services.

Click HERE to read an introduction to the script on MMV’s site

Click HERE to read the important documentation of the script

Last step:  change the MySQL root password:

sudo mysqladmin -u root -p'MMV-generated-password' password 'mypasswd'

MySQL should now be running properly.  I restarted the server to make sure MySQL started up on a reboot.  I also started and stopped MySQL a few times from the command line to make sure that the “unable to update PID…” problems were solved:

sudo /usr/local/mysql/support-files/mysql.server stop
sudo /usr/local/mysql/support-files/mysql.server start

I do NOT TRUST the MySQL preference-pane that is installed in OSX System Preferences and don’t use it – that may have been another source of dreaded “failure to update PID…” errors.  Just sayin’

import the databases:

I chose to rebuild my databases from the nightly dumps.  I tried various versions of “moving the data directory back and using the Update command” and had a rough time with all of them.  Besides, rebuilding the databases for the first time in many years seemed like a good housekeeping item.  I have about 35 databases — it took about an hour.  Note: change all the ‘mydb’ ‘myuser’ ‘mypasswd’ to values that match your environment.

  • Log into mysql as MySQL’s root user:
mysql -u root -p'mypasswd'
  • Create the databases in mysql using this command:
create database mydb;
  • Create a user for each database in mysql using this command (btw Sequel Pro 1.0.2 is crashing when it creates a user — a known bug, just do it from the command line).  Note: I’m assuming that you’re only using MySQL for WordPress sites like I am, and only need one user per database — this process will get a lot more tedious if you have multiple users per database.
grant all privileges on mydb.* to myuser@localhost identified by 'mypasswd';
  • Import the text-dump of each database into the newly-created empty one using Sequel Pro — File > Import


Two things have really helped with the brute-force attacks.  Opening up MySQL a bit and changing a setting in PHP.

To give MySQL a little more oxygen, I followed the guidelines in a sample .cnf file that came with the Mac Mini Vault script.  I slightly changed the settings, mostly to make them conform to MySQL standards (I’m not sure whether this matters).

  • edit /usr/local/mysql/my.cnf and add these lines at the very bottom (these are just a starting point, feel free to fiddle with them a bit):
  • edit /private/etc/php.ini and turn off persistent links.  Here’s the way my file looks right now:
; Allow or prevent persistent links.  
mysql.allow_persistent = Off

Image courtesy of Stuart Miles at



DSSA — DNS Security and Stability Analysis working group

I’ve been spending a fair amount of time working on an ICANN cross-constituency working group that’s taking a look at the risks to the DNS.  Our gang just posted a major report today and I thought I’d splash this up here so I can brag about our work on Twitter and Facebook.

That first picture is a summary of the methodology we built (we had to build a lot of stuff in order to get our work done).  It’s basically a huge compound sentence that you read from left to right in order to assess risk.  By the way, click on the pictures if they’re too small/fuzzy to read.

This second picture shows where we, the DSSA gang might fit in a much larger DNS security context.  We also had a lot of stuff to puzzle through about where we “fit” in the larger DNS security landscape.

And that last picture is a super high-level summary of what we found.  There’s lots more ideas and pictures in our report — but these three give you kindof a taste of what we’ve been working on.  I think it’s darn nifty.

If you’re interested in the whole scoop, head over to the DSSA web site.  You’ll find links to the full report, a cool Excel worksheet that crams the whole methodology on to one page (complete with scoring) and more.



Grinnell Reunion 2012 — a life of happy accidents

I gave a talk at my Grinnell College reunion last weekend and decided to build this post to share a bunch of links to things that I talked about.  This ain’t a’gonna make any sense to the rest of you.  But the stuff is interesting.  🙂

This is a story of rivers of geeks.  I described the rivers that I swam in during my career, but these are by no means all of the species of geeks that ultimately built the Internet.  I was lucky to be a part of a gang of 10’s maybe 100’s of thousands of geeks that came together in the giant happy accident that resulted in this cool thing that we all use today.  But don’t be confused — it was a complete accident, at least for me and probably for all of us.  Here’s a diagram…


The opening “bookend” of the talk was to introduce the idea of “retrospective sense-making” which I first learned about from Karl Weick when I was getting my MBA at the Cornell business school

I talked a little bit about what it was like as an Asperger guy showing up at Grinnell in the fall of 1968 — when everything was changing.  We Asperger folks have a pretty rough time dealing with changes.  Several people spoke with me about this part of the talk later in the weekend.  The really-short version of my reply was “just give us more runway.”  Many of the geeks that built the Internet are Asperger folks.

Another giant gaggle of geeks is the “community radio” gang that I was part of.  That part of the talk opened with a discussion of Lorenzo Milam, one of the folks who inspired many of us community-radio organizers to go out and do ridiculous impossible things.

  • These days Lorenzo hangs out at Mho and Mho Works (and Ralph Magazine)
  • He put the word “sex” in the title of his handbook about starting a community radio station, Sex and Broadcasting, just to get your attention and this was the book that got a lot of us going

Which led into a discussion of my involvement with the community radio movement — Tom Thomas, Terry Clifford and Bill Thomas are all still very much involved in public and community radio these days.

Then there was a musical interlude (you cannot believe how much the music went off the rails — almost all the technology failed — oh well).

The next series of accidents revolved around the “learn my chops in brand-name consulting organizations” part of the saga.  Another of the rivers of geeks — many people of the Internet construction workers came from big firms like Arthur Andersen and Coopers and Lybrand, the two places I worked.  Probably the biggest things I learned there were Structured Programming and project management.  And this…

The next accidents ran this Forrest Gump type guy through a couple of now long-dead mainframe companies , another BIG source of internet-building geeks.  First ETA Systems, the hapless wannabe competitor to Cray.  Then Control Data, where I learned how to do mass layoffs in an imploding manufacturing company.  Ugh.

I was an early personal computer enthusiast as were almost all Internet geeks.  I live in the Midwest, so I missed out on the Homebrew Computer Club in Silicon Valley.  Dang.  But relatively cheap modems showed up about that time which led to the rise of the Bulletin Board System (BBS) movement which provided the gathering places for a lot of us Internet geeks. Boardwatch Magazine, published by Jack Rickard, was the glue that held us together — Jack inspired me much the same way that Lorenzo Milam did.  The arrival of FidoNet allowed email to flow beyond the local boundaries of a BBS and brought a lot of us geeks together for the first time.

Another giant pile of Internet geeks came from the ham radio movement.  My call is KZ0C and I’m completely lame — I hardly do anything ham radio related these days.  But a whole giant tradition of “makers” comes out of that gang.  We hams were darn early adapters of the packet networking protocols that underpin the Internet.  We turned that stuff into packet radio.

So there’s the list of pre-Internet geek communities that I was a part of in one way or another.  No wonder some of my friends call me a Forrest Gump of Internet technology.  So what happened next?  This is what happened next…


That’s a picture of the first four-node ARPANET network in the late 60’s.  The network grew slowly over the next couple decades and by the mid-80’s had been opened up to include institutions of higher education.  I worked at the University of Minnesota which, when I was there, was home to the Gopher protocol and the POP3 email protocol — another great gaggle of geeks.  I was a Dreaded Administrator, there to fix a financial system problem, but I loved those geeks ’cause they were the ones that turned me on to the Internet.

The next kind of geeks that still play a huge role in the Internet are the folks that work at Internet Service Providers (ISPs).  Ralph Jenson and I started an ISP in my basement and called it  That project grew into an amazing gang that eventually got rolled up as the ISP market consolidated in the late 90’s and thereafter.  Lots of the geeks I’ve described in this post were involved in starting those early pioneering ISPs — what a time…

The last geek that I mentioned in my talk is Hubert Alyea, the role-model for the Disney films about the Absent Minded Professor.  Professor Alyea was another great Asperger geek who was quite emphatic in telling me about lucky accidents, great discoveries and the prepared mind.  Click HERE to see movies of some of his lectures on Youtube — they’re astounding.

What are Mike and Marcie obsessing about now?

The rest of this post is a series of links to projects that I mentioned during the talk.

The final thing I need to throw into this post is three little graphs I made up to describe the half life of knowledge — in which I choose to view the glass as half full.  As the half-life shortens, it takes less and less time to become an expert!








Mikey in the high branches.

This is a post that most readers of this blog are going to scratch their heads over.  I volunteered a fair amount of my time to ICANN (the organization that works on the domain-name and numbering systems that underpin the Internet).  Until yesterday.  I got pretty cranky over an email exchange that I (as a working-group member at the bottom of ICANN’s bottom-up policy-making process) had with a couple Big Kids on the Council that manages our working-group-based policy-making process.  I loudly resigned over this — here’s a link to my grouchy email to the community.

Kieren McCarthy wrote a great article that places my resignation in context and that article kinda went viral in the community yesterday afternoon.  A bunch of people have asked me “hey Mikey, what that heck put you up in the high branches like that??”  So I’ve decided to post the email dialog that so got me going.  Sorry to those of you regular readers who will be scratching your head over this weird post.

Cast of characters in the tragedy;

Mikey — that would be me

Tim Ruiz — one of the people who represents Registrars on the GNSO Council.  Tim works for, which is by far the largest registrar (essentially the Wal-Mart of domain-name registration outfits).  With those two hats, Tim pulls considerable weight in the organization.

Stéphane Van Gelder — another Registrar representative and also Chair of the GNSO Council.  Another heavy hitter.

The Dialog;

Mikey: hi all,

i’m just lobbing a suggestion into the “locking during UDRP”-recommendation discussion that’s going on in advance of the Council meeting coming up later today.  this note is primarily aimed at my Councilors, colleagues in the BC and fellow members of the IRTP-WG, but i’ve copied a few others just because i can.

as a member of a working group that’s wrapping up two years of work on this stuff, i am hoping that the Council will not rewrite our recommendations on its own.  this is a repeat of the “i’m trainable” comment i made in SFO.  what i’m hoping is that the Council will vote the recommendation up or down and, if it would like, sends the defeated recommendation back to the working group for refinement.  you can even include suggestions if you like.  but please don’t make changes to our recommendations without giving us a chance to participate in the process.

you can invoke all the historic “Council should be *managing* the policy process, not being a legislative body” arguments in this paragraph if you like.

i’m still trainable.  🙂

Tim Ruiz: My goal is not to derail the rest of the work over this since that rec was already acted on. The locking question has already been picked up in the UDRP issues report (done in response to the RAP report).

Mikey: yep — i get that Tim.  i’m really zeroed in on the process, though.  it would be fine to push it back to the WG with your comment as annotation.  this issue is the perfect one to use as a test-case for the very reasons you describe.  my worry is that some day we’ll get to a tough/complex issue  on a WG report and the Council will roar off and try to fix it on the fly rather than pushing it back to the people who’ve devoted the time to get up to speed on the nuances.

as a WG member i’d much rather hear “hey WG folks, can you fix this?” than “we fixed it for you.”

Tim Ruiz: There is nothing for the WG  to fix and the Council is not changing any recs. We just want to consider that one with the UDRP issue it is already tied in with. I am all for process, but we can protect that without duplicating efforts.

Mikey: you folks get to do whatever you want to do — but like i said, i’m trainable.  if you as the Council are going to make that call, without engaging the WG in the conversation, you’re setting precedents that the Council may come to regret when it is trying to recruit volunteers to devote years of their lives to efforts like that in the future.

all you have to do is ask us, rather than telling us.

Tim Ruiz: Mikey,

My record is pretty clear on process. I defend it fiercly. But you are really blowing this out of proportion. If you are trainable, let it show. Let’s discuss further F2F.


Mikey: Tim, i’d much rather have this conversation over a limited-scope test-case issue that’s relatively straightforward to resolve than a really hard one.

if working groups are the place where policy gets made, then let the WG fix this minor problem for you rather than fixing it yourselves.

Tim Ruiz: I’d rather not. I’ve explained it to you. You either don’t get it or don’t want to. If you want to discuss F2F let me know.

Stéphane Van Gelder: Mikey,

I think the GNSO Council has a clear understanding of its role in the policy development process.



Mikey: yep.  and so does this volunteer WG member.  i’m now fully trained.


I’m calming down (and was much appreciative of all of you who reached out to help me with that).  So I’m clambering down out of the high branches (while sitting in the Tokyo airport transit lounge on the trip home — not exactly the best place for reflective writing).  Thanks all of you who reached out.  I’ll write you direct notes tomorrow after I’m back in the midwest.

Online privacy tips

UPDATE March 2018: I just realized that I neglected to change this post to reflect my views about Facebook privacy.  Admonishment: delete the Facebook app on your phone – now.  Tip: turn off their apps, websites and plugins “platform” in their “Apps” settings section.  Don’t forget to use the tools listed below to block their pesky “share on Facebook” beacons and trackers on web sites — that’s how they collect personal data on people who aren’t Facebook subscribers.

UPDATE March 2016: This post was written in 2011, a more innocent time pre-Snowden.  I still run all this stuff, but recognize that these things won’t protect you from all the NSA attacks that have been revealed since then.  I’ve also added a broader workstation-security checklist at the very bottom of this post.

A friend asked Marcie about reducing her exposure to ads on Facebook and I decided to write up the answer as a blog post so it would be easy to send to others (and update with new stuff). So here is a list of stuff that I do — your mileage may vary.

I use Firefox as my primary web browser (and keep it up to date), mostly so I can add a gaggle of plug-ins.  Some of these are now available for Safari too.  Here’s the list (installing the first three will provide most of the benefit).

  • 1Password — a great way to manage a bajillion really-strong passwords on web pages, but costs (a little) money
  • NoScript — allows you to choose which pages you trust, and blocks Javascript on all the rest
  • Redmorph — my newest blocker, liking it so far [March 2018]
  • Privacy Badger — a good all-in-one blocker from EFF
  • BetterPrivacy — gets rid of “persistent” cookies that are used by lots of big companies (Google, Yahoo, etc) to track your behavior on the ‘net
  • Ghostery — same sort of thing that BetterPrivacy does, but gets rid of trackers that aren’t cookies
  • Adblock Plus — a plugin which, once you’ve subscribed to the EasyList USA filter, gets rid of all the ads on web pages
  • ShareMeNot — stops those Facebook/Twitter/etc. “sharing” buttons from sharing stuff until you click them
  • Web of Trust — take advantage of their huge database of “safe” and “unsafe” sites built by other Web of Trust users — like me.
  • HTTPS Everywhere — a project of the EFF to redirect to the SSH-encrypted version of popular web sites

I also have peculiar web-browser habits to further reduce the risk that corporations (or other bad-guys) are tracking me

  • I don’t log into any of the “big data” services (like Google, Yahoo, etc.) unless I absolutely have to and I log out when I’m done.  They track what you do while you’re logged in.  I just did a “What if Google Turns Evil?” podcast if you want to learn more about why I avoid Google services these days.  UDATE: See the “Divorcing Google” section below.
  • I don’t permit the web browser to “remember” any passwords — I use 1Password for that
  • I disable the “browsing history” feature, so the browser doesn’t remember where I’ve been in the past
  • I disable the “search” and “form” history features too
  • I allow the browser to “accept cookies” and “accept 3rd-party cookies” but I only keep them until I close Firefox, then all cookies are deleted
  • I have the browser open a blank page when it launches (just about every site plants a cookie when you arrive)
  • I disable Google and Yahoo in the “search” choices (they plant cookies when the browser starts)
  • I avoid putting cookie-planting sites (Google, Facebook, etc.) in the shortcuts bar (they plant cookies when the browser starts)
  • I elect to clear history when Firefox closes
  • I close and restart Firefox several times a day, especially after logging into Google, Yahoo, Facebook, etc.
  • I use the ICSI Netalyzr to check my DNS service-provider to see if they’re intercepting/redirecting some of my traffic (also good for all sorts of performance-improving stuff like identifying “buffer bloat”)

Facebook — DO NOT use their smartphone app.  If you have it, delete it.  It’s capturing all kinds of data about your phone calls and text messages on that phone.  I deleted that app almost immediately and have for years only used Facebook on my computer (and thus subject to all of the tips I’ve listed above).  Here are things you can do in your Facebook account.  As of this writing, these can be found in the “Privacy Settings” part of the “Account” menu — but they change things all the time, so look carefully.

  • I periodically run the “Scan for privacy” tool from
  • I’m pretty liberal with what people can see, but very conservative with what they can share about me with other people
  • I’m very aggressive in blocking applications — I try hard not to sign up for any applications and block them when they appear in my news feed.  UPDATE 2018: this is easier now — turn off the Facebook “platform” in Settings/Apps.
  • I am pretty aggressive about blocking “bozos” in my news feed.  I don’t unfriend them, I just block their inane posts.

Divorcing Google.  Inspired by this post about “Divorcing Google”, I decided to describe my replacements for all things Google — they’re very similar to his.  I too have pretty much completely weaned myself from Google, for the same reasons.  Here’s my “replacements” list.

  • Search — DuckDuckGo SSL
  • Mail/contacts/calendar — I run my own servers for these.  It’s a hassle but worth it to me.
  • Maps — Apple Maps
  • File storage — Dropbox

Broader topic: workstation security.  Tip of the hat to John Hoffoss for this link to a terrific workstation security checklist.

There.  That’s my list.

Whit Diffie is the new VP of info-security and cryptography at ICANN! Kewl!

Very neat news today out of ICANN.  Whit Diffie is this monster figure in the crypto world — he’s one of the founding folks in that circle.  He worked at Sun for ages and now he’s joining ICANN.

Click HERE for the ICANN press-release.

Click HERE for a starter-page at Wikipedia.

Click HERE to watch him on an episode of Cranky Geeks (with John Dvorak) to get a feel for what’s he’s like in person.

I’m really glad to hear that he’s joining the ICANN gang.  It’ll give us some depth that we badly need in this area.

Consensus decision making — WORT-FM, 1975

This is a piece by Jeff Lange in Volume One, Number Three of “Spread the WORT” — the newsletter of WORT-FM (Madison, WI) just as it was going on the air in 1975.  I’ve always loved this description of the consensus decision-making process we used to run the station.  All due apologies to Pogo…

The big deal?  The sentence that really catches it for me is “we ad WORT don wanna tred up on the wee miroridy vuponts, so we jus wade undill eberyone am finely agreed.”  Still works for me today, some 35 years later.  Thanks Jeff!

Here’s my translation, since many of you aren’t native-English speakers and might find this pretty tough to read in Jeff’s native Pogo-style language.  Apologies to Jeff for any mistranslations.

Yes, it’s a curious fact, that nobody is ever able to quite explain, how decisions get made at this particular radio station.  But they do.  This is a grievous hard and ticklesum thing for newcomers to digest.  Take, for example, the familiar caller who, in a fever pitch of excitement, has phoned up the station with his or her (or “it’s” for that matter) idea for a program.   Rnnng.  He (let’s just say it’s a “he”) says “My dog can bark heavy metal rock n’roll — can he have 5 hours on Tuesday nights?”   Well, the person at the station (say it is a person) says “Isn’t that the same thing as what’s on WBRK every night?”  The caller replies “Yes, but my dog barks badder!”  Then that, says the person, is a question for the Program Committee.

The best thing then is if the caller hangs up, thinking all is well for the Program Committee will do its duty.  But if the caller says “Oh, what’s the Program Committee?” then the person has to explain: The Program Committee are all the people that come to the Program Committee Meeting.  You can come.  So can your mother.  It’s Friday at 8pm.  No, they never vote on anything.  Voting is against the rules.  So is parliamentary procedure. They just talk about things until everyone is agreed, and that is consensus — the highest form of unanimity.

Then the caller says “oh.”

Then the person at the radio station should continue: “Yes, it’s a curious fact, but it seems to work.  So far, at least.  We at WORT don’t want to tread on the wee minority viewpoints, so we just wait until everyone is finally agreed.  Nope, it’s never failed yet…  which just goes to prove: you can make some of the decisions all of the time, and all of the decisions some of the time…”

Then the caller says, “can you put me through to the general manager?”

“No, there isn’t a general manager.  Would you like to talk to Sarah-Gene?”

“She the owner?”

“Nope.  She’s just another volunteer.”

New volunteer job — 37-word long title

I’m thinking another fold-out business card may be required;

Vice Chair of Finance and Operations (of the)
Commercial and Business Users Constituency (which is part of the)
Generic Name Supporting Organization (which is in turn part of the)
Internet Corporation for Assigned Names and Numbers

Can you see why ICANN has a bafflegab problem?

I’m quite excited about this one — it’s got lots of tasty issues and it’s the ops and finance stuff that I love to do. 

I had another fold-out business card job back in the early ’90’s.  That fold-out business card read;

Temporary Interim Acting Assistant Associate
Vice President (supervising)
Administrative Information Systems
Business Operations
Quality Management
Operations Improvement (for the)
University of Minnesota

or…  Vice President of Stuff that is Busted.  This new gig is a lot less complicated than that one was.

Infrastructure security – some useful ideas

I was on a panel talking to a bunch of infrastructure-security type people yesterday and came away feeling like we didn’t deliver on our promise to provide practical hands-on stuff.  So I’m tossing a couple Powerpoint slide decks up in this post by way of making amends.

This first one is the deck we used in Saint Paul to rally people around the “get ready for Y2k” initiative.  It’s an example of how to do non-scary, what’s-in-it-for-me? conversation around a pretty tough topic.  Maybe some of this kind of thinking can help the security folks when they’re pitching to their customers.  Click HERE for the file (no warrantees — scan it before you open it).

This next file is a huge deck I put together when I was first briefing the Big Kids at MnSCU about their enterprise security initiative.  This was the basis of selling senior management that this was a Good Thing and showed them how security could make them more money, make them more nimble, improve quality and oh by the way reduce costs.  This is an “everything including the kitchen sink” deck that might have a few ideas for people to steal.  Click HERE for the file (same warrantee as above).

There.  I feel like I’ve lived up to my advance-billing now.  Hopefully some security mavens will find some useful stuff in these.

High Dynamic Range (HDR) Photography — a hoot!


This is another one of those “document what I’m doing so I don’t forget” posts.  Thanks to Matt Walsh, I’ve joined the HDR cult.  This is some kinda fun!

First part of the project was to drop a copy of CHDK on my Canon SD 950 IS point and shoot camera.  Putting this free, open-source code on the camera is one of those projects I’ve been teetering on for a year or so.  But somehow it either felt Too Hard or Too Scary each time I approached it, so I procrastinated.  I finally did it and I wish I’d done it right off the bat.  Completely easy, completely safe, worked the first time.  So now my cheezy camera does all kinds of cool things — I can save RAW format files, I can put a histogram up on the screen, all kinds of neat stuff.

And one of the neat things I can do is have the camera “bracket” shots when it’s in continuous-shooting mode.  This is an essential part of the process of shooting these HDR photos — shooting a series of pictures that vary the exposure.

So here’s the series of pictures that went into that photo at the top of the page;

img_2275s img_2276s img_2277s img_2278s

img_2279s img_2280s img_2281s img_2282s

So there are eight photos, taken by holding the shutter button down and letting the camera just fire away.  The CHDK software takes the first photo at the best setting the camera can manage and then takes alternating lighter and darker shots until you stop holding the shutter button down.  You can tell the camera how much to increment the exposure — I have it set to 1 F-stop increments.

Click on the photo at the top of the page and you’ll see that there’s detail in the darkest spots and the lightest spots.  Pretty cool huh?  Well, I think so…

The software that does the magic is called Photomatix Pro.  You’ll see LOTS of cool photos and get lotsa info if you go to that site.  I think their stuff is way neat.  Here’s another one (I ran this one through the software before I bought it, so it’s got watermarks in it).


Same deal — click on the photo and you’ll get a bigger version.  Now here’s the deal — you’re supposed to take these pictures on a tripod (after all, you’re stacking 3 to 8 photos on top of each other, they better be lined up).  But the combination of the anti-shake in the camera and the image-aligning capability of the software means that I can get pretty good results from hand-held shots like these.  All of these pictures were shot without a tripod.  There’s a little trouble in there, but nothing that’ll bother me given what I do with my photos.

Here’s the sequence of shots that went into that one.

img_2254s img_2255s img_2257s

img_2258s img_2259s img_2260s

Chronicle of a banking-system collapse

I started watching these charts a couple months ago and, after the latest round got released, decided I’d post them here as a “canary in the coal-mine” alert.

These charts tell me that the folks at the Fed are in uncharted territory. Their own charts tell the tale…

This first one, “non-borrowed reserves of depository institutions” shows a quite startling plunge over the last three months. If I were running the family checkbook and looking at this chart I’d be saying “Marcie, I think we’re broke.” Here’s where I started;

Non-borrowed reserves

and here’s the latest version;

This is the latest...
Click HERE if you want to look at the current version of this chart. I sure hope it looks better when you visit their page.

Here’s another one — pretty much the converse of the first picture, but this series goes a lot further back. Yep, another huge swing. The thing that I like about this series is that it goes way back in time — to long before the great banking crash of 1930/31.
BORROW TotalBorrowingsOfDepositoryInstitutions

and here’s the latest version;

The latest...

A couple months ago, you could see all kinds of wiggles in the chart. Now it’s just a flat line with a giant upsurge at the end.

Click HERE to see the current version of this chart.

Here’s another view. This is the change in the size of the money supply, compared to last year. As you can see, the Fed is pushing this up pretty hard right now.

Click HERE for the page that I used to generate this chart — unfortunately, I can’t automate the “current version” display.

I’m interested in these charts because this represents a huge “tinkering” effort by the Fed. I hate those. I worry about unintended consequences (maybe a lot of inflation?). I worry that the banks are masking a huge weakness paving the problem over with money borrowed from the Fed.

Click HERE for a link to a bunch of charts like this from the St Louis Fed. Once you’ve clicked on a series that you’re interested in, look for a link that says “current series in FRED” to see the charts like these.

For the first time in my life, I’m buying gold. Click HERE for a link to BullionVault – that’s where I’ve landed after conducting all my usual obsessive-geek research. They’re a great gang, I love their systems, I appreciate being able to actually own the metal and I think the opportunity to choose between vaults in 3 countries rocks. I never ever thought I’d get to this ridiculous place. But this is nuts, people. I’ll keep adding on to this post as events unfold.

Related links

April 8, 2008 — Paul Volcker (Fed chair during Carter) blasts current Fed policy — click HERE — Summary: The Fed has lost sight of its mission to defend the dollar.

April 27th, 2008 — “Private Profits and Socialized Risk” — Ben Stein (NYTimes) highlights an April 8th speech by David Einhorn to Grant’s Spring Investment Conference. Click HERE for the NYTimes “Cliff Notes” and HERE for the full transcript (in PDF format). Summary: Wall Street firms are incented to take on unhealthy levels of risk, using capital reserves that are valued (and rated) by the firms themselves. Regulators snooze.

May 1, 2008 — Iran stops dollar-based oil trading, switches to Euros and Yen — Click HERE. Summary: Partly political of course, but also partly because of the continuing weakening of the dollar. This will be big trouble if the “walk away from the dollar” trend continues. Confirms Volker’s view.

May 1, 2008 — Kevin Phillips publishes “Numbers Racket, why the economy is worse than we know” in Harpers Magazine. Phillips contends that economic measures like the Consumer Price Index (CPI) and unemployment statistics have been gradually “sweetened” over the past 30 years, giving investors a too-positive view of the economy. Click HERE for the article.

May 2, 2008 — You recall those charts at the top of the page? Sure you do. Well today central banks (the Fed, Britain and Switzerland) expanded that program. Next month’s chart ought to be a hum-dinger. Click HERE Summary: The Fed added $25 billion to the $50 billion they’re already lending to non-bank banks, and also loosened the standards for what they’ll accept as collatoral for those loans.

May 7, 2008 — WSJ story about a Fed proposal to pay interest on required reserves. Interesting note; the article mentions that since last July the Fed has “replaced half the roughly $800 billion of Treasurys it held last July with loans to banks and securities dealers.” The pictures at the top of this page only show about $100 billion of that implied $400 billion. Click HERE

May 14, 2008 — Karl Denninger writes a story about these same charts. My observation in yesterday’s Geezercast is that this is starting to feel a lot like the Y2k crisis — there are “pollyannas” who are motivated to keep things calm, and “doom and gloom” people who are predicting disaster. And precious little credible information in the middle. Karl is definitely in the Doomer camp. I think his analysis is wrong, in that he doesn’t include *both* charts that I have here. But I’m not sure about the implications. Click HERE for Karl’s take.

May 14, 2008 — This American Life does a great episode called “The Giant Pool of Money” which describes the mortgage crisis in the voices of the participants. Absolutely fabulous radio. Click HERE for a link to the podcast.

May 28, 2008 — The New York Times comes out with a “looking back” piece. Click HERE for the story. One of the problems with situations like these is the chasm between the “doom and gloom” perspective and the “Pollyanna” perspective (reminds me of the Y2k crisis). Like Y2k, “muddling through” seems to work in many cases.

May 30, 2008 — click HERE for a gloomy FDIC (the people who insure banking deposits) report on the state of the banking industry. Report headlines: Industry Earnings Decline 46% from Year-Earlier Level, Loss Provisions Absorb a Higher Share of Revenue, Troubled Loans Accumulate in Real-Estate Portfolios, Lending Growth Slows, Fourth Quarter 2007 Earnings Are Revised Below $1 Billion. Click HERE for a chart that summarizes the trouble — Reserve Growth Has Not Kept Pace With Rising Noncurrent Loans.

June 8,2008 — The Economist publishes a gloomy story — their take is that the credit crisis is far from over, unlike the cheerful prognostications from Wall Street.  Interestingly they mention that the Fed is intending to end the the credit facility (reflected in the charts at the top of the post) by September.  Given where the total is at (over $100 billion at this writing), that seems an equally big disruption, in the opposite direction.  Click HERE for the article.

July 11, 2008 — Fannie Mae and Freddie Mac started to unwind this week.  This is a situation like the Bear Stearns run that triggered this article.  Except bigger — because these two companies back about half of all mortgage debt.  Click HERE for an early NYTimes story, and HERE for a grumpy reponse (Bloomberg) to the Fed’s proposed bail-out plan.

Sept 7, 2008 — Fannie Mae and Freddie Mac wind up in US government hands.  Click HERE.

Sept 15,2008 — Lehman Brothers is bankrupt after frenzied negotiations, Merril Lynch is bought by Bank of America, AIG is in trouble.  Click HERE.

Sept 17, 2008 — A bad day in the markets today.  Morgan Stanley and Goldman Sachs are in trouble, credit markets have seized up, international markets (like Russia) are hurting too, a major money-market fund “breaks the buck”, US T-bill yield is at levels not seen since Pearl Harbor — Click HERE to read a representative story from the Financial Times blog.  Gold was is up almost 10% for the day.

Oct 10, 2008 — My goodness what happens during a few weeks of vacation.  Marcie and I toured New Brunswick and Nova Scotia and just got back.  To a mess (stocks are down 20% for the week this week, and headed lower).  Click HERE to read Paul Volker’s thoughts about what we need to do.

More on ethanol…

Bah! I hate to read stories like this about my native Minnesota. Click HERE to read about Minnesota farm-lobby groups canceling the grant of Minnesota researchers who contributed to a recent study showing how ethanol may contribute more greenhouse gas than gasoline. The headline is “Reality Hurts Farmers Feelings.”

I loved that story when it came out, since it aligns with my views. And, I’m not surprised to see an industry group kick back. But you do wonder — will an industry group ignore such a profound warning just to make their constituents a buck or two? ‘Seems like there’s an ethical issue in there somewhere.


Just to drive the point home, click HERE to read a story about the other big problem with ethanol, water.

Mike’s Idiot’s Guide to the Truax Seed Drill

We planted the last prairie field this weekend — here’s Marcie’s post about the whole thing.

We rassled with the manual for the Truax Flex style drill — the model number of the one we were messing with was FLX-88, but I think these comments apply to any FLX model grass drill made by Truax. The problem I had was that the manual for that drill is written for people who aren’t idiots. The audience for the manual probably care about the stuff that’s in it, but all I wanted to do was hook up the drill, put seed in and plant. The manual doesn’t help with that at all, so this is my replacement — mostly for the next time I use the drill, but maybe it will help you too.

Puzzle Number One – What are the “transport locks” they’re talking about?

The manual has a dreadful picture (and only one) of the transport locks. So here are a few more, that make it obvious what’s going on.

This picture mirrors the one in the manual;


This picture shows the lock turned so that you can see it. Ok, so those have to come off before you start using the drill. Hook up hydraulics, raise the drill a little, take the blocks out. I get that.


Puzzle Number Two – Storing the trailer jack

Here’s just a stupid thing — I found it was a lot easier to rotate the jack and leave it on the drill than to pull it off. All the holes are set up for that. Here are two pictures to show you what I’m talking about.

Jack down;


Jack stored;


Puzzle Number Three – Making the drill actually work

I had a heck of a time figuring this out. Fortunately Dan Olson was home and clued me in. But the manual provided no hint. Here’s the deal.

There’s a little pin on the drive wheel that makes this happen. Here’s a photo of the pin the way we got the trailer. See? It’s pulled further out;


And here’s a picture of it in the position where the chains and gears and stuff will actually do something. It’s further in;


Here are pictures of the back side of the wheel that shows the opposite end of that pin. This first photo is in the non-engaged position (the one you would use to tow the drill to a new place). When the wheel goes around, the pin missed that ratchet thingy and as a result none of the gears turn;


Here’s a picture of the engaged position. Now the pin sticks out far enough to catch that ratchet thingy and the gears and bobbins and whatchacallits all go round and round;


Warning Number One — Check the seeding rate that the drill is set up for

Sheesh. We didn’t check this before we started. The lads at the DNR told us that the drill was set up for the lowest seeding rate, and we believed them. Not. It was set for the MAXIMUM rate, so we dumped about a jillion dollars worth of seed into about a quarter mile of furrows before we realized what was going on. Moral to the story — ALWAYS check.

Here’s where to look — behind this here silvery dingus. It swings forward;


Here’s the little decoder-ring that tells you which way the gears work. Ours came with the chain on the far LEFT set of sprockets. For planting prairies you’ll probably want them on the far RIGHT set of sprockets;


We tried the drill on the far right side for a while and concluded that it was too light, so we moved up one notch. Here’s a picture of the way we had it set up;


You change the chain by lifting that little idler-wheel up off the chain, fiddling with the chain until it’s on the right pair of sprockets and then moving the idler over so that it centers on the chain in it’s new position. This is a picture showing how the idler swings up away from the chain;


Warning Number Two — Beware of the “small seed” box, it overfeeds

We wound up using the drill ONLY on fluffy seeds. The small-seed box fed the seeds WAY too fast, even on the slowest gearbox setting. If I wanted to make a lifetime hobby out of this drill, I would have fiddled with that adjustment too. But I don’t. If anybody has messed with that set-up and wants to add a comment to this, feel free. But we just skipped it.

Editorial Comment — None of this information is in the manual

I know that for farm-equipment hotrods, this looks like the ravings of an idiot. But me I’m just a weekend warrior and figure maybe you are too. In which case, this little write-up might save you some time and aggravation.

The big reward — here’s a picture of the field as I’m dragging the drill behind TracDor.


Winterize your RV — the easy way, no fittings required

This is a “winterize the RV” reminder post for me, and a “how to” post for you.

There are lots of articles out on the ‘net that talk about winterizing an RV. Most of them are repeats of the same article and, while the approach will work, it’s a lot harder than it needs to be. I learned this approach from the old codger that was minding the RV portion of WalMart when I went in to buy anti-freeze and a few parts. I was after a fitting to hook my air compressor up to the city-water inlet — he talked me out of that approach and told me this one instead. So I thought I’d share it.

Conventional approach;

– Blow out the water in all your plumbing with compressed air (requires a special fitting, and great care not to over pressurize your lines with the air compressor — no more than 25 psi por favor)
– Use the water pump to suck antifreeze out of the jug, either by disconnecting the line to the fresh water tank or by installing a T-valve that allows you to do this without disconnecting. Either way is a hassle, involving plumbing changes.

Old Codger Approach;

– Drain all the tanks.
– Put anti-freeze in the fresh water tank (using the sophisticated device in the picture below)
– Use the water pump to distribute the anti-freeze, without messing about with your plumbing

It worked great — took about 5 gallons of anti-freeze, but I was being liberal in my use since WalMart had it on sale for 2 bucks a jug. It took about 3 gallons to prime the water pump.

Here are pictures of the various bits, and after that is a detailed checklist (mostly so I remember what I did)

Picture 1 — Sophisticated device to insert anti-freeze into fresh water system.

The funnel is all chopped up because I also use it to change the oil in the PowerTrac and it needs to fit into a really tight space. I may splurge and buy a new funnel.


Picture 2 — Hot Water Bypass

These valves on my RV are configured in the “normal” position in this photo. So water is supplied to the hot water tank from the fresh water supply, fills up the tank and the exits as hot water to the faucets. Turning all three valves 90 degrees bypasses the hot water tank (meaning that you don’t have to fill the water tank with anti-freeze before anti-freeze gets to the faucets). I let some anti-freeze go into the (previously drained) water tank before I bypassed it.


Picture 3 — The drain plug on the hot water tank

Just a reminder — drain the water tank, put the plug back in, put some anti-freeze in there.


Photo 4 — Water pump

See the connection between the water pump and the fresh water tank right behind it? That’s the one that I’d have to mess around with if I used the Traditional approach to all this. Some people would have you cut that short line, insert a T-valve that chooses between the fresh water tank and a hose that runs down to the anti-freeze jug. What a hassle. What a huge opportunity to introduce leaky joints in your plumbing system.
WaterPump1 01


Ok, here’s the step by step (so I remember next time).


– Drain everything (fresh water, hot water, gray water, black water) and then close them back up
– Pour 4 gallons of anti-freeze into fresh water tank
– Bypass hot water heater
– Turn on water pump
– Open faucets and toilet until they run pink (anti-freeze is pink)
– Drain gray and black water tanks again (probably don’t need to do this, but there’s a mix of fresh water and anti-freeze in there right now)
– Run some more anti-freeze into black water tank through toilet
– Run some more anti-freeze into gray water tank by pouring some into each trap
– Turn off the water pump for a second
– Un-bypass the hot water heater
– Fire up the water pump and let it put some anti-freeze in the hot water tank
– Turn off the water pump

Undoing the winterizing

should go something like this;

– Flush the anti-freeze out of the fresh water tank and, when the time seems right, shut off the drain valve and let fresh water accumulate in the tank — I dunno maybe 10 – 15 gallons
– Bypass the hot water heater
– Fire up the water pump
– Run all the taps and toilet until they don’t run pink any more
– Shut off the water pump
– Open the drain on the hot water heater, un-bypass it, fire up the water pump again, flush the anti-freeze out, shut off the water pump, re-install the drain plug, fire up the water pump and fill the water heater with fresh water (hot water taps should be open — they’ll start supplying water when the water heater is full).

Should be good to go at this point — I don’t see any reason to dump the gray and black water tanks, although I spoze you could as the last step. They’ll keep until the first time we’re at a dump station, at that point we can dump out the anti-freeze and fresh water that’s sloshing around in there.

Update — some months later (January, 2008)

All right!  It worked!  I just got back from a trip to Florida and can report that the plumbing worked fine after going through the “undo” checklist.

I can also report that this method sorta saved our bacon on the way back.  As we left 85-degree Florida, a huge cold snap rumbled through the Midwest and we realized it was going to be below freezing by the time we finished driving on the first day back (in Georgia!).  So in the mid-afternoon we pulled into an RV park, paid their $20 dumping fee, and did the “winterize” checklist before heading on up into frozen territory.   The nice thing about this approach is that you can do it without any tools, unlike the traditional approach.

So we’re all winterized again (which is good, ’cause the night we got back it hit 15 below) and looking forward to our next trip south.