A quick update for anyone who's peeking in here. Changed tactics a bit for my cabinet. Originally I was going with this person's version of the modular control panel. My biggest dilema with this route was the equipment I would have to buy to make this cabinet - specifically, a router and a drill press. Plus I ran into the problem of getting the correct angle for drilling and setting the dowels to have the individual panels to lock correctly.
Then I ran across this site, which is also doing a modular panel but uses rack mount server parts which seems a bit easier for me. Plus the controls can be mounted to metal panels, as opposed to routing and fitting the controls in thicker wood panels (Especially good with trackballs and spinners.) I'm currently researching how to mount these on the rack rails. I'd rather use something easy to 'hot swap' panels, without requiring a screwdriver each time I want to play a different game. Screws are out, thumb screws are a possibility, but I'd rather have something else. I seem to remember there's some type of locking device where if you press it in, it locks, if you depress it it unlocks. If these things exist and I can find them inexpensively, I want to go that route.