I know if I was to take a day off of work to get shopping done, then I would be darn sure to spend it shopping no matter what the weather is like. Days off are too hard to come by.
Did you offer to meet your wife for lunch? I rarely invite my hubby to lunch as I know his daily schedule is generally pretty busy - even when he is working from home.
In my opinion you have nothing to worry about, but if your on here asking us our opinion I'd say your already worked up. So I would take cinch314 advice and just let her know CALMLY how your feeling. I wouldn't start the conversation off as being suspected of anything. Just let her know that had she asked you'd have been glad to have met them for lunch. Then ask more about the co-worker.
Now if your asking if any of us ladies have met male coworkers for lunch before - the answer would be yes. 95% of my coworkers are male. Steve hasn't had the opportunity to meet most of my co-workers over the years, but I've always made a point of letting him know when we've had lunch or any off work activites (bowling, baseball, ect). Most of these activities were last minute plans, so he couldn't attend. I try to let him know before the fact, but if it's something simple as lunch then I don't see the need.
If she told you about meeting him for lunch and brought your son along, I'd say you have nothing to worry about.