I have to agree. #4's sure have the energy at that distance to get the job done, but the pattern will be sparse at best.
You may want to switch to a "hevishot" load if you shoot at those distances. You can then go to a smaller size shot and still have the energy levels, but with a much better pattern as well. (Shot sizes such as with #5's or #6's). Still have to pattern your gun, choke, and shell to be sure.