In recent work by Paschek (D. Paschek JCP 2004, 6674) and others (see H. Docherty et al. JCP 2006 074510 for a review) it has been suggested that, when coupled to a simple Lennard-Jones (LJ) model for various small non-polar solute molecules, the most common models of water (e.g. SPC/E and TIP4P) fail to reproduce quantitatively the solubility of small non-polar solute molecules in water due in part to failing to account for polarization of the solute molecule.  Given the importance of such systems as test-case prototype models of the solubility of proteins and biomolecules, in this work, we investigate the impact of using a polarizable solute model with the SPC/E, TIP3P, TIP4P, TIP4P-Ew and TIP4P/2005 rigid water models.  Specifically we consider Ne, Ar, Kr, Xe and methane as solutes.  In all cases we observe the use of a polarizable solute improves agreement between experiment and simulations, with the best agreement seen for the largest solutes, Kr, CH₄ and Xe and the recent reparameterizations of the TIP4P model, i.e. the TIP4P-Ew and TIP4P/2005 models.