We present predictions of the thermoelectric figure of merit (ZT) of Si nanowires with diameter up to 3 nm, based upon the Boltzman transport equation and ab-initio electronic structure calculations. We find that ZT depends significantly on the wire growth direction and surface reconstruction and we discuss how these properties can be tuned to select silicon based nanostructures with combined n-type and p-type optimal ZT. Our calculations show that only by reducing the ionic thermal conductivity by about 2 or 3 orders of magnitudes with respect to bulk values, one may attain ZT larger than 1, for 1 or 3 nm wires, respectively. We also find that ZT of p-doped wires is considerably smaller than that of their n-doped counterparts with the same size and geometry.