It's an error in the table. The second set of values for unheated and heated slabs should be for vertical insulation, not horizontal. See the errata sheet for ASHRAE 90.1-2007 from August 15, 2011. Your 12 inches of R-5 horizontal insulation has an F-Factor of 0.72.