Construction loans will go off whoever has the lower scores, doesn't matter if one person makes $0 and the other makes $1,000,000, but if you don't need both incomes (which in your situation you might not) then you don't need to put everyone on the mortgage. If you've owned the lot for 12 months or more then the current appraised value can be used as part of your down payment. Most construction loan programs require 680-720 scores, but there is a program from Fannie Mae called HomeStyle that can approve with lower scores... it's just like qualifying for a normal Fannie Mae loan program.
Free Mortgage Advice & Pre-Approvals (FHA, VA, USDA, Fannie, Freddie, Non-Prime, Construction, Renovation/Rehab, Commercial) since 2002
Located in Southern California and lending in all 50 states