I believe the federal government, for FHA, will allow anythinig down to a 580, but a lot of banks set a higher score, like 620. FHA is more lenient than most mortgages in that respect.
I was told when I applied last year that there could be no late payments in the last 12 months, no open collections, all judgments satisfied. They like to make sure nobody is going to come after you for debts right when you owe them money 
While there are some lenders that may underwrite at 580, most will not.
620-640 is the often the floor for FHA and community loan mortgages (specific loan product for low-moderate income buyers or urban/poor areas). Also every state has what's called a Housing Finance Agency (HFA) that generally offers more flexible underwriting for first time buyers or those who have not owned in 3 years. You can look up your state's HFA here: https://www.ncsha.org/housing-help
I always encourage people to attend a HUD certified Homebuyer Education class which covers the entire process of buying a home whether you are just thinking about it or well on your way. Some are free, some charge a nominal fee. Most of the housing counseling agencies are non-profit organizations (I work for one) and receive extensive training and certification. Many of them partner with lenders that have more favorable terms, incentives, and/or less stringent underwriting. Here's the locator tool to find one where you are: http://www.hud.gov/offices/hsg/sfh/hcc/hcs.cfm
There is a banking network called the Federal Home Loan Bank with regional offices, all of which offer some type of 1st time buyer incentives to low-moderate income households, the income limits are often pretty generous. The incentives range from being a straight closing cost grant to a matched savings program. In my area it's a 4:1 matched savings program which allows a family to save $1800 and they match the savings with $7200. Each regional office covers multiple states so start with the one nearest you.
Atlanta-http://corp.fhlbatl.com/services/affordable-housing-programs/homebuyers-and-homeowners/
Boston-http://www.fhlbboston.com/communitydevelopment/ebp/index.jsp
Cincinnati-https://web.fhlbcin.com/Housing/Programs/WelcomeHomeProgram/Pages/WelcomeHomeProgram.aspx
Dallas-https://www.fhlb.com/community/ahp_help.html
Des Moines-http://www.fhlbdm.com/strong-communities-fund/housing-programs/down-payment-assistance-programs/
Indianapolis-http://www.fhlbi.com/housing/HopProg.asp
New York-http://www.fhlbny.com/community/housing-programs/fhc/index.aspx
Pittsburgh-http://www.fhlb-pgh.com/housing-and-community/programs/first-front-door.html
San Francisco-http://www.fhlbsf.com/community/homebuyers.aspx
Seattle-http://www.fhlbsea.com/CommunityInvestment/OurPrograms/HomeStart/Default.aspx
Topeka-http://www.fhlbtopeka.com/s/index.cfm?aid=31
Current Score: TU: 674 | EQ: 722 | EX: 717 | Last Inq. 2/16/15...Locked Up in The Garden