How to Withdraw from 401k or IRA for the Down Payment on a. – Using Your 401k for a Down Payment. There’s no specific penalty exemption for home purchases when you pull money out of a 401k, so any money you take out will be classified as a “hardship exemption.”You’ll be assessed a penalty of 10% on the amount withdrawn and you’ll have to.Lowest Down Payment for Conventional Loans. Getting back to the question at hand: What is the lowest possible down payment for a conventional mortgage loan. Both Fannie Mae and Freddie Mac offer programs that allow for 97% financing. This means a borrower could make a down payment as low as 3% of the purchase price.