Using git on Windows git comes bundled with its own ssh. This is properly fine for most people, however I personally preferer using the Putty tools. Being an active user of Bazaar (uses plink by default) and I already have my ssh-keys setup using Putty Pageant.

To change git to use plink (the command line version of putty) just the the following environment variable

GIT_SSH=C:\Program Files\PuTTY\plink.exe

#Using powershell 
[Environment]::SetEnvironmentVariable("GIT_SSH", "C:\Program Files\PuTTY\plink.exe", "User")

Problem adding new host keys

For some unknown reason when using plink with git, It does not allow any keybord input, so you cant enter y to store the key.

The server's rsa2 key fingerprint is: <XXXXXXX>
[...]
Store key in cache? (y/n)  

To workaround you can simply connect to the server using putty and add the hostkey by pressing yes. I’m guessing this is a bug and will be fixed in a future version of putty or git.