Similar setup. In IIS Console on a Windows Server 2012 system, choosing the Browse link to open a web application, causes the error you are having. I fixed the issue by changing from "Default Host Application" set for the HTTP/HTTPS protocols in Default Programs, to "Internet Explorer". See the attached image.
-Joe