Installing YAF in a SharePoint sub directory is a lot simpler than you'd expect. I fumbled for hours trying to figure out the appropriate settings for getting YAF to work with SharePoint, and the following is what I discovered.

1- In your SharePoint site, create a virtual directory to host your YAF forums.
2- Make the virtual directory an application with the ability to run and execute scripts.
http://support.microsoft.com/kb/828810  to change the security settings of your SharePoint site's web.config.
4- Follow the YAF normal setup instructions. The instructions here:http://wiki.yetanotherforum.net/embeddedYaf.ashx do not work with SharePoint.

If using AD
5- Once the forums site is installed and configured, change the authentication mechanism from "Forms" to "Windows". This is done by editing YAF web.config.

IMPORTANT: The YAF virtual directory must have its own web.config. The yafnet.config must be in this directory also.

Use these instructions at your own risk. Test the setup in testing environment before going to production. I'd also recommend reading on the different security levels in the web.config.

Cheers!