Mail settings are dependent on the hosting provider. You will have to find out from them what server name or address to send to, and whether or not a login is required by the smtp client to send mail. YAF uses native ASP.NET sendmail function, which can send mail either via network (direct to remote mail server) or via "Pickup Directory" if the webserver has Microsoft SMTP server installed.
A PM inside YAF is not the same as an Email Notification. When you send a PM, YAF will send an Email Notification to the receiver of the PM to their email automatically. Same condition as if a topic is 'subscribed'. When a new post occurs, YAF will send Email Notification to any members subscribed to topic. Email notifications are also used for registrations and password resets.
YAF depend highly on proper email configuration. Once you have proper settings in mail.config, go to ~/forum-folder/install/default.aspx and after entering in Config Password, next screen offers "Test Email Configuration". When that test is successful, exit installer and email will be properly configured in YAF.
If you can't find it using the forum search, try my signature link -- searches this site using Google: Google is my Friend