For any messages (such as alerts, notifications and site invitations) to be sent out of SharePoint server, you need to configure SMTP server to send the messages. This server can be any SMTP-compliant server that you can connect to using port 25 from the SharePoint server. For SharePoint 2010/2013, it cannot be configured to send e-mail over encrypted port (such as Office 365 or Gmail). If you want to set outgoing mail settings for non-default port (25), you need to configure SMTP server on your server, and configure SMTP server settings in IIS 6.0 Manager.
Take Gmail as example:
1. Open Server Manager – Features tab, click Add Features. From the Features list, select SMTP Server and click Next button.
2. Follow the wizard, wait for installation complete.
3. And then Open Windows services, make sure that Simple Mail Transfer Protocol (SMTP) service has been started.
4. Open IIS 6.0 Manager, right-click SMTP Server and click Properties.
5. Under General tab, select your local IP address.
6. Under Access tab, click Connection and select All except the list below option. And select All except the list below option for Relay Restrictions.
7. Under Delivery tab, click Outbound Security. And click Basic authentication, enter user name and password. (enter your Gmail account) and enable TLS encryption.
Note that if your Gmail turn on 2-step verification, you need to enter the application specific password here. (https://support.google.com/accounts/answer/185833?hl=en) Follow these steps to generate the application specific password.
a. Click your name or photo near your Gmail inbox’s top right corner. Click Account link.
b. Open the Security tab, and go to the App-specific passwords tab. If prompted for your Gmail password, enter your password over Password and click Sign in.
c. Click Select device and choose the Windows Computer.
d. Click Select app and choose Mail.
e. Select Generate.
f. Copy the App password (the 16 character code in the yellow bar), and click Done.
8. Open Outbound Connections, change the TCP port to 587.
9. Open Advanced Delivery, type smtp.gmail.com in Smart host. And then save the settings.
10. Open SharePoint Central Administration, click Configure outgoing e-mail settings in System Settings.
11. In Outbound SMTP server, type your local IP address. In From address, type your Gmail address.
12. Click OK to save the settings.
Fortunately, SharePoint Server 2016 adds support for SMTP servers that use TCP ports other than the default port (25).
Also, SharePoint 2016 supports send email to SMTP servers that use STARTTLS connection encryption. The SMTP server must support the TLS 1.0, TLS 1.1, or TLS 1.2 protocol. (SSL 2.0 and SSL 3.0 protocols are not supported.) Follow the steps to configure the outgoing email settings in SharePoint 2016.
1. Open the SharePoint Central Administration and browse to System Settings > Configure outgoing e-mail settings.
2. In Outgoing E-Mail Settings page, configure the settings as followings:
Outbound SMTP server: SMTP server name
From address: your email address
Reply-to address: your email address
Use TLS connection encryption: Yes
SMTP server port: 587
3. Click OK button to save settings.
But unluckily, I cannot configure the outgoing email settings for Gmail SMTP server successfully. I will do more research, and introduce the how to figure it out later.