It uses what mail app is configured on the OS on the content server. Have you ever of heard Google? Searching "Documentum smtp server" returns this:
https://www.bluefishgroup.com/insights/articles/configuring-and-customizing-documentum-email-facilities/
Not sure what you are asking. You can SMTP/POP3 with MS Exchange.
The Bluefish link shows how to configure Documentum to use SMTP to automatically send emails to users when something is placed in their Documentum Inbox. If you want to send custom email, you will need to write your own method using java.mail.
1) This is done be default. You don't have to configure mail_method attributes as long as the SMTP service is running on Content Server 2) Java mail package is you want to send custom email independent of Documentum inbox event
What custom code? Its java class. If you want to know how to write java code, try Google.
If you are using BPS integration, then you don't need custom java mail method. The Process Builder provides activities that allows you to configure mail messages without having to write code.