Diagnosing E-mail Issues include SMTP auth

Step #1 - Check to see if your server is responding on port 25

You should repeat this test at least twice - once from the Modus server and once from another PC on the same network or, ideally, from a machine outside of your network (via a dialup or high-speed connection)


  • From a Command Prompt, type telnet <mail server IP> 25 <enter>
  • Your mail server’s banner should appear:

220 <domain> ModusMail ESMTP Receiver version x.xxx.xx ready

  • If the banner fails to appear within 30 seconds, you could be using RBL/DNSBL lookups or reverse DNS lookup and your DNS server or one of the blacklist servers is not responding quickly enough
  • Solution:


  • In the Console, go to Security – Properties – Sender Validation & Accreditation
  • If Perform a lookup for the SMTP host in the DNS is enabled, disable it
    • Stop/start the SMTPRS service to clear the cache
  • From a Command Prompt, type telnet <mail server IP> 25 <enter>
  • If the banner appears quickly, this problem has been resolved by disabling Reverse DNS lookups
  • If the problem persists, in the Console, go to Security – Properties – Real-Time Blacklist and click on RBL Servers
  • Stop/start the SMTPRS service to clear the cache
  • Remove one of the RBL servers you are using
  • Stop/start the SMTPRS service each time and replace the RBL servers that were removed during the testing (those not causing the problem)
  • From a Command Prompt, type telnet <mail server IP> 25 <enter>
  • Keep removing RBL servers until you determine which RBL server is causing the problem
  • If the banner appears immediately, it is possible that your IP address is whitelisted or cached
  • Repeat the telnet step from a PC outside of your network (if possible)
  • After you have tested that the SMTP banner responds quickly but people still do not receive email, proceed to the following step 
Step #2 - Use Telnet to Send an Email to a User

  • Check the recipient’s mailbox
  • If the user received the email, try sending an email for this user from an outside mail service (e.g. Hotmail)
  • If the person does not receive the email, something is preventing mail traffic from getting to your server from outside of your network OR the domain you are trying to send mail to from the outside world has a DNS configuration problem with regards to the MX record settings.
  • To rule out an invalid DNS setting, perform an nslookup of the domain that you are trying to deliver mail to from the outside world.
  • Example:

  • In the above example, we try to resolve vircom.com
  • set q=mx tells nslookup tells Windows to look for MX records
  • The result in the above example shows that mail goes to gate.vircom.com (pref level 0) and, if gate.vircom.com goes down, the mail goes to smtp.vircom.com (pref level 1)
  • It is always possible that your DNS server knows the destination domain but the outside world does not
    • You can do a lookup using a foreign DNS server by using nslookup - <remote DNS server> to see if the outside world sees your domains
  • Example:

nslookup –

  • This allows you do a lookup using Vircom’s DNS server instead of your own to find out if your domain resolves on outside DNS servers
  • Assuming that you reach this point and it is certain that DNS is properly configured for your domain, then there is a network issue between the outside world and your ModusMail server.  Chances are you have some sort of mail firewall that is causing the problems (e.g. Cisco Pix firewall)
  • If the user did not receive the email, proceed to the following step

Testing SMTP AUTH connections

When setting up a mail server, one of the things you should do before you "go live" is to test it- not only to make sure things which should work, do work, but to make sure things which shouldn't work, don't.

One of the things to test is whether or not your server correctly supports the AUTH command. This command is used when a remote client wishes to identify themself as an "authenticated" user, normally so that they can use your server as an outbound mail relay. This is very handy for companies with employees who travel, or for ISPs with clients who travel.

Find your authentication information

In order to use the AUTH command, you need to know the base64-encoded version of the userid and password you will be using to authenticate to the server. Normally this would be the same as the userid and password you would use to check your mail using IMAP or POP3. This perl command (which requires the MIME::Base64 module) will do the encoding for you:

% perl -MMIME::Base64 -e 'print encode_base64("\000jms1\@jms1.net\000not.my.real.password")'

Note: Make sure to use \0 both as the first character of what you're encoding, and as the separator between the userid and the password. There was an error with the original version of these directions- I had forgotten about needing a \0 at the beginning. Sorry all!

Another reader pointed out that perl silently interprets the "@" sign in the middle of a string and replaces it with the contents of an array with that name, if one exists... or with nothing, if not. I just did a full two-way test with my real password, and it turns out if you don't put a backslash in front of the "@" sign it won't work. Good call.

And JT Justman pointed out that if you use \0 as the separator, and the userid or password happens to start with a digit, perl will try to find and use a three-digit octal character code instead of a one-digit null byte with two normal digits behind it. Using \000 instead of just \0 prevents this from happening.

Connecting to the server

Depending on how the server is configured, you may need to use SSL or TLS before you are able to use the AUTH command. In fact, if you are able to use the AUTH command without using either SSL or TLS, you are in fact sending your userid and password over the internet in clear text. Anybody with a packet sniffer in the right spot will be able to read the base64-encoded string you send to authenticate, and it's really easy to decode that stuff- in fact the same command above will work if you change "encode_base64" to "decode_base64" (and put the encoded string between the double quotes, obviously.)

  • To connect to a normal, non-secured SMTP server on IP address, you would use this command:

    % telnet 25

  • To connect to a server which should support TLS, you may wish to verify that it does support TLS first. When you send the EHLO command, the server will respond with a list of the items it supports. If you see STARTTLS on the list, it means the server will allow you to send the STARTTLS command. Example:

    % telnet 25
    220 a.mx.jms1.net NO UCE ESMTP
    ehlo testing
    250-a.mx.jms1.net NO UCE
    250 8BITMIME

    Once you have verified that the server supports the STARTTLS command, you can use the "-starttls smtp" option of openssl s_client to connect. This makes openssl connect normally (without encryption), send a STARTTLS command, negotiate the SSL encryption, and then allow you to interact with the encrypted session. For example, to connect to a TLS-enabled SMTP servers on IP address, you would use this command:

    % openssl s_client -starttls smtp -crlf -connect

  • And for an SSL server (where you connect to a different port number and have to establish an SSL connection before the SMTP conversation even starts) on IP address port 465, you would use this command:

    % openssl s_client -crlf -connect

Make sure the server supports AUTH

When you first connect to an SSL or TLS server, you will see the key-exchange information fly by on the screen, and the last line you see when it stops scrolling text will be the server's "banner" message, which tells the client that the server is ready to accept commands. For a non-secured connection, the first thing you see will be the banner.

When the banner is received, a normal SMTP client would send an EHLO command to the server in order to identify the client machine, as well as ask for a list of the capabilities supported by the server.

If you are using an openssl command to connect to an SSL or TLS server, make sure to enter your SMTP commands in lowercase as shown here. The openssl "s_client" command watches what you type- if you send a line of text starting with a capital "R", it will re-key the SSL layer instead of sending your command to the server... and if you send a line of text which starts with a capital "Q", it will terminate the SSL connection and exit.

220 a.mx.jms1.net NO UCE ESMTP
ehlo testing
250-a.mx.jms1.net NO UCE


Look at the response from your EHLO command, make sure AUTH is on the list, and that PLAIN is one of the options it supports. If it's not listed, the server will not let you send an AUTH command. This may be because the connection is not secured and the server is protecting you from sending your authentication information across the net in plain text...

Sending the AUTH command

Assuming the server supports AUTH, we will send the actual AUTH command to try and authenticate.

AUTH PLAIN AGptczFAam1zMS5uZXQAbm90Lm15LnJlYWwucGFzc3dvcmQ=
235 ok, go ahead (#2.0.0)

If you see this message, you are authenticated. If you see this one instead...

AUTH PLAIN AGptczFAam1zMS5uZXQAbm90Lm15LnJlYWwucGFzc3dvcmQ=
535 authorization failed (#5.7.0)

... then obviously it means you are not authenticated. If you were not able to authenticate, you can try another AUTH PLAIN command- although if the server is logging the traffic or running an intrusion detection system, having multiple AUTH commands in a single SMTP session is enough to raise a red flag. Be careful not to ban your test client's IP address.

Sending the message

Once you are authenticated, you may continue with a normal SMTP conversation and the server should accept any message from you, whether you are relaying to an outside domain or not. Even if you don't authenticate, the server will still accept messages from you- it just won't relay (it will act the same as if you had never entered an AUTH command at all.)

mail from: <nospam@jms1.net>
250 ok
rcpt to: <nospam@jms1.net>
250 ok
354 go ahead
From: John <nospam@jms1.net>
To: Nobody <nospam@jms1.net>
Subject: fnord

hail eris!

250 ok 1113954693 qp 29052
221 a.mx.jms1.net NO UCE