Protect Your Business Against High Risk Security Threats

A Deep Dive into Email Server Security and Must-have Policies

Telecom Systems Provider, LLC, Hassan Salama


    No doubts, Email is important tool for every business. It is used heavily for internal communication between employees and also externally towards other companies and consumers depending on the business type. In its basic form, It is a very powerful tool that asynchronously or indirectly carries your message to the recipient and allow him to respond later at his convenient. 

    Due to weaknesses in email transport layer, which have been revealed through the years since its first use, attackers have been using emails as the number one entry point for many different types of 
    attack such as phishing, malware, business email compromise - BEC, and ransomware. 

    According to Blackhat survey 2017, phishing was number one security threat that concern the most. Second was the directed attack such as spear phishing. Ransomware was rank five. It is interesting that the entry point for the three of them is email system. So, what is the business impact? Well, FBI reported last year (2017) $5.3B damage only from business email compromise - BEC.

    It is clearly crucial for business entities to secure their email system and accordingly their users and business against this kind of damage.

    OK, what to do then? 

    You can simply contact us for helping in auditing/securing your system or else to continue reading.

    There are three key areas that need to be covered to avoid challenges that come along with email system utilization. 

    • End-user awareness.

    • Using two-layer email security mitigation techniques for email service hosted internal or on cloud.

    • Implying, enforcing, and updating security policies for email system and its use.

    End-user Awareness is one of the pillars required for mitigating security challenges that comes along with email usage. On brief, businesses must consider enrolling their end-user into several awareness programs about email use and challenges arising from end-user mis-use. 

    Hold you gears on as in the following lines, we are going to deep dive into the other two areas.

    Two-layer Email Security Mitigation Technique.

    In this architecture, you design your email systems in such a way of having two servers one internal for local users and second one (email gateway) will be in charge about communicating with the outside world. This design allows you to mitigate and isolate incoming threat prior even entering your local network and assign this task to your external email gateway. On the other hand, you should enforce other policies on the internal email server to secure lateral and outgoing communication. This technique can be achieved in either cases of having an on-premises email server or if it is hosted on the cloud. 

    One of the commercial solution available is Trend Micro Hosted Email Security, which can be positioned as an external gateway. Schematic diagram hereafter shows the way it operates. HES offers flexible email policy configuration and also an exterior tier that mitigate lots of threat prior entering your domain. 

      Security Policies for Email Systems

      Here we are going into the anatomy of different policies required to secure your email systems and accordingly your end-user and business. It is classified into two main groups, which are: -

      • Domain Based Policies.

      • Miscellaneous Policies.

      Domain Based Policies 

      Those policies are published by the sending domain end interestingly via another service namely domain name service or DNS for short. It is a best practice that those policies are getting verified and validated from the receiving end; however, many email-servers have poorly configured to overlook or ignore those best-practices. Generally speaking, email-servers are playing the two rules of sending and receiving emails, accordingly, System Admin personnel should consider publishing their Sender Domain Based Policies (send role) and also to verify and validate other domains policies (receive role). Hereafter we are going to discuss those policies and its benefits in enhancing server performance and mitigating many security threats such as Spamming, Spoofing, and Malware.

      • Reverse/MX Record Policy
        It is a common policy applied on most mail-server available today to ensure that the sending email server name matching the configured mx record for that particular sending domain and also its MX record has a reverse lookup entry. That means MX record resolved IP address is resolvable back and matching the MX record qualified domain name. 

      • Open Relay Policy
        Most Email-servers software relay emails for non-local domains by default, in such situation mail-server is known to be an open-relay. No doubts that could harm server performance and it is identified as a security threat. Email-servers must imply a non-open relay policy to prevent its misuse.

      • IP Reputation Policy
        This policy rely on different external sources, which forms different lists for known bad reputation sources. One example, which is very well know is the various spammers black-list for known to do spam trafficking. So this policy instruct your mail server to check the contacting server to make sure it is not listed and make an action (reject) if it is listed in one of those lists.

      • Anti-Spoofing Policies
        Spammer usually tries to impersonate a valid email account or domain when sending their spams and they as they have many old and newer tricks to do so. We can't eliminate spamming issue completely; but no worries because there are several policies that can be used to mitigate spamming to reasonable levels. Hereafter, we will discuss several important policies which as promised should limit the spoofing. 

        • Self Anti-Spoofing
          Email server has to have the ability to distinguish between local message originated from a valid user account and a spoofed message that originating from an outside server impersonating a local user account. As the later should be rejected or deleted accordingly.

        • Sender Policy Framework - SPF (RFC 4408) - 2006
          It is an IETF standard that allow sender domain to publish a policy via a TXT dns record, which announces its eligible email servers. Accordingly, SPF-capable email servers check incoming emails from that particular domain against its SPF record to make sure it is being received from its eligible email servers(s) listed in the published policy. Hence, email delivery action will be based on that check result.

        • Domain Keys Identified Mail - DKIM (RFC 5585) - 2009
          DKIM is similar to SPF as it publishes a TXT dns record and it relies on DKIM-capable email servers to check the integrity of the message being sent. DKIM went for an extra mile as it signs the message so that the receiving end will validate the message using the published TXT DNS record, which is basically the validation public key. Hence, it prevents message modification during transit and decide email delivery action based on validation.

        • Domain-based Message Authentication, Reporting and Conformance -  DMARC (RFC 7849) - 2015
          As you can see from the RFC number and release date, DMARC is the latest standard and it is a kind of framework for both or either of SPF and DKIM. It allows the sending domain to set a broader policy for the usage of SPF and/or DKIM policies. It tells the receiving end what to do if the policy gets violated (none, quarantine, or delete). It also shares an email address with the receiving end so it might send a periodical summary report -based on the server configuration- indicating the type of treatment for the sending domain messages.

      Domain Policies

      Reverse/MX record

      • Sending Domain
        Ensure your email server is eligible.

      • Receiving Domain
        - Check sending server is eligible.

      • Handling
        - Success: Relay the message.
        - Failure : Reject the message.


      • Sending Domain
        Limits your domain from spoofing.

      • Receiving Domain
        - Ensure that the received message is not a forged email

      • Handling
        Success : Deliver the message.

        - Hard Fail : Don't Deliver.
        - Soft Fail : Local server policy.


      • Sending Domain
        Limits your domain from spoofing.

      • Receiving Domain
        - Ensure that the received message is not a forged email and validate its origin domain
      • Handling
        Decipher the header and ensure it is originated from sender domain.
        Success: Deliver the message.
        Fail: delete the message


      • Sending Domain
        Action policy provided by the sending domain.
      • Receiving Domain
        - Action policy is executed when configured on the receiving domain

      • Handling
        Hard and Soft SPF Fail = SPF Fail.
        - policy=none: deliver the message.
          policy=delete/quarantine : don't deliver.
        - delete or quarantine policy is based on the check if either of SPF or DKIM failed.

          Considering above policies eliminates tremendous amount of scavenger emails. It protect your network against attackers trying to compromise your network or users via ransomware, sextortion, phishing, or malware. Having the fact that one of the ways an attacker could penetrate your network is by spoofing your email server either by impersonating your domain or other trusted domains. So, end-user awareness is pivotal in your mitigation endeavor.

          Miscellaneous Policies

          • High-risk attachment Policy
            High risk attachment policy can be customized to delete a whole message or the attachment only that contains certain file types. It can also count for the number of recipients in the 'To' field header and the message size as well. Also, Admin must consider either deleting emails with password protect attachment, warning the user about them, or let the local anti-virus to scan such files. Certain anti-virus could use sandboxing mechanism to test the file prior to allow its use on local computer.

          • Malware and 0-Day Threats Protection - Anti-Virus Policy
            those policies get into play at the anti-virus software layer integrated into your email server - inbuilt or via 3rd party - where it scans email content for viruses or malware, this quite important to be done in both tiers internal email server and external one or email gateway.

          • Spam Policy/ Phishing Policy / Newsletter or spam-like Policy / Social Engineering Attack
            Most of email-server has a build in filter or it relies on different spam filter providers that checks the sender address and message content to classify the message either as a normal message or as a spam message. For this policy, you could add a message header, to instruct your mail client to move messages classified as a spam into the junk or spam folder.

          • Business Email Compromise - BEC
            BEC is when attacker impersonate high profile individual within an organization sending email for accounting department asking for money transfer. Such policy should count for two-tier mitigation by considering an external email gateway and make sure that external email received should be coming from high profile users. in other words, internal emails shouldn't be sent to external gateway and should remain local to internal email servers.

          • Web Reputation Policy
            This policy considers scanning the message content to see if any embedded url with bad reputation and mask it. Accordingly, end-user won't reach risky url or at least get warned prior to accessing that url.

          • Recipient Filter Policy
            This policy informs the external email server - email gateway - about existing local users, in such a way to prohibit emails not intended for local users. that eliminate the processing overhead on internal server and possible DDOS attack for internal email server.

          Finally, here are some guidelines when you apply those policies.

          • Apply each policy individually, so you could track if it is getting trigged or not.

          • Try to set a notification mechanism such as sending an email to a certain mailbox i.e mail-policy-violation, it will allow admin personnel to detect false positive, to review policies, and to tweak them accordingly; if needed.