Configuring a Spam and Attachment Filtering SMTP Relay on the ISA Server 2000 Firewall - Part 1: Installing and Configuring the SMTP Services and Remote Domains

by Thomas Shinder [Published on 1 Dec. 2003 / Last Updated on 1 Dec. 2003]

Spam is a major threat to the Internet and corporate networks today. Spam clogs Internet routers and gateways, usurps bandwidth on corporate Internet links, and consumes processor cycles and disk space on corporate SMTP relays and mail servers. Criminal spammers are using increasingly sophisticated techniques to circumvent spam filtering applications. The more complex the spammer’s attack methodologies used to avoid detection, the more processor intensive the spam whacking applications must be. These processor cycles are robbed from other applications and services running on the same machine and often require that you dedicate a single box to spam filtering. This two part article shows you with detailed step by step instructions on how to make your ISA Server 2000 firewall into a front-end spam filtering SMTP relay.

Configuring a Spam and Attachment Filtering SMTP Relay on the ISA Server 2000 Firewall
Part 1: Installing and Configuring the SMTP Services and Remote Domains


By Thomas W Shinder M.D.

Note:
Part 2 of this article can be found at
http://isaserver.org/tutorials/messagescreeneronfirewallpart2.html

Spam is the major threat to the Internet and corporate networks today. Spam clogs Internet routers and gateways, usurps bandwidth on corporate Internet links, and consumes processor cycles and disk space on corporate SMTP relays and mail servers. Criminal spammers are using increasingly sophisticated techniques to circumvent spam filtering applications. The more complex the spammer’s attack methodologies used to avoid detection, the more processor intensive the spam whacking applications must be. These processor cycles are robbed from other applications and services running on the same machine and often require that you dedicate a single box to spam filtering.

Get the Book!

Spam isn’t the only problem that travels over SMTP. Email attachments represent a major risk to corporate network security. Many viruses and worms move through the Internet and corporate networks using SMTP as their transport mechanism. Unwary users find salacious subject lines compelling and open attachments containing viruses, worms and other dangerous payload. In addition to viruses and worms, attachments can contain information that you do not want allowed into or out of your network. For example, Word documents can contain details of your latest Research and Development efforts. You may want to block and retain these documents for further evaluation before allowing them out of your company’s control.

Given the above observations, its clear that all companies requires comprehensive spam and virus filtering. Not only do all organizations require comprehensive spam and virus filtering, they must use the same Defense in Depth principles they use for mitigating other attacks. For example, you install host based virus software on the desktops and your application layer filtering firewalls (like ISA Server 2000 firewalls) scan and block viruses downloaded from the Internet. In the same way, you need multiple levels of spam and attachment blocking to provide an effective anti-spam Defense in Depth solution.

Figure A shows an example of how an effective anti-spam Defense in Depth solution works. Internet users send mail to their own SMTP servers and then Internet SMTP servers forward mail for your domains to your published SMTP servers through the ISA Server 2000 firewall.

The ISA Server 2000 firewall is configured as a front-end firewall and spam filtering SMTP relay and removes some spam and virus attachments at the perimeter. The front-end ISA Server 2000 firewall and spam filtering SMTP relay forwards the SMTP messages to a dedicated spam filtering relay on the internal network. Filtering on the ISA Server 2000 firewall filtering SMTP relay offloads some of the processing requirements on the back end SMTP filtering relay and improves overall performance.

Figure A

Figure B shows a scenario where a front-end ISA Server 2000 firewall filtering SMTP relay solution is even more important. Spam filtering is very processor intensive and can reduce performance of the Exchange Server when co-located on the Exchange Server box. The front-end ISA Server 2000 firewall spam filtering SMTP relay can make a big difference by offloading some of the processing requirements from the co-located spam filtering Exchange Server machine.

Figure B

Installing and Configuring the SMTP Message Screener and SMTP Filter on the ISA Server 2000 Firewall

The ISA Server 2000 firewall can be used as a spam and virus filtering gateway. The following steps are required:

  • Install the IIS 6.0 SMTP Service on the Windows Server 2003 ISA Server Firewall Computer
  • Disable SMTP Service Socket Pooling
  • Configure the IIS 6.0 SMTP Service Relay Properties
  • Create Remote Domains to Support Your Email Domains and Enable Relay for Those Domains
  • Install ISA Server 2000 onto the Windows Server 2003 Firewall Computer
  • Configuring Server Publishing Rules on the ISA Server Firewall
  • Configure the SMTP Filter and SMTP Message Screener Properties

In part 1 of this two part article we will cover the first four steps. Part 2 of this series will provide the detailed procedures required to carry out the final three steps.

Install the IIS 6.0 SMTP Service on the Windows Server 2003 ISA Server Firewall Computer

The SMTP Message Screener requires the IIS SMTP service. You need to install the SMTP service because Windows Server 2003 does not install the SMTP service by default. Perform the following steps to install the IIS 6.0 SMTP service:

  1. Click Start, point to Control Panel and click the Add or Remove Programs command.
  2. Click the Add/Remove Windows Components button on the left side of the Add or Remove Programs window.
  3. In the Windows Components dialog box, click on the Application Server entry (do not put a checkmark in its checkbox!). Click on the Details button.

  1. In the Application Server dialog box, click on the Internet Information Services entry (do not put a checkmark in its checkbox). Click on the Details button.

  1. On the Internet Information Services (IIS) page, put a checkmark in the SMTP Service checkbox. The Internet Information Services Manager checkbox will be automatically selected for you. Click OK.

  1. Click OK in the Application Server dialog box.

  1. Click Next on the Windows Components page.

  1. The Windows Components Wizard installs the IIS SMTP service.
  2. Click Finish when the Wizard completes.

Get the New Book!

Disable SMTP Service Socket Pooling

The SMTP service listens on all IP addresses on all adapters installed on the ISA Server firewall by default. You must disable socket pooling to prevent the SMTP service from listening on all IP addresses on all adapters. Socket pooling prevents Server Publishing Rules from working correctly.

Its good practice to disable socket pooling for any IIS service installed on the ISA Server firewall. Perform the following steps to disable socket pooling for the IIS 6.0 SMTP service:

  1. Click Start and then click the Command Prompt link. In the Command Prompt window, switch to the Inetpub\AdminScripts folder. Type the following command and press ENTER:

      Adsutil.vbs set /smtpsvc/1/DisableSocketPooling 1

  1. If the SMTP service is installed and you entered the command correctly, you should see what appears in the figure below.

  1. Close the command prompt window.

At this point the SMTP service continues to listen on all IP addresses on all interfaces. You must configure the service to listen on specific IP addresses to limit the server to listening on a subset of addresses. In the next section you will configure the SMTP service to listen on the internal IP address of the ISA Server 2000 firewall computer.

Get the New Book!

Configure the IIS 6.0 SMTP Service Relay Properties

The Default Virtual SMTP Server listens for incoming messages to email domains you host. Perform the following steps to configure the Default Virtual SMTP Server:

  1. Click Start, point to Administrative Tools and click on Internet Information Services (IIS) Manager. In the Internet Information Services (IIS) Manager window, expand your server name and click on the Default SMTP Virtual Server entry in the left pane. Right click on Default SMTP Virtual Server and click on the Properties command.
  1. In the Default SMTP Virtual Server Properties dialog box, click on the General tab. Click the down arrow in the IP address drop down list box. Note the list of IP addresses in the list. You should see entries for your external addresses, internal addresses, and (All Unassigned).

Select an internal IP address. The Server Publishing Rule will forward the incoming SMTP packets to this address.

Click Apply after selecting an IP address.

  1. Click the Access tab. There are a number of options available on this tab. Click on Relay button located in the Relay Restrictions frame.

  1. The default setting in the Relay Restrictions allows no relay through this virtual SMTP server except for authenticated users. This is a global setting for the SMTP virtual server. We will over ride this global setting by configuring a Remote Domain on this SMTP virtual server later.

We do not want anyone to have "open relay" access to this machine, regardless of their ability to authenticate. Remove the checkmark from the Allow all computers which successfully authenticate to relay, regardless of the list above. Removing this option prevents this virtual server from being able to relay to all mail domains except for mail domains you create Remote Domain entries for.

Click OK.

  1. Click on the Messages tab. You have the option to limit the size of messages moving through the server, the number of messages per connection, and the number of recipients per message. You can also set a location for the badmail directory, which is the directory where messages not destined for any of your remote domains are deposited. Place this directory on a volume with a generous amount of free space so that your disk does not fill up in the event of a spam flood.

  1. Click on the Delivery tab. On this tab you can configure how long the SMTP relay waits before retrying to send messages to your Exchange SMTP service. This allows "queuing" of SMTP messages on this SMTP virtual server when the Exchange Server is not available. If the SMTP relay cannot immediately deliver messages to your Exchange SMTP server, it will place them in a queue and attempt to redeliver the messages based on intervals set on this tab.

Note that the SMTP relay will continue to resend the mail indefinitely. After the third retry, subsequent delivery attempts are done at an interval based on the Subsequent retry interval (minutes) entry. Even if your Exchange Server is unavailable for a day or longer, the SMTP relay will queue mail for you. Once your Exchange Server becomes available, you can restart the SMTP service on the SMTP relay computer and the mail will be delivered to your Exchange Server’s SMTP service immediately.

  1. Click on the Outbound Security button. In the Outbound Security dialog box, you have the option to configure credentials the SMTP relay can use to authenticate with the SMTP service on the Exchange Server. This feature confers an additional level of secure because then you can configure the SMTP service on the Exchange Server to block unauthenticated connection requests.

Click Cancel in the Outbound Security dialog box. We want to allow the SMTP relay to anonymously access the Exchange Server. You do not need to worry about spammers using your Exchange Server’s SMTP service as an "open relay". The SMTP relay on the ISA Server 2000 firewall only relays messages are destined for domains you host on your Exchange Server.

  1. Click Apply and then click OK in the Default SMTP Virtual Server Properties dialog box.

Get the New Book!

Create Remote Domains to Support Your Email Domains and Enable Relay for Those Domains

The SMTP relay server is now configured to block all incoming SMTP messages. All incoming messages to the SMTP relay server are dropped. However, you do want to relay SMTP messages destined for domains hosted on the Exchange Server. This is accomplished by creating remote domains.

A Remote Domain is an email domain hosted on the Exchange Server. For example, if you host the email domain internal.net, then you want all email messages destined for users in the internal.net email domain to be relayed by the SMTP relay server to the Exchange Server’s SMTP service on the internal network.

Note that email domains do not need to be the same as your internal network’s Active Directory domain or domains. The email domains hosted by the Exchange Server’s SMTP service can be configured in the Recipient Policy of the Exchange Server. For example, the Exchange Server may be a member of the internal.net domain, but it can be configured to receive email destined for users in the domain.com and domain.net domains.

You need to create a Remote Domain for each email domain you want your Exchange Server to receive email for. In the current example, we want to host mail for a single email domain, internal.net.

Perform the following steps to create a Remote Domain for the internal.net domain:

  1. Click Start, point to Administrative Tools, and click on Internet Information Services (IIS) Manager. In the Internet Information Services (IIS) Manager console, expand your server name and then expand the Default SMTP Virtual Server node. Click on the Domain node and then right click on it. Point to New and click on Domain.

  1. On the Welcome to the New SMTP Domain Wizard page of the New SMTP Domain Wizard, select the Remote option. Click Next.

  1. On the Domain Name page, type the name of your email domain in the Name text box. Click Next.

  1. The new Remote Domain appears in the right pane of the console. Right click the Remote Domain and click on the Properties command.

  1. In the Remote Domain’s Properties dialog box, click on the General tab. On the General tab, put a checkmark in the Allow incoming mail to be relayed to this domain checkbox. This option allows mail addressed to users in this domain to be relayed to the Exchange Server’s SMTP service.

You have two options in the Route domain frame:

Use DNS to route to this domain This option allows your DNS infrastructure to route requests to your mail domains based on the MX record entries for these domains. In order for this to work correctly, you must have a split DNS infrastructure that allows the ISA firewall 2000 machine to resolve names of your email domains to the internal IP address of the Exchange Server computer. If the ISA Server 2000 firewall resolves email domains to the external address of the ISA Server 2000 firewall, then relay will fail.

Forward all mail to smart host This option allows you to enter the IP address of your Exchange Server and have mail for your domains relayed to this IP address. You must put brackets around the IP address. If you do not put brackets around the IP address, the SMTP relay server attempts to resolve the IP address to an IP address [sic].

The Outbound Security button allows you to configure authentication methods the SMTP relay server can use to authenticate with the SMTP service on the Exchange Server. In this example we will not configure the Remote Domain to authenticate with the Exchange Server because only mail destined for domains under your administrative control are relayed to the server.

Click Apply and then click OK.

  1. In the Internet Information Services (IIS) Manager, right click on the Default SMTP Virtual Server node and click the Stop command.

  1. In the Internet Information Services (IIS) Manager console, right click on the Default SMTP Virtual Server node and click the Start command.

The SMTP relay is now ready to relay mail to your mail domain. You will need to create a remote domain for each of your email domains If you have multiple email domains.

Get the Book!

Summary

In this article we started by discussing the dangers of spam and attachments present to the corporate network. We then took at look at ISA Server 2000 firewall topologies that allow the front-end ISA Server 2000 firewall to act as a spam filtering front-end SMTP relay. The remainder of the article described step by step instructions on how to install and configure the IIS SMTP service, disable socket pooling and create remote domain for your organization.

Part two of this article is found at http://isaserver.org/tutorials/messagescreeneronfirewallpart2.html  

I hope you enjoyed this article and found something in it that you can apply to your own network. If you have any questions on anything I discussed in this article, head on over to http://forums.isaserver.org/ultimatebb.cgi?ubb=get_topic;f=6;t=002187 and post a message. I’ll be informed of your post and will answer your questions ASAP. Thanks! –Tom

If you would like us to email you when Tom Shinder releases another article on ISAserver.org, subscribe to our 'Real-Time Article Update' by clicking here. Please note that we do NOT sell or rent the email addresses belonging to our subscribers; we respect your privacy!

Advertisement

Featured Links