Exchange 2007 Message Queues

Brief

This article investigates message queues in Exchange 2007. I begin by highlighting the differences in architecture between Exchange 2003 and 2007 in particular, discussing the fact that Exchange 2007 uses a queue database. I then discuss the new look queue viewer in Exchange 2007 and what it actually does! Finally I take a look at how the queue viewer is built on PowerShell and suggest some ways in which that could be useful!

Message Queues, an Introduction

Exchange has always had a way of viewing the messages it was processing right back to the early days of Exchange 5.x, and possibly even Exchange 4.0. However, the ease with which this is possible and the functionality available to administrators has changed throughout the versions. This is again the case with the transition from Exchange 2003 to Exchange 2007. In Exchange 2007, the way queues work has changed fundamentally. We have moved away from the Exchange 2003 method where each SMTP virtual server had its own queue directory on an NTFS partition to Exchange 2007 using a standard Extensible Storage Engine (ESE) Database for its queue information. On top of that the user interface (UI) has changed completely in Exchange 2007 as it is now based on a new Microsoft Management Console (MMC) v3 snap-in. To highlight the UI difference, take a look at the screenshots below;

Figure 1: The location of Exchange 2003 Queues

In Exchange 2003 the UI for viewing queues made things fairly easy to find however, it had the drawback of only being able to monitor one server’s queues at one time.

Figure 2: Viewing queues in Exchange 2003

In Figure 2 you can easily see the queue type and its status. In this example you can see that there are several mails waiting to be sent which most likely are non-delivery reports (NDRs) from spam mails!

With Exchange 2007, the queues are viewed in a new tool called “Queue Viewer” which you can find alongside other utilities in the new Toolbox area, shown in Figure 3.

Figure 3: The new Toolbox area in Exchange 2007

When you open Queue Viewer, you can see that it is based on an MMC v3.0 snap-in as shown in Figure 4:

Figure 4: The Queue Viewer UI

The major benefit of this is that you can now create your own custom MMCs using the standalone viewer to monitor multiple Exchange 2007 servers at once:

Figure 5: Adding an MMC snap-in for Queue Viewer

Having looked at the UI changes, it is nearly time to move on to the fundamental theory of the queues in Exchange 2007; however, before I do there is one other piece of info which you should be aware of. Not all Exchange 2007 servers have queues! This is a big difference to Exchange 2003 where, because of the fact that all servers were involved in message transport, they all had an SMTP queue. In Exchange 2007 only Hub Transport and Edge Transport servers have queues.

Queue Theory

So where does this database fit in? Well as mentioned briefly above, all queue activity now occurs in a new ESE database. The main database file is called mail.que and by default can be found here:

C:\Program Files\Microsoft\Exchange Server\TransportRoles\data\Queue

Figure 6: Folder containing queue database files

The other files are in the locations as described below:

• Trn.chk - The checkpoint file.
• Trn.log - The current transaction log file.
• Trntmp.log - The next provisioned transaction log file that is created in advance.
• Trnnnn.log - Other transaction log files that are created when Trn.log reaches its maximum size.
• Trnres00001.jrs - The Reserve log file.
• Trnres00002.jrs - The Second Reserve log file.
• Temp.edb – Temp DB used to verify database schema on start-up.

You might wonder what happens with the logs in this scenario. Well, they are configured for circular logging with transaction logs being deleted after they have been committed.

Just before we move on to another area, it is worth stating how to move the queue databases. One important reason for moving the Queue DB and logs is performance. Another slightly less well known reason is that the drive on which the Queue DB and logs are stored must have 4GB or more free space otherwise the server will apply back pressure and start slowing the flow of messages!

When moving the DB, the usual rules for splitting transaction logs and DB files apply. To move the databases you must edit the EdgeTransport.exe.config file which by default is located at the location below and then stop and restart the msexchangetransport service:

C:\Program Files\Microsoft\Exchange Server\Bin\EdgeTransport.exe.config

The key thing to bear in mind before editing the config file is that the parent directory has the correct permissions as set up below; that way the directory will be created for you:

• Network Service: Full Control
• System: Full Control
• Administrators: Full Control

The relevant lines are shown below. To move the database, you should edit the line containing “QueueDatabasePath” and to move the logs, you should edit the line containing “QueueDatabaseLoggingPath”. You can see in Figure 7 that I have moved my DB and logs to H:

Figure 7: Editing the EdgeTransport.exe.config file (click to view a larger image)

Having looked at the Database it is now time to understand what it contains. There are various different queues:

• Submissions: Used by the categorizer to gather all messages that have to be resolved, routed, and processed by Transport agents.
• Poison Message: The poison message queue is a special queue that is used to isolate messages that are detected to be potentially harmful to the Exchange 2007 system after a server failure.
• Remote Delivery: Remote delivery queues hold messages that are being delivered to a remote server by using SMTP.
• Mailbox Delivery: The mailbox delivery queues hold messages that are being delivered to a mailbox server by using encrypted Exchange RPC.
• Unreachable Destination: Each transport server can have only one Unreachable queue. The Unreachable queue contains messages that cannot be routed to their destinations. 

Using Queues

Now that MMC v3 is used for queue viewer the UI is very simple. By default the queue viewer opens and displays the queues for the transport server that you are logged on to. To connect to another server use the Connect to Server task in the right hand action pane:

Figure 8: Showing Connect to Server

The main queue viewer window opens with two tabs along the top which show all queues and all messages by default. When you open a queue by double clicking, another tab appears showing only the messages in that queue:

Figure 9: Showing tabs in Queue Viewer

To manage the queues all you have to do is highlight the object to operate on and view the actions pane on the right hand side of the window as shown in Figure 9.

A key new Exchange 2007 queue viewer feature is message filtering. An example of how to use filtering is in the case of a spam attack. As an administrator you can take advantage of the "Bulk Action" feature, which applies an action to all messages that meet the parameters specified by the filter to remove spam messages with or without NDR.

Figure 10: The filtering UI shown in the messages tab

Figure 11: Some more filtering options

The other actions which you can perform in queue viewer are shown below:

• Suspend queue This action temporarily prevents delivery of messages that are currently in the queue.
• Resume queue The opposite of Suspend queue.
• Retry queue When a connection to the next hop for a queue fails, a retry timer is set. This forces an immediate connection attempt.
• Suspend message This action temporarily prevents delivery of a single message.
• Resume message The opposite of Suspend message.
• Remove message This action permanently prevents delivery of a message.
• Export message This action copies a message to the file path that you specify. The messages are not deleted from the queue, but a copy of the message is saved to a file location. Before you export a message, you must suspend the message in the queue.

Queues and PowerShell

Whilst I know that the whole of Exchange Management Console is based on PowerShell, something that was really highlighted even further was the error message I got when I had the queue viewer open and stopped the msexchangetransport service! You can see in Figure 12 the PowerShell commands that run to provide the output to the Queue Viewer UI.

Figure 12: PowerShell command error

It got me thinking about using PowerShell to manipulate the queues. To start off, I used the PowerShell command below to show all commands with Queue in the name:

get-command *queue*

Figure 13: get-command *queue*

Then I tried the same command this time looking for commands with “message” in them:

Figure 14: get-command *message*

Armed with a knowledge of available commands I began by running a simple get-queue command

Figure 15: get-queue

I then moved on to locate any queue with a message count of less than 100. In this example three queues are shown all with 0 messages. All but the submission queue will shortly be removed as their messages have been delivered. The submission queue is persistent and is therefore always present, waiting for mail to be categorized.

Figure 16: get-queue with a filter

As you can see, the simplicity of the PowerShell commands make manipulating the queues via script much easier than when using VBScript. Obviously the examples above are simple but they could be taken a lot further. For example, you could run the following to remove messages from queues which come from anything@spammer.com with an SCL rating higher than 5:

Conclusion

Hopefully this has given you an insight into the new way message queues work in Exchange 2007. For more information about the inner workings of queues, check out the Exchange 2007 help file which can be downloaded here:

http://www.microsoft.com/downloads/details.aspx?familyid=555F5974-9258-475A-B150-0399B133FEDE&displaylang=en