Overview of IPSec Rules in Windows Firewall with Advanced Security

IPSec is becoming increasingly popular and as a result we are seeing more support calls on it.  I wanted to give a brief overview of the IPSec wizard in the Windows Firewall with Advanced Security as well as provide some additional references.  IPSec is a great tool that can help in securing your network, however many people find it difficult and confusing to deploy.  If you are new to IPSec and just wanting to know more about it or are preparing for your first deployment, I suggest skipping to the additional references at the end of this blog post.  If you are already familiar with the legacy IPSec wizard in Windows XP and Windows Server 2003, then my hope is that this blog post will help you in getting more comfortable with the new Windows Firewall with Advanced Security console for creating your IPSec rules in Windows Vista and Windows Server 2008.

When creating IPSec rules for Windows Vista and Windows Server 2008 on a Windows Vista or Windows Server 2008 server, you should now be using the Windows Firewall with Advanced Security console.  While the legacy IPSec policy console does still exist, this is not the preferred method.  Also the legacy wizard presents the option to select the Default Response Rule (DRR); this rule is no longer supported in Windows Vista or Windows Server 2008 and should not be selected when creating rules for these operating systems, as noted in KB article 942964.

In the legacy IPSec console shown below, you would right click on "IP Security Policies on Local Computer" to start the IP Security Policy Wizard.  This wizard still exists in Windows Vista and Windows Server 2008 for the creation IPSec policies in mixed environments.

Legacy IPSec console

Legacy IPSec Policy Wizard

Note: As mentioned earlier the Legacy wizard presents the Requests for Secure Communication page.  While this wizard can be used to creates IPSec policies for all Windows operating systems it is best to get in the habit of not selecting the Activate the default response rule in this wizard, since Windows Vista and Windows Server 2008 no longer support this rule.  The Default Response Rule can be enabled in the IPSec Management console after the rule is created.

Requests for secure communication

When you open the Windows Firewall with Advanced Security console you will see an option for Connection Security Rules. If you select this you will see a list of the currently configured rules.

Windows Firewall with Advanced Security console

In the Windows Firewall with Advanced Security console a new IPSec rule is created by right clicking on Connection Security Rules and selecting New Rule.  This will bring up the New Connection Security Rule Wizard.

New Connection Security Rule Wizard

There are various rule types that can be selected.  All rules include the Profile and Name Steps in the wizard.

The Isolation rule includes the Requirements and Authentication Method.

The Authentication exemption rule adds the Exempt computers option.

The Server-to-Server and Custom rules include the Endpoint option.

The Tunnel rule adds the Tunnel Endpoint option.  It is important to note that the Tunnel Endpoint and Endpoint options are slightly different.

Notice that you can move through the wizard by selecting Next or you can skip to different parts of the wizard by using the Steps menu on the left.  Also, the Steps menu on the left will change depending on the type of connection you select in the wizard. 

Requirements

It is also important to realize that different rule types will change options in other parts of the wizard.  For example, while the Authentication Method will be displayed in the menu on the left for both the Isolation and the Server to server rules, the available Authentication Methods will change.  This is to be expected as different options are appropriate depending on the type of rules being selected.

Authentication method for the Isolation rule type

The advanced option allows for more granular control of the authentication methods that will be used in the IPSec negotiation.

Advanced Authentication Methods

Below are the options that will be available to add.  Notice that the available authentication methods are not identical.  It is important to note, as mentioned in the dialog on the first authentication method, that if you select the Preshared key you cannot select a second authentication method.  The Preshared key is considered less secure and really is only supported for testing.

First authentication method

Second authentication method

Once the authentication method is selected you will need to select the profile that the rule will apply to.

Profile

Connection name

For the Tunnel rule type you will see the following options.

Tunnel End points

Notice how this differs from the Server-to-server Endpoints.

Server-to-server Endpoints

As I mentioned this is just an overview of the IPSec portion of the Windows Firewall with Advanced Security console and what it contains. My hope is this will make administrators more comfortable with this interface as we move from the legacy wizard to the new Windows Firewall with Advanced Security console.

For additional information see the following links:

TechNet article - Security: Managing the Windows Vista Firewall

This article has a description of the Profiles and a little more depth on rules and filtering.

http://technet.microsoft.com/en-us/magazine/cc510323(TechNet.10).aspx

Security rules for Windows Firewall and for IPsec-based connections in Windows Vista and in Windows Server 2008
http://support.microsoft.com/default.aspx?scid=KB;EN-US;942957

How IPSec Works
http://technet2.microsoft.com/windowsserver/en/library/8fbd7659-ca23-4320-a350-6890049086bc1033.mspx?mfr=true

- Clark Satter