Step 6: create a fully cross-signed zone

Written by Roland van Rijswijk in category: Architecture, Procedures, Technical

The purpose of this step is to create a fully cross-signed zone that includes the key set from the source as well as the destination signer. The situation at the end of this step is shown in the diagram below:

To reach this situation, the following sub-steps need to be taken:

  1. Edit the output (i.e. the signed) zone on the source signer and include the RRSIG made with the active KSK from the destination signer over the DNSKEY set saved at the end of step 4
  2. Publish this edited zone on the authoritative name servers
  3. Do NOT restart the signer component on the source signer
  4. Wait TTL(DNSKEY) for the DNSKEY set and the associated signatures to propagate to caches

No Comments

Step 5: introduce destination keys on source signer

Written by Roland van Rijswijk in category: Architecture, Procedures, Technical

The purpose of this step is to introduce the DNSKEY records for the active keys from the destination signer on the source signer. The situation at the end of this step is shown in the diagram below:

To reach this situation, the following sub-steps need to be taken:

  1. Stop the signer software on the source signer and ensure that zone publication is temporarily halted (i.e. no new versions of the zone are pushed from the signer to the authoritative name servers)
  2. Alter the input zone on the source signer to include the DNSKEY record for the active KSK from the destination signer
  3. Alter the input zone on the source signer to include the DNSKEY record for the active ZSK from the destination signer
  4. Set the SOA serial number in the input zone such that it is higher than the SOA serial number of the zone that was published in step 2
  5. Allow the signer software to run once on the source signer and ensure that the output zone includes a valid signature over the entire DNSKEY set, which should include the active KSK and ZSK from both the source and the destination signer
  6. Do NOT resume zone publication (i.e. do not push new zones from the signer to the authoritative name servers)

No Comments

Step 4: introduce source keys on destination signer

Written by Roland van Rijswijk in category: Architecture, Procedures, Technical

The purpose of this step is to introduce the DNSKEY records for the active keys from the source signer on the destination signer and to get an RRSIG signature for the new DNSKEY set this will result in using the active keys on the destination signer. The end situation of this step is shown in the diagram below:

To reach this situation, the following sub-steps need to be taken:

  1. Stop automated uploads of the input zone to the destination signer
  2. Save an unmodified copy of the input zone on the destination signer; this unmodified copy will be used at a later stage in the process
  3. Alter the input zone on the destination signer to include the DNSKEY record for the active KSK from the source signer
  4. Alter the input zone on the destination signer to include the DNSKEY record for the active ZSK from the source signer
  5. Allow the signer software to run on the destination signer and ensure that the output zone includes a valid signature over the entire DNSKEY set, which should include the active KSK and ZSK from both the source and the destination signer; stop the signer after the zone has been generated (or, alternative, run the signer once if possible)
  6. Save the RRSIG made with the active KSK on the destination signer over the DNSKEY RRset; you will need this record in the following steps

No Comments

Step 3: configuring the destination signer

Written by Roland van Rijswijk in category: Architecture, Procedures, Technical

The purpose of this step is to configure the destination signer and to transition it to the situation as shown in the diagram below:

To reach this situation, the following sub-steps need to be taken:

  1. Configure the zone to be migrated on the new signer
  2. Launch automated key management on the new signer but do not yet launch zone signing (when using OpenDNSSEC this means launching the “enforcer” component but not yet launching the “signer” component)
  3. Once the signer configuration has been created by the automated key manager, you must stop automated key management again
  4. Edit the signer configuration such that no new ZSK is pre-published if the key manager generated and configured such a key
  5. Run the signer and allow it to generate an output zone with the trust chain as shown in the diagram above; stop the signer afterwards (or, alternatively, run the signer once if this is possible)

No Comments

Step 2: cleaning up the source signer

Written by Roland van Rijswijk in category: Architecture, Procedures, Technical

The purpose of this step is to clean up the source signer and to transition it to the situation shown in the image below:

To reach this state, the following sub-steps need to be taken:

  1. Stop automated uploads of the input zone to the destination signer
  2. Stop active key management (when using OpenDNSSEC this means stopping the “enforcer” component)
  3. Alter the signer configuration such that only the active ZSK is published as part of the DNSKEY set (when using OpenDNSSEC this means manually editing the signer configuration for the zone that is being migrated)
  4. Force the signer to generate a zone that only contains fresh signatures for the active ZSK (when using OpenDNSSEC this means deleting the signed zone and all intermediate files and re-running the signer)
  5. Set the SOA serial number in the input zone such that it is higher than the currently published zone’s SOA serial number
  6. Re-start the signer and make sure that a new zone with the new DNSKEY set and fresh signatures is published
  7. Wait maxTTL(zone) after all authoritative name servers have received the new zone data for the new signatures to propagate to caches

Note: a side effect of this step is that all signatures have the maximum validity time at the start of the migration process.

No Comments

Step 1: starting situation

Written by Roland van Rijswijk in category: Architecture, Procedures, Technical

The diagram above shows the starting situation on the source signer. The starting situation shown is a typical snapshot of the state of a signer that uses the “ZSK pre-publication” rollover strategy in which a ZSK is pre-published before it is made active and in which old signatures are gradually rolled to the new key during a key rollover.

The diagram shows the active DS in the parent zone at the top. This DS “certifies” the active KSK for the zone, depicted below it. In turn, this KSK signs the entire DNSKEY set, which will include an active ZSK (depicted in the middle, directly below the KSK), and may include an old ZSK (which is still being published because there are still signatures in use that depend on it) and may include a new pre-published ZSK.

The authoritative name servers are publishing zone data that is output by the source signer at this moment in the process. This continues to be the case until the document explicitly mentions that this changes.

Record the characteristics such as the maximum zone TTL and the SOA serial number of the zone. You can use this worksheet to do this.

No Comments

Signer migration: our goals and assumptions

Written by Roland van Rijswijk in category: Architecture, Procedures, Technical

Introduction

As mentioned in the previous post, we will be publishing a series of articles on signer migration. In this article, we will address the goals we set ourselves for the signer migration as well as outline the assumptions we made. We’ll end the article with a brief explanation of the diagrams we will be using to illustrate the steps in the migration process.

Terminology

In the articles on the migration process we will talk about two systems:

  • The source signer – by this we mean the system that is currently responsible for serving out a signed version of the zone that will be migrated
  • The destination signer – by this we mean the system that will be serving out the signed version of the zone to be migrated after the migration completes

Goals

We set ourselves the following goals for the migration process:

  • The DNSKEY RRset may only consist of keys that are active on one or both signers
  • Manual zone editing shall be kept to a minimum
  • The migration must take place as quickly as possible

Assumptions

At the start of this process, the following conditions are assumed to be true:

  • There is no KSK rollover in progress on the source signer
  • The validity of the signatures is significantly longer than the maximum TTL in the zone that is to be migrated (at least 4 times as long)
  • Source and destination signer function as a hidden primary (i.e. are not authoritative name servers that are part of the NS-set for the domain); consequently, no change to the NS-set for any signed domains is required to facilitate the migration
  • There is a means to switch signers (e.g. reconfiguring the public primary name server to source the zone data from a different signer)
  • Key management and zone signing are separate processes (as is for instance the case when using OpenDNSSEC)
  • A split-key policy is used (i.e. with separate ZSK and KSK)
  • The parent zone allows a single DS for a secure delegation

Notation

Each step is illustrated using a diagram. For these diagrams, we have used consistent arrow types and colour coding:

  • Blue indicates (part of) the chain of trust that uses the active key set on the source signer
  • Red indicates (part of) the chain of trust that uses the active key set on the destination signer
  • Grey indicates (part of) a chain of trust that is no longer used or published
  • An arrow indicates a validation based on the object at the origin of the arrow for the object that the arrow points to (in most cases this implies a signature)
  • A dashed line in an arrow indicates that this relationship may exist but does not necessarily have to exist
  • “Self-references” (i.e. KSK signatures over the KSK) are omitted for reasons of clarity

To further clarify the diagrams independent of the colour coding we have included subscripts for each object. The following subscripts are used:

  • src – indicates that this object belongs to or originates from the source signer
  • dst – indicates that this object belongs to or originates from the destination signer
  • old – indicates that this object is an old key that is no longer actively being used to sign data
  • act – indicates that this object is a key that is actively being used to sign data
  • pre – indicates that this object is a key that is being pre-published and may be the next in line for a key rollover

Finally, we always show the situation on both the source as well as the destination signer. The signer that is authoritatively serving the zone has thicker lines and a bold heading at the top of the diagram. To illustrate what these diagrams look like, we have included an example from later in the series below.

1 Comment

Signer migration: a step-by-step guide (introduction)

Written by Roland van Rijswijk in category: Architecture, Procedures, Technical

One of our goals with this blog has been to share what we have learned from our DNSSEC deployment with our constituency and the wider Internet community. Last month we performed a complicated operation on our DNSSEC signer deployment: we migrated from the existing signer setup to a completely independently running new signer. We had prepared this migration beforehand and the migration was a big success; all our signed domains were migrated safely while remaining secure.

Over the next couple of days we are going to post a step-by-step guide on how we achieved this, starting today with the motivation for our migration and, in a separate post, the goals we set ourselves during the migration and the assumptions we have made. At the end of the series we will post an integral document that you can use if you want to perform a similar migration.

Motivation

In 2010 SURFnet deployed automated DNSSEC signing in its “SURFdomeinen” managed DNS service. One of the design decisions that we made during the implementation of DNSSEC in this service was to use “shared keys” (i.e. sharing the same set of cryptographic keys for multiple DNS zones, instead of using separate keys for each individual zone).

There were two reasons for this. First, the HSMs that we selected for secure key storage only support storing a limited number of objects. Secondly, we saw this as a way to simplify key management by only using one set of keys per user (in our case connected institutions such as a university or teaching hospital). Another decision that we made early on was to use OpenDNSSEC as software suite for our deployment.

Using shared keys has a number of drawbacks:

  • It is harder to migrate zones to a different DNS operator due to the fact that this requires stopping key rollover for the zone; in the case of shared keys this affects all zones that share a key set
  • The state of a zone cannot develop independently; the zone with the highest overall TTL determines the speed at which key rollovers can proceed and adding a zone with a higher overall TTL may pose a problem
  • Using shared keys creates an artificial tie between zones that are otherwise independent of one another
  • KSK rollover is problematic for the case where a large number of zones shares one key set because of the number of parent interactions and possibly because of differing policies on the parent side when different top-level domain registries are involved
  • Synchronised key rollover as is the case with shared keys may impose additional load on the infrastructure because caches will be sending more requests for all zones for which the keys are rolled

On top of that, the OpenDNSSEC version that we are currently using (version 1.1) does not have optimal support for shared keys.

For this reason, we have decided to upgrade our HSMs to a version that supports storage of a much larger number of keys and to simultaneously migrate away from using shared keys. To do this in one go, we are going to migrate to new signers systems that are also running a newer version of OpenDNSSEC (version 1.3). Effectively, this means that we are performing a signer migration or signer rollover.

Links to the articles

Click on one of the links below to jump straight to the article

No Comments

Researching UDP fragmentation issues

Written by Roland van Rijswijk in category: General

The last time we posted on this blog (earlier this year), we discussed issues we had encountered because of firewalls blocking IP fragments on port 53. At the moment, we have a student working on his master thesis who is researching this issue in detail and will investigate ways of mitigating this issue in situations where the firewall cannot be reconfigured or removed. Read more about his work in his research proposal.

No Comments

MTU woes again…

Written by Roland van Rijswijk in category: Resilience, Technical

We recently experienced some more MTU woes as a result of our main zone being DNSSEC signed that I’d like to share with you in the hope that this can help prevent this problem for others. Last week I got an e-mail from our IT department about mail issues that some of my colleagues were experiencing. They had received phone calls from companies we do business with that they were unable to send e-mail to my colleagues. Diving deeper into the problem revealed that all these companies had one thing in common: they were using the hosted MS Exchange servers of a large ISP. The errors they got looked something like this (with privacy sensitive information blanked out):

Unable to deliver message to the following recipients, due to being
unable to connect successfully to the destination mail server.
Reporting-MTA: dns;********************
Received-From-MTA: dns;macpro.lan
Arrival-Date: Thu, 17 Mar 2011 14:54:18 +0100
Final-Recipient: rfc822;*************@surfnet.nl
Action: failed
Status: 4.4.7
From:  ***********************************
To:  ******************@surfnet.nl

The rather unclear error code pointing out the problem is “4.4.7”. Apparently this means that the mail server was unable to connect to the recipients mail exchanger.

I talked to the people at the ISP about this and they said that the cause of the error message was a DNS resolving problem. When we dove into it a bit deeper it turned out that they had recently upgraded the resolvers used by their hosted Exchange environment from Windows Server 2008 to Windows Server 2008R2. And apparently Windows Server 2008R2 has EDNS0 enabled by default with the DNSSEC OK (DO) bit set to true. In addition to that, it turned out (after some packet tracing) that their servers are behind a firewall that drops UDP fragments and somewhere along the route between their servers and our authoritative name servers the MTU drops to 1472 bytes. Since the largest answer we could produce for our MX set is over 2000 bytes, this meant that packets were getting fragmented. And when these fragments then get dropped, the resolver never receives a proper answer.

The workaround was very simple: since they do not validate DNSSEC answers and are not planning to start doing so very soon, they decided to disable EDNS0. This is done by executing the following command:

C:\Windows\System32\> dnscmd /config /EnableEDNSProbes 0

Once that was done the problem was solved. This does however, teach us an important lesson: MTU troubles show up as very unclear problems in secondary systems…

There was also an interesting spin-off from the search we performed to find the problem: their resolver sent back ICMP message to indicate that fragment reassembly had failed:

11:01:59.849643 IP *.*.*.* > ns3.surfnet.nl: ICMP ip reassembly time exceeded, length 92
11:01:59.849655 IP *.*.*.* > ns3.surfnet.nl: ICMP ip reassembly time exceeded, length 92

I’m going to work some more with this because I usually receive these ICMP messages back on the authoritative server. That means I now have a way of detecting – with some probability – MTU problems to certain resolvers that query our authortitative name servers. If I get some results from this I will publish that information here on the blog.

No Comments