Paper Notes: Redirecting DNS for Ads and Profit

Redirecting DNS for Ads and Profit is one of the collection of papers from the ICSI team, with the results from the Netalyzr, network diagnosis tool. This paper focuses on the 66K session traces where DNS error traffic has been monetization and calls out Paxfire, for their role in this area, the paper focuses on NXDOMAIN wildcarding and search engine proxying (see my past post on how middleboxes interfere with DNS for an introduction to these techniques). The authors acknowledge the unrepresentative sample of Netalyzr users and the high number of sessions using OpenDNS or Comcast DNS resolvers.

NXDOMAIN wildcarding is not encouraged by ICANN and can have serious implications for non web browser DNS traffic (some resolvers only rewrite lookups starting with www. to try to prevent this). In many cases, redirection servers do not simply use HTTP 302.

The highlight of this paper was the fake NXDOMAIN opt-out offered by Paxfire, where the ad server simply served the user’s browser’s error page.

DNSSEC may provide authenticated denial of existence but this doesn’t necessarily fix the problem, for example Xerocole offers DNS resolvers with the option to simply rewrite DNSSEC signed NXDOMAIN responses without a signature, thus assuming the client will not validate DNSSEC.

OpenDNS wildcards NXDOMAIN and SERVFAIL errors as well as directing users to the redirection server if there server supports only IPv6. This is provided as an option in D-Link routers.

The study observed >12 ISPs using squid proxies to redirect search engine traffic. The study did not observe resolver independent NXDOMAIN redirection but did see NATs redirecting all DNS requests (regardless of resolver) to the configured recursive resolver, thus creating in-path NXDOMAIN rewriting if the new resolver uses NXDOMAIN wildcarding.

This paper is a fun, light read that I would recommend, though its results are a bit out of date now, as it used data from Jan 2010 to May 2011.

Part 2: Running your own DNS Resolver with MirageOS

Last time, we wrote a simple “dig like” unikernel. Given a domain and the address of a nameserver, the unikernel resolved the domain by asking the nameserver and returned the return to the console.

Today, we will look at another way to resolve a DNS query, being a DNS server. This is useful in its own right but also allows us to cool things with our local DNS resolver such as locally overwriting DNS names and resolving .local names, both of which we will add to our DNS resolver another day.

Today we use features only added to ocaml-dns library in version 0.15 (currently PR #52), so if you do not have this version or later, then update OPAM or pin the master branch on github.

Building a DNS server with MirageOS is simple, look at the following code:

open Lwt
open V1_LWT
open Dns
open Dns_server

let port = 53
let zonefile = "test.zone"

module Main (C:CONSOLE) (K:KV_RO) (S:STACKV4) = struct

  module U = S.UDPV4
  module DNS = Dns_server_mirage.Make(K)(S)

  let start c k s =
    let t = DNS.create s k in
    DNS.serve_with_zonefile t ~port ~zonefile
end

The above code will serve DNS requests to port 53, responding with the resource records (RR) in test.zone. We have provided an example zone file in the repo with the code from this guide. To use this unikernel, we also need to edit the config.ml file from yesterday.

open Mirage

let data = crunch "./data"

let handler =
  foreign "Unikernel.Main" (console @-> kv_ro @-> stackv4 @-> job)

let ip_config:ipv4_config = {
  address= Ipaddr.V4.make 192 168 1 2;
  netmask= Ipaddr.V4.make 255 255 255 0;
  gateways= [Ipaddr.V4.make 192 168 1 1];
}

let direct =
  let stack = direct_stackv4_with_static_ipv4 default_console tap0 ip_config  in
  handler $ default_console $ data $ stack

let () =
  add_to_ocamlfind_libraries ["dns.mirage";"dns.lwt-core"];
  add_to_opam_packages ["dns"];
  register "dns" [direct]

We are using crunch to access the zone file in the data directory. As explain in part 1, this config file is specific to my network setup for xen backends and can easily be generalised.

You can now test your DNS server and see it work

$ dig @192.168.1.2 ns0.d1.signpo.st.

 

Middleboxes considered harmful: DNS Edition

This article is brief overview of how middleboxes interact with DNS traffic. In particular I’m interested in finding out the answers to the following: Will middleboxes drop/modify DNS traffic and what is the purpose of this: stopping abuse, security, buggy implementations, advertising or censorship? Therefore does using your own stub resolver and recursive nameserver free you from the above issues? Do DNS recursive nameservers with caching respect the TTL? And ultimately how does the all this affect the deploy of DNS extensions such as DNSSEC, DNSCurve, DynDNS, EDNS?

My particular interest in DNS is how will research projects for naming edge network devices (e.g. HIP, UIA, UIP, MobilityFirst, CoDoNS, FERN) actually fair in the wild and is using or extending DNS a way around such issues. The title of this article is play on the title of the paper describing Delegation-Oriented Architecture.

Applications & Stub Resolvers

Stub resolver are in essence the clients to the Domain Name System (DNS), they sit between applications and DNS, usually ran locally by the OS and interfaced with by gethostbyname. The stub resolver is responsible for forming and parsing DNS packets for the application, offering a simple API to application for resolving domain names into IP address. The simplicity of this API is also its downfail, for example, gethostbyname has few error codes compared to DNS’s RCODEs. Proponents of DNSSEC hope that web browsers will present DNS validation failures to users in the same way that TLS failures are presented. At the moment however, for many stub resolvers the only possible error codes (often called h_errno) are HOST_NOT_FOUND, TRY_AGAIN, NO_RECOVERY and NO_ADDRESS. The application may not ever get this much information depending on the language API, such as Unix.gethostbyname in OCaml’s standard library.

A common linux default is to request AAAA records as well as A records even if the host doesn’t have a IPv6 address. Kreibich et al found that 13% of all sessions requested AAAA records: 42% of linux session requested AAAA records, compared to 10% of non-linux sessions, backing up this theory.

Some stub resolvers and client applications cache DNS responses, interestingly some do not respect TTLs. For example, the default cache time for ncsd (enabled by default on some linux distros) is 15 mins regardless of TTL, whereas internet explorer caches all records for 30 mins. It is important the caches respect short TTLs as they are increasingly utilised by content distribution networks and dynamic DNS. A quick check on my own browser (go to chrome://net-internals/#dns in chorme) shows that the browser cache contains 73 A/AAAA active records and 263 expired records.

Weaver et al. and Kreibich et al. studied how middleboxes interact with DNS traffic using the Netalyzr tool. Weaver et al. concluded that application wishing to use non-standard resource records (RRs) including TXT resources or DNSSEC should use their own DNS resolver and bypass the stub resolver provided by the host. It is often not possible for an application to overwrite the stub resolver’s choice of DNS resolver, which is normally a DNS resolver at the gateway, with a host of problems (see next section). The study also concluded that host stub resolvers often lack failovers (e.g. trying requests over TCP) to common issues such as: the gateway resolver not supporting the full DNS protocol, the gateway resolver cannot be trusted, the gateway resolver may be slow and the network gateway/middleboxes may filter UDP traffic.

In-Gateway Resolvers

The gateway resolver is a common (but not necessary) stage in DNS resolution (there may also be multiple stages of gateway resolvers). The stub resolver running on local host will usually forward the DNS query to the resolver(s) whos address it was given by DHCP lease when connecting to the local router. This address is normally a DNS resolver running at the gateway (at the .1 or .254 address in the local subnet e.g. 192.168.1.x) . I say “usually” as this can be overwritten, for example some people instead opt to use a public DNS server such as Google’s or OpenDNS, or run their own resolver, this is of course rare. Furthermore, not all gateways run DNS resolvers, in this case they typically refer hosts straight to the ISPs resolvers. Gateway resolvers have the advantage that they can enable the local resolution of domain such as .local or domain name for router adminisation (e.g. www.routerlogin.net for Netgear devices).

Weaver et al. tested the whether in-gateway resolvers correctly processed various DNS queries, they found that following: AAAA lookup (96%), TXT RRs (92%), unknown RRs (91%) and EDNS0 (91%). They also found that a significant number of in-gateway resolver are externally usable, opening the gateway to DoS attacks.

ISPs (& Other) Resolvers

The ISP’s resolver is a common (but not necessary) stage in DNS resolution (there many also be multiple stages of ISP resolvers). The ISP resolver is often the resolver responsible for begin to the actual resolution instead of just forwarding/proxying queries.

Despite there widespread deployment and dedicated management, these resolvers are not without there problems. Weaver et al found that 4% of sessions did not implement source port randomisation, only 55% of sessions exhibit EDNS0 usage, 4% of sessions implemented 0x20 whilst 94% propagate capitalisation unmodified. Kreibich et al found that 49% of sessions used DNSSEC enabled resolvers.

https://www.vs.uni-due.de/wander/20121229_Secure_Name_Resolution.pdf
DNSSEC capable resolvers by Matthäus Wander

NXDOMAIN wildcarding is where resolvers replaces responses with the NXDOMAIN error (for example, when a user mistypes a domain) with valid DNS responses point to another site, often with advertising. Weaver et al observed this in 24% of the sessions surveyed. This should only be done on queries from web browsers, though this is not always the case. This may also interact with web browsers who treat NXDOMAIN errors specifically, e.g. if the query fails due to NXDOMAIN, then suggest some likely alternatives. Worryingly, Weaver et al also observed a few cases of SERVFAIL wildcarding, IPv4 addresses in responses where IPv6 only was requested and ignoring additional answer RRs. Some resolvers redirect queries for some search engine, whilst other have malware to inject adverting. Kreibich et al found that essentially all resolver respected a 0 and 1 second TTL.

Another interesting area is the treatment of RRs from the Authority and Additional RR sets. For example, glue records are A RRs in the Additional section added to an answer with NS RRs which put the name servers under the domain they control, without these additional RR’s we would have a circular dependency. Kreibich et al found that 61% of sessions accept glue records when the glue records refer to authoritative nameservers, 25% accept A records corresponding to CNAMEs contained in the reply and 21% of sessions accepting any glue records present in the Additional field, and those only doing so for records for subdomains of the authoritative server.

Other ISP controlled middleboxes

It is clear that resolvers (stub, in-gateway and ISP/Public) do not reliability handle all DNS traffic and all its extensions. Users could opt to run there own resolver and stub resolvers, would this mean that their traffic be free from modification by middleboxes? Of course not.

ISPs have been know to highjack traffic to port 53 to their own DNS resolvers or simply drop it, blocking use of third party DNS resolvers. Some public resolvers support alternative ports (e.g. OpenDNS supports port 5353), but this can be difficult to configure as its cannot be easily expressed in /etc/resolv.conf. There is some evidence of gateways provided by ISPs, redirecting traffic to port 53 to the ISP’s DNS resolvers

TLDs and Root Servers

The root DNS server (or actually the 504 servers, 13 addresses) is the heart of the DNS. The root has supported DNSSEC since 2010, will not support DNSCurve. Likewise many of the TLD’s support DNSSEC and will not support DNSCurve. On the whole, these seems to fairly well managed and free of major issues.

DNS question: Avoiding circular dependencies without glue records?

Can someone help me the understand the following:

When the authoritative name server for a domain (e.g. ns1.example.com) lies within the domain over which it has authority (e.g. example.com), a query (e.g. for example.com) to the parent domain (e.g. .com) will include both the NS RRs, to delegate authority of the domain to the nameservers, in the answer section and the corresponding A RRs in the additional section, this is know as glue records. These glue records are essential to avoid a circular dependency, yet the Netalyzr study found that only 61% of resolvers accept glue records when the glue records refer to authoritative nameservers. How do the other 39% of resolvers actually work then, given that its very common for the authoritative name server for a domain to lie within the domain over which it has authority ?

Comcast blocking NASA.gov

Today, people love to hate their ISPs, they have a public image problem. A great example of this when Comcast apparently blocking NASA’s website in 2012. In fact, Comcast was the only major US ISP to be using DNSSEC validating resolvers thus where the only ones affected when NASA’s website failed to properly sign their DNS responses. Poor Comcast.

On January 18, 2012, the NASA.GOV domain had a DNS Security Extensions (DNSSEC) signing error that blocked access to all NASA.GOV sites when using DNS recursive resolvers performing DNSSEC validation. As one of the largest ISPs in the world utilizing DNSSEC validation, users of Comcast noticed a problem when attempting to connect to the website. This caused some people to incorrectly interpret this as Comcast purposely blocking access to NASA.GOV and recommending users switch from Comcast security-aware DNS resolvers to resolvers not performing DNSSEC validation … Instead, the administrators of the NASA.GOV domain had enabled DNSSEC signing for their domain, and the security signatures in their domain were no longer valid. The Comcast DNS resolvers correctly identified the DNSSEC signature errors and responded with a failure to Comcast customers. This is the expected result when a domain can no longer be validated, and this protects users from a potential security threat

source: http://www.internetsociety.org/deploy360/blog/2012/01/comcast-releases-detailed-analysis-of-nasa-gov-dnssec-validation-failure/

Video: An overview of secure name resolution [29c3]

Here is an excellent talk by Matthäus Wander, introducing DNSSEC, DNSCurve and few other DNS extensions.

 

A few points of interest:

  • stub resolvers need new API’s to report DNSSEC validation failures, then browsers can provides users with “TLS like” failure messages
  • AD flag is useless as there is no validation, yet windows 7/8 still read it
  • Comcast name servers support DNSSEC, though this hasn’t work out great for them
  • Some ISP redirect NXDOMAIN responses, another reason to run your own DNS resolver or use a public one.
  • The root server and big TLDs will not deploy DNSCurve
  • DNSSEC cannot be used directly to validate the DNSCurve public key, stored in the domain name of the parent NS record, as DNSSEC does not sign the domain name.

Matthäus’s slides are online.

Part 1: Running your own DNS Resolver with MirageOS

The following is the first part in a step-by-step guide to setting up your own DNS resolver using MirageOS. I will be running this on a low power, low cost ARM device called the Cubieboard 2. Up to date code for each version of the DNS resolver is on Github. This guide assumes some basic experience of lwt and MirageOS, up to the level of the Hello World Tutorial.

Feedback on this article and pull requests to the demo code are welcome.

Part 1.1 – Setting up the cubieboard with MirageOS

Plenty of information on setting up a cubieboard with Xen and MirageOS is available elsewhere, most notability:

For debugging I am a big fan for wireshark. I run a full wireshark sesson on the machine which is connection sharing to my cubieboard network, to check all external traffic.

For this guide, I will always be compiling for Xen ARM backend, with direct network connection via br0 and a static IP for all unikernels. My test network router is configured to give out static IP of the form 192.168.1.x to hosts with the MAC address 00:00:00:00:00:0x. As a result, my config.ml file look like:

open Mirage

let ip_config:ipv4_config = {
  address= Ipaddr.V4.make 192 168 1 2;
  netmask= Ipaddr.V4.make 255 255 255 0;
  gateways= [Ipaddr.V4.make 192 168 1 1];
}

let client =
  foreign "Unikernel.Client" @@ console @-> stackv4 @-> job

let () =
  add_to_ocamlfind_libraries [ "dns.mirage"; ];
  register "dns-client" 
[ client $ default_console $ direct_stackv4_with_static_ipv4 default_console tap0 ip_config]

Since the IP address of the unikernel is 192.168.1.2, before launching the unikernel, I do:

echo "vif = [ 'mac=00:00:00:00:00:02,bridge=br0' ]" >> dns-client.xl

I build unikernel using the usual commands:

mirage configure --xen
make depend; make; make run
# edit file.xl
sudo xl create -c file.xl

Part 1.2 – Getting Started

The following is the complete code for a unikernel which queries a DNS server for a DNS domain and prints to console the IP address returned.

open Lwt
open V1_LWT

let domain = "google.com"
let server = Ipaddr.V4.make 8 8 8 8

module Client (C:CONSOLE) (S:STACKV4) = struct

  module U = S.UDPV4
  module DNS = Dns_resolver_mirage.Make(OS.Time)(S)

  let start c s =
    let t = DNS.create s in
    OS.Time.sleep 2.0 
    >>= fun () ->
    C.log_s c ("Resolving " ^ domain)
    >>= fun () ->
    DNS.gethostbyname t ~server domain
    >>= fun rl ->
    Lwt_list.iter_s
      (fun r ->
         C.log_s c ("Answer " ^ (Ipaddr.to_string r))
      ) rl

end

This unikernel will query a DNS server at 8.8.8.8 (google public DNS resolver) for a domain google.com. Here we are using the simple function, DNS.gethostbyname, with the following type sig:

  val gethostbyname : t ->
    ?server:Ipaddr.V4.t -> ?dns_port:int ->
    ?q_class:Dns.Packet.q_class ->
    ?q_type:Dns.Packet.q_type ->
    string -> Ipaddr.t list Lwt.t

This returns a list of IP’s, which we then iterative over with Lwt_list.iter_s and print to the console.

Part 1.3 – Boot time parameters

Hardcoding the server and domain is far from ideal, instead we will provide them at boot time with Bootvar, the interface for bootvar is below:

type t
(* read boot parameter line and store in assoc list - expected format is "key1=val1 key2=val2" *)
val create: unit -> t Lwt.t

(* get boot parameter *)
val get: t -> string -> string option

(* get boot parameter, throws Not Found exception *)
val get_exn: t -> string -> string

We can now use this to provide domain and server at boot time instead of compile time

let start c s =
    Bootvar.create () >>= fun bootvar ->
    let domain = Bootvar.get_exn bootvar "domain" in
    let server = Ipaddr.V4.of_string_exn (Bootvar.get_exn bootvar "server") in
    ...

Part 1.4 – Using Resolve

Now, a real DNS resolver will need to make many more parameters (any DNS query) and return full DNS responses not just IP address. Thus we need to move on from DNS.hostbyname to using the less abstract resolve function, resolve:

  val resolve :
    (module Dns.Protocol.CLIENT) ->
    t -> Ipaddr.V4.t -> int ->
    Dns.Packet.q_class ->
    Dns.Packet.q_type ->
    Dns.Name.domain_name ->
    Dns.Packet.t Lwt.t 

We can achieve same result of hostbyname as follows:

...
    DNS.resolve (module Dns.Protocol.Client) t server 53 Q_IN Q_A (string_to_domain_name domain)
    >>= fun r ->
    let ips =
    List.fold_left (fun a x ->
      match x.rdata with
      | A ip -> (Ipaddr.V4 ip) :: a
      | _ -> a ) [] r.answers in
...

We are now explicit about parameters such as port, class and type. Note that we have opened the Dns.Name and Dns.Packet.t modules. The return value of resolve is a Dns.Packet.t, we fold over answers in the produce an IPaddr.V4 list as with hostbyname. We can also use the to_string function in Packet to print

I’ve taken a break to do some refactoring work on the ocaml-dns library. In the next post, Part 2, we will expand our code to a DNS stub resolver.

 

Lack of Love for DNSSEC

It’s more than two years since I wrote my introduction to DNSSEC and the internet is let to open its loving arms to DNSSEC and DANE.

Don’t just trust my work for it:

Thanks to Hannes for originally bringing these articles to my attention

Dreaming of a new life on the edge network

Motivation

The internet has abandoned the end-to-end principles on which it was established. With IPv4 addresses depleted, devices are left behind NATs, with the transition to IPv6 yet to restore their public identity. Users have been left isolated by their ISPs, they are pushed to depend on opaque centralised services boosting usability and availability. However, data breaches, DDoS attacks, censorship and mass-surveillance have made individuals re-evaluate their decisions and look for alternatives, a search hindered by data lock-in and network externalities.

The infrastructure exists for building secure distributed systems over a user’s personal cloud of devices. Current approaches require intricate configuration to deal with the diversity of devices, middleboxes and network environments. Developers each try to re-implement solutions to establishing authenticated identities, distributed consensus and availability in the face of mobile nodes, pervasive network partitioning, asymmetric channels and Byzantine failures. Applications sit on top of an unstable stack, which without modification and violation, falls down in the face of everyday challenges, fails to utilise the resources available and slow at deploying new protocols. For example, without Explicit Congestion Notifications wireless traffic is unnecessarily throttled in the face of interference and without Multi-path TCP multiple NICs offer no resilience/speedup for a connection.

With trust in internet services wavering and ever more private data becoming available from the Internet of Things, we must improve on today’s opaque terms of service which minimise legal responsibility and offer few availability guarantees. Can we build a new representation for legally binding contracts between applications and their users, which provides upfront guarantees that are understandable to the user and provably enforced by the application?

State of the Art

Most of the time, devices are underutilised: CPUs idle, storage to spare and bandwidth unused. The premise that the required physical infrastructure already exists, relies in part, on people being willing to share their resources given a good incentive model. BitTorrent will reward you for sharing files with faster downloads, Bitcoin will trade your computation and storage on the blockchain for cryptocurrency and BOINC allows you to contribute to scientific research. Project Tor allows you to share your bandwidth with people around the world seeking anonymity or bypassing censorship, whilst the Public Access WiFi Service (PAWS) allows you to share bandwidth with your local community.

Giving data back to its owners allows individuals to make informed decisions about how exactly to distribute their data. Even if the owner chooses to utilise cloud storage for their data, they can still remain in control with systems like Priv.io which allow the user to provide their own cloud storage and grant 3rd party apps access via their browser. Community efforts to address the usability challenges often involve packaging a collection of P2P alternatives into a plug and play solution such as Freedom Box and arkOS

Approach

Inspired by the previous work in SLAs and financial contracts, I dream of replacing opaque terms of service with a formally defined contract in a domain specific language (DSL). This would allow it to be easily understood by users, stand up in a court of law and be dynamically enforced by the verified applications. But why would service provider choice to adopt such as scheme? Perhaps to minimise expensive legal battles with customers who argue that they didn’t give informed consent} and the poor publicity that follows. Or to difference themselves from the competition, by bowing to users pressure. Ultimately if adopted by a sufficient minority, then regulatory changes could make it the new norm.

Building a personal cloud of devices, ultimately depends on establishing and revoking layers of trust between devices. A popular technique is public key infrastructure, as used in SSL and DNSSEC, but this relies heavily on a trusted certificate authority and sensible key management. I intend to develop an alternative such as utilising a web of trust scheme such as PGP, authenticating a host’s public key by observing it from a range of network vantage points as used in Perspectives or authenticating hosts by consensus as used in Unmanaged Internet Architecture.

I dream we will put aside many of the assumptions which have dominated the discussion on distributed systems, to focus on life at the edge, to build a new federated layer for applications. One which provides consensus algorithms, so data will always be consistent no matter where it is accessed from, even if malicious agents try to gain control of the system. One which puts users first and manages their data responsibly. Unifying an individual’s collection of devices into a secure resilient personal cloud with incentive systems to stimulate fair sharing of excess resources, improving utility and fault tolerance.

Evaluating the project will begin with building applications such as social networking, content distribution or micro blogging over the personal cloud and testing there performance on typical set-ups, against that of centralised services and popular P2P alternatives. Followed by, formal verification of many of the components such as the enforcement of the term of service, as defined by the DSL and the consistency, availability and fault tolerance of distributed system. While a threat model will consider the authentication, encryption and confidentiality properties.