D-Link Firmware Abuses Open NTP Servers

DES writes "FreeBSD developer and NTP buff Poul-Henning Kamp runs a stratum-1 NTP server specifically for the benefit of networks directly connected to the Danish Internet Exchange (DIX). Some time last fall, however, D-Link started including his server in a hardcoded list in their router firmware. Poul-Henning now estimates that between 75% and 90% of NTP traffic at his server originates from D-Link gear. After five months of fruitless negotiation with a D-Link lawyer (who alternately tried to threaten and bribe him), he has written an open letter to D-Link, hoping the resulting publicity will force D-Link to acknowledge the issue. There are obvious parallels to a previous story, though Netgear behaved far more responsibly at the time than D-Link seem to be."

List of Affected Products:

From TFA: "A number of D-Link products, so far I have at least identified DI-604, DI-614+, DI-624, DI-754, DI-764, DI-774, DI-784, VDI604 and VDI624, contain a list of NTP servers in their firmware and using some sort of algorithm, they pick one and send packets to it."

Re:List of Affected Products:

From the RFC website: http://www.rfc-archive.org/getrfc.php?rfc=4330 [rfc-archive.org]

10. Best Practices

      NTP and SNTP clients can consume considerable network and server
      resources if they are not good network citizens. There are now
      consumer Internet commodity devices numbering in the millions that
      are potential customers of public and private NTP and SNTP servers.
      Recent experience strongly suggests that device designers pay
      particular attention to minimizing resource impacts, especially if
      large numbers of these devices are deployed. The most important
      design consideration is the interval between client requests, called
      the poll interval. It is extremely important that the design use the
      maximum poll interval consistent with acceptable accuracy.

      1. A client MUST NOT under any conditions use a poll interval less
              than 15 seconds.

      2. A client SHOULD increase the poll interval using exponential
              backoff as performance permits and especially if the server does
              not respond within a reasonable time.

      3. A client SHOULD use local servers whenever available to avoid
              unnecessary traffic on backbone networks.

      4. A client MUST allow the operator to configure the primary and/or
              alternate server names or addresses in addition to or in place of
              a firmware default IP address.

      5. If a firmware default server IP address is provided, it MUST be a
              server operated by the manufacturer or seller of the device or
              another server, but only with the operator's permission.

      6. A client SHOULD use the Domain Name System (DNS) to resolve the
              server IP addresses, so the operator can do effective load
              balancing among a server clique and change IP address binding to
              canonical names.

      7. A client SHOULD re-resolve the server IP address at periodic
              intervals, but not at intervals less than the time-to-live field
              in the DNS response.

      8. A client SHOULD support the NTP access-refusal mechanism so that
              a server kiss-o'-death reply in response to a client request
              causes the client to cease sending requests to that server and to
              switch to an alternate, if available.

-daedone

Re:List of Affected Products:

I don't get why D-Link doesn't just solve the problem. All they need to do is put up an ntp.dlink.com with a simple mock DNS server that checks the requesting IP, and returns the closest known, public (or authorized for that network) NTP server as a CNAME. In most of the cases, that's going to be the IP's ISP-provided NTP server, which D-Link could easily compile a list of from ISP Web-sites. It's like 2 weeks of one person's work to write the server, gather data, and solve 80% of the problem (and avoid doing this to companies that CAN afford to sue in the future). This would also allow organizations to request special listings in D-Link's table.

Even in the case where the request comes from a recursive lookup, it should (in almost all cases) come from a DNS server which indicates the rough location (in terms of Internet topography) of the client.

Of course, they could also obey DHCP responses (either to the device or to a directly connected IP) as a fallback, solving even more of the problem.

Re:List of Affected Products:

I asked for details and this is what he provided to me. I haven't gotten to do this yet:

"If you download the firmware from DLink and run unarj on it
you get a file called something like nml.mem.

Run strings on that and grep for GPS.dix.dk to make sure it is not
listed in there."

Re:List of Affected Products: - ERR Wrong Answer

Can you please show me where the Source MAC address exists in an IP packet that has been forwarded over the internet from (for example) the United States - to a server in Denmark?

Now that you look at your ethernet sniffs (I assume you just went running off and ran ethereal) look at the source ethernet address... Hmmmmm - doesn't that look familiar, like maybe it looks kinda like your first hop routers MAC address.

Nice try -

Thank you, Come Again

And please read either Stevens or Comer before posting on networking topics again

Moochers

Give people an inch and they take a mile. I don't see why D-Link and Netgear couldn't just make their own stratum-1 NTP servers. I mean, if you trust the brandname enough for your routing, don't you trust them enough for your time as well?

Re:Moochers

I'm not considering good will, appreciation, or the right thing to do. None of these things apply to a business unfortunately.

Eh hem, at the risk of sounding like a troll, they apply to my business damnit and don't you forget that.

The problem is, when you do the right thing, like enforcing security over convience, customers don't always appretiate it.

Re:Moochers

I mean why in the hell does cheap dlink crap need to connect to stratum-1 servers? Seriously these things should be running on stratum-3 or lower. I doubt the FBI will come into your home with national security at stake and the whole world ENDS because your $40 dlink router is off by half a second. Why doesn't dlink run their own damn ntp server off of the stratum-1 (making them stratum 2 - stratum 1 is sortof expensive). There is no need for these things to have this level of time precision - they just need ballpark correct time.

Re:Moochers

It's cheaper for D-Link to freeload off other people.

That being said, D-Link has acquired quite a bad reputation in my book. The last time they were prominently mentioned on Slashdot was when their routers were randomly silently redirecting a small chunk of HTTP traffic to D-Link advertisements, and causing the obvious mayhem in non-human-readable HTTP traffic.

I'm also wondering just how much mayhem this guy could cause on various networks by playing with the time he returns. I'm not advocating that...I'm just pointing out that D-Link is rather leaving the owners of their routers open to whatever he chooses to do to them. Adding NTP support to a product is one thing -- hardcoding it to reference an NTP server that you can't guarantee is trustworthy is another thing. Suppose, for instance, this guy drops the name due to the expenses and someone else picks it up...

To be blunt, buying D-Link hardware at this point means that you're kind of, well, asking for whatever the hardware does to you.

Re:Moochers

That was Belkin [theregister.co.uk].

Path to Justice

1. Buy the domain name off this poor guy / arrange for alternate hosting if it can't be sold.

2. Take a collection from the /. community to set up an alternate server.

3. Wait a month for all the legitimate users to switch to a new URL.

4. Fire up a server at the old URL reporting Midnight, Jan 1, 1900

5. Let D-Link deal with users accusing D-Link of failing to sell a Y2K compliant product in 2006.

Re:WTF???

It doesn't seem like a moral crusade to me.

He discovered a problem.
He contacted the company causing the problem.
He explained the problem, and simply asked them to fix it.
They didn't.
They put him off.
They threw a lawyer at him to threaten him.
They offered 'compensation' that didn't come close to covering his costs.

He was trying to do it all quietly and nicely, not crusading, and they wouldn't have it.

So instead of going through the often extremely troublesome and lengthy legal procedings (which are even worse than normal since this is an international case), he was hoping to publically embarrass the company into fixing the problem they caused. Seems like a reasonable attempt at a speedy solution, not a crusade.

Re:WTF???

I would have contacted a lawyer right after step four

Right, because lawyers are cheap... right.

I like how he doesn't mention any numbers.
He already has dedicated hosting, do they charge him $1 per megabyte or something?

If you'd bother to RTFA, once again, he answers how much the hosting is costing him. He talks about numbers all over the place.

" because I offer this service free of charge and NTP is a low bandwidth protocol, the organization behind the DIX has graciously waived the normal DKR 27.000,00 (approx USD 4,400) connection fee."

" the current theory is that I will have to close the GPS.DIX.dk server or pay a connection-fee of DKR 54.000,00 (approx USD 8,800) a year as long as the traffic is a significant fraction of total traffic to the server."

" I owe $5000 to an external consultant who helped me track down where these packets came from."

" I have already spent close to 120 non-billable hours (I'm an independent contractor) negotiating with D-Link's laywers and mitigating the effect of the packets on the services provided to the legitimate users of GPS.dix.dk."

" Finally I have spent approx DKR 15.000,00 (USD 2,500) on lawyers fees trying to get D-Link to negotiate in good faith."

" If I closed the GPS.dix.dk server right now, wrote off all the time I have spent myself, then my expenses would amount to between DKR 45.000,00 and DKR 99.000,00 (USD 7,300 to 16,000) and several hundered administrators throughout Denmark would have to spend time reconfiguring their servers.

If on the other hand we assume I leave the service running and that the unauthorized packets from D-Link products continue for the next five years, the total cost for me will be around DKR 115.000,00 + 54.000,00 per year (approx USD 18,500 + USD 8,800 per year) or DKR 385.000,00 over the next five years (USD 62,000). " block the NTP traffic from anything outside his network if it is sooooo expensive for him. You can do that at the ISP level in most cases.

He also mentions how blocking traffic is not feasible, and why, IF YOU'D BOTHER TO READ THE FUCKING ARTICLE. Learn how to read or STFU about him being an asshole.

Re:Couldn't they filter

A good idea, but not easily doable, since the allowed networks include most of Denmark. He would have to filter traffic based on the AS of the sender; this would require a full BGP feed and probably also a continuously updated mirror of the RIPE database.

Easy fix

If he can detect that the majority of connections are from D-Link products, then he can detect which connections are from D-Link products. The easy solution? Whenever a D-Link product connects, report a very very wrong time. :)

Re:Easy fix

He says that such a solution is hard to implement on Cisco, and would be too CPU intensive. FTFA: "Filtering the D-Link packets requires inspection of fields which are not simple to implement in Cisco routers, and in particular such filtering seems to send all packets on the interface through the CPU instead of fast switching, so ingress filtering the packets at the ingress of AS1835 is totally out of the question."

wrong easy fix. try this...

send a private communication to the authentic users (not the robot moochers from D-Link) that on date X, the new IP service address will be unhacked.gps.dix.de or whatever suits him.

on date X, send bogus packets in response... not just wrong time, but seriously wrong time, like a packet with time of 9s in all fields, which would be most seriously wrong.

hopefully, it would lock up the offending junkpiles, and clear the problem right smartly.

the general idea in engineering an end to these things is to find a way to blow up the crooked machine by a seriously wrong entry that will screw up the internals. since they took an ugly and cheap shortcut by using firmware tables, they probably don't error-check their inputs from NTP and other services. so there should be a memory jump and a crash in those pirate boxes someplace.

and that puts the onus back where it belongs, on supercheap designers for obnoxious companies that don't give a shit about network etiquette. the market will punish them. that's how it should be for slap-happy outfits.

Hasn't anybody at D-Link heard of

pool.ntp.org?

Re:Hasn't anybody at D-Link heard of

The pool guys have been talking of implementing a $company_name.vendor.ntp.pool.org setup. Having the $company_name specificity would allow them some leeway if an individual vendor does something silly. I don't know if any vendors have bought into this though.

Re:Splendid admins over there at pool.ntp.org

Someone else replied, but let me actually EXPLAIN.

pool.ntp.org is a collection of volunteer NTP servers, served up via DNS. You should not expect to get meaningful results from pointing a Web browser at such a host name, but because it is random, you could end up hitting Amazon.com (assuming they volunteered) or some guy that just set up an Apache server.

http://www.pool.ntp.org/ [ntp.org] is what you meant, as a simple google search for "pool ntp" would have told you.

Repost of Digg comment

If there's one thing I hate more than incompetence, it's people who don't care that they are incompetent and carry on churning out crap regardless of the problems it causes others.

According to this page [dlink.com], D-Link have an office operating in Denmark. This makes them subject to Danish law whether they like it or not. I don't know whether Denmark's computer crime laws cover this, but it wouldn't surprise me.

No...

Consider this. To use NTP, they have to use it to spec.

open specifications are still the property of the creators. (kinda like the GPL)
they are licensed to 'the world' to use, so long as the specification is followed.
the spec in this case, includes disallowing certain services to certain levels of useage

So, the creators of NTP spec can (in an extreme beyond all belief example)
deny d-link further permission to use NTP at all.

Further, if they are not following the spec (honoring requests by the NTP server not to be used
in this manner) you could as the owner of one of the devices(one again, extreme example)
sue d-link for advertising/listing on the box of the products in question,
for saying they are ntp capable- when it's proven they are not compatible with the spec.
(the spec that includes respecting requests not to be used in this manner)
what are your damages? at least the cost of the affected hardware.

pool.ntp.org

Should be using pool.ntp.org surely........

or am I being daft again..

Blacklist time

Time to add D-Link to the hardware vendor blacklist. Whenever you're asked by your non-tech friends what hardware they should buy, recommend anything BUT D-Link, and tell them to actively AVOID D-Link.

I just bought a DI-624+

The DI-624+ is not on the list and it is possible to manually change the NTP server which the router uses.

They're clearly wrong here

So why didn't they just own up to the mistake, update the firmware and cut him a check for his expenses plus a 5% or so to apologize for the inconvenience? Bureaucrats and lawyers who cannot admit that they are wrong only end up creating more public disgust with their behavior. When you find yourself digging a hole, stop digging!

Re:They're clearly wrong here

No, he can't "just firewall the server" and "tell the few people that would affect". There are thousands of legitimate users distributed across thousands of ASes covering thousands of IP ranges which may change from day to day or even hour to hour. His server is directly connected to the core switch at the Danish Internet Exchange, where all major Danish networks exchange BGP routing information and domestic IP traffic, and its purpose is to provide a stratum-1 reference for NTP servers on these networks. To determine which IP ranges may legitimately access his server, he would need a full BGP feed and a continuously updated copy of all as-block and aut-num records in the RIPE database.

D-Link is just a bad net citizen

It's not the first time that D-Link's crappy programming has affected a service. DynDNS.com [dyndns.com] last year started blocking all update requests [dyndns.com] that match a user-agent of client/1.0, beleived primarily to be several D-Link routers. D-Link has been mum on a response last I heard.

Why not rename the server

Change the DNS name. Granted, he gives reasons for not wanting to do this, but the only practical alternative is to shut down the server entirely. This will still require 2000 or so system administrators to reconfigure their servers, so he might as well provide a logical alternative.

Stupid idea....

...why don't you change the one they (D-Link) use to (basically) lie about the time! Deliberatly send out the wrong information. Altered the config for the customers of dix and