Slashdot Log In
D-Link Firmware Abuses Open NTP Servers
Posted by
ScuttleMonkey
on Fri Apr 07, 2006 09:36 AM
from the frustration-in-a-box dept.
from the frustration-in-a-box dept.
DES writes "FreeBSD developer and NTP buff Poul-Henning Kamp runs a stratum-1 NTP server specifically for the benefit of networks directly connected to the Danish Internet Exchange (DIX). Some time last fall, however, D-Link started including his server in a hardcoded list in their router firmware. Poul-Henning now estimates that between 75% and 90% of NTP traffic at his server originates from D-Link gear. After five months of fruitless negotiation with a D-Link lawyer (who alternately tried to threaten and bribe him), he has written an open letter to D-Link, hoping the resulting publicity will force D-Link to acknowledge the issue. There are obvious parallels to a previous story, though Netgear behaved far more responsibly at the time than D-Link seem to be."
Related Stories
[+]
Developers: Netgear Routers DoS UWisc Time Server 447 comments
numatrix writes "For the last few months, hundreds of thousands of netgear routers being sold had hardcoded values in their firmware for ntp synchronization, causing a major denial of service to the University of Wisconsin's network before it was filtered and eventually tracked down. Highlights how not to code embedded devices." A really excellent write-up of the incident.
[+]
D-Link Settles Danish Time Dispute 192 comments
igb writes "The Register reports that DLink has settled the time server dispute described a little over a month ago here on Slashdot. They're going to stop using an NTP server they're not really authorized to chime with, and they've reached an amicable settlement over the use by existing products. The details of the settlement are, not unsurprisingly, somewhat vague, but let's hope that the good guys aren't out of pocket any more."
This discussion has been archived.
No new comments can be posted.
D-Link Firmware Abuses Open NTP Servers
|
Log In/Create an Account
| Top
| 567 comments
(Spill at 50!) | Index Only
| Search Discussion
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
(1)
|
2
(1)
|
2
List of Affected Products: (Score:5, Informative)
(http://www.caseyandanna.com/)
Re:List of Affected Products: (Score:4, Informative)
10. Best Practices
NTP and SNTP clients can consume considerable network and server
resources if they are not good network citizens. There are now
consumer Internet commodity devices numbering in the millions that
are potential customers of public and private NTP and SNTP servers.
Recent experience strongly suggests that device designers pay
particular attention to minimizing resource impacts, especially if
large numbers of these devices are deployed. The most important
design consideration is the interval between client requests, called
the poll interval. It is extremely important that the design use the
maximum poll interval consistent with acceptable accuracy.
1. A client MUST NOT under any conditions use a poll interval less
than 15 seconds.
2. A client SHOULD increase the poll interval using exponential
backoff as performance permits and especially if the server does
not respond within a reasonable time.
3. A client SHOULD use local servers whenever available to avoid
unnecessary traffic on backbone networks.
4. A client MUST allow the operator to configure the primary and/or
alternate server names or addresses in addition to or in place of
a firmware default IP address.
5. If a firmware default server IP address is provided, it MUST be a
server operated by the manufacturer or seller of the device or
another server, but only with the operator's permission.
6. A client SHOULD use the Domain Name System (DNS) to resolve the
server IP addresses, so the operator can do effective load
balancing among a server clique and change IP address binding to
canonical names.
7. A client SHOULD re-resolve the server IP address at periodic
intervals, but not at intervals less than the time-to-live field
in the DNS response.
8. A client SHOULD support the NTP access-refusal mechanism so that
a server kiss-o'-death reply in response to a client request
causes the client to cease sending requests to that server and to
switch to an alternate, if available.
-daedone
Re:List of Affected Products: (Score:4, Insightful)
(http://www.ajs.com/~ajs/)
Even in the case where the request comes from a recursive lookup, it should (in almost all cases) come from a DNS server which indicates the rough location (in terms of Internet topography) of the client.
Of course, they could also obey DHCP responses (either to the device or to a directly connected IP) as a fallback, solving even more of the problem.
Re:List of Affected Products: (Score:5, Informative)
(http://www.caseyandanna.com/)
"If you download the firmware from DLink and run unarj on it
you get a file called something like nml.mem.
Run strings on that and grep for GPS.dix.dk to make sure it is not
listed in there."
Re:List of Affected Products: - ERR Wrong Answer (Score:5, Informative)
Now that you look at your ethernet sniffs (I assume you just went running off and ran ethereal) look at the source ethernet address... Hmmmmm - doesn't that look familiar, like maybe it looks kinda like your first hop routers MAC address.
Nice try -
Thank you, Come Again
And please read either Stevens or Comer before posting on networking topics again
Moochers (Score:5, Insightful)
(http://suso.suso.org/ | Last Journal: Tuesday March 09 2004, @12:03AM)
Re:Moochers (Score:4, Insightful)
(http://suso.suso.org/ | Last Journal: Tuesday March 09 2004, @12:03AM)
Eh hem, at the risk of sounding like a troll, they apply to my business damnit and don't you forget that.
The problem is, when you do the right thing, like enforcing security over convience, customers don't always appretiate it.
Re:Moochers (Score:4, Insightful)
Re:Moochers (Score:5, Interesting)
(Last Journal: Thursday February 23 2006, @02:47AM)
That being said, D-Link has acquired quite a bad reputation in my book. The last time they were prominently mentioned on Slashdot was when their routers were randomly silently redirecting a small chunk of HTTP traffic to D-Link advertisements, and causing the obvious mayhem in non-human-readable HTTP traffic.
I'm also wondering just how much mayhem this guy could cause on various networks by playing with the time he returns. I'm not advocating that...I'm just pointing out that D-Link is rather leaving the owners of their routers open to whatever he chooses to do to them. Adding NTP support to a product is one thing -- hardcoding it to reference an NTP server that you can't guarantee is trustworthy is another thing. Suppose, for instance, this guy drops the name due to the expenses and someone else picks it up...
To be blunt, buying D-Link hardware at this point means that you're kind of, well, asking for whatever the hardware does to you.
Re:Moochers (Score:5, Informative)
(http://machinae.lionsanctuary.net/)
Path to Justice (Score:5, Interesting)
(http://www.onlineconfessional.com/confess | Last Journal: Tuesday June 06 2006, @02:10PM)
2. Take a collection from the
3. Wait a month for all the legitimate users to switch to a new URL.
4. Fire up a server at the old URL reporting Midnight, Jan 1, 1900
5. Let D-Link deal with users accusing D-Link of failing to sell a Y2K compliant product in 2006.
Re:WTF??? (Score:5, Interesting)
He discovered a problem.
He contacted the company causing the problem.
He explained the problem, and simply asked them to fix it.
They didn't.
They put him off.
They threw a lawyer at him to threaten him.
They offered 'compensation' that didn't come close to covering his costs.
He was trying to do it all quietly and nicely, not crusading, and they wouldn't have it.
So instead of going through the often extremely troublesome and lengthy legal procedings (which are even worse than normal since this is an international case), he was hoping to publically embarrass the company into fixing the problem they caused. Seems like a reasonable attempt at a speedy solution, not a crusade.
Re:WTF??? (Score:5, Informative)
Right, because lawyers are cheap... right.
I like how he doesn't mention any numbers.
He already has dedicated hosting, do they charge him $1 per megabyte or something?
If you'd bother to RTFA, once again, he answers how much the hosting is costing him. He talks about numbers all over the place.
" because I offer this service free of charge and NTP is a low bandwidth protocol, the organization behind the DIX has graciously waived the normal DKR 27.000,00 (approx USD 4,400) connection fee."
" the current theory is that I will have to close the GPS.DIX.dk server or pay a connection-fee of DKR 54.000,00 (approx USD 8,800) a year as long as the traffic is a significant fraction of total traffic to the server."
" I owe $5000 to an external consultant who helped me track down where these packets came from."
" I have already spent close to 120 non-billable hours (I'm an independent contractor) negotiating with D-Link's laywers and mitigating the effect of the packets on the services provided to the legitimate users of GPS.dix.dk."
" Finally I have spent approx DKR 15.000,00 (USD 2,500) on lawyers fees trying to get D-Link to negotiate in good faith."
" If I closed the GPS.dix.dk server right now, wrote off all the time I have spent myself, then my expenses would amount to between DKR 45.000,00 and DKR 99.000,00 (USD 7,300 to 16,000) and several hundered administrators throughout Denmark would have to spend time reconfiguring their servers.
If on the other hand we assume I leave the service running and that the unauthorized packets from D-Link products continue for the next five years, the total cost for me will be around DKR 115.000,00 + 54.000,00 per year (approx USD 18,500 + USD 8,800 per year) or DKR 385.000,00 over the next five years (USD 62,000). " block the NTP traffic from anything outside his network if it is sooooo expensive for him. You can do that at the ISP level in most cases.
He also mentions how blocking traffic is not feasible, and why, IF YOU'D BOTHER TO READ THE FUCKING ARTICLE. Learn how to read or STFU about him being an asshole.
Re:Couldn't they filter (Score:5, Informative)
(http://www.des.no/)
Easy fix (Score:4, Funny)
(http://brianm.org/)
Re:Easy fix (Score:5, Informative)
wrong easy fix. try this... (Score:5, Interesting)
(http://slashdot.org/ | Last Journal: Monday April 16 2007, @01:18PM)
on date X, send bogus packets in response... not just wrong time, but seriously wrong time, like a packet with time of 9s in all fields, which would be most seriously wrong.
hopefully, it would lock up the offending junkpiles, and clear the problem right smartly.
the general idea in engineering an end to these things is to find a way to blow up the crooked machine by a seriously wrong entry that will screw up the internals. since they took an ugly and cheap shortcut by using firmware tables, they probably don't error-check their inputs from NTP and other services. so there should be a memory jump and a crash in those pirate boxes someplace.
and that puts the onus back where it belongs, on supercheap designers for obnoxious companies that don't give a shit about network etiquette. the market will punish them. that's how it should be for slap-happy outfits.
Hasn't anybody at D-Link heard of (Score:5, Insightful)
(Last Journal: Tuesday September 25, @04:26AM)
Re:Hasn't anybody at D-Link heard of (Score:4, Informative)
Re:Splendid admins over there at pool.ntp.org (Score:5, Informative)
(http://www.ajs.com/~ajs/)
pool.ntp.org is a collection of volunteer NTP servers, served up via DNS. You should not expect to get meaningful results from pointing a Web browser at such a host name, but because it is random, you could end up hitting Amazon.com (assuming they volunteered) or some guy that just set up an Apache server.
http://www.pool.ntp.org/ [ntp.org] is what you meant, as a simple google search for "pool ntp" would have told you.
Repost of Digg comment (Score:5, Informative)
If there's one thing I hate more than incompetence, it's people who don't care that they are incompetent and carry on churning out crap regardless of the problems it causes others.
According to this page [dlink.com], D-Link have an office operating in Denmark. This makes them subject to Danish law whether they like it or not. I don't know whether Denmark's computer crime laws cover this, but it wouldn't surprise me.
No... (Score:4, Interesting)
(http://www.ocean7motel.com/ | Last Journal: Monday May 07 2007, @07:50AM)
open specifications are still the property of the creators. (kinda like the GPL)
they are licensed to 'the world' to use, so long as the specification is followed.
the spec in this case, includes disallowing certain services to certain levels of useage
So, the creators of NTP spec can (in an extreme beyond all belief example)
deny d-link further permission to use NTP at all.
Further, if they are not following the spec (honoring requests by the NTP server not to be used
in this manner) you could as the owner of one of the devices(one again, extreme example)
sue d-link for advertising/listing on the box of the products in question,
for saying they are ntp capable- when it's proven they are not compatible with the spec.
(the spec that includes respecting requests not to be used in this manner)
what are your damages? at least the cost of the affected hardware.
pool.ntp.org (Score:3, Insightful)
(Last Journal: Wednesday August 08, @03:46AM)
or am I being daft again..
Blacklist time (Score:4, Insightful)
(http://www.spamgourmet.com/)
I just bought a DI-624+ (Score:4, Informative)
They're clearly wrong here (Score:5, Insightful)
(http://www.codemonkeyramblings.com/)
Re:They're clearly wrong here (Score:5, Informative)
(http://www.des.no/)
D-Link is just a bad net citizen (Score:5, Interesting)
(http://slashdot.org/)
Why not rename the server (Score:3, Insightful)
(Last Journal: Friday June 11 2004, @11:15AM)
Stupid idea.... (Score:3, Insightful)
Brutal but (in theory) affective....
Jaj
cname to the rescue (Score:4, Insightful)
someone proof read my letter plz (Score:3, Insightful)
(http://www.theworldwidewebguy.com/)
17595 Mt. Herrmann St
Fountain Valley, CA 92708
I have recently read an open letter to D-Link available at the following URL:
http://people.freebsd.org/~phk/dlink/ [freebsd.org]
I must say that I am disgusted with D-Link's poor choice of action. D-Link may
think that abuse such as this will go un-noticed, but that is not the case.
While I don't expect my actions to bring your corporation to its knees, I am the
"geek" of my family, and I have taken a personal stand by ordering Linksys
products to replace any and all of the D-Link networking gear that my parents,
siblings, cousins, and roomates are using. I hope that my sacrifice puts a dent
in the damage your corporate negligence has caused Mr. Kamp.
Poul-Henning clarifies (Score:5, Informative)
(http://people.freebsd.org/~phk)
1. My server has not replied to the packets sinde the CodeRed virus/worm abused NTP servers to coordinate attacks. That was a couple of years ago. I doubt D-Link ever even tried to test this.
2. NTP is a timing protocol. You do not want to do expensive and timeconsuming filtering on the packets because that disturbs your timing performance.
3. If I have to sue D-Link, it will be either in USA or Taiwan. Both their Danish marketing office and the UK european office will be able to deflect a lawsuit to their mothership.
4. If you download a firmware file from D-Link, it is often a ARJ archive. unpack that and run strings. If you see GPS.dix.dk in there, please use another version. If the firmware you run is older than about a month, please update it.
5. The list of products in my open letter is unlikely to be complete, those are the only ones I have been able to positively identify (using the method above). If you find out other products are affected, please email me.
6. We do have a number of very interesting sections of our penal code here in Denmark that are very likely to apply. Only problem is, they havn't been tried in a court yet. So I have to persuade an overworked criminal inspector to raise a criminal case against a foreigner over a, lets face it, quite small monetary amount. Then I have to spend a lot of time making sure that we convince a judge who have never heard of NTP that they are guilty and then if I win, I can see some D-link manager make a checkmark in their pocket book: "Remember to not visit Denmark under true name". I have better things to use my life for.
I can see a couple of hits from a C-class belonging to "D-Link Irwine": please escalate this guys, your bosses don't read slashdot.
Thanks for all the supportive email.
Poul-Henning
Re:Poul-Henning clarifies (Score:4, Insightful)
Can't that easily be re-written to "Remember not to visit the European Union"?
Osama Bin Laden (Score:3, Funny)
(http://linuxhomepage.com/)
D-Link must be run by Osama Bin Laden. That's why no one can be reached (hiding in the mountains of the Afghanistan and Pakistan border). Obviously, this attack has something to do with that cartoon thing.
D-Link Business Development (Score:5, Interesting)
Ok, let's do some good. Are we slashdot, or what?
D-Link Business Development and Strategic Partnerships, E-mail: bdm@dlink.com
>>>
To whom ever it may concern:
Hello.
I just learned of you companies notably persistent inability and unwillingness to deal with a serious design flaw in a growing range of your products. This flaw is severly disrupting internet services for a large amount of internet participants and even though you have been informed in detail of these effects your products are having, you have done nothing of substance to resolve the issue and compensate for the damage done.
Until I learn that the issue described in the open letter do D-Link, available under http://people.freebsd.org/~phk/dlink/ [freebsd.org], was resolved in a professional and mutualy satisfying manner I will not purchase any D-Link products and will strongly discourage anybody asking for my expertise as a professional in the IT field from buying D-Link products or from engageing in any sort of business relationship with D-Link.
Sincerely
An Internet User
Mistakes in this one? Please post corrected version below and then add a 'mailto' link to the address.
Grammar Nazis, it's your turn!
Poul-Henning clarifies more (Score:3, Insightful)
(http://people.freebsd.org/~phk)
The place where the service restriction is clearly written out, the "stratum 1 list" is the only place where DLink can have found the name of the NTP server in the first place.
As several posters have pointed out: consumer devices like these have no need to query stratum 1 servers.
As I said clearly in my letter: filtering will not prevent me from getting hit with bandwidth charges of $8800/year.
I have not tried sending any bogus return packets because that would hit innocent consumers who bought D-Links defficient products.
And for the people who could have identified the source of these packets so much faster and easier: Drop me an email, I'll be sure to ask for your help next time.
Finally, I can see that more than 40 people at D-Link Irwine (192.152.81.0/24) have read the open letter now, please guys: get somebody to call me or email me so we can get this matter settled. (both email and phone# is in the open letter)
Poul-Henning
Here's what I'd do (Score:3, Interesting)
(http://www.crypt.gen.nz/)
He's correct that performing complex packet matching on a Cisco router would load it too much - they just don't have the CPU to do that function for any significant traffic load.
I would configure the switch that the NTP server is on to have a SPAN port - a port to which all traffic is copied. Most Cisco switches will do this without any problem. On that SPAN port, connect a Linux box with a bit of CPU power - 2GHz would be tons. On the Linux box, setup tcpdump to match the packet patterns that D-Link routers are sending ( from TFA he has this as detected by a network consultant ).
From the output of tcpdump, extract the source IP addresses. A fairly small perl script would probably do it. Take these IP addresses and massage them into access-lists for the upstream router to block, again perl or TCL/Expect would be reasonable tools. Routers are good at blocking large lists of IP addresses - its not such a load for them as the list gets compiled and pushed onto the hardware. Depending on his router model a few thousand ACL lines would be fine.
Alternatively, he could use the same approach to detect the non-D-Link source IPs - permit these and block anything else. From his stats of legit -vs- D-Link sources this would result in a shorter access list.
The only issue here is that a D-Link behind a shared-NAT'd IP address would result in that address being blocked, but there shouldn't be too many of these. And legally he can block anything he wants - his service has no written guarantee to he should be legally safe (yeah, IANAL).
To keep costs and time down, he can probably get help from the local University ( a cool project for any CompSci students ) to do the code and Linux setup, or help from the local LUG - I'd bet there would be plenty of volunteers to set it up, and I could imagine it being done within a couple of days.
Kerry
This is how bad it was (Score:3, Informative)
(Last Journal: Tuesday June 26, @08:41AM)
Here he explains how he traced down who was behind, what he calls a DDoS attack: His blog [lightbluetouchpaper.org]
Re:Im confused (Score:5, Informative)
You don't use the root DNS servers for all your DNS requests, right?
Re:Im confused (Score:5, Informative)
(http://www.spamgourmet.com/)
Re:Im confused (Score:5, Insightful)
(http://www.borkbork.org/~bigjoe | Last Journal: Tuesday December 30 2003, @03:11PM)
You can learn all this and check the list to be sure you comply within 10 minutes thanks to the power of Google. Any responsible company would know this and do so. D-Link made a big mistake (not in terms of the impact on them, sadly) and is evidently refusing to own up.
As others have pointed out, it's not easy to implement the restrictions that would enforce the access policy. It's also sad, though not surprising, that one would have to. It'd be one thing if the server was the target of script kiddie DOS attacks, but a legitimate company selling network products really ought to know better (and care).
Re:Im confused (Score:5, Informative)
Sorta like how server admins get pissed when an article posted on their site causes them to be Slashdotted.
And honestly, the fact that D-Link is acting in the way it is while he trys to get them to resolve the issue probably isn't helping matters.
Then again, as a former owner of a D-Link product which rebooted itself anytime I went over 50 simultaneous connections (think P2P), I don't doubt they'd be too cheap to actually just run their own.
Re:Im confused (Score:5, Insightful)
So let me get this straight... this guy hosts an NTP server and is pissed because... its being used as an NTP server?
If I set up an NTP server, say for my university, and left it open for others, I also might think it a bit unorthodox if a multinational corporation hardcoded all there gear (which was deployed internationally) to query it. This is for several reasons. First, it generates unneeded bandwidth and violates convention by not using a local NTP server. Second, it means thousands of people are relying on one person for their gear to work properly, a person the company did not even bother to consult. What if he decides to change the time by five hours, just for fun? It is bloody irresponsible of the manufacturer to give him that option. And what happens if the server is deprecated or the hostname and IP changed in a reworking of the network? Tons of wasted traffic as they ping his IP space.
Re:Fishy (Score:3, Insightful)
Either this is a very weak attempt at a troll, or an incredible demonstration of ignorance.
Re:just change the DNS (Score:3, Insightful)
That is one of the dumbest things I have ever heard.
Using your twisted logic there is nothing wrong with spammers sending people hundreds of thousands of unsolicited commercial email a day. If people don't want spam then they should not have set up an email address right?
Re:Why didn't he take the "bribe"? (Score:5, Informative)
(Last Journal: Thursday April 06 2006, @10:50AM)
Sorry to correct your rant, but he does say in TFA that the offer was so low that it didn't even cover his costs. That would be a good enough reason to say no wouldn't it?
Re:Fishy (Score:3, Funny)
(http://www.vandemar.org/)
Re:Im confused (Score:5, Interesting)
(Last Journal: Thursday February 23 2006, @02:47AM)
* To keep the network working, the NTP system is tiered. Anything other than a time server used to redistribute time to other machines should probably access a Tier 3 system, or a Tier 2 if that is not possible. It should never hammer a Tier 1 -- this can screw up the rest of the NTP network.
* There are large lists of NTP servers, and they list access restrictions. As pointed out in the letter, this guy explicitly stated in his access rules that this server was not for client use.
* As pointed out in the letter, this guy explicitly stated in his access rules that this server was not for use outside of Denmark.
You may not be used to this sort of thing, because no such set of agreements exists for, say, webservers. However, in the NTP world, network administrators respect these, and it is why the time system continues to work.
What D-Link is doing hurts all Danish NTP users, and freeloads off a volunteer (D-Link is selling the product and profiting from it -- let *them* handle the traffic and factor any bandwidth costs into their product cost). It opens their product to potential abuse if the server becomes malicious (a properly-designed router would allow the user to specify an NTP server, or if the user is unable to configure a router, to do what the letter suggested and use a D-Link-controlled name.). It violates agreements that have been generally respected by the NTP-using administrator community for many years.