Huge Survey of Firmware Finds No Security Gains In 15 Years (securityledger.com) 61
A survey of more than 6,000 firmware images spanning more than a decade finds no improvement in firmware security and lax security standards for the software running connected devices by Linksys, Netgear and other major vendors. The Security Ledger reports: "Nobody is trying," said Sarah Zatko, the Chief Scientist at the Cyber Independent Testing Lab (CITL), a non-profit organization that conducts independent tests of software security. "We found no consistency in a vendor or product line doing better or showing improvement. There was no evidence that anybody is making a concerted effort to address the safety hygiene of their products," she said.
The CITL study surveyed firmware from 18 vendors including ASUS, D-link, Linksys, NETGEAR, Ubiquiti and others. In all, more than 6,000 firmware versions were analyzed, totaling close to 3 million binaries created from 2003 to 2018. It is the first longitudinal study of IoT software safety, according to Zatko. CITL researchers studied publicly available firmware images and evaluated them for the presence of standard security features such as the use of non-executable stacks, Address Space Layout Randomization (ASLR) and stack guards, which prevent buffer overflow attacks.
The results were not encouraging. Time and again, firmware from commonly used manufacturers failed to implement basic security features even when researchers studied the most recent versions of the firmware. For example: firmware for the ASUS RT-AC55U wifi router did not employ ASLR or stack guards to protect against buffer overflow attacks. Nor did it employ a non-executable stack to protect against "stack smashing," another variety of overflow attack. CITL found the same was true of firmware for Ubiquiti's UAP AC PRO wireless access points, as well as DLink's DWL-6600 access point. Router firmware by vendors like Linksys and NETGEAR performed only slightly better on CITL's assessment. CITL researchers also "found no clear progress in any protection category over time," reports The Security Ledger. "Researchers documented 299 positive changes in firmware security scores over the 15 years covered by the study... but 370 negative changes over the same period. Looking across its entire data set, in fact, firmware security actually appeared to get worse over time, not better."
On the bright side, the survey found that almost all recent router firmware by Linksys and NETGEAR boasted non-executable stacks. "However, those same firmware binaries did not employ other common security features like ASLR or stack guards, or did so only rarely," says the report.
The results were not encouraging. Time and again, firmware from commonly used manufacturers failed to implement basic security features even when researchers studied the most recent versions of the firmware. For example: firmware for the ASUS RT-AC55U wifi router did not employ ASLR or stack guards to protect against buffer overflow attacks. Nor did it employ a non-executable stack to protect against "stack smashing," another variety of overflow attack. CITL found the same was true of firmware for Ubiquiti's UAP AC PRO wireless access points, as well as DLink's DWL-6600 access point. Router firmware by vendors like Linksys and NETGEAR performed only slightly better on CITL's assessment. CITL researchers also "found no clear progress in any protection category over time," reports The Security Ledger. "Researchers documented 299 positive changes in firmware security scores over the 15 years covered by the study... but 370 negative changes over the same period. Looking across its entire data set, in fact, firmware security actually appeared to get worse over time, not better."
On the bright side, the survey found that almost all recent router firmware by Linksys and NETGEAR boasted non-executable stacks. "However, those same firmware binaries did not employ other common security features like ASLR or stack guards, or did so only rarely," says the report.
Shocking! (Score:5, Funny)
Not having any penalties for having bad security has not improved security.
I know I'm surprised.
Re: (Score:1)
Well, for one thing, you gotta keep the doors open for the spies, so "good" security is not really possible.
Re:Shocking! (Score:4, Insightful)
Unregulated markets always put chalk in the bread because it is assumed that this makes more money. Japan conquered the global market by using the quality improvement paradigm invented by American professors. Companies find it cheaper to damage a few customers and take the risk that the costs of doing so is cheaper than making a defective product. A competitive market always rewards the first to market with a monopoly. In life it is not possible to eliminate all risks because you would never leave the house otherwise. Criminal actors, particularly well funded state actors exploit defective products to achieve their goals.
This is a complicated issue with no obvious solution but it is moving towards an issue of state security. I am guessing that politicians will only wake up to this fact when public utilities start being destroyed and people die as a result. What they will do about it is anybodies guess.
Re: (Score:2)
About first to market, there are lots of obvious counter-examples. Apple is rarely first. When Google Search came along there were plenty of established search engines already.
Re: (Score:1)
Has Apple ever had or even aspired to a monopoly? No. They aspire to be the premium brand that all others look up to and a lot of people (but not everybody) will pay a stupid amount of money for. Profit! As long as they get their piece of the market, they're happy, all the way to the (offshore) bank. Even for PCs, Apple was not first to market. They were arguably first to market with a nicely packaged computer that could sell (at high prices, even for the time) to non-nerds with cash. There were other PCs a
Suprises me because other things are more secure (Score:5, Interesting)
As a long-time professional in the field, it suprises me. Why?
Because IT security spend is about a hundred times as much as it was a few years ago. Apparently router manufacturers aren't doing what other companies are doing.
Twenty years ago a security professional like myself would have a hard time finding work; today I can name my price. I currently have three offers, with one company paying me for month before doing any work. They're paying several thousand dollars without me doing any work, just to reserve me for future work.
It's a different world for security today than 10 or 20 years ago.
Re: (Score:3)
Hiring you is their "spend".
They spent X to get you. When you tell them "OK, you've got a problem, and you need to do Y", then they're gonna say "But that will cost Z, and we already spent X this year".
They probably have some programmer who already wanted to do Y, but that person isn't a security expert, and they're sure as hell not going to spend Z on some peon's say-so. And now that they spent X, they don't have Z to blow any more.
So they'll send you off to do audits against ISO 123456 and make lists of h
They'd get that for half the salary (Score:3)
> They spent X to get you. When you tell them "OK, you've got a problem, and you need to do Y", then they're gonna say "But that will cost Z, and we already spent X this year".
That absolutely does with a company who hires someone at half my salary. I just did a presentation on precisely that topic, to a group of about 100 security people. Then the security person is frustrated and the company isn't as secure as they could be.
What I taught in my presentation to security pros last week is a different app
Re: (Score:2)
... but they WOULDN'T get that for half the salary. What they're buying is the ability to say that they've proven their commitment by spending some specific amount of money, or more likely by getting some specific set of credentials. Spending X/2 (or getting some lesser set of credentials) isn't good enough.
As for the method you use, it's very old and it does work. The methods for estimating of both costs and probabilities are usually totally indefensible, and yes I have read many attemtps to justify such m
Re: (Score:2)
Typical costs / losses from an intrusion into our tier 1 systems bases on SQL injection would be about $10 million.
If we don't do anything to protect ourselves, we estimate a 10% chance of *one or more* of our tier 1 applications being breached each year.
Therefore the expected cost of breach, if we do nothing, is $1 million / year.
Mitigation* through the proposed SAST program for tier 1 apps will cost $100,000.
I therefore propose that we save save $1 million in expected losses by investing $100,000 in protection.
In addition, SAST will likely catch bugs that result in approximately $100,000 in down time, making the net annual cost zero, to save $1 million expected losses.
* Residual risk due to exploit despite SAST is $100K.
Well north of 90% of all breaches have shit to do with exploitation of programming defects.
Assume you spend that 100k and as a result _ALL_ security vulnerabilities are magically removed from your code. You've taken less than 10% of the overall risk off the table.
Re: (Score:2)
The failures we're seeing in corporations today are structurally the same as the failures seen in militaries in centuries past. We live in corporate feudalism and management are the nobility. They're given positions of leadership they're utterly unqualified for as a sinecure and the hard divide between the know nothing nobility and the do everything peasantry prevents any meaningful improvement or competent leadership.
Re: (Score:3)
I've spent my career writing firmware. There is a hell of a lot of really bad firmware because writing it is hard and few people have the necessary skills. There is a huge shortage of people who can do a good job of it, and in some markets companies are very reluctant to pay for good people so they all just leave anyway.
For that reason a lot of firmware is written by contractors working to a spec. The spec rarely says "must be secure", and even if it does I've never seen anyone specify to what standard it s
Re: (Score:1)
The person/user/brand buying a huge firewall in front of the "product".
That allows the product to work, for a happy user and for "professional" firewall sales.
Re: (Score:1)
Re: (Score:2)
As long as we're talking about the 90s, a lot of these iot devices ship with open telnet ports. Mainly because it's easier on a small embedded system.
Re: (Score:2)
That is surprising (Score:2)
Re: (Score:2)
Those are consumer devices as well, and those typically have lousy security. I don't think it should be treated as symptomatic of the entire firmware industry.
Re: (Score:1)
They are devices that, because of their lack of security, are preferentially attacked as the first stage of a larger attack on other devices that *should* (but often don't) have better security. So it's all part of the same ecosystem. Improving consumer-device security improves security for all devices, and even with consumer Terms of Use that disclaim everything and force arbitration, somebody will find a loophole and sue the company into an early demise eventually.
That's a real problem with IoT, whether a
there is no money to made in security (Score:1)
"Nobody is trying" (Score:1)
"Nobody is trying," said Sarah Zatko, the Chief Scientist at the Cyber Independent Testing Lab (CITL)
That is basically what I found as well myself looking at custom made applications and publicly available applications. Patching as you can after the application is developed is generally what happens only after being hit. Security training and concern should be part of application or firmware development right from day one, not after the application is put in production.
The real reason? (Score:2)
What's the average age of a programmer today. Is it any older than it was 15 years ago?
So no increase in experience leads to no better outcomes.
Nothing has ever been sold on security (Score:4, Insightful)
That's the sad truth. Being secure is simply no selling point, nobody gives a fuck about the security of an appliance. What matters is price first and foremost.
This isn't only truth for consumers, it is what you find in commercial settings, too. Fine companies for lax security standards and you see a change.
Re:Nothing has ever been sold on security (Score:5, Insightful)
I don't think that nobody gives a fuck about security. The problem is that the buyer has no way to know whether you're lying about the security or not.
Your product costs $85. The competitor costs $75. You both CLAIM to have good security. The buyer doesn't know enough to know what good security looks like, and either can't learn at all, or can't learn without it costing a lot more than the equivalent of the price of the product in time and work.
Re: (Score:2)
Oh, yeah, and not only does the buyer have no way to recognize better security, they also have no way to know whether it's good ENOUGH or exactly how much value to put on it.
Re: (Score:1)
Re: (Score:3)
Did I say it was? My point is that the buyer has NO realistic way to measure it.
Re: (Score:1)
Re: (Score:2)
Certification might actually make things worse, because of Goodhart's law.
Look at FIPS 140. It was originally aimed at some very specific hardware modules. It covers a really, really narrow set of concerns. It's possible for your crypto to be totally breakable and still be completely FIPS 140 certifiable. But because it's a Government Security Standard (TM), it's used to sell all kinds of snake oil. The illusion of having a useful measure of security is worse than knowing that you don't have one.
The same ap
Something like the UL listing, but for security (Score:3)
.
If the insurance companies got together and said they won't pay out cyber insurance if there is a non certified IoT device, or software, on the network, then this problem would go away quickly. "We're not paying for your ransomware recovery, because you had a non-S
Re: (Score:2)
It would also exclude open source and small vendors from the market, you would hand the market off to one or two large incumbent vendors who would produce even worse products because of the reduced competition and as you pointed out, the cost/delay of certifying patches.
There have been such schemes in the past, whenever you looked at the accredited products list they were all old versions with known vulnerabilities and patching them meant you're no longer compliant.
Re: (Score:1)
UL is great for hardware that has a long life and is potentially harmful. Its original purpose as noted was for electrical fire protection, and it still serves that purpose well (computer power supplies really should be UL certified). But for products that for marketing or even real reasons have to constantly change, UL certification is impractical if not impossible. So what's the alternative? Other than having an expensive staff on board to vet everything (and which still won't catch the zero-days) you hav
Re: (Score:2)
This is probably true in some arenas, particularly the consumer market. However we have industrial customers that insist we get outside security penetration testers. So for some customers, security is important and important enough to make sure it's baked into the products and not just an afterthought.
Re:Nothing has ever been sold on security (Score:4, Interesting)
I work in a field where security is (or at least should) be one of the top priorities. Do you think that the security record of a company even comes into play when we are looking for new routers? Nope. We just complain endlessly after every penetration test and demand fixes for their blunders, but as long as even PCI-DSS doesn't give a fuck, why should we?
Re: (Score:2)
Seems like a good case for getting the EU involved. The EU rates a lot of products for things like energy efficiency and how well they work, then mandates that the ratings have to be on the box. They could add a security rating, as well as things like how many years guaranteed cloud service you can expect and any monthly subscription costs.
Re: (Score:2)
They could. But why should they?
Environment is something that is tangible to people and something politics can win brownie points for. Security is something nobody gives half a shit about, and something politicians don't even understand if you explain it to them.
Re: (Score:1)
After Ford tried to sell the safety of the "deep dish" steering wheel in the mid-1950s ... and saw everybody buying the slighly cheaper and much fancier-looking Chevys, they (and the industry) decided quite firmly that "safety doesn't sell" and stopped promoting it. In fact, often doing the opposite of offering safety. It took a salesman like Ralph Nader, with a tool like the early Corvair, almost 10 years to get some government safety standards even started. Later, when things like anti-lock brakes and air
Re: (Score:2)
The buyer COULD know if he cared. It's not like it's hard to find the various reports about glaring security holes in various routers and modems. But when it comes to computer equipment (or pretty much anything in the area of entertainment electronics), people go for price and tick boxes.
They stand at their electronics retailer and look at the little cards next to the item. This one has 6 of the feature boxes ticked, this one has 8 ticked, so the one with the 8 ticked is better. That they don't even underst
openwrt (Score:2)
That's why I don't trust any vendor's firmware for my router and use openwrt. They seem to do a good job at least for public issues such as KRACK.
Expected from hardware vendors... (Score:3)
But I'd really like to hear how more software-savvy firms do, in particular, Apple and Google have similar consumer grade hardware but I'd expect their software security to be significantly more advanced.
Re: (Score:2)
Re: ASLR, NX, and stack guards are what you do... (Score:2)
No shit, Sherlock. (Score:3)
Nobody is going to care until companies are held liable, either under civil law, criminal law, or both, for shitty security enabling cybercrime.
At the rate we're currently going I wouldn't at all be surprised if there's at least one criminal organization that can play all the worlds' computerized devices like a musical instrument, making them do whatever they want -- and they just haven't had sufficient reason to bring every digital device on the planet crashing down around our ears yet.
Re: (Score:3)
The problem is (Score:2)
Mozilla is making an OpenWRT flavour (Score:5, Informative)
Mozilla is working on a router platform. Could be interesting:
https://iot.mozilla.org/gatewa... [mozilla.org]
Off-topic: it often feels like the user interfaces of routers haven't improved much either...
Re: (Score:3)
Off-topic: it often feels like the user interfaces of routers haven't improved much either...
Well, that strikes me as a bit of a catch-22.
For many people, a router is just the thing that gives them Wi-Fi, and that's about it. Many people don't spend much time inside their router. People that do, are people like you and me, and are generally intelligent enough to figure out where they have the control we happen to be looking for.
Next, you have the inertia problem. My boss is a die-hard Sonicwall guy; the running gag is that he swears by them, I swear at them. They've kept their UI consistent enough
Re: (Score:2)
Re: (Score:1)
Linuxbios was hindered by gneder politics (Score:1)
There have been attempts to open source, or "free software", critical firmware. The libreboot project was a notable one, effectively providing compilable, tunable access to the BIOS to provide completely "free software" based hardware. I tried to work with it some years back, and it was a promising project. Sadly, it was crippled by some very strange gender political accusations from its primary maintainer, Leah Rowe. Slashdot mentioned, the followup from the "Libeboot foundation" is at.
I'm very sad to say
Re: (Score:2, Insightful)
Libreboot has greatly failed not because of gender politics, but lack of documentation from motherboard vendors. Keep babbling though.
Re: (Score:2)
Well for one, this is even in theory offtopic, as this refers to the operational firmware of these devices, not the boot firmware (essentially, the operating system and software stack).
I don't know of political issues, but open source boot firmware is blocked because the prevailing security story at the moment requires that the manufacturer to block custom firmware with signed firmware checking. In that environment, it is not viable to even entertain the possibility of customer built boot firmware images.
Re: (Score:2)
I appreciate the distinction between device firmware and bios firmware: the BIOS is an example of firmware that has normally been closed source and difficult, though not impossible, to open source. The political issues involved its primary developer, Leah Rowe, withdrawing the software from the GNU project because of perceived transgender abuse, abuse which did not happen. This was covered in Slashdot, the original mailing list from Leah is available at:
https://lists.gnu.org/archive/... [gnu.org]
An apology from Leah
Those are mitigations (Score:2)
non-executable stacks, Address Space Layout Randomization (ASLR) and stack guards, which prevent buffer overflow attacks.
Those are techniques that apply when you've already screwed up. They don't prevent buffer overflow attacks, they make attacks harder, slow them down. They're mitigations.
So do these products use memory-safe languages? Do the vendors perform security-minded code walkthroughs? Do they fix bugs before they add features? These are the things that matter. These are the things that prevent buffer overflows from occurring in the first place. But this survey doesn't look at any of that.
It's a worthy project loo
Re: (Score:2)
I generally agree, but they are decent at indicating how much general security concern they have. Anecdotally similar protections that are more meaningful are generally missing from consumer grade firmware I can access.
For example, not one of the devices I could check had any apparent CSRF protections, CSP header, cookies did not include SameSite.
At consumer level, there is absolutely zero sales benefit to security measures. There's never been a liability issue for their devices getting hacked.
In corporat
Not my problem (Score:2)
I'm running pfSense on PC Engines hardware. Have been for more than a decade.
No incentive (Score:2)
CITL researchers also "found no clear progress in any protection category over time," reports The Security Ledger. "Researchers documented 299 positive changes in firmware security scores over the 15 years covered by the study... but 370 negative changes over the same period. Looking across its entire data set, in fact, firmware security actually appeared to get worse over time, not better."
Really this should surprise no one. There is absolutely no incentive for these companies to invest in improving the quality of their devices. There's very little real consequences for having gaping security holes in your products. The general public just has no clue. They just buy whatever, with no research.
Without actual market share and bottom line consequences to poor security and the bad PR revolving around that (which again, the General Public is completely unaware), well, this is totally the expec
A bit of a history lesson (Score:2)
Because nobody takes it seriously (Score:2)