NHTSA Has No Software Engineers To Analyze Toyota 459
thecarchik writes "An official from the National Highway Traffic Safety Administration told investigators that the agency doesn't employ any electrical engineers or software engineers, leaving them woefully unable to investigate correctly what caused the most recent Toyota recall. A modern luxury car has something close to 100 million lines of software code in it, running on 70 to 100 microprocessors. And according to consultant Frost & Sullivan, that number will rise to 200 to 300 million lines within a few years. And the software that controls the 'drive-by-wire' accelerators of Toyota and Lexus vehicles is one potential culprit in the tangled collection of issues, allegations, and recalls of many of those vehicles for so-called 'sudden acceleration' problems."
consultants (Score:4, Insightful)
Surely it would be a serious inefficiency for NHTSA to maintain on staff a large number of specialists to handle this kind of problem? Isn't that exactly what (properly qualified) consultants are for?
Computer Engineers needed (Score:5, Insightful)
100 million lines of code?? (Score:2, Insightful)
I find that extremely hard to believe. Jurassic Park ran on just two million lines of code. I doubt all the lifetime output of all the readers of this thread, combined, equals 100 million. I further doubt that such complexity is remotely necessary to run a car, and that it is remotely possible to debug that much complexity to the standards of, say, the airline industry. And that NHTSA could audit that code in any respectable amount of time. I hope beyond hope the number is wrong.
100 million lines? Sure, we will get right on it (Score:5, Insightful)
What exactly would the NHTSA do with a set of engineers? Audit all 100 million lines of code for each and every car they suspect has a safety issue with the computer system? Yeah, that sounds like a worthwhile endeavor. How about they do it the old fashioned way; collect the reports, identify the risk, and sanction the manufacturer to find/fix the problem. Thinking that an NHTSA coder (or a hundred) would have gotten to the bottom of this Toyota issue in any reasonable amount of time is a joke!
How many microprocessors was that again? (Score:2, Insightful)
Re:Computer Engineers needed (Score:2, Insightful)
Speaking as someone with a CMPE degree, employers see me as under-qualified to do EE work and over qualified to do programming work. What they need is either EEs with heavy embedded programming experience or software engineers with (guess what) embedded programming experience. The title isn't that important.
Re:With all the recent US layoffs ... (Score:3, Insightful)
I know plenty who are laid off for other reasons- such as their C-level executives being slackers and the whole bloody company going under.
Re:Heads better roll (Score:3, Insightful)
Re:Heads better roll (Score:5, Insightful)
In Canada, the province where I live (Alberta), derives a major part of its revenues from oil and gas. In the same conservative government 35 years ago, we had 2 independent arms of the government who could determine how much royalties were owed to the government from the oil and gas producers.
Today, we have no one in our government who is able to determine how much we should be collecting and therefore have to rely upon the oil and gas companies to tell use how much they are supposed to remit. Our own government auditor believes we have been bilked out of billions yet somehow we have a leaner and, ahem, more efficient government.
Just remember that the only thing to stand up to a big business nowadays is big government, and the goal of any big business is to convince everyone that a small government can watch over big business just like a big government can.
Re:Heads better roll (Score:3, Insightful)
It has to be deeper than just the President. The NHTSA lacking EE's and SE's is institutionalized fail. They don't even have the talent to meet their mandate. It required a full blown Congressional investigation into dozens of fatalities for someone to stand up and basically say, "By the way, we can't do our job."
Re:consultants (Score:3, Insightful)
Surely it would be a serious inefficiency for NHTSA to maintain on staff a large number of specialists to handle this kind of problem? Isn't that exactly what (properly qualified) consultants are for?
I agree that it'd be inefficienct to have a large number of EEs & SEs on staff, but they have no one to do even a simple sanity check on the hardware and software that is being certified for public roads. And that strikes me as a failure of their organizational mission.
Re:With all the recent US layoffs ... (Score:3, Insightful)
Re:consultants (Score:5, Insightful)
Given that there isn't a car made today whose safety can be properly evaluated without the skills of EE and software engineers, why would it be inefficient for the agency responsible for that evaluation to have people with those skills on staff? It's not like next years cars won't have even more of the same complete with modified firmware to examine.
Given that the safety evaluation will involve interactions between mechanical, electrical and software systems, you'd want a cohesive multi-disciplinary team, not a revolving door.
Re:Heads better roll (Score:5, Insightful)
List of software powered cars (Score:1, Insightful)
Slashdot, please provide a list of software-powered cars so I know which cars to avoid like the plauge.
Seriously, most software out there is so poor quality I don't want to run it outside of a VM. I really do not want my life to depend on software...
Re:Heads better roll (Score:5, Insightful)
One would be a good start. Oh hell, let's get wild and crazy and say.. 2.
Certainly more than zero.
Re:Heads better roll (Score:2, Insightful)
The problem, of course, is that nothing can stand up to big government. That's a tiny problem though, it's not like the government would ever abuse its power to grab control of the citizenry, right?
Re:Heads better roll (Score:5, Insightful)
Like every other safety certification organization. The car companies pay for a certificate, NHTSA takes some of the blame when something happens, and the general population feels safe knowing their is an entire organization dedicated to protecting them.
Re:Heads better roll (Score:2, Insightful)
I'm sorry, but if you think the antidote to big business is big government, you're delusional. Big government is big business's *partner*. It's always been that way, and it'll always be that way. Handing government more power means that there will be plenty of regulations. You *do* know that a regulation-heavy environment favors big business, not small business, right? Small business can't afford the compliance department you need.
This is the government, not an engineering firm (Score:3, Insightful)
I totally disagree: the NHTSA shouldn't hire engineers. NHTSA should not do the job of Toyota's engineers and testers; they were created to set policy and propose safety laws. The NHTSA should hire economists, policy makers, and maybe some scientists. But the job of ensuring the nuts and bolts of a car are safe should fall on the car-maker, with strict repercussions if they fail.
My biggest problem with all this is what people on Slashdot should already know: looking through and understanding millions of lines of code would take an engineer a few lifetimes - how many engineers are we proposing NHTSA hires? They could learn Toyota's software system, but then what about Ford cars? Or BMW? All for a government organization with 600 employees...
In cases like this, NHTSA should force Toyota to hire a third party (objective) consultant to create a technical report. Maybe a small team of engineers could remain on staff to read and understand those reports.
Re:consultants (Score:4, Insightful)
Under the circumstances, you pretty much have two options. The radical, future-looking one is to say "Ok, clearly complex software is the future. We are going to do whatever it takes, build up a serious software engineering team, impose standards that would make medical device makers cry, sponsor research in automated verification, whatever. Yeah, it sucks that we have do deal with that complexity; but so it goes." The traditional conservative(and, much more likely to fit within your budget and not ruffle feathers) option is to throw up your hands and treat the software as a black box. Have your existing test engineers use their existing techniques, or limited variants, to run the vehicles through test conditions, hoping that, if the test conditions effectively model the real world, any real world critical bugs will appear in testing, at which point you can kick it back to the people who wrote the code and tell them to fix it.
It seems pretty clear that the NHTSA has pretty much gone with option two. And, frankly, it is hard to blame them under the circumstances. Even at the best of times, technical regulation is a pretty unsexy legislative priority, and tends to be funded accordingly. It wouldn't take an actively antiregulatory corporatist to raise an eyebrow at a request for the sort of resources that you'd need to seriously audit the code in each new car coming off the line. And, if you don't have the resources to properly evaluate code from a CS or formal verification perspective, empirical black-box testing under real world-ish conditions is about the best you can do.
Re:Heads better roll (Score:4, Insightful)
Re:Here come the shackles. (Score:5, Insightful)
Dunno.
My kids were runover by an out-of-control Mustang about four years ago. There was nothing mechanically wrong with the car. Maybe it was driver error. I don't know, but apparently the accelerator was still stuck to the floor when the police got there. I remember how the cruise control on the cars I've owned will lower the accelerator when the CC is accelerating.
I've always blamed the firmware. Maybe that's because I'm an EE who used to write firmware for a living. (Firmware that's been in use in life-critical applications for five years with a 0% failure rate.) Odds are the code is shit and there's an edge case that nobody thought about. Maybe there's an uninitialized variable in there. I've seen it happen before. Of course, I'm not Woz-brand, so my opinion doesn't mean a thing.
For some reason, the various regulatory agencies (i.e. Engineering Associations) have been rolling over and letting the manufacturers put any code they want into public use without any thought that hey, maybe we should get someone with some credentials to look into it. I've tried to mention it to mine, no results. Maybe they're dinosaurs who think that engineering is about roads and sometimes other things, like buildings and handrails. Software can't hurt people, can it?
This problem is not limited to Toyota, and we've only just seen the beginning. I guarantee that other manufacturers are clenchinging their butts hoping that nobody in the media wonders about all the intermittent "floor mat" problems.
Re:Heads better roll (Score:4, Insightful)
Just remember that the only thing to stand up to a big business nowadays is big government, and the goal of any big business is to convince everyone that a small government can watch over big business just like a big government can.
- I mean, really? Wake up, is there anyone home? The government that you like so much consists of a system of people, who like to remain in power. To do so takes money. Lots and lots of money. Where do you get the money? It's the system - the bribes real and implied etc.
Government today is in it with the large corporations. They are one government. In Canada it is a bit different from the US but the principles are the same. Big money wants more money, to do so it needs to corrupt the government and it works on that day and night. Big government wants to stay in power, to do so it needs contributions and various other things money can buy, they do this day and night.
It's like that Alien vs Predator: no matter which one of them wins, who do you think is going to lose?
Re:100 million lines of code?? (Score:1, Insightful)
Well, that's just one guy throwing a wild-ass guess out there. I find it incredibly difficult to believe the number is anywhere close to 100 million; that seems like it is off by an order of magnitude. I understand there are some complex systems in a modern car, but aside from something like a media/navigation console I don't see where 20 million lines are going to come from (much less 100 million lines).
Re:With all the recent US layoffs ... (Score:3, Insightful)
Well I don't know if "talent" is the right word. The people who get laid off are the worst 10%. Usually the real slackers.
I thought the most expensive got laid off first
call the FAA or NASA and ask for help (Score:1, Insightful)
They have been dealing with avionics and space software certification for decades.
Why? (Score:3, Insightful)
Fire all the bridge and tunnel inspectors, too (Score:3, Insightful)
If you think that the government should not get involved in engineering.
Re:Heads better roll (Score:1, Insightful)
Just remember that the only thing to stand up to a big business nowadays is big government, and the goal of any big business is to convince everyone that a small government can watch over big business just like a big government can.
You want to stand up to big business?
Demand that Big Government stop propping up giant corporations with tax breaks, tax shields, regulations that strangle small business' ability to effectively compete, and generous anti-trust exemptions.
Corporations, after all, are merely a artificial legal entities created by the government.
Government is part of the problem; more of it won't solve the problem!
Don't reinvent the wheel (Score:3, Insightful)
The NHTSA does not need to evolve a new set of standards out there to address part of this problem. Just require that all automobiles meet the FCC Part 15, Class B standards for electromagnetic susceptibility. It is stupid that this is not done already.
There are plenty of critical pieces of equipment that cannot turn up their noses and fail because of electromagnetic interference. Medical equipment is tested to at least this standard every day. There are hundreds of testing laboratories throughout the world who manufacture products that have to meet these specifications. There are thousands of engineers who already do this type of testing.
Now lines of code and software is a different animal. In a hundred million lines of code there are certainly bugs and flaws.
Re:Heads better roll (Score:2, Insightful)
The Federal Reserve would not exist in a libertarian society. I guess someone modded you up as "funny" because "ignorant" isn't an option. The first and second banks of the US served their functions (until the second became corrupt and President Jackson rightfully killed it), but the Fed is corrupt, enigmatic, and detrimental to the republic. Since FDR, it has allowed the USA to essentially print money at will and rob the people through inflation.
Regulation != Bad (Score:5, Insightful)
Most of the financial regulators are former high level executives from Goldman Sachs...
Some are but most are demonstrably not. Many are financial industry insiders but that's by necessity. Do you really want an financial regulator who has no knowledge of the industry he/she is regulating? The only place to get people with the appropriate financial experience is from the finance industry.
I don't understand why we need so many useless regulators who are usually wolves being put in charge of the hen house when the courts could easily handle this.
While I admire your faith in the court system, in truth the courts are woefully ill-prepared to deal with the sorts of issues the SEC and other regulating bodies deal with. The court system is sloooooowww, expensive and can only effectively deal with misconduct after it has occurred. The courts are a poor monitoring system. The court system also is not heavily staffed with financial experts who understand the issues involved. Trust me, you REALLY don't want financially illiterate judges deciding financial regulations.
The reason the industry insiders often end up as regulators is precisely because they are the only ones who really understand what is going on. Finance is really, really complicated. Yes it's not perfect but that's why the regulators are accountable to other bodies including the President and Congress. If anything the problem with the regulators isn't (usually) that they do poor quality work but rather that they aren't given enough resources to really do a great job. The SEC for instance is badly understaffed given it's mandate. If you really want to keep a better watch on the finance industry, lobby congress to increase funding to the SEC and other watchdog agencies.
It's going to end up being prosecuted in a court of law anyway and not solved by some magic regulation hand-waving.
Spoken like someone who has no experience whatsoever in the financial industry. I won't argue that all regulations are good or well enforced but relying on the court system alone to solve the issues that regulators deal with daily would be insanity. If you really want to screw up the financial system, get rid of the regulators. Our current financial mess is due in significant part to a lack of regulation.
Borrow some from the FDA (Score:2, Insightful)
It's time... (Score:3, Insightful)
a) for a global safety-critical standard for drive-by-wire software.
b) for an open industry standard for interfacing for servicing, fault codes, etc, to end the scam of lock-in to specific manufacturers servicing tools and dealers.
c) to open source it.
Re:Heads better roll (Score:5, Insightful)
Wrong. There's a reason Theodore Roosevelt is on Mt. Rushmore. Go back in history and read up. In the USA, we are re-living many of the same issues, roughly 100 years later.
Re:With all the recent US layoffs ... (Score:3, Insightful)
This is the same problem faced by businesses who need a 'software' person. Without having a good software person in the company already, how can they tell the difference between candidates? They can't.
If you're an organization looking to hire your first expert or two, you do it the old-fashioned
way. You consider their degree and the institution that granted it. You consider their work experience. And perhaps you rely on a referral from a trusted contact who knows more about the field than you do.
If you're looking for some fine-grained specialization in a particular technology, there are a number of certification programs out there. If you're looking for broader skills sets, there are both BS and MS programs available in disciplines such as Computer Science, Computer Engineering, and Software Engineering. What does a licensing regime get you that certificates and degree programs do not?
Re:100 million lines? Sure, we will get right on i (Score:4, Insightful)
It would be a pretty crappy car if it engaged the seat belt PREtensioners POST-impact.
Pretensioners are fired after the initial contact, whilst the very front of the vehicle is still crumpling away. How the hell do you think the computer knows that it has hit something otherwise? Radar? Not on your $10K cheapo. Magic? No, a little ball + spring combo live underneath your front bumper and the last thing they tell the vehicle before they are crushed in an accident is "something big is heading your way".
I'd also rather it didn't "kill engine power" every time I hit the brakes.
We're not just talking about 'every time', we're talking about the two-feet-on-the-brake-pedal-jesus-christ-I-want-to-stop-NOW kind of braking that will activate ABS. Once ABS (and it's cousin, stability control) are running the show, engine power can (and will) be modulated as they see fit in attempt to keep the vehicle going where you want it to go. If you think you can simultaneously control brake force and engine power separately to each wheel whilst in an emergency to do the same, than you go right ahead. I'll take the bus.
although "traction control" systems might retard timing if severe wheel slip is detected.
Traction control is a lot smarter than you seem to think now, and retarding timing went out of fashion about 15 years ago. Now if the traction control system wants less power it simply requests the engine computer to reduce power output by X percent and the engine computer will choose between:
- Simply closing the throttle body, if it has control of it.
- Killing fuel injection on a few cylinders to drop power.
- Dropping boost if it's a turbo'd vehicle.
- Cutting (or yes, retarding) ignition. Bit of a last resort due to unburnt fuel getting out the other side of the engine.
And what $20,000 compact automatically turns on hazard blinkers, mutes the stereo, and opens windows?
My Peugoet 307 turned on the hazards and muted the music if you hit the brakes hard enough to activate its electronic brake force assist system. I did it a couple of times in the two years I had the car, but never got into a collision to find out about the windows.
Legitimate checks to power (Score:5, Insightful)
The government doesn't have to do anything complicated. It just has to have the ability to strike fear into the hearts of the business community it's supposed to regulate.
This requires a few things: an independent media, which we don't have; a civically informed populace that takes it's democratic duties seriously, which we don't have; and a culture that values human dignity over profits, which we don't have.
In cultures that do have all of these things, government regulation works very well and fosters progress, since you don't have to constantly worry about getting screwed over, you don't have to wonder if you'll have access to medical care, or a good public school, or a good safety net to get you back on your feet if your fall ill, get in an accident, or whatever.
Clear and concise regulation with real penalties for breaking those regulations fosters competitive markets. Diminishing the government to the point where it can be bought and sold by businesses usually leads to fascism. The markets destroy themselves with greed, destabilize the economy (and eventually the whole society), and further concentrate wealth and power until you have a virtual oligarchy sprinkled with political theater.
Re:Huh! (Score:3, Insightful)
Regulations are to prevent safety problems in the first place. The reason why regulation is not that regulation *never works*, but instead the regulators are not independant, there are too many conflicts of interests, and it will be the case until we implement some real compaign finance reform so that politicians are not helped to be elected by corporations through all of their generous donations. The fact is, without regulations, matters would not be any better, in fact they would likely be worse, as you want. When we have broken regulators we do not get rid of all regulation but fix the problem as to why its broken. We need more independant experts in regulatory agencies, they need to be run by people never employed by corporations and not paid or influenced by them in any way.
Computer vs car industry (Score:1, Insightful)
Computer vs car industry
Bill Gates reportedly compared the computer industry with the auto industry and stated "If GM had kept up with the technology like the computer industry has, If we would all be driving $25 cars that got 1,000 miles to the gallon."
In response to Bill's comments, General Motors issued a press release stating "If GM had developed technology like Microsoft, we would all be driving cars with the following characteristics:
l. For no reason whatsoever, your car would crash twice a day.
2. Every time they repainted the lines in the road, you would have to buy a new car.
3. Occasionally your car would die on the freeway for no reason. You would have to pull over to the side of the road, close all of the windows, shut off the car, restart it, and reopen the windows before you could continue. For some reason you would simply accept this.
4. Macintosh would make a car that was powered by the sun, was reliable, five times as fast and twice as easy to drive - but would run on only five percent of the roads.
5. The oil, water temperature, and alternator warning lights would all be replaced by a single "General Protection Fault" warning light.
6. The airbag system would ask "are you sure" before deploying.
7. Occasionally, for no reason whatsoever, your car would lock you out and refuse to let you in until you simultaneously lifted the door handle, turned the key and grabbed hold of the radio antenna.
8. Every time GM introduced a new car, car buyers would have to learn to drive all over again because none of the controls would operate in the same manner as the old car.
9. You'd have to press the "Start" button to turn the engine off.
Re:This is the government, not an engineering firm (Score:4, Insightful)
Why not simply require that any software in an automobile be OSS (not FOSS). In fact that requirement should seem to be an extension of mechanic laws that required car makers to provide parts and knowledge to service vehicles outside dealerships. All software in such a critical item should be OSS so it can be reviewed for errors and be reprogrammed by mechanics who wish to offer such services.
More needed than just source code review (Score:3, Insightful)
This drive-by-wire stuff is very serious. I seriously doubt that any car manufacturer validates their computer software and hardware as rigorously as the Dept of Defense; in fact they probably don't do compiler or chip logic validation at all. I bet the aviation industry could give them guidance in this arena.
Re:With all the recent US layoffs ... (Score:4, Insightful)
Re:With all the recent US layoffs ... (Score:2, Insightful)
So you are blaming Reagan for the Cold War? I grant he was alive when it started, but that's giving a B-grade actor a lot more credit than Hollywood ever did.
Or are you blaming Bush the First for not keeping all those Federal employees who had been in or supporting the military on the payroll once the cold war was over?
One good snark deserves another :-)
Re:With all the recent US layoffs ... (Score:3, Insightful)
Outsourcing!
The "reductions" in federal head count are just politicians beating their chests - the employees all get replaced by contractors.
The government is not smaller today than it was in the 70s or 80s... just look at the budget!
Re:With all the recent US layoffs ... (Score:3, Insightful)
Hint: There exists state jobs, which are massively in excess at the moment - compared to lossage in every other field.
Basically, the "stimulus" has been used to shore up failing state budgets to avoid public employee layoffs. Then these jobs are listed as "saved or created", and Obama takes a bow. Meanwhile, productive jobs in the private sector are experiencing 10% unemployment - that's people looking for work, the official Unemployment Rate. Alternative measures reaches as high as 18% in the month of January 2010.
http://www.bls.gov/webapps/legacy/cpsatab15.htm [bls.gov]
Select U-6.
Public sector jobs experience "only" a 4% unemployment rate.
http://mercatus.org/publication/public-vs-private-unemployment [mercatus.org]
Shouldn't the least productive, public tax fed jobs be pruned first?
Oh but wait, those jobs are unionized - primarily - and the system allows the union to get their representatives on both sides of the negotiation table.
I just HAVE to ask this question... (Score:3, Insightful)
Bear with me for a second here...
The three laws of robotics:
1. A robot may not injure a human being or, through inaction, allow a human being to come to harm.
2. A robot must obey any orders given to it by human beings, except where such orders would conflict with the First Law.
3. A robot must protect its own existence as long as such protection does not conflict with the First or Second Law.
I know that a car is not a robot. But the same rules should apply for ANY computer system that, in case of a serious bug, could result in any of those 3 laws being broken.
This computer literally controls a rather large piece of metal that can travel at speeds sufficient to kill someone. So why is there no subroutine that ensure that brake pedal input will ALWAYS override the gas pedal input? It seems that even on the absolute most basic of level, adding this extremely basic concept could seriously mitigate these issues. Not to mention all of the legal responsibilities, public outcry, and other consequences of not having software or hardware with these "basic" concepts built in.
Even when making a car and using this system on a test site somewhere. Wouldn't you want to have LOADS of extra code in there to make sure a bug in the software doesn't kill the driver at the test site? It seems to me Toyota's definition of "safety" is practically non-existent.
Honestly, when seeing something like this, I have to question what kind of work ethic Toyota has and how much they value me as a customer.
Re:Heads better roll (Score:1, Insightful)
We didn't see deregulation of the banking industry... we saw government meddling in the banking industry.
Deregulation means hands-off, not Barney Frank (D) "encouraging" them to make loans "affordable" to people who couldn't afford them while accepting bribes from Fanny and Freddie.
We don't need to regulate the banking industry, we need to regulate the federal government and get rid of all the shady dealing going on behind the scenes.
Throttle software causing sudden acceleration? (Score:5, Insightful)
I can't believe I'm the first one on this thread to make that joke. I'm not even a programmer.
You should all be ashamed of yourselves.