NHTSA Has No Software Engineers To Analyze Toyota 459
thecarchik writes "An official from the National Highway Traffic Safety Administration told investigators that the agency doesn't employ any electrical engineers or software engineers, leaving them woefully unable to investigate correctly what caused the most recent Toyota recall. A modern luxury car has something close to 100 million lines of software code in it, running on 70 to 100 microprocessors. And according to consultant Frost & Sullivan, that number will rise to 200 to 300 million lines within a few years. And the software that controls the 'drive-by-wire' accelerators of Toyota and Lexus vehicles is one potential culprit in the tangled collection of issues, allegations, and recalls of many of those vehicles for so-called 'sudden acceleration' problems."
With all the recent US layoffs ... (Score:5, Informative)
Re: (Score:2, Funny)
Re:List of software powered cars (Score:5, Informative)
Anything street legal without a needing a special waiver for emissions.
Re:List of software powered cars (Score:5, Informative)
I had an '05 Chevy Cobalt that had "computer assisted" electromechanical power steering. Basically, what I found out from the dealer after the computer controlling it failed (and I lost all power steering) is that the computer (BCM, Body Control Module) takes inputs from the ABS system, Traction control (if equipped), speedometer, accelerometers and about a dozen other sensors and computes the way it thinks you want to be steering. Then it provides an "intelligent" boost in that direction. I must say, it worked really well in the snow and when fishtailing (it made if VERY difficult to over-correct and put it into a spin). But when it failed, I'd be in the middle of a curve on the highway when all power steering went out... Luckily they were smart enough to put a kill switch in to prevent it from coming back on while the car was moving (I could just imagine struggling through a corner when all of a sudden it came back)... It turns out that it was a software issue in the first place (they updated the software, and it never happened again). I got rid of the car a few years later for other, more significant reasons...
The benefits of computer control are good, but there needs to be intelligent fail-safes put in place to prevent disaster when something does go wrong (not if, when)...
Re:List of software powered cars (Score:5, Informative)
2. There is a wide range of speed and turn radius conditions between straight freeway and parking lot.
Re:With all the recent US layoffs ... (Score:5, Informative)
[citation needed]
Are you aware that there were more federal government employees in the 1980s under Reagan than there are today?
Are you aware that there were more government employees in the 70s under Nixon, Ford, and Carter than there are today?
Go take your horseshit somewhere else.
Sources: Article on Bush increasing the federal employment rolls [washingtonpost.com], just to point out your misplaced ire.
All fed employees, 1962 to 2008 [opm.gov] Here you go. What's that? Federal employment peaked at the end of Reagan's term and decreased under Clinton, only to increase again slighlty under Bush? How can that be, in your misinformed little world?
An article pointing out the increase in federal employees due to Obama's stimulus packages [usatoday.com] as of last September. It was newsworthy that 25k federal employees were added from Dec 08 to Aug 09. FYI, more have been added since, with 33k added in Jan 2010 as an example. Still far under what we had in the 80s under Reagan.
Get a clue. Dig into the numbers before you make erroneous claims parroting your stupid right-wing ideological leaders.
Re: (Score:3, Insightful)
Outsourcing!
The "reductions" in federal head count are just politicians beating their chests - the employees all get replaced by contractors.
The government is not smaller today than it was in the 70s or 80s... just look at the budget!
Re: (Score:3, Insightful)
Hint: There exists state jobs, which are massively in excess at the moment - compared to lossage in every other field.
Basically, the "stimulus" has been used to shore up failing state budgets to avoid public employee layoffs. Then these jobs are listed as "saved or created", and Obama takes a bow. Meanwhile, productive jobs in the private sector are experiencing 10% unemployment - that's people looking for work, the official Unemployment Rate. Alternative measures reaches as high as 18% in the month of Janu
Re: (Score:3, Insightful)
I know plenty who are laid off for other reasons- such as their C-level executives being slackers and the whole bloody company going under.
Re: (Score:2, Funny)
Simple AI: If a question ends in a vowel, the answer is no. Otherwise the answer is yes.
Question: Is your signature true?
Re:With all the recent US layoffs ... (Score:4, Funny)
No
Re: (Score:3, Funny)
Re: (Score:3, Funny)
yes
Re:With all the recent US layoffs ... (Score:5, Funny)
But if he is lying then he is telling the truth
and if he is telling the truth he is lying...
NORMAN COORDINATE!!!!!
Re: (Score:3, Funny)
yes- This exact comment has already been posted. Try to be more original...
Re: (Score:3, Insightful)
Re:With all the recent US layoffs ... (Score:4, Insightful)
Re: (Score:3, Insightful)
Well I don't know if "talent" is the right word. The people who get laid off are the worst 10%. Usually the real slackers.
I thought the most expensive got laid off first
Re: (Score:3, Insightful)
This is the same problem faced by businesses who need a 'software' person. Without having a good software person in the company already, how can they tell the difference between candidates? They can't.
If you're an organization looking to hire your first expert or two, you do it the old-fashioned
way. You consider their degree and the institution that granted it. You consider their work experience. And perhaps you rely on a referral from a trusted contact who knows more about the field than you do.
If you're looking for some fine-grained specialization in a particular technology, there are a number of certification programs out there. If you're looking for broader skills sets, there are both BS and MS progr
Huh! (Score:2)
Re:Huh! (Score:5, Interesting)
If the NHTSA didn't exist Toyota would have had to spend money to fix the problem instead of paying ex-regulators [dailyfinance.com] to quash multiple investigations.
Toyota (TM) hired ex-government regulators to kill at least four investigations into problems with its cars in the U.S. That's the conclusion of an investigation by Bloomberg. The news service reports that, "Christopher Tinto, vice president of regulatory affairs in Toyota's Washington office, and Christopher Santucci, who works for Tinto, helped persuade the National Highway Traffic Safety Administration to end probes including those of 2002-2003 Toyota Camrys and Solaras, court documents show. Both men joined Toyota directly from NHTSA, Tinto in 1994 and Santucci in 2003. "
The same goes for Wall Street. Most of the financial regulators are former high level executives from Goldman Sachs or strong ties to them and other financial institutions.
I don't understand why we need so many useless regulators who are usually wolves being put in charge of the hen house when the courts could easily handle this. It's going to end up being prosecuted in a court of law anyway and not solved by some magic regulation hand-waving.
Regulation != Bad (Score:5, Insightful)
Most of the financial regulators are former high level executives from Goldman Sachs...
Some are but most are demonstrably not. Many are financial industry insiders but that's by necessity. Do you really want an financial regulator who has no knowledge of the industry he/she is regulating? The only place to get people with the appropriate financial experience is from the finance industry.
I don't understand why we need so many useless regulators who are usually wolves being put in charge of the hen house when the courts could easily handle this.
While I admire your faith in the court system, in truth the courts are woefully ill-prepared to deal with the sorts of issues the SEC and other regulating bodies deal with. The court system is sloooooowww, expensive and can only effectively deal with misconduct after it has occurred. The courts are a poor monitoring system. The court system also is not heavily staffed with financial experts who understand the issues involved. Trust me, you REALLY don't want financially illiterate judges deciding financial regulations.
The reason the industry insiders often end up as regulators is precisely because they are the only ones who really understand what is going on. Finance is really, really complicated. Yes it's not perfect but that's why the regulators are accountable to other bodies including the President and Congress. If anything the problem with the regulators isn't (usually) that they do poor quality work but rather that they aren't given enough resources to really do a great job. The SEC for instance is badly understaffed given it's mandate. If you really want to keep a better watch on the finance industry, lobby congress to increase funding to the SEC and other watchdog agencies.
It's going to end up being prosecuted in a court of law anyway and not solved by some magic regulation hand-waving.
Spoken like someone who has no experience whatsoever in the financial industry. I won't argue that all regulations are good or well enforced but relying on the court system alone to solve the issues that regulators deal with daily would be insanity. If you really want to screw up the financial system, get rid of the regulators. Our current financial mess is due in significant part to a lack of regulation.
Re: (Score:3, Insightful)
Regulations are to prevent safety problems in the first place. The reason why regulation is not that regulation *never works*, but instead the regulators are not independant, there are too many conflicts of interests, and it will be the case until we implement some real compaign finance reform so that politicians are not helped to be elected by corporations through all of their generous donations. The fact is, without regulations, matters would not be any better, in fact they would likely be worse, as you w
Here come the shackles. (Score:5, Interesting)
Here comes DO-178B for cars.
I wonder what the cost is per line of code?
Re:Here come the shackles. (Score:5, Insightful)
Dunno.
My kids were runover by an out-of-control Mustang about four years ago. There was nothing mechanically wrong with the car. Maybe it was driver error. I don't know, but apparently the accelerator was still stuck to the floor when the police got there. I remember how the cruise control on the cars I've owned will lower the accelerator when the CC is accelerating.
I've always blamed the firmware. Maybe that's because I'm an EE who used to write firmware for a living. (Firmware that's been in use in life-critical applications for five years with a 0% failure rate.) Odds are the code is shit and there's an edge case that nobody thought about. Maybe there's an uninitialized variable in there. I've seen it happen before. Of course, I'm not Woz-brand, so my opinion doesn't mean a thing.
For some reason, the various regulatory agencies (i.e. Engineering Associations) have been rolling over and letting the manufacturers put any code they want into public use without any thought that hey, maybe we should get someone with some credentials to look into it. I've tried to mention it to mine, no results. Maybe they're dinosaurs who think that engineering is about roads and sometimes other things, like buildings and handrails. Software can't hurt people, can it?
This problem is not limited to Toyota, and we've only just seen the beginning. I guarantee that other manufacturers are clenchinging their butts hoping that nobody in the media wonders about all the intermittent "floor mat" problems.
Re: (Score:3, Interesting)
Re: (Score:3, Informative)
I've seen that feature, basically it helps when switching from cruise control to manual. You put your foot on the gas and release CC, and you can maintain speed. I'm not sure if the CC presses the accelerator in place of a human, or if the CC controls fuel flow and then adjusts the accelerator to match.
What I do want to know is how many crashed cars had the cruise control "on" but not set. My CC light can be on but not controlling speed until I hit "set". And if I hit the brake or clutch (it's a manual)
Re: (Score:3, Interesting)
My kids were runover by an out-of-control Mustang about four years ago. There was nothing mechanically wrong with the car. Maybe it was driver error. I don't know, but apparently the accelerator was still stuck to the floor when the police got there. I remember how the cruise control on the cars I've owned will lower the accelerator when the CC is accelerating.
I had a Mustang with an out of control acceleration problem. I was driving down a country road when all of a sudden it kept accelerating. I stomped
Re: (Score:3, Interesting)
None of these tests are entirely effective when dealing with embedded applications.
Bluntly, software tests can only prove the existence of a software bug relative to the specification. For an embedded application, toss the specification out, and start looking at real-world failure scenarios. Glitches on the reset line c
unlikely, given most networks are separated (Score:5, Interesting)
Here comes DO-178B for cars.
The vehicle drivetrain network is very often, if not always, separate from the "entertainment" network; Audi, for example, runs two separate CAN busses for them. The original story hypes things a bit; there may be 70-100 microCONTROLLERS, but half or more of them are "body" (ie windows, sunroof, etc) or "entertainment"(audio, navigation) related and thus don't really need to be reviewed.
The vast majority of them do very, very simple things, mostly sending CAN bus messages or responding to CAN bus commands. Ie, you move the wiper stalk. The microcontroller for the steering wheel controls says "the stalk moved" either to the wiper motor interface or a 'body control' computer, which then sends a command to the wipers.
The code review for most of the modules, as a result, is extremely simple- they're just (mostly digital) I/O boxes. Some of them are things like fuel pump modules, which at most have some diagnostic capabilities (like current draw from the pump, pressure sensor, etc.)
The code review will not be very problematic for engine computers, because (gasp!) they're not made by car manufacturers. Bosch, Magnetti Marelli, Hitachi, and a couple of other companies are the primary producers. And guess what? The code is largely the same car-to-car. Parameters are changed- code doesn't, so much. And car companies share "platforms", which further simplifies things.
It's not nearly as scary as it sounds.
Re: (Score:3, Informative)
While there is truth in what you are saying on complexity, as someone who has invested a lot of time understanding why Bosch has some fuel pumps failing in a non-passive fashion on stationary engines... there are a lot of assumptions built in, and many problems are only found by trial and error.
Re: (Score:3, Informative)
No clue, but I very much doubt the figure.
100 million lines is more than in a normal Linux installation (with OS, openoffice, gnome/kde, firefox, etc.)
consultants (Score:4, Insightful)
Surely it would be a serious inefficiency for NHTSA to maintain on staff a large number of specialists to handle this kind of problem? Isn't that exactly what (properly qualified) consultants are for?
Re:consultants (Score:5, Informative)
Given how much of our vehicles are run by computer, I don't think there should ever be a lack of demand for software engineers at the NHTSA.
Re: (Score:3, Insightful)
Surely it would be a serious inefficiency for NHTSA to maintain on staff a large number of specialists to handle this kind of problem? Isn't that exactly what (properly qualified) consultants are for?
I agree that it'd be inefficienct to have a large number of EEs & SEs on staff, but they have no one to do even a simple sanity check on the hardware and software that is being certified for public roads. And that strikes me as a failure of their organizational mission.
Re:consultants (Score:4, Insightful)
Under the circumstances, you pretty much have two options. The radical, future-looking one is to say "Ok, clearly complex software is the future. We are going to do whatever it takes, build up a serious software engineering team, impose standards that would make medical device makers cry, sponsor research in automated verification, whatever. Yeah, it sucks that we have do deal with that complexity; but so it goes." The traditional conservative(and, much more likely to fit within your budget and not ruffle feathers) option is to throw up your hands and treat the software as a black box. Have your existing test engineers use their existing techniques, or limited variants, to run the vehicles through test conditions, hoping that, if the test conditions effectively model the real world, any real world critical bugs will appear in testing, at which point you can kick it back to the people who wrote the code and tell them to fix it.
It seems pretty clear that the NHTSA has pretty much gone with option two. And, frankly, it is hard to blame them under the circumstances. Even at the best of times, technical regulation is a pretty unsexy legislative priority, and tends to be funded accordingly. It wouldn't take an actively antiregulatory corporatist to raise an eyebrow at a request for the sort of resources that you'd need to seriously audit the code in each new car coming off the line. And, if you don't have the resources to properly evaluate code from a CS or formal verification perspective, empirical black-box testing under real world-ish conditions is about the best you can do.
Re:consultants (Score:5, Insightful)
Given that there isn't a car made today whose safety can be properly evaluated without the skills of EE and software engineers, why would it be inefficient for the agency responsible for that evaluation to have people with those skills on staff? It's not like next years cars won't have even more of the same complete with modified firmware to examine.
Given that the safety evaluation will involve interactions between mechanical, electrical and software systems, you'd want a cohesive multi-disciplinary team, not a revolving door.
Re:consultants (Score:5, Interesting)
Rise of the machines (Score:2)
Clerk: I'm sorry, sir, the computer controls the doors too.
Welp (Score:4, Interesting)
Such is the cost of more complicated technology. Although, I will admit, this problem seems awfully widespread for Toyota to have not caught this at some point in their QC/QA process.
I'm reminded of the "recall" speech in Fight Club...
Re:Welp (Score:5, Funny)
Which car company do work for?
A major one.
Heads better roll (Score:5, Funny)
If the statement in the article is true then this country is in even worse shape than I thought. It seems like rarely a handful of months can go by without the realization that yet another Federal department is completely incompetent. How in the hell does the NHTSA even do their job?! They are supposed to ensure that vehicles are safe but they don't even have the staff to do that.
What the hell is wrong with our country?
Re: (Score:2)
I never even know NHTSA existed.
Re: (Score:2)
ever heard of a "4 star" or "5 star" crash rating? Then you've heard of the NHTSA.
Re: (Score:3, Insightful)
Re:Heads better roll (Score:5, Insightful)
Re:Heads better roll (Score:5, Insightful)
One would be a good start. Oh hell, let's get wild and crazy and say.. 2.
Certainly more than zero.
Re:Heads better roll (Score:4, Insightful)
Re:Heads better roll (Score:5, Insightful)
In Canada, the province where I live (Alberta), derives a major part of its revenues from oil and gas. In the same conservative government 35 years ago, we had 2 independent arms of the government who could determine how much royalties were owed to the government from the oil and gas producers.
Today, we have no one in our government who is able to determine how much we should be collecting and therefore have to rely upon the oil and gas companies to tell use how much they are supposed to remit. Our own government auditor believes we have been bilked out of billions yet somehow we have a leaner and, ahem, more efficient government.
Just remember that the only thing to stand up to a big business nowadays is big government, and the goal of any big business is to convince everyone that a small government can watch over big business just like a big government can.
Re:Heads better roll (Score:4, Insightful)
Just remember that the only thing to stand up to a big business nowadays is big government, and the goal of any big business is to convince everyone that a small government can watch over big business just like a big government can.
- I mean, really? Wake up, is there anyone home? The government that you like so much consists of a system of people, who like to remain in power. To do so takes money. Lots and lots of money. Where do you get the money? It's the system - the bribes real and implied etc.
Government today is in it with the large corporations. They are one government. In Canada it is a bit different from the US but the principles are the same. Big money wants more money, to do so it needs to corrupt the government and it works on that day and night. Big government wants to stay in power, to do so it needs contributions and various other things money can buy, they do this day and night.
It's like that Alien vs Predator: no matter which one of them wins, who do you think is going to lose?
Re:Heads better roll (Score:5, Insightful)
Wrong. There's a reason Theodore Roosevelt is on Mt. Rushmore. Go back in history and read up. In the USA, we are re-living many of the same issues, roughly 100 years later.
Legitimate checks to power (Score:5, Insightful)
The government doesn't have to do anything complicated. It just has to have the ability to strike fear into the hearts of the business community it's supposed to regulate.
This requires a few things: an independent media, which we don't have; a civically informed populace that takes it's democratic duties seriously, which we don't have; and a culture that values human dignity over profits, which we don't have.
In cultures that do have all of these things, government regulation works very well and fosters progress, since you don't have to constantly worry about getting screwed over, you don't have to wonder if you'll have access to medical care, or a good public school, or a good safety net to get you back on your feet if your fall ill, get in an accident, or whatever.
Clear and concise regulation with real penalties for breaking those regulations fosters competitive markets. Diminishing the government to the point where it can be bought and sold by businesses usually leads to fascism. The markets destroy themselves with greed, destabilize the economy (and eventually the whole society), and further concentrate wealth and power until you have a virtual oligarchy sprinkled with political theater.
Re: (Score:2)
Are you kidding? NHTSA sanctions the testing, develops *some* of the test protocols and performs *some* investigative work to identify problems. Their best strategy to create/keep cars safe in the US is to make sure the manufacturers go through the right processes in creating them. Does that mean having code auditors at the NHTSA looking over the shoulders of programmers at all the car manufacturers? I don't think it does. Does it mean the NHTSA should mandate auto makers to do rigorous code audits of
Re: (Score:2)
How are they developing effective tests without engineering talent to guide the creation of those tests? How are they validating simulated tests if they don't even have the theoretical and practical knowledge that engineers would give them? It isn't like the NHTSA should be doing all of the testing or code audits for the auto makers. However they should have some talent on hand so that when Toyota says, "It isn't the electronics.", someone at the NHTSA can begin to verify it.
Re:Heads better roll (Score:5, Interesting)
My parents both worked for the FDA and if the NHTSA operates in any similar way to the FDA, it's a shadow of itself in the 1970s. For the FDA that means that there are less food inspectors and no surprise, there is a rise in food poisoning incidents. I wouldn't be surprised if NHTSA is also chronically understaffed. Additionally, even if individual government workers wanted to do their jobs, they are often prevented by doing so because that is not perceived as "business friendly". The political appointees who run the show are in the thrall of private industry, in fact, they are often people taken directly from private industry (e.g. big pharma lobbyists often run the FDA). This "government capture" is the fault of the democrats just as much as the republicans, e.g. Obama lied about hiring lobbyists in his campaign. Basically, we have a non-functioning government and one party's answer to this is the get rid of the thing all together. That is one solution but that wouldn't prevent things like this incident with Toyota.
I'm sure Toyota will do the right thing though, because that would be in its interests as a good corporate citizen. *snicker*
Re:Heads better roll (Score:5, Insightful)
Like every other safety certification organization. The car companies pay for a certificate, NHTSA takes some of the blame when something happens, and the general population feels safe knowing their is an entire organization dedicated to protecting them.
Re:Heads better roll (Score:5, Informative)
Years of deregulation and resource starvation have strangulated our regulatory agencies to the point where they are unable to act.
Much of this based on Greenspan-style Libertarian philosophies that market forces can correct any problem including fraud and crime, a position which he himself has now renounced and we as a people have yet to heed.
Since the late 80s we have been riding on a giant ponzi scheme and its all coming crashing down right now. And yet, nothing. I expect things to get much worse.
"Resource starvation"? (Score:4, Interesting)
Here's some recent data about the resources available to the DoT, the parent agency of the NHTSA: When the recession started, the Transportation Department had only one person earning a salary of $170,000 or more. Eighteen months later, 1,690 employees had salaries above $170,000 [usatoday.com]. Plus the juicy benefits and pension plan. I'll bet all those managers and supervisors raking in the big bucks would agree that their agencies are "resource starved" and that if they only had more money and more power, they could hire two or three software engineers (for the cost of one manager).
Re: (Score:3, Insightful)
It has to be deeper than just the President. The NHTSA lacking EE's and SE's is institutionalized fail. They don't even have the talent to meet their mandate. It required a full blown Congressional investigation into dozens of fatalities for someone to stand up and basically say, "By the way, we can't do our job."
Computer Engineers needed (Score:5, Insightful)
Re: (Score:3, Funny)
Even better, this one [mattel.com] only costs $12.99!
Re: (Score:2, Insightful)
Speaking as someone with a CMPE degree, employers see me as under-qualified to do EE work and over qualified to do programming work. What they need is either EEs with heavy embedded programming experience or software engineers with (guess what) embedded programming experience. The title isn't that important.
Re: (Score:2)
100 million lines of code?? (Score:2, Insightful)
I find that extremely hard to believe. Jurassic Park ran on just two million lines of code. I doubt all the lifetime output of all the readers of this thread, combined, equals 100 million. I further doubt that such complexity is remotely necessary to run a car, and that it is remotely possible to debug that much complexity to the standards of, say, the airline industry. And that NHTSA could audit that code in any respectable amount of time. I hope beyond hope the number is wrong.
Re:100 million lines of code?? (Score:5, Informative)
sadly, it appears to be true:
http://spectrum.ieee.org/green-tech/advanced-cars/this-car-runs-on-code [ieee.org]
Re: (Score:2)
What a revealing article:
The F-22 Raptor has 1.7 million lines, the F-35 about 5.7 million, and a 787 has 6.5 million lines, but somehow a consumer automobile needs 100 million?
I'm honestly surprised this is the first major incident.
Re:100 million lines of code?? (Score:4, Informative)
Why? (Score:3, Insightful)
Re: (Score:2)
I doubt all the lifetime output of all the readers of this thread, combined, equals 100 million.
Surely you jest ... or you've been favorably sheltered from our endless verbosity, pedantic ramblings and self-serving diatribes.
Dr Zoidberg: Loot at me, I'm helping!
100 million lines? Sure, we will get right on it (Score:5, Insightful)
What exactly would the NHTSA do with a set of engineers? Audit all 100 million lines of code for each and every car they suspect has a safety issue with the computer system? Yeah, that sounds like a worthwhile endeavor. How about they do it the old fashioned way; collect the reports, identify the risk, and sanction the manufacturer to find/fix the problem. Thinking that an NHTSA coder (or a hundred) would have gotten to the bottom of this Toyota issue in any reasonable amount of time is a joke!
Re: (Score:2)
Re: (Score:2)
Well, if you don't know what you're asking for, how would you ever know if the answer they give you is even close to reality?
"Hey, I need you to investigate x, I have no idea how to even analyze x, but I trust you will investigate it exhaustively!"
"Sure, we fully investigated x and it's fine."
"Oh, ok, we'll take your word for it, thanks!"
You have to at least be able to understand what's going on to a certain degree before you can tell someone to fully investigate it _and_ then trust their results.
So yes, th
Re:100 million lines? Sure, we will get right on i (Score:5, Interesting)
And they said in a modern luxury car.
So that's all the code in the following computers:
Engine (controls throttle and such)
Transmission
Collision avoidance (ABS, traction control, etc. TPMS is usually here, too, because it's sometimes part of the ABS system to save costs)
Safety (airbags, seatbelt pretensioners, etc.)
Central convenience (security system, power locks, power windows, cabin illumination, in some cars even the exterior lighting goes through central convenience)
HVAC
Instrumentation (yep, there's a computer dedicated to that - and some security functions are sometimes in there)
Entertainment (navigation, stereo, DVD, etc., etc.)
And all these systems are interconnected.
You get in your car (central convenience deactivates security upon receiving the signal, and when you open the door, it illuminates the cabin, alerts the engine computer that a start is imminent, possibly starting fuel pumps, on diesel cars turning on the glow plugs, etc., etc., and notifies the instrument cluster that the door is ajar.)
You insert your key into the ignition (yes, I know about push-button start,) and start the engine (engine computer starts up, after which the instrument cluster polls the RFID chip on the key. If it can't get a read, it immediately requests that the engine computer shut down.)
You decide that you want a little heat before you set off, so you use your steering wheel controls (which go through instrumentation) to set HVAC settings, and then you figure some music won't hurt (entertainment.) Then, you remember that you don't know where you're going, so you punch the address into the navigation system, and it feeds directions back to the instrument cluster.
Now, you put the car into gear. The transmission computer notifies the other computers about this, and the engine computer adjusts the idle fueling to compensate. The instrument computer reflects the gear change. The central convenience module turns on the daytime running lights. The entertainment system might prevent you from using the touchscreen interface. The safety computer may become more persistent about reminding you that you didn't put on your seat belt, and will notify the instrument cluster of this, to annoy you more.
After you put your seatbelt on, you let off the brake and pull out of your parking space. Obviously, the engine computer and transmission computer are working together here, the instrument cluster is constantly updating the status of those (and the entertainment computer, which is noting the changes in vehicle position.) After you hit 10 MPH, the engine or transmission computer sends a request to the central convenience module to lock the doors.
Now, you're going down the freeway, and right in front of you, a semi truck loses control, and flips onto its side. You jam on the brakes, which kills engine power immediately (engine computer, and the transmission computer is affected as well, and this all gets fed back to the instrument computer.) Collision avoidance computer activates ABS and (as you're attempting to swerve out of the way) stability control, and notifies the central convenience computer that you're undergoing a panic stop, and to activate the hazards.
Unfortunately, you don't have enough time and room to stop, and you hit the semi. The safety computer notices this, and fires the seatbelt pretensioners and the appropriate airbags. Once that's done, there's some less immediate concerns. It would be a bad idea to leave the engine running, so the safety computer requests an engine shutdown. The transmission computer may be requested to shift to neutral, to make moving the wreck easier. The entertainment system will be told to stop playing music, and if it's got a system like OnStar (which used to be yet another TWO separate computers off of the entertainment system,) an emergency call initiated. Instrumentation is of course updating the status of all of this. HVAC may be set to off. The collision avoidance computer will still be trying to keep t
Re:100 million lines? Sure, we will get right on i (Score:4, Insightful)
It would be a pretty crappy car if it engaged the seat belt PREtensioners POST-impact.
Pretensioners are fired after the initial contact, whilst the very front of the vehicle is still crumpling away. How the hell do you think the computer knows that it has hit something otherwise? Radar? Not on your $10K cheapo. Magic? No, a little ball + spring combo live underneath your front bumper and the last thing they tell the vehicle before they are crushed in an accident is "something big is heading your way".
I'd also rather it didn't "kill engine power" every time I hit the brakes.
We're not just talking about 'every time', we're talking about the two-feet-on-the-brake-pedal-jesus-christ-I-want-to-stop-NOW kind of braking that will activate ABS. Once ABS (and it's cousin, stability control) are running the show, engine power can (and will) be modulated as they see fit in attempt to keep the vehicle going where you want it to go. If you think you can simultaneously control brake force and engine power separately to each wheel whilst in an emergency to do the same, than you go right ahead. I'll take the bus.
although "traction control" systems might retard timing if severe wheel slip is detected.
Traction control is a lot smarter than you seem to think now, and retarding timing went out of fashion about 15 years ago. Now if the traction control system wants less power it simply requests the engine computer to reduce power output by X percent and the engine computer will choose between:
- Simply closing the throttle body, if it has control of it.
- Killing fuel injection on a few cylinders to drop power.
- Dropping boost if it's a turbo'd vehicle.
- Cutting (or yes, retarding) ignition. Bit of a last resort due to unburnt fuel getting out the other side of the engine.
And what $20,000 compact automatically turns on hazard blinkers, mutes the stereo, and opens windows?
My Peugoet 307 turned on the hazards and muted the music if you hit the brakes hard enough to activate its electronic brake force assist system. I did it a couple of times in the two years I had the car, but never got into a collision to find out about the windows.
Re: (Score:2)
A line-by-line audit is silly, and nobody is suggesting this. However, I can't see why the department that oversees embedded systems (automobiles) has no electrical engineering talent on hand.
How many microprocessors was that again? (Score:2, Insightful)
Re: (Score:2)
Microsoft Hotline (Score:2, Funny)
Can't they just call Microsoft's toll-free number and ask someone over there to look at it?
Sounds like the government needs to be updated (Score:2)
Seriously. How did they not see this coming. They have been hearing cases about secret codes and OBD standards and the like for quite some time now. The fact that cars are running with the added use and assistance of digital computational systems is well known. If they are not equipped to do testing for safety purposes, they are simply not equipped to do their jobs. And I'm afraid to ask about air vehicle safety testing now...
Why would they? (Score:2)
They respond to problems, they don't reverse engineer things. Does the FDA or the Surgeon General's office have engineers to paw through the lines of code in MRI machines or CT scanners, or anesthesia machines, or respirators, or any other number of computerized medical machines? No... they get tested emperically, just like cars do. It's very difficult to prove that some of these flaws exist.... remember the Audi "sudden acceleration" problems in the late '80s that almost killed the brand? That was pre-c
So What? (Score:2)
Safety related functionality should have a redundant overriding mechanism that isn't subject to the vagaries of software failure. For example, if the engine computer suddenly wants to run an explode subroutine, the fuel valve should limit the outcome to chitty chitty bang bang.
Then you don't have to check every line of code, you just have to check the overrides.
This is the government, not an engineering firm (Score:3, Insightful)
I totally disagree: the NHTSA shouldn't hire engineers. NHTSA should not do the job of Toyota's engineers and testers; they were created to set policy and propose safety laws. The NHTSA should hire economists, policy makers, and maybe some scientists. But the job of ensuring the nuts and bolts of a car are safe should fall on the car-maker, with strict repercussions if they fail.
My biggest problem with all this is what people on Slashdot should already know: looking through and understanding millions of lines of code would take an engineer a few lifetimes - how many engineers are we proposing NHTSA hires? They could learn Toyota's software system, but then what about Ford cars? Or BMW? All for a government organization with 600 employees...
In cases like this, NHTSA should force Toyota to hire a third party (objective) consultant to create a technical report. Maybe a small team of engineers could remain on staff to read and understand those reports.
Re: (Score:3, Interesting)
Re:This is the government, not an engineering firm (Score:4, Insightful)
Why not simply require that any software in an automobile be OSS (not FOSS). In fact that requirement should seem to be an extension of mechanic laws that required car makers to provide parts and knowledge to service vehicles outside dealerships. All software in such a critical item should be OSS so it can be reviewed for errors and be reprogrammed by mechanics who wish to offer such services.
advice for anyone with a runaway gas pedal (Score:4, Funny)
Re: (Score:3, Interesting)
There are two major problems with the "shift to neutral" solution:
1. It doesn't always work.
2. Only a few auto-mechanic and maybe some race car drives have the reflex to shift the car into neutral.
Most people will not think of shifting to neutral when a problem is encountered, simply because they never need to do it. I'm an engineer, and if my car takes off, it will take me a while to think of shifting to neutral. A car at full acceleration can cover much ground in less than 1 second.
The other problem
Not news to me (Score:3, Interesting)
I can't say I find this surprising. Anyone who has ever worked on software for a US government contractor, or US military contractor, knows the government/military has no one who can analyze the product they pay for. Nearly every software product I've seen delivered is of absurdly poor quality. It would be laughable if the implications of the software's use weren't so disturbing.
Fire all the bridge and tunnel inspectors, too (Score:3, Insightful)
If you think that the government should not get involved in engineering.
Don't reinvent the wheel (Score:3, Insightful)
The NHTSA does not need to evolve a new set of standards out there to address part of this problem. Just require that all automobiles meet the FCC Part 15, Class B standards for electromagnetic susceptibility. It is stupid that this is not done already.
There are plenty of critical pieces of equipment that cannot turn up their noses and fail because of electromagnetic interference. Medical equipment is tested to at least this standard every day. There are hundreds of testing laboratories throughout the world who manufacture products that have to meet these specifications. There are thousands of engineers who already do this type of testing.
Now lines of code and software is a different animal. In a hundred million lines of code there are certainly bugs and flaws.
It's time... (Score:3, Insightful)
a) for a global safety-critical standard for drive-by-wire software.
b) for an open industry standard for interfacing for servicing, fault codes, etc, to end the scam of lock-in to specific manufacturers servicing tools and dealers.
c) to open source it.
100 MLOC? They should have used Lisp. (Score:3, Funny)
The car function is built in.
100 million lines of code on 70 processors? (Score:3, Interesting)
I would be more interested in the process of how
Toyota develops/maintains code. Do they rewrite code for every car?
When they reuse code, how do they retest assertions?
How do they do code verification?
What is their culture when coding problems interfere w/deadlines ?
Is there a whole crap load of unused code in there because
they are scared shitless to remove it ?
etc.
More needed than just source code review (Score:3, Insightful)
This drive-by-wire stuff is very serious. I seriously doubt that any car manufacturer validates their computer software and hardware as rigorously as the Dept of Defense; in fact they probably don't do compiler or chip logic validation at all. I bet the aviation industry could give them guidance in this arena.
Standards, not analysis. (Score:4, Interesting)
And safety, not peformance.
Instead of testng code, evaluating the design process, pretending the NHTSA can even begin to become expert in software design, how about applying the old standards to the new systems?
For instance, braking safety. I was listening to and reading the testimony from Rhonda Smith [msn.com], where she even describes shifting her Lexus into neutral. Neutral?
A simple test, and I'm not an engineer, but shouldn't a car come to a stop with 'maximum' brake effort, despite the acclerator position? This is solvable in software - if the brakes are going into lock, and ABS is engaged, engine power and/or transmission state have to be compelled to answer the driver's command to stop. Traction control is already being used in many cars; NHTSA should be able to make a test capable of verifying that even multiple malfunctions are overcome.
Crap, my wife's 1995 Saab 900SE has a mode where the ECU shuts down the fuel pump if the engine stops running, on the assumption that something is terribly wrong, and spewing gas to a stopped engine is pointless if not dangerous. How do I know this? Her car developed a habit of stalling at stops. The real cause was a defective vapor recovery canister, causing loss of vacuum and low RPMs, and the ECU saw that as a stopped engine and made sure it stopped.
Certainly there are other states that can be tested for performance and safety, not some quality of performance standard. Most cars have 'safe' or 'cripple' modes to protect the drivetrain if something seems wrong, like the transmission in a gear that should not permit the indicated speed. My '95 Explorer does that, and it's only an OBD-I system. Acclerator position, wheel speed, and transmission mode should all correlate, and if something is wrong the system needs to cripple - slow down, set a max speed, etc.
Aircraft flight control systems are held out as an example of safety and reliability. Most of these, if not all, have to at least ensure the aircraft doesn't exceed the flight envelope and exceed safety limits. This is the sort standard and evaluation the NHTSA needs to focus on.
Maybe NHTSA needs to borrow a few investigators from the FAA and the military? They should be looking to Boeing, McDonnell, Electric Boat, General Dynamics for expertise in verifying safety in vehicles. Maybe even some NASA people. At least NASA seems to have turned the Shuttle program around a little too late. They certainly have a cautionary tale to tell, and a jaundiced eye towards the assurances of the 'experts' and trusting management.
Which would go a long way to reinstating a somewhat adversarial relationship between the regulators and the industry. There should be some tension there. Hiring your industry's former employees is not the way to go.
We can do so much better. We just need to solve the real problems.
I just HAVE to ask this question... (Score:3, Insightful)
Bear with me for a second here...
The three laws of robotics:
1. A robot may not injure a human being or, through inaction, allow a human being to come to harm.
2. A robot must obey any orders given to it by human beings, except where such orders would conflict with the First Law.
3. A robot must protect its own existence as long as such protection does not conflict with the First or Second Law.
I know that a car is not a robot. But the same rules should apply for ANY computer system that, in case of a serious bug, could result in any of those 3 laws being broken.
This computer literally controls a rather large piece of metal that can travel at speeds sufficient to kill someone. So why is there no subroutine that ensure that brake pedal input will ALWAYS override the gas pedal input? It seems that even on the absolute most basic of level, adding this extremely basic concept could seriously mitigate these issues. Not to mention all of the legal responsibilities, public outcry, and other consequences of not having software or hardware with these "basic" concepts built in.
Even when making a car and using this system on a test site somewhere. Wouldn't you want to have LOADS of extra code in there to make sure a bug in the software doesn't kill the driver at the test site? It seems to me Toyota's definition of "safety" is practically non-existent.
Honestly, when seeing something like this, I have to question what kind of work ethic Toyota has and how much they value me as a customer.
Throttle software causing sudden acceleration? (Score:5, Insightful)
I can't believe I'm the first one on this thread to make that joke. I'm not even a programmer.
You should all be ashamed of yourselves.
Re: (Score:2)
Previously I would have suspected that absurd bloat like that would have been the result of bureaucratic NHTSA regulations, but obviously that's not the case...
Re: (Score:3, Interesting)
I've seen the comment about a modern car having something like 100 million lines of code in articles before. Now, I am not in any way qualified to say that number is to large or to small. But as an embedded systems software developer, that seems like an INSANE amount of code.
Someone posted a link to this article [ieee.org] that confirms it. I can't find the comment with the link; someone must have modded him down past my threshhold. But the article linked itself confirms that it is indeed an insane amount of code, ins
Re:100 million lines of code? (Score:4, Interesting)
Most of that code is auto generated. Except for some low level stuff, nothing is written by hand in assembly or C. It's all auto coded from some sort of control toolbox. Most likely Matlab/Simulink.
Sure enough this is one of the first hits [mathworks.com] on Google.
Writing that many lines of code would be damn near impossible in the relatively short development cycle.
Even a simple PID controller could take up a few dozen lines of code even though on screen it's simply represented by 3-4 blocks.
Re:This all story starts to look like swine flu (Score:5, Interesting)
http://abcnews.go.com/Blotter/RunawayToyotas/toyota-acceleration-problems-new-evidence-imprisoned-minnesota-toyota-camry-owner/story?id=9903455 [go.com]
This guy apparently killed a few people and got put in jail for it. Now it looks like he was telling the truth when he said the car wouldn't stop.