NHTSA Has No Software Engineers To Analyze Toyota 459
thecarchik writes "An official from the National Highway Traffic Safety Administration told investigators that the agency doesn't employ any electrical engineers or software engineers, leaving them woefully unable to investigate correctly what caused the most recent Toyota recall. A modern luxury car has something close to 100 million lines of software code in it, running on 70 to 100 microprocessors. And according to consultant Frost & Sullivan, that number will rise to 200 to 300 million lines within a few years. And the software that controls the 'drive-by-wire' accelerators of Toyota and Lexus vehicles is one potential culprit in the tangled collection of issues, allegations, and recalls of many of those vehicles for so-called 'sudden acceleration' problems."
Here come the shackles. (Score:5, Interesting)
Here comes DO-178B for cars.
I wonder what the cost is per line of code?
Welp (Score:4, Interesting)
Such is the cost of more complicated technology. Although, I will admit, this problem seems awfully widespread for Toyota to have not caught this at some point in their QC/QA process.
I'm reminded of the "recall" speech in Fight Club...
Re:How many microprocessors was that again? (Score:1, Interesting)
Certainly not a direct connection but the automobile industry does use CAN (Controller Area Network) to link many systems in the car, it is a shared bus to so say that a bug in one system couldn't effect another may not be entirely accurate.
One would hope mission critical system have separate buses but with a safety administration with no ability to check, who knows?
Re:Heads better roll (Score:5, Interesting)
My parents both worked for the FDA and if the NHTSA operates in any similar way to the FDA, it's a shadow of itself in the 1970s. For the FDA that means that there are less food inspectors and no surprise, there is a rise in food poisoning incidents. I wouldn't be surprised if NHTSA is also chronically understaffed. Additionally, even if individual government workers wanted to do their jobs, they are often prevented by doing so because that is not perceived as "business friendly". The political appointees who run the show are in the thrall of private industry, in fact, they are often people taken directly from private industry (e.g. big pharma lobbyists often run the FDA). This "government capture" is the fault of the democrats just as much as the republicans, e.g. Obama lied about hiring lobbyists in his campaign. Basically, we have a non-functioning government and one party's answer to this is the get rid of the thing all together. That is one solution but that wouldn't prevent things like this incident with Toyota.
I'm sure Toyota will do the right thing though, because that would be in its interests as a good corporate citizen. *snicker*
Re:Huh! (Score:5, Interesting)
If the NHTSA didn't exist Toyota would have had to spend money to fix the problem instead of paying ex-regulators [dailyfinance.com] to quash multiple investigations.
Toyota (TM) hired ex-government regulators to kill at least four investigations into problems with its cars in the U.S. That's the conclusion of an investigation by Bloomberg. The news service reports that, "Christopher Tinto, vice president of regulatory affairs in Toyota's Washington office, and Christopher Santucci, who works for Tinto, helped persuade the National Highway Traffic Safety Administration to end probes including those of 2002-2003 Toyota Camrys and Solaras, court documents show. Both men joined Toyota directly from NHTSA, Tinto in 1994 and Santucci in 2003. "
The same goes for Wall Street. Most of the financial regulators are former high level executives from Goldman Sachs or strong ties to them and other financial institutions.
I don't understand why we need so many useless regulators who are usually wolves being put in charge of the hen house when the courts could easily handle this. It's going to end up being prosecuted in a court of law anyway and not solved by some magic regulation hand-waving.
Re:This all story starts to look like swine flu (Score:5, Interesting)
http://abcnews.go.com/Blotter/RunawayToyotas/toyota-acceleration-problems-new-evidence-imprisoned-minnesota-toyota-camry-owner/story?id=9903455 [go.com]
This guy apparently killed a few people and got put in jail for it. Now it looks like he was telling the truth when he said the car wouldn't stop.
Re:consultants (Score:5, Interesting)
Re:100 million lines of code? (Score:3, Interesting)
I've seen the comment about a modern car having something like 100 million lines of code in articles before. Now, I am not in any way qualified to say that number is to large or to small. But as an embedded systems software developer, that seems like an INSANE amount of code.
Someone posted a link to this article [ieee.org] that confirms it. I can't find the comment with the link; someone must have modded him down past my threshhold. But the article linked itself confirms that it is indeed an insane amount of code, insanely implimented.
It gets worse.
Re:100 million lines of code? (Score:4, Interesting)
Most of that code is auto generated. Except for some low level stuff, nothing is written by hand in assembly or C. It's all auto coded from some sort of control toolbox. Most likely Matlab/Simulink.
Sure enough this is one of the first hits [mathworks.com] on Google.
Writing that many lines of code would be damn near impossible in the relatively short development cycle.
Even a simple PID controller could take up a few dozen lines of code even though on screen it's simply represented by 3-4 blocks.
Re:100 million lines? Sure, we will get right on i (Score:5, Interesting)
And they said in a modern luxury car.
So that's all the code in the following computers:
Engine (controls throttle and such)
Transmission
Collision avoidance (ABS, traction control, etc. TPMS is usually here, too, because it's sometimes part of the ABS system to save costs)
Safety (airbags, seatbelt pretensioners, etc.)
Central convenience (security system, power locks, power windows, cabin illumination, in some cars even the exterior lighting goes through central convenience)
HVAC
Instrumentation (yep, there's a computer dedicated to that - and some security functions are sometimes in there)
Entertainment (navigation, stereo, DVD, etc., etc.)
And all these systems are interconnected.
You get in your car (central convenience deactivates security upon receiving the signal, and when you open the door, it illuminates the cabin, alerts the engine computer that a start is imminent, possibly starting fuel pumps, on diesel cars turning on the glow plugs, etc., etc., and notifies the instrument cluster that the door is ajar.)
You insert your key into the ignition (yes, I know about push-button start,) and start the engine (engine computer starts up, after which the instrument cluster polls the RFID chip on the key. If it can't get a read, it immediately requests that the engine computer shut down.)
You decide that you want a little heat before you set off, so you use your steering wheel controls (which go through instrumentation) to set HVAC settings, and then you figure some music won't hurt (entertainment.) Then, you remember that you don't know where you're going, so you punch the address into the navigation system, and it feeds directions back to the instrument cluster.
Now, you put the car into gear. The transmission computer notifies the other computers about this, and the engine computer adjusts the idle fueling to compensate. The instrument computer reflects the gear change. The central convenience module turns on the daytime running lights. The entertainment system might prevent you from using the touchscreen interface. The safety computer may become more persistent about reminding you that you didn't put on your seat belt, and will notify the instrument cluster of this, to annoy you more.
After you put your seatbelt on, you let off the brake and pull out of your parking space. Obviously, the engine computer and transmission computer are working together here, the instrument cluster is constantly updating the status of those (and the entertainment computer, which is noting the changes in vehicle position.) After you hit 10 MPH, the engine or transmission computer sends a request to the central convenience module to lock the doors.
Now, you're going down the freeway, and right in front of you, a semi truck loses control, and flips onto its side. You jam on the brakes, which kills engine power immediately (engine computer, and the transmission computer is affected as well, and this all gets fed back to the instrument computer.) Collision avoidance computer activates ABS and (as you're attempting to swerve out of the way) stability control, and notifies the central convenience computer that you're undergoing a panic stop, and to activate the hazards.
Unfortunately, you don't have enough time and room to stop, and you hit the semi. The safety computer notices this, and fires the seatbelt pretensioners and the appropriate airbags. Once that's done, there's some less immediate concerns. It would be a bad idea to leave the engine running, so the safety computer requests an engine shutdown. The transmission computer may be requested to shift to neutral, to make moving the wreck easier. The entertainment system will be told to stop playing music, and if it's got a system like OnStar (which used to be yet another TWO separate computers off of the entertainment system,) an emergency call initiated. Instrumentation is of course updating the status of all of this. HVAC may be set to off. The collision avoidance computer will still be trying to keep t
unlikely, given most networks are separated (Score:5, Interesting)
Here comes DO-178B for cars.
The vehicle drivetrain network is very often, if not always, separate from the "entertainment" network; Audi, for example, runs two separate CAN busses for them. The original story hypes things a bit; there may be 70-100 microCONTROLLERS, but half or more of them are "body" (ie windows, sunroof, etc) or "entertainment"(audio, navigation) related and thus don't really need to be reviewed.
The vast majority of them do very, very simple things, mostly sending CAN bus messages or responding to CAN bus commands. Ie, you move the wiper stalk. The microcontroller for the steering wheel controls says "the stalk moved" either to the wiper motor interface or a 'body control' computer, which then sends a command to the wipers.
The code review for most of the modules, as a result, is extremely simple- they're just (mostly digital) I/O boxes. Some of them are things like fuel pump modules, which at most have some diagnostic capabilities (like current draw from the pump, pressure sensor, etc.)
The code review will not be very problematic for engine computers, because (gasp!) they're not made by car manufacturers. Bosch, Magnetti Marelli, Hitachi, and a couple of other companies are the primary producers. And guess what? The code is largely the same car-to-car. Parameters are changed- code doesn't, so much. And car companies share "platforms", which further simplifies things.
It's not nearly as scary as it sounds.
Not news to me (Score:3, Interesting)
I can't say I find this surprising. Anyone who has ever worked on software for a US government contractor, or US military contractor, knows the government/military has no one who can analyze the product they pay for. Nearly every software product I've seen delivered is of absurdly poor quality. It would be laughable if the implications of the software's use weren't so disturbing.
Re:This all story starts to look like swine flu (Score:2, Interesting)
Wow, if this is true, then Toyota with-held evidence that could have kept an innocent man out of jail...
Re:Here come the shackles. (Score:3, Interesting)
100 million lines of code on 70 processors? (Score:3, Interesting)
I would be more interested in the process of how
Toyota develops/maintains code. Do they rewrite code for every car?
When they reuse code, how do they retest assertions?
How do they do code verification?
What is their culture when coding problems interfere w/deadlines ?
Is there a whole crap load of unused code in there because
they are scared shitless to remove it ?
etc.
Re:Here come the shackles. (Score:3, Interesting)
None of these tests are entirely effective when dealing with embedded applications.
Bluntly, software tests can only prove the existence of a software bug relative to the specification. For an embedded application, toss the specification out, and start looking at real-world failure scenarios. Glitches on the reset line can cause all sorts of interesting results ... and that is just one possible failure mode.
On a well designed embedded system, most of the dangerous failure modes involve complex unexpected system level interactions.
Re:This is the government, not an engineering firm (Score:3, Interesting)
Re:Here come the shackles. (Score:3, Interesting)
My kids were runover by an out-of-control Mustang about four years ago. There was nothing mechanically wrong with the car. Maybe it was driver error. I don't know, but apparently the accelerator was still stuck to the floor when the police got there. I remember how the cruise control on the cars I've owned will lower the accelerator when the CC is accelerating.
I had a Mustang with an out of control acceleration problem. I was driving down a country road when all of a sudden it kept accelerating. I stomped on the brakes and managed to bring it to about 10 mph, pulled off the road, then turned off the ignition.
The culprit? I had stored it all winter long (this was the spring) and squirrels had used my engine compartment as their own winter storage. An acorn had lodged itself in the throttle cable and held it wide open.
Sometimes what sounds like it might be something more complicated is simple.
Re:advice for anyone with a runaway gas pedal (Score:3, Interesting)
There are two major problems with the "shift to neutral" solution:
1. It doesn't always work.
2. Only a few auto-mechanic and maybe some race car drives have the reflex to shift the car into neutral.
Most people will not think of shifting to neutral when a problem is encountered, simply because they never need to do it. I'm an engineer, and if my car takes off, it will take me a while to think of shifting to neutral. A car at full acceleration can cover much ground in less than 1 second.
The other problem is that I doubt that auto-transmissions will consistently disengage under *fault* conditions.
Your best chance is to be driving a manual transmission. Every manual transmission driver knows to hit the clutch and brakes at the same time, and will do it instinctively. Additionally, the manual transmission is less vulnerable to simultaneous failure modes than the modern computer controlled automatic transmission. For instance, if you are high gear in a manual transmission, it won't automatically down-shift to apply more torque to the wheels when you brake the car to slow it down. Additionally, if the manual transmission is in low gear, it won't up-shift automatically if the car engine takes off. The engine may rev-high, but in low gear, at least you won't be going fast. The manual transmission is much safer in runaway engine conditions.
Standards, not analysis. (Score:4, Interesting)
And safety, not peformance.
Instead of testng code, evaluating the design process, pretending the NHTSA can even begin to become expert in software design, how about applying the old standards to the new systems?
For instance, braking safety. I was listening to and reading the testimony from Rhonda Smith [msn.com], where she even describes shifting her Lexus into neutral. Neutral?
A simple test, and I'm not an engineer, but shouldn't a car come to a stop with 'maximum' brake effort, despite the acclerator position? This is solvable in software - if the brakes are going into lock, and ABS is engaged, engine power and/or transmission state have to be compelled to answer the driver's command to stop. Traction control is already being used in many cars; NHTSA should be able to make a test capable of verifying that even multiple malfunctions are overcome.
Crap, my wife's 1995 Saab 900SE has a mode where the ECU shuts down the fuel pump if the engine stops running, on the assumption that something is terribly wrong, and spewing gas to a stopped engine is pointless if not dangerous. How do I know this? Her car developed a habit of stalling at stops. The real cause was a defective vapor recovery canister, causing loss of vacuum and low RPMs, and the ECU saw that as a stopped engine and made sure it stopped.
Certainly there are other states that can be tested for performance and safety, not some quality of performance standard. Most cars have 'safe' or 'cripple' modes to protect the drivetrain if something seems wrong, like the transmission in a gear that should not permit the indicated speed. My '95 Explorer does that, and it's only an OBD-I system. Acclerator position, wheel speed, and transmission mode should all correlate, and if something is wrong the system needs to cripple - slow down, set a max speed, etc.
Aircraft flight control systems are held out as an example of safety and reliability. Most of these, if not all, have to at least ensure the aircraft doesn't exceed the flight envelope and exceed safety limits. This is the sort standard and evaluation the NHTSA needs to focus on.
Maybe NHTSA needs to borrow a few investigators from the FAA and the military? They should be looking to Boeing, McDonnell, Electric Boat, General Dynamics for expertise in verifying safety in vehicles. Maybe even some NASA people. At least NASA seems to have turned the Shuttle program around a little too late. They certainly have a cautionary tale to tell, and a jaundiced eye towards the assurances of the 'experts' and trusting management.
Which would go a long way to reinstating a somewhat adversarial relationship between the regulators and the industry. There should be some tension there. Hiring your industry's former employees is not the way to go.
We can do so much better. We just need to solve the real problems.
"Resource starvation"? (Score:4, Interesting)
Here's some recent data about the resources available to the DoT, the parent agency of the NHTSA: When the recession started, the Transportation Department had only one person earning a salary of $170,000 or more. Eighteen months later, 1,690 employees had salaries above $170,000 [usatoday.com]. Plus the juicy benefits and pension plan. I'll bet all those managers and supervisors raking in the big bucks would agree that their agencies are "resource starved" and that if they only had more money and more power, they could hire two or three software engineers (for the cost of one manager).