US Air Force Scraps ERP Project After $1 Billion Spent 362
angry tapir writes "The U.S. Air Force has decided to scrap a major ERP (enterprise resource planning) software project after spending $1 billion, concluding that finishing it would cost far too much more money for too little gain. Dubbed the Expeditionary Combat Support System (ECSS), the project has racked up $1.03 billion in costs since 2005, 'and has not yielded any significant military capability,' an Air Force spokesman said in a statement. 'We estimate it would require an additional $1.1B for about a quarter of the original scope to continue and fielding would not be until 2020. The Air Force has concluded the ECSS program is no longer a viable option for meeting the FY17 Financial Improvement and Audit Readiness (FIAR) statutory requirement. Therefore, we are canceling the program and moving forward with other options in order to meet both requirements.'"
New project (Score:5, Funny)
Re:New project (Score:5, Funny)
I'd like to see them implement a CRM system instead
Are the victims of drone attacks complaining much about the quality of service?
Re:New project (Score:5, Funny)
Re:New project (Score:5, Interesting)
Are the victims of drone attacks complaining much about the quality of service?
Most drone attacks are done by the CIA, not the Air Force. If the Air Force launched the attacks, the results could be second guessed by CIA analysts evaluating satellite photos. But if the CIA both launches the attacks and evaluates the results, it is all wrapped up in a neat little package with no loose strings of accountability.
There IS accountability (Score:3, Insightful)
Obama is in charge... the buck stops with him; he's the one who brags about his "kill list"... Oh, wait, this is Slashdot... Obama has a GREAT smile and a cool attitude and nobody is to blame for the drone strikes. Move along, nothing to see here. Dick Cheney is retired so there is no evil to be denounced.
Re:There IS accountability (Score:5, Insightful)
Liberalism and leftism are not a fan clubs. Many of us might prefer the big O over the alternative whilst also deeply disaproving drone strikes against allied countries.
Re:There IS accountability (Score:5, Informative)
disaproving drone strikes against allied countries.
Huh ?!
CIA launched drone strikes on Israel?!
Oh, c'mon ! Pakistan isn't an "allied country". Them Pakis actively support the Talibans.
Re:There IS accountability (Score:4, Interesting)
Oh, c'mon ! Pakistan isn't an "allied country". Them Pakis actively support the Talibans.
We have active drone campaigns against our "allies" Yemen and Somalia.
Our "allies" in Saudi Arabia, Quatar, and the UAE are notorious for funding terrorism.
That should tell you a lot about the quality of our "allies" in the Middle East and Asia.
Re:There IS accountability (Score:5, Interesting)
Re:There IS accountability (Score:5, Insightful)
Well you know...its ok to compromise on such a little detail like that. I mean, its not like people are going to die over it....oh wait...
Yah, thats why I never voted for him. Being the scum floating on top of the other scum doesn't make it any more appetizing to me.
Re:There IS accountability (Score:5, Insightful)
Re:New project (Score:5, Funny)
CRM? Good idea. Then the Air Force can keep better track of its customer data like which Buckwhupistani wedding received a Predator strike package and which one got a JDAM shower and so on.
Re:New project (Score:5, Funny)
Re:New project (Score:4, Funny)
Re:New project (Score:5, Insightful)
Re:New project (Score:5, Funny)
Re:New project (Score:5, Funny)
Are you a consultant?
http://www.despair.com/consulting.html [despair.com]
Ouch. (Score:5, Interesting)
Seems that this is a common theme with ERP rollouts-- scope creep tends to get them all in the end. Granted, most organizations seem to wave off long before the $1 billion mark...
Re:Ouch. (Score:5, Informative)
Oh wow, it gets worse. Oracle won this with a $88.5 million bid; what the hell took the Air Force so long to pull the plug with that kind of overrun?
Re:Ouch. (Score:5, Interesting)
I love how each branch of the DoD gets to pick it's own ERP solution. It says Oracle won it over SAP, not that i have a preference but SAP has a showing of being successful in the market via is use in the Navy. With all ERP solutions there are going to be issues, but overall the Navy has been very successful with their SAP deployment.
Again, why isn't this pushed from the top of the DoD vs. every branch figuring it out and reinventing the wheel each time?
Re:Ouch. (Score:5, Interesting)
One has to wonder if the Navy was all that successful or just willing to handle a portion of the job, or willing to settle for half the result.
You will never know, because those who do have too much ass to cover, and they will be slipping in fixes and upgrades for decades, before deciding the whole thing is too top heavy.
Systems of this size are grandiose and seldom successful. Not only government fails at systems this big, private industry does as well. But private industry learns from their costly mistakes faster. Google is a good example. They hold a house cleaning each spring and just arbitrarily kill off projects that have no chance of a ROI.
Its amazing that two world wars were fought with this kind of stuff being handled by people.
Re:Ouch. (Score:4, Informative)
I fully understand where you are coming from, and i can't answer for the Navy on the system as a whole, but i will say their implementation PM (Plant Maintenance) portion of SAP is a very good example of a very functional implementation that is very effective at doing it's job.
Re: (Score:2)
I'd settle for half the result within budget over getting to 1200% of budget and being told that you're not even half way towards getting a quarter of the result.
Re:Ouch. (Score:5, Informative)
But private industry learns from their costly mistakes faster.
lol.
Let me restate that again: lol.
This comment makes me think that you've worked in neither government nor industry. Or you've been very, very lucky with your employers. Or never worked at a very large company.
Part of the reason (possibly the main one) they fail is due to people. That is the same for both sectors.
With a project that large, it's a big embarressment if it fails, so it's in the interest of the people in charge of the project to force it through at all costs no matter what. Because they don't care about their host organisation (be it government or industry), they care about their own career. Having a big failure like that is a blot. So, instead some half-asses expensive, buggy and minimally functional heap of shit is usually foisted onto the hapless minions of the organisation, usually with a large loss of productivity.
Oracle is usually the cause, and the event should be known as getting "Oracled".
It happens in the public, private and education sector. Oracle knows no limits. They will screw anyone they can get their hands on with crap products. There is no escape.
At least the USAF pulled the plug. After $1bn and a 10x overrun, there is not a single change in hell that the system would every be a net gain. It was a huge fuckup. But given where it was at that time, this was the only sane solution.
The problem is inherent to large organisations. It's not a public versus private problem. It's a big versus small one. That means that the public sector experiences the problems more often due to its size. But basically, large companies suffer exactly the same problems too.
Re: (Score:3)
A a small college I worked at (3500 students, 200 staff/faculty) I watched a 1 year People soft transition roll into a 7 year project, with both consultant and wife hired on as full time db administrators (with complimentary condo on the beach).
Re: (Score:3, Informative)
Yep. And who owns Peoplesoft.
You've
been
ORACLED!!!
Sipping the Kool-Aid (Score:4, Interesting)
I wouldn't be surprised that the DoD is encouraging this. In this way, each branch picks their own solution because they need to satisfy so many domestic "interests". (Yes, SAP America contributes to political campaigns and PACs, just like every other large ERP company in the US). Besides, the only reason that anyone has been successful is probably because they are sipping more Kool-Aid and sitting in a circle "reassuring" one another.
Re: (Score:3)
Re: (Score:2)
I just wish they'd pay me $1 billion to tell them something's not going to work out. Plus I bet I could do it in half the time.
Re: (Score:2)
Re:Ouch. (Score:5, Funny)
Oh wow, it gets worse. Oracle won this with a $88.5 million bid; what the hell took the Air Force so long to pull the plug with that kind of overrun?
What's an order of magnitude between friends. :p
Re:Ouch. (Score:5, Informative)
Re:Ouch. (Score:4, Insightful)
Actually, having 1000 developers working on one project is an excellent explanation for the cost, time taken and failure.
Re: (Score:3)
Let's say you pay a developer $100K/year, and the project lasted 10 years.
Your numbers are way off. Each cleared contractor probably costs the government 200+$/hr. Then there's hardware. If the project has been going on for 10 years, they probably went through at least one refresh cycle. Then there are other costs like admin overhead, facilities, travel etc.
Re: (Score:3)
Re: (Score:3)
ERP is dead! (Score:2, Insightful)
ERP is dead--especially for very large, agile institutions. The only people that don't think so are companies, like Oracle, that are pretending that it can scale to large institutions with some sort of economy of scale, let alone ones that probably make many changes. The fact that it took the Air Force an extra $900+ million to realize this is shameful. Especially since institutions like the Air Force are probably better off looking at agile and adaptive front-end software (it's not just the Marines that
Re:ERP is dead! (Score:4, Informative)
I agree that some decisions can be made to break it up into manageable pieces and accept less efficiency, but with an organization of that size you still have a problem of complexity whether using an ERP package or creating point solutions and integrating them.
Why? Becasue people know it sucks. (Score:4, Interesting)
ERP is dead because word is on the street: Too many failed or seriously delayed implementations.
I have seen (first hand) too many institutions decide to implement ERP, pay a tremendous amount of cash, and watch it fail. If it ever does get fully implemented (in a way that was originally envisioned) the institutions have spent so much time and effort to get it running that the institutions have lost their focus because senior management was distracted or the cost of full implementation has affected the bottom line. In some cases, the institution was irreparably damaged or failed.(often surpassed by their competition).
In theory, ERP is a wonderful thing. In actuality, it can kill.
Re:Why? Becasue people know it sucks. (Score:5, Insightful)
Bullshit. I have been involved in dozens of ERP implementations over the years. The software works. When implementations fail it is always, in my experience, because of the people (i.e. management) making the decisions on how to implement the product.
Me: "Let me show you how Product X handles Accounts Payable"
Client: "That's not how we do it"
Me: "This might be a good opportunity to take a look at your current business practices and see if they can be done in a more efficient way"
Client: "But we've always done it this way"
Me: "Why?"
Client "Dunno...just always have. And I doubt that the team is willing to change"
Me: "Ok, we can customize the product to make it work the way you want but it's going to take more time and money. And when you do an upgrade later on there will be implications as well"
Client: "Fine. Just make it work the way we do it now"
And so it goes. Time and again I see clients go out and buy an expensive ERP system only to customize the bejezus out of it to make it look exactly like the systems they are retiring. They are not open to better business practices. Too many political headwinds.
What does this say about these clowns in the Air Force? It takes them 10 years and $1.03B to realize that the project is going to fail? On an original budget of $88M? One of the big problems with trying to shoehorn a best practice ERP system into a large government institution is that often they employ worst practices. They won't, or can't, change them so you have to end up rewriting the product to fit their ass backwards ways. The whole purpose of implementing an ERP system is to replace aging, stove-piped systems with modern integrated systems. It can work well if it's implemented properly and the right decisions are made along the way. But it's not a magic pill.
Re:Why? Becasue people know it sucks. (Score:4, Insightful)
We have a winner. I am seeing this very poli-drama being played out right now at my institution. The multi-decade tenured staff will not change from business processes implemented to fit a bad system bought 3 decades ago; and will not listen because they don't have to.
Re:Why? Becasue people know it sucks. (Score:4, Insightful)
While I hear what you're saying, government entities, and especially the military, are also subject to legal requirements that they not do things in certain ways, or have unique requirements not accounted for in a 'best practices' system.
Re: (Score:3)
Just as often it's the contractors trying to cut corners, over promising, getting lawyers to weasel them out of contract agreements.
You're assumption the ERP = better business process is wrong. Sometime entrenched process are there for a reason, often a legal reason. Sadly the people who knew that reason have left and no one wanted to spend the money to hire someone to properly record it so they don't know. And they continue to not know until the begin to replace it. Once the agree to replace it they start
Re: (Score:3)
It isn't because it sucks... it is the fact that it needs a champion to be successful, and in a large organization, that champion needs to be a large number of people.
We deployed an ERP system for our small business last year. The core functionality was done previously in Quickbools and various Excel spreadsheets. We spent about $4k per employee on it.
Now we have a system that requires more ongoing money and effort than our old workflow, and for at least 40% of the process still needs to be done in Excel.
B
Re: (Score:3)
Re: (Score:3)
Re:Ouch. (Score:5, Informative)
To my dismay, I worked on this project. The project started with controversy -- the Oracle bid that beat out SAP like seven years ago was surrounded by complaints. The article skips some details. CSC (Computer Sciences Corp, who is quoted) was the main driver of about $800-million of that spending. It is accurate to say that this change didn't affect them, but that's because hundreds of people had already been laid off or moved of the project between last September and last March.
There's enough blame to go all over the place. Years spent in requirements that weren't turned into code; time spent passing blame back and forth across development teams who were so large and segregated that they rarely communicated properly, both within the Air Force and within CSC and between the other teams. At it's peak I believe the project had roughly 800 people on it. I don't know what the maximum size a development project should have, but it's got to be smaller than that. That number includes everyone, trainers, managers, and some key initial users and testers, but still it's a very high number.
The Air Force tried several times to realign the project, but there were contractual disputes or, once that was over, difficulty deciding what to keep and what to scrap, which lead to a death spiral where everything went back on the drawing board and I think ultimately leadership just lost hope.
It wasn't a complete loss, though. A few small teams, including the one I was previously on, have survived. We built a robust data quality system and are working on some enterprise data dictionary and master data tools, which will help the systems that are left behind. With hundreds of systems supporting a half million users, $1billion probably isn't off the chart -- at least not had this been a successful project, but the worst part is that there's still much work that needs to be done, and now someone will have to start over... again.
Re:Ouch. (Score:4, Funny)
Re: (Score:3)
Wow. I guess that this is a new record, eclipsing even the FBI's failure from a couple of years ago. Have to say, I am impressed. Leave it to the Pentagram to do things bigger and worse than anyone else on the planet.
In fairness, in terms of employees the US Air Force is 10X the size of the FBI, and bigger than any but a small handful of corporations. In terms of assets and materiel, military forces have one or two orders of magnitude more than a comparable corporation. It's pretty much guaranteed that their expensive failed efforts are going to be bigger and worse than just about anyone's.
Re:Ouch. (Score:4, Informative)
Granted, most organizations seem to wave off long before the $1 billion mark...
Most organisations aren't connected to the DoD's endless money spigot.
Re: (Score:2)
Re:Ouch. (Score:5, Insightful)
Seriously, has there EVER been an ERP implementation that was anything other than a colossal fuckup? Way behind schedule, overbudget, and not functioning properly are the general themes of ERP. And businesses continue to fall for this scam.
Re:Ouch. (Score:4, Insightful)
That's because it's usually the head of the accounting department that gets to approve it. Farking ridiculous.
Re:Ouch. (Score:5, Insightful)
There are big failures, typically in situations where the size of the project exceeds the experience and capabilities of the people managing them. With something as big as the DoD, there just aren't too many opportunities for anyone to gain the proper experience to know how to make it successful. Something like that needs to be broken into much smaller pieces and you just have to forego some of the efficiencies of a completely integrated non-redundant system in favor of more manageable pieces.
Re:Ouch. (Score:5, Informative)
Re: (Score:3)
No they don't, because such a thing would be impossible.
Individuals change jobs and when they do, they generally mention their prior experience. Consulting firms have resumeés too, even if they call them "portfolios".
Re:Ouch. (Score:4, Interesting)
Once you have modelled the 5th separate way to order stionary and the umpteenth vacation policy for a department of 5 you know that you are screwed. I wouldn't speak of a system as such but rather a set of specific exceptions.
It is always the same pattern. And since you never start small and you never start flexible you will end up with a bloated, slow hairball that approximately does was the customer wants. Not what he needs.
Re: (Score:3)
The Air Force propably has a lot more red tape than that. Red, white and blue tape tangled up in a fine mess.
I remember when my country(Germany) wanted to introduce a unified system for our state police(we've got 16 stat
Let me guess... (Score:3, Funny)
jobs program (Score:3, Interesting)
I know lots of programmers who can get the same result for half the price.
Re: (Score:2)
Yeah no shit. I wish I could bid and actually have a chance at winning one of the contracts. Then it would at least get done and not cost 1 billion dollars.
Re: (Score:2)
Re: (Score:2)
Maybe a pattern here (Score:5, Insightful)
From my observations, I've concluded that no organizational group works toward reducing its size, reducing the amount of its discretionary budget, or increasing its accountability for the preceding.
Any exceptions?
those billions (Score:5, Insightful)
Re:those billions (Score:4, Interesting)
Re:those billions (Score:5, Insightful)
Because civilization is built with taxes.
There a plenty of countries that have no to little taxes, you are welcome to move their and enjoy the squalor and disease.
BTW, you benefit from social programs. Less crime, more industry, more entertainment, better beer.
Re:those billions (Score:5, Insightful)
"Those" billions? It's one billion, singular.
The US government spends 19% on defense, 19% on social security, and 20% on healthcare [reason.com]. The last two items are expected to grow much faster than the first.
Useless? Do you know what a "contested sea zone" is and how it affects commerce? No? Yeah, that's what I thought, and the reason why is overwhelming dominance. Assuming, of course, you like imported coffee at the hip indie coffeeshop and hipster fruits like the Durian instead of that crap domestically made junk.
Re: (Score:3, Funny)
A billion here, a billion there. Pretty soon, you're talking about real money.
--(sic)
Re: (Score:3)
The US government spends 19% on defense, and refunds 19% on social security and 20% on healthcare to recipients among whom many have past contributions in excess of benefits received in total. (This is the nature of insurance, you know. Insurance is a communist plot.)
Wrong. Social Security and friends almost invariably gives back much more than the recipient paid in contributions. Which is why it is not insurance, but a Ponzi scheme backed by general tax revenues (I remember the uproar a while back when s
Re:those billions (Score:4, Insightful)
The only thing you proved is you don't know what a Ponzi scheme is.
It's missing 3 crucial elements
1) A 'undisclosed' way of making money
2) A handful of people collecting the majority of the money.
3) Unsustainable under any condition. Meaning no adjustment can be made without collapse the whole thing.
In a Ponzi scheme, the people 'late' the the investment(which is everyone who didn't start it) won't get anything out of it.
Social Security is running really well, it has minimal overhead, it's accountable, has money set aside, and is designed to allow for adjustments along the way.
It should be held op up as one of Americans crowning achievements. Right there with the Hoover Dam, Golden Gate Bridge, the Interstate highway and putting a men on the moon.
But republicans don't like it so they keep lying about it.
This is wrong... (Score:2)
There should be a criminal negligence investigation into this.
With at least eight full-lifecycle development projects under my belt as both a Software Engineer and a Development Team Lead I cannot even wrap my mind around the amount of irresponsible waste that would be required to throw away that much money.
The real news here (Score:2)
The real news here is that a branch of the military actually avoided the sunk cost fallacy. I know it's probably not the first time. Nevertheless, I can't help but wonder if they will use the money they save for porcine pilot training.
Re: (Score:3)
Too Much Time In Pocket D and The Busted Flagon (Score:2)
I think I must have spent too much time idly hanging out in RP areas like Pocket D in City of Heroes and The Busted Flagon in Guild Wars 2. Shamefully, I first saw the headline "US Air Force Scraps ERP Project..." as "US Air Force Scraps Erotic Role Play Project..."
Someone care to explain what this is exactly? (Score:2)
Since this thread is just going to be a bunch of "zomg wasted muney!" why don't you educate academics like me about what exactly "ERP" systems are and what you do with it and why its so great?
The university I work at gets new crazy "enterprise" software sometimes and usually it ends up offloading some of the work the bureaucrats used to do on me (purchasing paperwork) meanwhile they take 51% of my grant money.
So tell me, WHY?
Re:Someone care to explain what this is exactly? (Score:4, Insightful)
Answered my own question:
http://www.youtube.com/watch?v=PVRgIXLWDHs [youtube.com]
Re: (Score:3)
It manages stuff. People, salaries, suppliers, inventories, clients, payments and whatever else you can think of. In the company I work for we just did a whole school management system (students, teachers, evaluations, etc) on top of OpenERP [openerp.com] (python based, AGPL licensed ERP).
Re: (Score:2)
It's "great" because you can't really run
in other news... (Score:2)
That's nothing! (Score:2, Interesting)
Re: (Score:2)
Compare to our Canadian (1/10 population) gun registry it cost up to $2B and scrapped.
$2B that would have been better allocated to teaching you punctuation and grammar!
Re: (Score:2)
Re: (Score:3)
According to your link, it was estimated at 2M and ended up running 66M.. a far cry from 2B but, still a 33x increase in cost so very respectable fail there.
Naturally (Score:5, Insightful)
ERP is a bunch of disparate functions mashed together then held in place with a metric assload of duck tape. It's only natural that if you try to tacle the whole thing at once the result will be a sort of dynamic paralysis where you run back and forth in a nearly random pattern burning money all the way.
Just as well, if you ever manage to build the thing, you'll create paralysis across the entire company if you suddenly drop this chimera on people's desks.
Note, I am NOT claiming that the individual functions aren't necessary nor am I claiming that they shouldn't support common data formats.I am claiming that trying to build the whole thing at once and as a single 'solution' is wrong headed and doomed to failure.
Re: (Score:2)
You shouldn't need to build the whole thing at once. A decent ERP system is modular, and can be easily upgraded in place. And while there's always some duck tape, it's still much better than an assorted collection of programs, often times written in different languages and running on different machines (e.g. client vs web based). One of our clients was doing "IPC" by manually adapting files in Excel!
(Disclosure: the company I work for does projects based on a free AGPL licensed ERP system)
Re: (Score:2)
Correct, you shouldn't do the whole thing at once. Unfortunately, too many (including the USAF apparently) try anyway. The ability to run distributed is a good thing. That doesn't necessarily (and shouldn't) mean a mis-mash written in different languages and to different coding standards.
Re: (Score:2)
I've seen this happen repeatedly. Trying to implement a Grand Plan usually results in nothing.
Almost every large project I've dealt with and was delivered without any major catastrophes was rolled out piecemeal. We picked the core/representative functionality, nailed the scope down with a sledgehammer and built it as "phase 1". All scope creep got pushed to "phase 2". Once phase 1 was done we just pick stuff out of the phase 2 bucket that can be done in the allotted time and just repeated the process. Obvio
Worst of Both Worlds (Score:3, Insightful)
Perfect application of Hanlon's Razor [wikipedia.org]: Not so much a conspiracy to waste money as the worst combination of both world (defense acquisition and enterprise software development). Both fields are very prone to overruns, scope creep, and repeated waste of funds as manager after manager--or contractor after contractor--throws away work to start over again. Another great example is the FAA's version of enterprise software [wikipedia.org], which is currently at $63.4 BILLION and counting (though, to be fair, it's quite possible the most complicated software project in the world).
Still, there are worse examples--specifically, when these kinds of overruns, violations, and program restarts are done deliberately to ensure continued funding to entrenched players in a limited field and / or to pursue minor permutations on someone's pet dream of a project. This can occur at the cost of throwing away many years and billions of dollars of decent work while never really getting closer to a functioning system. Space Launch System [wikipedia.org], anyone? (Not a software example, but the line between software and aerospace engineering is a lot thinner than most people realize.)
Enterprise resource planning? (Score:5, Funny)
If only they'd had a better ERP system, they could've planned this project more carefully, and put all those resources to better use.
Re:Enterprise resource planning? (Score:5, Funny)
Why big projects fail (Score:2)
In der Beschränkung zeigt sich der Meister.
A total success then .. (Score:3)
Bullet dodged (Score:3)
Its only virtual money (Score:3)
When a country is trillions in debt its all just fictional amounts of money anyways.
entitlement spending (Score:3)
Re: (Score:2)
The Thief In Chief blew over an extra *trillion* *every* *year* and people weren't' smart enough to fire him.
Well, that didn't take long.
Re: (Score:3, Informative)
How is that partisan? *All* recent Thieves-in-Chief blow trillions, what changes is who the main beneficents are.
Dubya: big oil, military contractors
B. Hussein: wall street, big media, big pharma
[would be] Mittens: wall street, wall street, wall street
And Obama's bailout has been more harmful that all recent wars put together. It ensured no financial companies not connected to the main mafia can thrive: they were either bankrupted, bought out or marginalized, while investors received a clear message that
Re: (Score:2)
Re: (Score:3)
We tried, but they swift-boated the other guy. I guess we got the president we deserved for that one...