Computers May Thwart 2010 Census

smooth wombat writes "With the Constitutionally mandated census of 2010 just around the corner, it appears the Commerce Department's attempt to use handheld computers to gather census information may not come to fruition. Originally, the contract was awarded at a cost of $596 million to Harris Corporation. However, the GAO has now estimated the revised contract, now costing $647 million, could grow to $2 billion and the equipment may still not work properly. There is consideration that the paper and pencil method might have to be employed to complete the census."

Any history buffs out there?

[pegr (46683)]

Recall that Herman Hollerith came up with punched cards for the 1890 census. He founded the company that became IBM. Here's some linky goodness [wikipedia.org].

Re:Any history buffs out there?

[xav_jones (612754)]

Yeah punched cards seem to have worked well for Florida ...

Re:Any history buffs out there?

[Creepy Crawler (680178)]

Yo Chad, whats hangin?

Re:

[Lilith's Heart-shape (1224784)]

I remember now. I remember how it started. I can't remember yesterday. I just remember doing what they told me.

Re:

[Kozz (7764)]

I remember now. I remember how it started. I can't remember yesterday. I just remember doing what they told me.

Politicians say "no" to drugs, while we pay for wars in Saudi Arabia.

Re:

[An ominous Cow art (320322)]

But now the holy dollar rules everybody's lives, gotta make a million doesn't matter who dies.

Re:

[KevinKnSC (744603)]

They'll still use computers for all the tabulation at census offices. They're just talking about replacing the traditional paper-and-pencil questionnaires used by Non-Response Follow-Up Enumerators with a handheld computer. If the computers aren't ready, they'll go back to the optical scan forms like they used in 2000. As an aside, I was a NRFU Enumerator for the 2000 census, and suggest it as an excellent part-time job for students, retirees, those with flexible schedules, etc. It's good pay for being

Anyone have any idea...

[Darundal (891860)]

...what accounts for the differences in the estimate and the cost? What cost(s) were underestimated?

Re:Anyone have any idea...

[petermgreen (876956)]

Government departments often have regulations that require them to put contracts out to tender and give them to the lowest bidder to prevent people handing out government contracts to thier friends.

The problem is if they put them out for bidding as fixed price contracts they probablly wouldn't get any bids and if they did those bids would be very high. So the bids are only estimates. Of course this makes the bidding a farce as everyone tries to put in the lowest estimate they can and sponge more money later once the governement department is locked in.

Re:Anyone have any idea...

[shadow349 (1034412)]

So the bids are only estimates.

I've done work on construction estimates that were of similar (~$250M) size that were submitted as "GMP" - Guaranteed Max Price. Yeah, Change Orders will pad that somewhat, and there is some contingency, but together they won't be more than 15%.

Of course this makes the bidding a farce as everyone tries to put in the lowest estimate

Which is why I've always felt that the process should pick the second lowest bid. It's trivial to shoot for the bottom ... it's impossible to shoot for second from the bottom.

Re:

[Dahamma (304068)]

Which is why I've always felt that the process should pick the second lowest bid. It's trivial to shoot for the bottom ... it's impossible to shoot for second from the bottom.

Unless there are only two bidders.

Re:

[Urza9814 (883915)]

True story I was just told today: Girl working in Procurement at a US Army Depot is sent a list of parts needed for a radio. The radio is supposed to cost $30 each. She brings the list to her superior officer to get it OKed, and he notices the resistors listed are +/- 5% tolerance. He decides that the army deserves the very best and changes it to +/- 2%. It then has to be sent to his superior too, who makes the same decision and changes all the resistors to +/- 1%. In the end, the radios cost $250 each.

Re:Anyone have any idea...

[NewbieProgrammerMan (558327)]

Yay for non-technical decision makers making technical decisions!

My favorite procurement/supply story is from a guy I worked with in the Navy; we needed a switch for some system that required replacement parts to be some super-special reliability grade, so he got the part number and called up the company from home. "Hi this is Billy Joe Ray Bob's electronics supply in Podunk, Lousiana; I need a BR-549 limit switch, how much are they?" The answer was something like $12.

He calls back a week later as Petty Officer so-and-so from the USS Neveryoumind, and then the BR-549 limit switch costs $349. Apparently the super-special reliability grade sticker they put on it after they take it out of the bin costs $337.

Re:

[DerekLyons (302214)]

That would be a bullshit story - as the Army buys complete radios, not components. It would also be bullshit because when the Army does buy components, its buys them to MIL-SPEC, no more and no less.

Re:

[porkchop_d_clown (39923)]

...And, since this is the government, instead of fixing the problem by requiring individuals who are qualified and competent to administer the census...

And, what, exactly, do you have against equal opportunity?

Re:

[SourceVisigoth (141614)]

I could buy that argument if it were possible to recruit, train, and manage 800,000+ people who have decent technical skills for a low paying, short term job. Keep in mind that that most non-response follow-up enumerators work for 2-3 months tops, make around $12/hour, are part-time, and have no particular interest in data integrity or security.

The decennial census absolutely needs to be a turn-key operation for the tens of thousands of local recruiters and trainers. When you have an organization that exp

Open source

[Missing_dc (1074809)]

The bloat is occurring because the project is not open-source, it could be done for pennies, but would take 20 years to complete.

Re:

[KevMar (471257)]

I have this gut feeling that it would only compound the issue.

The article says they only had a 1% failure rate in field tests. I bet the crew of 20 to 30 year old tech guys had no issues with it. They under esimated the end users. Yes, some systems are very simple but you still find people that can't figure them out. Not only were more computers "breaking", the support calls would have been greater then expected.

With electronic, you now have to pay for the support of that electronic component.

1% error

[Red Jesus (962106)]

He also said the computers actually are easy to use, with a failure rate of less than 1 percent when tested in the field.

One percent of three hundred million is three million.

Re:1% error

[eepok (545733)]

Ya, people are really bad when it comes to big numbers. I was a part of a research study as an undergraduate with the following premise:

You're on a jury for a murder case with the scenario that a tan/brown man seen running away from a murder scene on a college campus. There was not enough of the attacker's DNA at the scene, but they were able to extract a DNA derivative that has matched that of a tan man in custody. Given that this derivative has a 99.9% successful rate, do you feel comfortable convicting the man in custody.

I was the only one in my group of 12 to say "No, I will not convict based on this evidence." No one else understood that .1% = 1/1000. Nor did they realize that our university alone had 20,000+ people on the campus at any time let alone that it was in the middle of a city of 200,000+.

Most people know what "fifty" is. Many know what "one hundred" is. Few understand what "one thousand" is. Too few understand the effects of millions, billions, and trillions.
There's no way I'd convict with a .1% error, there's no way I'd accept a 1% error in the business of millions.

How to get big numbers across

[sconeu (64226)]

Many people don't understand big numbers... My favorite way to get across just how big ONE BILLION (pinky-to-mouth) is, is as follows:

A rich man wanted his wife to stop bugging him for money. So he gave her $1Million, and told her to spend $1000 a day.

Three years later, she came back, and said she was out of money. So he gave her $1Billion.

She didn't bug him again for 3000 years.

Re:How to get big numbers across

[Shakrai (717556)]

Uhhh, $1 billion is 100 times $1 million

Google [google.com] doesn't seem to think so....

People in stone houses shouldn't throw bricks moron.

Indeed.

Re:

[SatanicPuppy (611928)]

1% is pretty good in the real world...Most hardware failure rates for hand held devices are much higher than that.

As far as 99.9% certainty...It's almost impossible to get that good in the real world. What would be your standard for guilt? Eyewitnesses, fingerprints, photos; for the most part they're not 99.9% accurate for identification purposes.

It's an ugly inductive world. You're never going to be 100%

wow... ideal role for the XO

[advocate_one (662832)]

of course do a more businesslike version with a larger keyboard... but the XO with custom census gathering application saving the data off onto flash drives would have been perfect... pity the timescale is a bit short now...

Horrible...

[Stanislav_J (947290)]

How could we possibly do a census with paper and pencil? I mean, we've never done it that way before, right?

Re:

[JonTurner (178845)]

I agree with your sentiment, however the problem with the "modern" census is, for many citizens, it goes far beyond the simple enumeration of all citizens proscribed by the Constitution and has become a multipage survey asking questions about plumbing, commute times, what languages are spoken at home, who raises your children, where you work, etc. It's so extensive, the government estimates it will take a person 38 minutes to complete the survey!
See for yourself: http://www.census.gov/dmd/www/pdf/d02p.pdf [census.gov]

Are you serious?

[TripMaster Monkey (862126)]

Originally, the contract was awarded at a cost of $596 million to Harris Corporation. However, the GAO has now estimated the revised contract, now costing $647 million, could grow to $2 billion and the equipment may still not work properly.

1.4 billion is one hell of an overrun...and after all that, the equipment may still not work properly?

Is the Harris Corporation currently hiring? I'd like to get me some of that boondoggle.

Re:Are you serious?

[moosesocks (264553)]

The more pressing question on my mind is why they haven't been sued into oblivion. How can you seriously get away with that?

Re:Are you serious?

[Hatta (162192)]

Corruption.

Re:

[Tablizer (95088)]

The more pressing question on my mind is why they haven't been sued into oblivion. How can you seriously get away with that?

Something this large and complex probably results in "gripe room" for both sides. The company can probably cite delays or problems with information, personal, staff testing, etc. that the gov't was supposed to supply but was either late with or bungled. And there's probably some vague contract wording that can be used as a weapon by both sides. It's an age-old dance with these kinds

Another waste of money

[Harmonious Botch (921977)]

Between driver's licenses, utilities, medicare, social security, public school enrollment, arrests, and other records, a good statistician should be able to get an answer that is close enough. To double check the results, canvass a few dozen randomly chosen counties, then adjust accordingly.

But there is no reason that counting people should cost over half a billion dollars.

We should be able to contract this out. Offer maybe a mere 50 million dollars to the entrant that can produce the best results. Anyone can enter. They do their counting by whatever legal method they choose. THEN the census dept does their random counties, and whoever is closest on those counties gets paid, and their results for the whole country are used.
BTW, I'm assuming here that a census should be just counting heads; that all of the other questions that the census people ask, such as level of education, are none of their business. The constitution requires that people be counted. The goal was to ensure proportional representation. It does not require all of the intrusive questions that they ask now.

Re:Another waste of money

[drinkypoo (153816)]

BTW, I'm assuming here that a census should be just counting heads; that all of the other questions that the census people ask, such as level of education, are none of their business. The constitution requires that people be counted. The goal was to ensure proportional representation. It does not require all of the intrusive questions that they ask now.

You are not required to answer any other question on the census, either.

You can really just say "nine people live here, go away" and they will.

All that information IS necessary for the government to provide all the services they provide today in a reasonable and efficient manner.

Unfortunately, it would also require that those in charge be interested in reason or efficiency. All they want to do is separate you from money.

So, I agree, but only in that the government should get their nose out of places it doesn't belong in a more general sense. Unfortunately, we could probably argue about what those things are all day.

Re:

[Archangel Michael (180766)]

". Many Americans are concerned that the lowest income quintile keeps getting poorer."

This is because of the fallacy of believing the pie is only so big, and has to be divided equitably. However, the pie is actually variable in size, and all one has to do is increase the size of the pie by working to increase one's proportion of the pie.

There are plenty of unemployed people unwilling to work at any wage under a certain amount, even though they aren't qualified to do any work that provides such a wage as the

Re:Another waste of money

[Harmonious Botch (921977)]

....the lowest income quintile keeps getting poorer. If the Federal government is going to do anything to help these people along, it needs to know a little more about them.

Why are we implicitly assuming that the feds should do anything in particular to help these people? The last thing they need is more government 'help'. The 'war on poverty' has been going on for 40+ years now, and - as noted - the poorest group is getting poorer, not wealthier.

If you think Congress makes dumb decisions, then complain about the dumb decisions. But don't cut off their access to information that could help them make better decisions.

Parent - and congress - seem to assume that if you just knew enough about people, you could make good decisions for them. The latest atack on poverty didn't work? Well we just need more data.

But the fact is that you can never know enough about a person's needs and circumstances as the person himself. Leave him alone, quit meddling in his life, lower his taxes ( by not wasting his money on cencus boondogles ) and he will probably get rich on his own.

Is Diebold making the computers?

[Tired and Emotional (750842)]

Lets hope not. The population of the West Coast and the North East combined would come out at zero!

Foreign census experience

[Bombula (670389)]

I was living in a Gulf country a few years ago when the government there conducted the census. They just sent out an army of 20-somthings with PDAs to do the surveys. I believe most of the survey was multiple-choice, but there were some numeric entries (how much I earned per month and what y rent cost, for example). You could do those with multiple-choice too, obviously, with a selection of ranges.

The whole census survey took about 15 minutes. They collected a lot of data - I'd say there were between 60 and 80 questions. Since I'm a geeky sort of person, I asked the kid how it worked and he showed me - the PDA (a Compac Pocket PC) just ran a macro in MS Office which dumped each survey as a file into a folder. That folder synced via wireless/mobile-phone link to where the main data center was.

The country has a population of about 4 million, and he said there were 200 people doing the survey for several months. Seemed pretty straightforward, and I can't imagine it cost that much - certainly labor and not the PDAs was the primary expense.

Re:

[Sentry21 (8183)]

I worked for a company that did mobile data collection software, including actual survey design (flowing questions, ask this question based on this question's answer, filter these responses, ask this set of questions once for each child listed in previous questions, etc.).

This stuff is trivial to implement if you do it right, and all it takes is commodity Palm hardware (or PocketPC hardware running an emulation layer, or Windows tablets). It's trivial to do, syncs automatically, and can export all the data

Paint me Blue...

[whisper_jeff (680366)]

Paint me blue and call me stupid, but really, how hard is it to make a hand-held computer designed to take and store census data? It's not like these machines need to calculate pi. It's data entry and retention. Right? How could that possibly require $2 billion dollars to implement? What am I missing? (beyond the obvious corruption and inflation of budgets to line the pockets of fat cats)

Major IT failures seem so common

[by Anonymous Coward]

Bet if you did a study, a serious one, you'd find there's an irrefutable inverse relationship between the amount of money bid for a project and the success of that project. I know it sounds like a flippant witticism, but I'm sure of it, do the research and the figures will prove a direct *causal link* between the amount of money put in and project failure.

I mean, what is it with these large scale IT projects? They take a simple problem and turn it into a money pit. Here in the UK we've had several high profile massive budget IT failures in the last 10 years, air traffic control, national health patient record databases, in fact the more critical it is the more of a spectacular unqualified fuck-up it becomes.

Now, if you got a couple of average hacker nerds and gave then the same specs, but didn't tell them it was for a large scale project, or for whom, they would give you a faultless solution using commodity hardware, stock methods and free software in a few months at one *millionth* the cost we're looking at here. Every one of you here knows it to be true. So, my question is, what goes wrong? How can it possibly go so wrong? Are the people involved complete idiots? Or corrupt?

What are the factors that turn a simple software project into an impossible task? Is it the stress of high budgets? Too many crooks spoiling the broth? And more to the point, when is some bright person going to break from this pattern of failure and realise that to award a major government IT contract to *more than one* complete no-name outsiders bidding a fraction of the cost makes more sense than giving billions of dollars to one contractor and putting all your eggs in one basket?

Re:Major IT failures seem so common

[DerekLyons (302214)]

Bet if you did a study, a serious one, you'd find there's an irrefutable inverse relationship between the amount of money bid for a project and the success of that project.

I can refute it without even breaking a sweat. The Manhattan Project. The Apollo Project. The creation of the Polaris, Atlas, and Titan missiles... The creation of nuclear powered ships... Etc... Etc... Big ticket projects all - unqualified successes all.
 
 

I mean, what is it with these large scale IT projects? They take a simple problem and turn it into a money pit. Here in the UK we've had several high profile massive budget IT failures in the last 10 years, air traffic control, national health patient record databases, in fact the more critical it is the more of a spectacular unqualified fuck-up it becomes.

Mostly because we really don't have all that much experience building huge monolithic IT projects from scratch and to spec. The vast majority of the [truly tremendously] big IT projects to date (the telephone system, the networks big banks use, etc...) have been built piecemeal and grown from small beginnings.
 
 

Now, if you got a couple of average hacker nerds and gave then the same specs, but didn't tell them it was for a large scale project, or for whom, they would give you a faultless solution using commodity hardware, stock methods and free software in a few months at one *millionth* the cost we're looking at here. Every one of you here knows it to be true.

I know it's a common conceit of IT workers to believe so. I don't believe for a single second that it's true. 'Average Hacker Nerds' have essentially zero experience in building large systems, triply so for distributed ones.
 
 

So, my question is, what goes wrong? How can it possibly go so wrong? Are the people involved complete idiots? Or corrupt?

Or, just maybe, the projects are Really Hard in extremely specialized project domains.
 
 

What are the factors that turn a simple software project into an impossible task?

The persistent belief that these projects are 'simply software' and thus easy to do. Especially among people with essentially zero knowledge of the problem domain(s) and the issues involved.

Who can you call?

[by Anonymous Coward]

I'm pretty sure MacGyver did his own 1985 census with a paperclip, a piece of scotch tape, and 3 guys he found standing outside a Home Depot in Tucson.

My Other Computer is a Pencil...

[mkcmkc (197982)]

One thing that 20+ years in the computer industry has taught me is that if you don't need a computer for a task, don't use one.

Coincidentally, my first paying job was working as a US Census enumerator for the 1980 census. Paper worked fine. The real problems were with my fellow citizens who didn't want to be enumerated (which I can understand, though calling the police on me seemed like overkill).

Finally, apropos of this topic, I recently discovered that the best "organizer" in the world is an empty file folder (or perhaps several) and a supply of sticky notes. Portable, easy to reorganize, no problem if you run your car over it, easy to back up, etc.

Canada had census forms online in 2006

[LordEd (840443)]

Our last census had the option of filling out the census forms online. I didn't find out how many actually did it, but they were originally estimating 20% usage. Instead of getting the full booklet to fill out, you got an access code.

http://www.cbc.ca/story/canada/national/2006/04/27/online-census060427.html [www.cbc.ca]

While searching for a reference article, i found that there were some issues with Linux users, although they attempted to correct it.

http://www.linux.com/articles/54366 [linux.com]

Republicans are "Computers"?

[Doc Ruby (173196)]

So how does yet another Republican boondoggle contract for an essential government service mean that "computers" will thwart the 2010 Census? Are these incompetent Republicans really just a computer simulation?

Maybe this really is all just some kind of Y2K bug VR nightmare. Would someone please reboot Gore, so I can go back to watching _the Simpsons_ when it was still funny?

It is done elsewhere

[Thelasko (1196535)]

This is not new and pioneering technology. There are companies [acnielsen.com] that take similar surveys for market research purposes. Have you ever been asked to take a survey at a mall? Have you ever been at a bar when a beautiful woman with a tablet computer asks you to take a survey about cigarettes? I have. The Government is wasting billions of dollars to develop technology that has existed for years.

Why doesn't the government just outsource the whole census to a market research company and be done with it?

Break the Law

[Jherek Carnelian (831679)]

Census data was used to round up japanese-american citizens for interment camps during WWII:
http://www.usatoday.com/news/nation/2007-03-30-census-role_N.htm [usatoday.com]

With the current "war on the unexpected" who knows how current census data will be used to abuse citizens like yourself.

Good old government waste.

[MaWeiTao (908546)]

They could have purchased hundreds or thousands of off-the-shelf PDAs and had a company develop a basic piece of census-tracking software for a mere fraction of what this project costs. Instead they'll argue they need some elaborate, over-priced piece of hardware under the pretense that only something so fancy can reliably handle the government's needs. The best part is that the devices might not even work properly. What in the hell are these companies doing that even with this much money thrown at them they can't do anything right?

Still, that doesn't excuse the government's stupidity. It's like that stimulus package. As if enough money hasn't already been dumped into that some halfwit decided they needed to send out letters informing recipients that they were going to be receiving these checks. In many cases these notices will be arriving barely a month before the check arrives. Sending these letters out has cost the government over $40 million.

It's time the government's budget were capped at the rate of inflation making allowances only for population growth. It's time they learned how to manage their expenses like the rest of us have to.

Re:

[steveg (55825)]

In 52 years I've been counted once, to my knowledge. It was either the 1980 or 1990 census. Never before or since.

Canada's been there, done that

[WebCowboy (196209)]

can we not allow people to register for a census on the internet?

Sure you can. I submitted my census questions via secure website during the last census in 2006...but that was in Canada. It was easier, and certainly less expensive to process (didn't save paper though, because everyone still got the mailer; you could fill it in and mail it back or log in with the information provided in the mailer).

I'm not sure about how it goes in the US, but sending out canvassers only covers about two percent of data co

Re:

[statemachine (840641)]

Read the original version of the Constitution. There are provisions for fractional people.