Is The DOJ Using Obsolete Software To Subvert FOIA Requests? (theguardian.com) 85
"A new lawsuit alleges that the U.S. Department of Justice intentionally conducts inadequate searches of its records using a decades-old computer system when queried by citizens looking for records that should be available to the public," reports The Guardian. Slashdot reader Bruce66423 writes:
An MIT PhD student has filed a suit in Federal court alleging that the use of a 21-year-old, IBM green screen controlled search software to search the Department of Justice databases...constitutes a deliberate failure to provide the data that should be being produced.
Ryan Shapiro's lawsuit alleges "failure by design," saying that the Justice Department records are inadequately indexed -- and that they fail to search the full text of their records when responding to requests "When few or no records are returned, Shapiro said, the FBI effectively responds 'sorry, we tried' without making use of the much more sophisticated search tools at the disposal of internal requestors." The FBI has a $425 million software system to handle FOIA requests, but refuses to use it, saying that would be "needlessly duplicative...and wasteful of Bureau resources."
Ryan Shapiro's lawsuit alleges "failure by design," saying that the Justice Department records are inadequately indexed -- and that they fail to search the full text of their records when responding to requests "When few or no records are returned, Shapiro said, the FBI effectively responds 'sorry, we tried' without making use of the much more sophisticated search tools at the disposal of internal requestors." The FBI has a $425 million software system to handle FOIA requests, but refuses to use it, saying that would be "needlessly duplicative...and wasteful of Bureau resources."
Re:intent or consequence? (Score:5, Insightful)
When the Federal government is invloved, don't blame on intentional malice that which can be explained by consequences of Republican budget cuts.
No. This is wrong. They already have the tools to do better. They simply refuse to use them. If they had to buy some new software, you'd have a point. They don't, so you don't.
Re: intent or consequence? (Score:3)
Having previously worked for a large corporation, I found that the costs are always limited to the ticketed price. There are others including employees to run and manage the system; process change; new audits; infrastructure; politics; expertise; training; software limitations leading to longer execution time.
Sometimes saving money leads to spending more money and creating more headaches. Not always, but you do need to understand the nature of the environment you are dealing with.
In our particular case we h
Re: intent or consequence? (Score:1)
if it ain't broke don't fix it
Is it just me or is anyone else getting pissed off with this excuse for shitty ancient systems still in use today?
Sure, by all means do not try to 'fix' something already working well.
But there's the keyword of the day: well.
This system is beyond well, another universe of not well.
A lot of these older systems are also usually maintained by awful sysadmins that are lazy, and even geared the whole system in weird exotic ways only they can understand, on purpose, so that they can use this to secure their job b
Re: (Score:1)
Is it just me
Yes
Re: (Score:2)
It's always a matter of priorities. The FBI would rather chase down 'pirates' than complete FOIA requests.
Re: (Score:2)
It's always a matter of priorities. The FBI would rather work for their corporate overlords than do something for the public.
FTFY
Re: (Score:1)
...because you want ...
As your message so aptly illustrates, you have no idea what I want. :)
Re: (Score:2)
Re: (Score:2)
Why yes, I do!
Re: (Score:2)
Voting bloc (Score:2)
... intentionally...
When the Federal government is invloved, don't blame on intentional malice that which can be explained by consequences of Republican budget cuts.
It's too bad we don't have another party, a group of people who could come together and vote as a bloc to prevent these sorts of outrages.
Re: (Score:3, Insightful)
Political flamebait works both ways. The other side of the coin is that Democrats set up overly complicated systems that can't work without an ever-increasing price tag, then complain (loudly) that they just aren't getting the support they need.
Let's move on.
When the Federal government is [involved], don't blame on intentional malice that which can be explained by...
...anything else.
Bureaucracy in general is a breeding ground for unintentional malice. There are literally thousands of people in the federal government with the ability to influence programs like this, and they often have conflicting priorities. Some a
Re:intent or consequence? (Score:5, Informative)
Political flamebait works both ways. The other side of the coin is that Democrats set up overly complicated systems that can't work without an ever-increasing price tag, then complain (loudly) that they just aren't getting the support they need.
The problem is that that assertion doesn't line up with reality. Go down to your DMV some time, and observe the kinds of systems that they're using. They're using databases built in the 80s and 90s on top of DOS, running on ancient computers with CRT monitors (at least around here).
What reasonable business do you know of that hasn't upgraded their systems since that time to allow for more efficiency savings, faster processing, reduced staff costs etc?
There's a lack of investment in this kind of system, plain and simple, being disguised as "government efficiency" by the republicans.
Re: (Score:2)
Re: (Score:2)
Re:intent or consequence? (Score:4, Informative)
The problem is that that assertion doesn't line up with reality. Go down to your DMV some time, and observe the kinds of systems that they're using. They're using databases built in the 80s and 90s on top of DOS, running on ancient computers with CRT monitors (at least around here).
...And is that a problem? Does the thickness of the monitor really impact how legibly they can print your drivers' license?
What reasonable business do you know of that hasn't upgraded their systems since that time to allow for more efficiency savings, faster processing, reduced staff costs etc?
As one example, in the mid-2000s, I worked at a company whose main computer was built in 1988, with only minor upgrades (disk capacity, and a modem that was occasionally plugged in so it could be maintained remotely) since its construction. It had survived the obsolescence of its product line, the rise of DOS and Windows, and had only a minor stumble for Y2K. For a system whose primary purpose was tracking orders moving through departments, and tracking employees' time cards, it did the job perfectly well. That particular company was in the top 10% of the industry by order volume and profits, so it seems to have done just fine by most standards of "reasonable".
There's a lack of investment in this kind of system, plain and simple, being disguised as "government efficiency" by the republicans.
Again, to show the other perspective, there is grossly excessive spending in other kinds of systems, being disguised as "upgrades" by the Democrats.
I'm not promoting any particular political party here. Rather, my point is to illustrate that every partisan criticism in this thread has an equally-valid counterpoint that is too-often glossed over. When the Republicans shout about "spending", the Democrats shout "obsolescence". Nobody ever seems to want "get what's useful and nothing more", or "review the cost/benefit analysis for every component in the system".
I've worked for the federal government before, notably on one particular system whose lifespan was about 20 years. The system was designed and built to be state-of-the-art, using top-of-the-line COTS hardware available at the time (as a cost-saving measure, naturally). Ten years into the system life, those original components were obsolete, and being replaced with new top-of-the-line hardware, with the promises you mentioned: efficiency savings, faster processing, reduced costs, et cetera.
However, the basic workflow hadn't changed at all, and the software hadn't been rewritten (as that'd be prohibitively expensive), but only ported up to newer technologies. Even though each part of the process was indeed faster, the system as a whole hadn't changed significantly. It could run perfectly fine on modern (for the day) mid-grade or even low-end hardware, but because "upgrades" were seen as desirable, the system continued to be built with top-of-the-line parts, for about triple the cost.
Towards the end of the project lifespan, there was an effort to re-engineer it using minimal hardware, but by that point the idea had grown into something of a legend. The managers (and bureaucrats) who had seen the system's early versions and knew its original cost couldn't believe the system could actually run on such a low hardware budget. Every actual test was successful, but the mantra that "you get what you pay for" had become such an integral part of common sense that actually getting approval for a cost-efficient system was impossible. Eventually, my team ended up inflating our quoted costs to get approval, then delivering a working system under budget and getting extra praise.
That tale doesn't meet my idea of "reasonable", but it was definitely the reality that I saw.
Re: (Score:2)
...And is that a problem? Does the thickness of the monitor really impact how legibly they can print your drivers' license?
Yes, it is. It indicates that their systems are so old as to require special purpose hardware. The only reason that places use this kind of monitor and computer any more is because they don't have enough money to replace everything outright. In the long run, this causes higher costs, not lower, because they end up needing to seek out compatible hardware to keep things running, rather than being able to wholesale upgrade. If you go and look at companies, you'll find that almost every company has a 3 yea
Re: (Score:2)
Please try reading.
It indicates that their systems are so old as to require special purpose hardware.
A VGA monitor is now considered "special-purpose hardware"?
Having a CRT monitor indicates only that the system is compatible with a CRT monitor. If you're making further assumptions about the system's capabilities based on the age of a peripheral device, that's your fault, not the system's.
As one example, in the mid-2000s, I worked at a company whose main computer was built in 1988...
Sure, one single system in the back of one company did not get upgraded.
No, that was the main system running the whole industry-leading company.
I'd be willing to bet that the reason it didn't get upgraded was simple - it had got so old that it was at this point a major pain, and a major cost to upgrade.
That's only half of it. The other half was that it wouldn't bring any benefit. The company's production was limited by physical pr
Re: (Score:2)
Re: (Score:3)
Re: (Score:2)
As if Democrats don't cut budgets? Why would you think that Republicans cut the DOJ's budget so that they can't afford to run the better system that is already paid for?
CRT (Score:2)
Re: CRT (Score:3)
Re: (Score:2)
what is a green screen..is that something to do with dumb terminals?
IBM 3270 [wikipedia.org]
Re: (Score:2)
From the description my first thought was IBM 5100.
Re: (Score:2)
That screen could connect to a brand new current technology million dollar mainframe for all they know.
Re: (Score:2)
That screen could connect to a brand new current technology million dollar mainframe for all they know.
Re: (Score:2)
Show a millennial a rotary phone and ask him to report on it. Wonder what terms he'd use to describe it.
Re: (Score:2)
he's got a point (Score:3)
More recently, the FBI’s own investigation into the September 11 attacks found that “[o]n September 11, 2001, the Bureau’s information technology was inadequate to support its counterterrorism mission”, noting further that “[t]he FBI’s legacy investigative information system, the Automated Case Support (ACS), was not very effective in identifying information or supporting investigations”.
kinda bad when even you are on record as saying the system sucks.
Clueless... (Score:2)
Not only have the state of the art in searching normal databases not changed in the last 21 years (specialized cases like web searches excluded), the use of "green screens" rather than a web interface have nothing to do with the quality of searches - the _real_ complaint is that the full capabilities of the existing system isn't used!
Re: Clueless... (Score:3)
Especially if the web interface was essentially a vt100 interface wrapped in a REST service.
Not a surprise- (Score:5, Insightful)
The DoJ is one of many departments these days that are run for the benefit of the of the administration, not to serve justice or even the American people.
Clever Grad Student (Score:5, Interesting)
Re:Clever Grad Student (Score:5, Insightful)
By requesting a search that should have netted his own earlier FOIA requests, and didn't, Ryan Shapiro was able to demonstrate the inadequacy of the index-term search the FBI is using. Clever.
He'd better hope they don't decide to make an example of him and charge him under CFAA or some other law, regulation, or Act for exposing the inadequacy of a government computer system or something along those lines. It wouldn't even matter if they knew beforehand that they couldn't get a conviction, the process is the punishment.
The Rule of Law is dead in the US. The "law" now depends on who you are, who you know (and what you know about them), and how much money and power you have.
Strat
Re: (Score:2)
If I embraced this standard of discourse, I'd probably agree with you.
But I don't. Your post is indistinguishable from FUD. That's a serious problem, regardless of the validity of your argument. Nothing is ever that pure. Not even pessimism or despair.
Re: (Score:1)
HSBC comey and lynch, feel free to look it all up, they show up just in time to not prosecute AGAIN.
BlueStrat is correct. He failed to say, The Rule of law is Dead unless your poor, then your
Re: (Score:2)
So, do you really have a coherent argument, or are you just gonna go full pseudo-philosopher here? Because the only thing I got from your post that's even faintly on-point is that the Rule of Law is only "mostly dead". And therefore could be resuscitated for a noble cause.
You probably didn't mean that, though. You probably didn't mean anything.
But,but, but.... (Score:2)
But, but, but, they don't intend to break the law, so it is okay.
Clinton's emails? (Score:2, Insightful)
The Guardian again ... (Score:5, Insightful)
Frequent posts from The Guardian and BBC cover important events in the US that local media fail to report. This should make us wonder why American news media aren't on top of these stories.
Of course tight budgets constrain many traditional news outlets and restrict the ability to really investigate anything. The de-funding of Public Broadcasting was a disaster in American history, forcing a dependence on advertising and fund raising.
But those of us who entertain conspiracy theories may suspect that the 'free press' in the US is heavily influenced by various pressures from government and advertisers. For instance, many media are now forbidden at Trump rallies because they have offended The Donald by asking serious questions. Some media are unwelcome at White House briefings. Your local city/state politicians also have preferred, cooperative, outlets for their announcements. Cooperation with big advertisers is also important for American media to survive financially. Evil Monsanto stories go on page 3 or nowhere at all.
Most US publishers share with their readers the political posturing of government officials and the promotional 'news' of advertisers but fail to investigate anything. The remainder of US news is crime, weather, celebrities, a smattering of drama about terrorist activity, and no mention of large parts of the world like Latin America.
So, thanks Guardian and BBC, for a fresh look at the world and my own country.
Re: (Score:3)
Re: (Score:2)
Frequent posts from The Guardian and BBC cover important events in the US that local media fail to report. This should make us wonder why American news media aren't on top of these stories.
One thing I'd suggest any American try: Find yourself a local broadcast of BBC America and listen to them interview someone. They actually follow-up their questions if the subject isn't answering, and if the subject is BS'ing will freely tell them so to their face. Its like reporters are supposed to be.
The best is when they interview an American populist politician. One of those people who is used to spouting coded racist/classist language, or even flat out lies, without the interviewee calling them on it
Looks like a good decision (Score:3)
The allegation is that only searches of an incomplete index are ever performed for FOIA purposes, and such searches are (1) archaic and unusual nowadays, (2) rarely find the requested material. That's more likely misfeasance than innocent.
The DOJ is an agency that ought to find compliance with law of primary interest, and arguments of 'needlessly duplicative' ring false. The plaintiff's test was amusing: he asked DOJ to find his previous FOIA requests, and was told there was no record...
The articles make it clear.... Limited searches (Score:1)
This isn't about the hardware. The most important aspect is that the FOI searches only look at an index hand-built by whatever person entered the record. There appears to be no full text search for these external requests. Internal requests take advantage of modern techniques. Public requests are limited, by design. Kudos for exposing this by well crafted requests. "We tried" is not sufficient.
Re: (Score:2)
I've worked with government IT and it's behind the time everywhere. It's not a conspiracy. It's a constantly budget conflict.
When you have budget X and it's near impossible to get funding. This kind of stuff happens on a regular basis. IT winds up just making it work.. good enough for as long as they possibly can. Like a poor person or minimalist trying to get 50 years out of their old pickup.
Bullshit!
They'd just prefer spending our money that they confiscate under threat of deadly force on spying on the private communications of regular US citizens (particularly those who oppose the current regime and/or their policies/actions) and not on systems to keep them accountable to those citizens.
Strat
"Your Honor, would you accept...." (Score:3)
"Your Honor, if you approved a subpoena for records and the response was 'We searched and found nothing responsive,' would you accept that response if you knew that the search consisted of nothing but looking at a list of filenames? After all, that's a search - a very poor one, but a search nonetheless."
If IRS Uses Kennedy Era Mainframes... (Score:2)
21 years ago was 1997 (Score:1)
Re: 21 years ago was 199*5* (Score:1)