New Bill Would Let Defendants Inspect Algorithms Used Against Them In Court (theverge.com) 47
Lauren Feiner reports via The Verge: Reps. Mark Takano (D-CA) and Dwight Evans (D-PA) reintroduced the Justice in Forensic Algorithms Act on Thursday, which would allow defendants to access the source code of software used to analyze evidence in their criminal proceedings. It would also require the National Institute of Standards and Technology (NIST) to create testing standards for forensic algorithms, which software used by federal enforcers would need to meet.
The bill would act as a check on unintended outcomes that could be created by using technology to help solve crimes. Academic research has highlighted the ways human bias can be built into software and how facial recognition systems often struggle to differentiate Black faces, in particular. The use of algorithms to make consequential decisions in many different sectors, including both crime-solving and health care, has raised alarms for consumers and advocates as a result of such research.
Takano acknowledged that gaining or hiring the deep expertise needed to analyze the source code might not be possible for every defendant. But requiring NIST to create standards for the tools could at least give them a starting point for understanding whether a program matches the basic standards. Takano introduced previous iterations of the bill in 2019 and 2021, but they were not taken up by a committee.
The bill would act as a check on unintended outcomes that could be created by using technology to help solve crimes. Academic research has highlighted the ways human bias can be built into software and how facial recognition systems often struggle to differentiate Black faces, in particular. The use of algorithms to make consequential decisions in many different sectors, including both crime-solving and health care, has raised alarms for consumers and advocates as a result of such research.
Takano acknowledged that gaining or hiring the deep expertise needed to analyze the source code might not be possible for every defendant. But requiring NIST to create standards for the tools could at least give them a starting point for understanding whether a program matches the basic standards. Takano introduced previous iterations of the bill in 2019 and 2021, but they were not taken up by a committee.
How long did it take last time? (Score:5, Insightful)
The validation of Truecrypt took what, nearly two full years by a team of experts? It's nice to have the option, but the reality is you're not going to be able to fund a campaign to fully investigate the software or algorithms used against you.
Re: (Score:1)
Lolol rsilvergun is a better poster than you and creimer is even a better troll.
Re: (Score:2)
If those names had an R in front of them you’d be on your knees with your mouth open in thanks.
LOL what I love most about ACs posting is how clueless you are. Now if you replace R with D you may start seeing something resembling my posting history, despite the fact I'm not American and don't give a shit.
I love you guys. You remind me that no matter how bad modern discourse gets, the bar can always be set lower and more nonsensical.
Re: (Score:2)
The validation of Truecrypt took what, nearly two full years by a team of experts?
If it's even possible any more. The algorithms are going to be neural nets like LLMs. "Here's a 100 billion weights. Knock yer socks off deciphering why they computed to not make you a job offer."
Re: (Score:2)
Using an ANN to analyze evidence and identify suspects is reasonable.
Allowing their output to be admitted as evidence in court is certainly not.
ANNs are nearly as unreliable as humans.
ANN= Artificial Neural Network
Re: (Score:2)
This.
Some (many?) people have a profoundly delusional belief in the ability of humans, and that is a straw man which is eventually going to fail.
I saw telly adverts for an upcoming programme where two different "juries" will be presented with the same evidence in a (made up) criminal case ... with access to the "juries" decision-making processes. That is going ot pierce some delusions of competence, and is going to up set a lot of people. I didn't note which channel
Re: (Score:2)
That's easy then. Don't use it. If you can't explain the algorithm, or if it can't output its reasoning in a human-interpretable manner, then it should be forbidden.
Re: (Score:2)
Here's a 100 billion weights. Knock yer socks off deciphering why they computed to not make you a job offer
So then maybe we shouldn't allow technologies we can't control, or even understand, to play a role in making life altering decisions
You just need to find one bug (Score:2)
The validation of Truecrypt took what, nearly two full years by a team of experts? It's nice to have the option, but the reality is you're not going to be able to fund a campaign to fully investigate the software or algorithms used against you.
Just one bug that could give a false positive related to your case, that would be enough to give reasonable doubt. No need to review the entire program.
Re: (Score:2)
No it wouldn't. Causality applies. You'd need to find a bug and demonstrate the the outcome is affected. Also it's not a given that you find X bugs in Y amount of review hours. To keep with my example, Truecrypt went through 1.5 years of validation without finding a bug, and then a year later a bug was discovered in a Windows DLL related to drive mounting.
If bugs were so easy to find we wouldn't have bugs. It takes serious time and effort.
Re: (Score:2)
The difference is I guarantee that the LEO programs won't be nearly as clean.
Re: (Score:1)
This is about standards and also the threat of exposure by people selling bunk products to law enforcement.
Right now they’re selling field drug test kits that have been well known to show positive on a random assortment of common materials and plants. They’re still accepted by the courts even though a simple wikipedia search would tell you they’re snake oil.
There was also the joke golf ball detector rebranded as a bomb and drug detecting device. That was big hit throughout the law enforc
Re: (Score:3)
In practice it will take only one wealthy suspect or one interested philanthropic party to intervene and do a complete investigation. Even if the results are not ready for the trial, once they have the source code they can keep looking for issues.
Besides, most of this software is crap, so it will only take a few minutes with an automated source code scanner to find enough flaws to discredit them.
Didn't that happen with laser speed guns some years back? Once they had the source code, it was shown to be horri
Re: (Score:2)
Why would anyone disagree? (Score:3)
Re: (Score:2, Insightful)
Re:Why would anyone disagree? (Score:5, Interesting)
Example: I'm a suspect in a murder trial. They found titanium dioxide on the murder weapon and I was apprehended wearing white sun screen. If this becomes a law, I can subpoena the source code and algorithms for the mass spectrometer and x-ray fluorescence machines used to identify the titanium dioxide.
The prosecutors would need to contact the manufacturers of the test equipment, negotiate a release of the source code and algorithms used by the devices, and provide it to the defense team. Every piece of equipment used by the forensics team would have to have a point of contact at the manufacturer to handle these requests. If the manufacturer does not agree to release than information (trade secrets) does that mean the forensics team cannot use that equipment any more?
While having access to the source code and algorithms sounds good on paper, in real life it would just give those trying to game the justice system another tool to confuse and drag out criminal proceedings.
Re: (Score:2)
>in real life it would just give those trying to game the justice system another tool to confuse and drag out criminal proceedings.
Doesn't it do both? Or are you saying there aren't shady devices (lie detectors, anyone) or algorithms out there that are just bunk?
Re: (Score:2)
Shouldn't it be the norm for these tools to be open source, or at least subject to continual security review under NDA?
Many forensic techniques have been shown to be bunk or at least deeply flawed, and to have resulted in numerous false convictions. By now we really should have realized that when a new one comes along, we need to be extremely sceptical.
Re: (Score:2)
> If the manufacturer does not agree to release than information (trade secrets) does that mean the forensics team cannot use that equipment any more?
What is there to agree with ? There's a court case going, and he's being subpoenaed. If they don't agree, they get held in contempt and go to jail.
This isn't a negotiation.
Re: (Score:2)
That's not how it works. You would have to convince the judge that there is sufficient cause for the subpoena*. If you can't show the probability of the resultant information having a material effect on the outcome of the trial, the judge will likely not grant it.
* Technically, your lawyer can simply submit the paperwork to the court clerk and the subpoena is issued as a matter of course, but the manufacturers and/or the prosecution would likely object, and then the judge would have to rule.
Judges are not
Re: (Score:2)
In the past (maybe still) juries were told that the probability if a false DNA match was
Th
Re:Why would anyone disagree? (Score:5, Informative)
What business would turn down free testing?
Errr most of them, if it involves handing over source code and algorithms.
Re: (Score:2)
Simple question. Simply answered : one which knows their product is utter crap. That includes people who have tried to make a good product, and fucked up - and those who tried to make a profitable product without any regard to evidence in the case. And that sort of fraud does go on. We've had enough people selling snake oil here (outside America) that I'm sure it happens in America too. Plenty of racist cops and judges with prejudices that can be profitably pandere
Does this mean that I can... (Score:2)
you want the calibration logs (Score:2)
you want the calibration logs.
Re: you want the calibration logs (Score:3)
The calibration logs don't mean anything if you don't know how the thing actually works.
It could easily have code that makes it more accurate when it detects it's being tested.
Re: (Score:2)
> The calibration logs don't mean anything if you don't know how the thing actually works.
AIUI those calibration logs usually don't exist.
And you can subpoena the person who ran the calibration to testify.
They really hope you won't flex your rights.
Re: (Score:2)
Re: (Score:2)
There were a series of posts about people trying to get the source code for the breath analyzer machines for DUI cases about 15 years ago.... Same thing.
Your honor, please excuse me (Score:2)
they will just use per crime and keep you in jail (Score:2)
they will just use per crime and keep you in jail forever where you don't event get to the court room.
Re: (Score:2)
Your guess is as good as mine folks.
Machine learning (Score:4, Insightful)
I feel like the logic will be hidden in a machine learning model - the authors don't even understand.
Re: (Score:3)
I feel like the logic will be hidden in a machine learning model - the authors don't even understand.
If you are being convicted based on the use of a system that nobody understands and nobody can prove works... maybe that's a problem.
Re: Machine learning (Score:2)
Agreed. And that's why the company selling the software wouldn't want it examined. Their marketing material won't match reality. It's only "correct" for it's exact training set - every thing else is a guess. It should only be used as one data point in a larger argument.
Question the Witness, Including Software (Score:3)
I think everyone on /. has at least heard of a data-driven bug. If software is the witness, it should be questioned.
Re: Sounds right but (Score:2)
Seems to me that either a) if the prosecutors want to rely on the software, they should be expected to pay whatever the cost for the inspection and certification, or b) if the developer company wants their software to be used in court proceedings, they should be expected to pay whatever the cost for certification.
I agree with the sentiment that software should be treated as a witness, like a coroner who performed an autopsy, and the software should be questioned.
Relevant story in the UK (Score:2)
Check out the Post Office scandal relating to sub post offices, and the Horizon software. There is a dramatization coming soon on PBS.
In UK courts, "evidence" provided by computers is (or was) presumed correct, leading to a huge number of convictions that should never have happened.
The scandal continues to this day: the government and Fujitsu (the software vendor, formerly ICL) are dragging their feet on compensation and on reversing the convictions.
what source code? (Score:2)
Source code obfuscation is easy enough.
Malicious compliance (Score:4, Informative)
I have actually worked in "explainable AI" for a while, and yes even for the "deepest" deep neural networks that are supposed to be black boxes there are techniques to explain "why" a recommendation was made.
However, it requires additional engineering effort, also usually along with preparation of the training for this task.
What is more? Most "Machine Learning Engineers" do not actually understand how the model works in the first place. Yet alone tackle this additional task ("I downloaded this sample model from Github, ran it on Google cloud with Colab on our CSV file, and yes it is working with 80% accuracy")
The question is, which one will a "lowest bidding contractor" for software do:
a. Spend time and effort, and also hire capable engineers to make their algorithms and AI explainable?
b. Dump a gigabyte of data in random floating points and say "that is all we can do"?
Two caveats ... (Score:2)
#1 : If the company refuses to supply source-code then it is assumed to be wrong ...
#2 : If the company cannot show it is correct, as most machine learning systems cannot be shown to be correct, then it is assumed to be wrong ..
watch as most cases where the evidence is based on AI systems just fall apart ...