Please create an account to participate in the Slashdot moderation system

 



Forgot your password?
typodupeerror
The Courts Crime Software

When Sentencing Criminals, Should Judges Use Closed-Source Algorithms? (technologyreview.com) 196

Some judges in America have recently started using a closed-source algorithm that predicts how likely convicts are to commit another crime. Mosquito Bites shared an article by law professor Frank Pasquale raising concerns about the algorithms: They may seem scientific, an injection of computational rationality into a criminal justice system riddled with discrimination and inefficiency. However, they are troubling for several reasons: many are secretly computed; they deny due process and intelligible explanations to defendants; and they promote a crabbed and inhumane vision of the role of punishment in society...

When an algorithmic scoring process is kept secret, it is impossible to challenge key aspects of it. How is the algorithm weighting different data points, and why? Each of these inquiries is crucial to two core legal principles: due process, and the ability to meaningfully appeal an adverse decision... A secret risk assessment algorithm that offers a damning score is analogous to evidence offered by an anonymous expert, whom one cannot cross-examine... Humans are in charge of governments, and can demand explanations for decisions in natural language, not computer code. Failing to do so in the criminal context risks ceding inherently governmental and legal functions to an unaccountable computational elite.

This issue will grow more and more important, the law professor argues, since there's now proprietary analytics software that also predicts "the chances that any given person will be mentally ill, a bad employee, a failing student, a criminal, or a terrorist."
This discussion has been archived. No new comments can be posted.

When Sentencing Criminals, Should Judges Use Closed-Source Algorithms?

Comments Filter:
  • No (Score:2, Insightful)

    by Anonymous Coward

    No

    • Agreed. Algorithms are made by man, and as such, are fallible; that they involve math / programming (Magic) merely obscures their origin.

      • Your sarcasm is misaimed. The problem is not that they are algorithms--it of course has always been algorithms, though not always machines; "laws, not men". The problem is that they are closed source: sentences are being meted out by a set of rules that those being sentenced are not allowed to know. That's not acceptable.

        • by kenh ( 9056 )

          The problem is that they are closed source: sentences are being meted out by a set of rules that those being sentenced are not allowed to know. That's not acceptable.

          Then we're agreed, sentences should be determined by sentencing guidelines set at the whim of politicians attempting to look 'tough on crime' (like, during the crack cocaine epidemic in the 80s/90s), and then sympathetic when our prisons are over-crowded with victims of 'three strike' regulations?

          Frankly, I'd like it to be in the hands of judges, with very limited exceptions, but that will never fly in today's litigious society.

          • by AK Marc ( 707885 )
            This algorithm isn't changing the maximum and minimum. That's what the politicians mess with. This is giving the judge more information. The judge always arbitrarily set the sentence. This way, he has an additional piece of information. The sentencing basis is *always* closed source and arbitrary. Now, if it's closed source and non-arbitrary, how is that worse?
          • Saying that its acceptable to use algorithms doesn't mean you think the algorithms in use are the right ones.

            Frankly, I'd like it to be in the hands of judges, with very limited exception

            Which ensures that the unfairness is better covered and harder to uproot (not that it's easy now).

          • by cas2000 ( 148703 )

            > Frankly, I'd like it to be in the hands of judges

            That makes sense almost anywhere but in America - you have elected judges. Judges are politicians themselves. Judges can be complete fucking morons with no comprehension of the law, just their own collection of prejudices. Judges can be elected just by being popular - or perceived as "tough on crime" - in some dumb-fuck inbred county.

            But yeah, regardless of that, sentencing algorithms, like the laws themselves, should be open source whether they're ru

      • The algorithms themselves are actually the least important aspect. As I have said before [slashdot.org], even if the algorithms are 100% open and transparent, that means nothing if the data feed into them is poor. If the bank uses an algorithm to determine if it want to lend money to you, how is the data about you collected? Who decided to classify you as a say medium risk person? What cirterias did he/she/they use for that? How thorogh were he/she/they in gathering decition material? What did he/she/they miss/ignore/mis

  • Long answer Nooooooooooooooo!!!!!!

    This is as crazy as having closed source software determine guilt (think breathalyser, radar gun etc) -- which means it is probably inevitable.

    In the legal system, on every encounter with an object running code, (firmware or otherwise) an accused party should move for discovery and expert analysis of said code, and how it was developed.

    Were they using best-practices? Running valgrind? (lets see the exception list) Address-sanitizer, theread-sanitizer, clang static analyser? etc..
    Do they have unit tests? Do they track code coverage of those tests? (it had better be 100% across the board)

    Even using all the latest tools and with plenty of unit tests with good coverage, there are still bugs.
     

    • Re: (Score:2, Interesting)

      by Anonymous Coward

      think breathalyser, radar gun etc

      Those are not really comparable, they measure something by objective metrics, and the reliability of the results produced by those tools can be verified independently, even if the tools themselves are closed source. How likely someone is to commit a crime again is not something that can be reliably calculated by using some well defined formula, thus, a secret algorithm that tries to do that is far worse than the quoted examples.

    • At least with breathalysers and radar guns, the algorithm is known. (Parts alcohol per unit of blood, or a doppler formula calculation, or time to travel a certain distance). People have succesfully challenged these [telegraph.co.uk]. This is worse. Even with access to all the same data, you can't query the result.

      At the very least, they could reveal the algorithm used to determine the risk. If that's a trade secret, it calls into serious question how accurate it actually is.
    • The software is providing expert opinion against you, it's not a camera submitting facts but an intelligent system giving opinion (probabilities). You should clearly be able to question this expert and get full justification of its testimony. A mathematical proof of its accuracy would also be justified. The lie detector is not allowed in court because sometimes it doesn't work, why should this device be allowed unproven?
    • This is as crazy as having closed source software determine guilt (think breathalyser, radar gun etc) -- which means it is probably inevitable.

      Not the same thing. The problem isn't so much that that software is closed source as that the algorithm is. Breathalyzers and radar guns may have closed source software, but what they measure is a well-publicized, independently verifiable fact. Not the same thing.

    • This is as crazy as having closed source software determine guilt

      Or indeed using software at all, when it comes to doling out justice. The law is imperfect - one might almost say by design - and is only ever meant to be a guideline or framework for the judge, who is there to add some human insight and understanding. I think it is a very disturbing idea, using computers to "calculate" justice; before we know it, we will have courts run by call centers.

  • by Anonymous Coward on Sunday June 04, 2017 @07:03AM (#54545405)

    Government, as being tax funded, should use entirely open source software and open formats. Anything otherwise is favoring certain corporations (Microsoft formats for example) or having potential to be abused (FBI backdoors in government software).

    We the people elected them. We the people should be able to inspect them.

  • it means a "diminutio" of defendant rights... if it will be used extensively we (well... you in the US) will have unfair processes
  • This is tricky because most people don't get the issues related to open/closed source. Remember that the majority of people now have smart phones and tablets with their walled garden of app ecosystems. The idea of libre software simply does not occur to them. They likely didn't grow up learning computers in the era when source code commonly shipped with the computer and you could inspect/tweak it if something went wrong.

    That said, I firmly believe that any software which is used in determining innocence/

  • by Anonymous Coward

    "I'm sorry, Citizen - the reason for your incarceration is not available at your security clearance level. Please report to you closest walled garden for mandatory fun."
    -The Computer.

  • by Hognoxious ( 631665 ) on Sunday June 04, 2017 @07:21AM (#54545457) Homepage Journal

    Imagine a judge just picks a number out of the air based on his own experience, opinions and yes, prejudices.

    That's using a closed source algorithm, except it runs on a wet carbon platform rather than silicon.

    • Re: (Score:2, Insightful)

      by mhkohne ( 3854 )

      Yes, and it's trivially easy to get people to believe the judge might have made a 'bad call' in your case because we all know that people make mistakes.

      The problem with the black box algorithm is that in spite of years of things blowing up in their faces, a number that came out of a computer is still imbued with an air of 'rightness' that it most assuredly doesn't deserve. But that air of 'rightness' will make it much harder for a defendant to challenge inappropriately harsh sentencing because people think

      • by Anonymous Coward

        People's irrational belief in the accuracy of technology.

        "Garbage in; garbage out" has been forgotten by the current generations.

        AND - you can just bet some very sharp person will figure out how to game the algorithm.

        CONSULTANT: Let's see, several drug offenses, rape, homicide - order the defendant to go to Church and "find Jesus".

        COMPUTER: "200 hours of public service and time served."

      • But that air of 'rightness' will make it much harder for a defendant to challenge inappropriately harsh sentencing because people think the computer must know something.

        It also makes it harder to challenge inappropriately lenient sentencing (in jurisdictions where that's permitted).

      • by nomadic ( 141991 )

        "Yes, and it's trivially easy to get people to believe the judge might have made a 'bad call' in your case because we all know that people make mistakes."

        Doesn't really matter what people believe or not, if the judge has the discretion to select a number that number is almost always going to stand.

      • by tlhIngan ( 30335 )

        The problem with the black box algorithm is that in spite of years of things blowing up in their faces, a number that came out of a computer is still imbued with an air of 'rightness' that it most assuredly doesn't deserve. But that air of 'rightness' will make it much harder for a defendant to challenge inappropriately harsh sentencing because people think the computer must know something.

        Why not? People already appeal their sentences for being "too harsh" - whether it was made by a judge, or whether it wa

        • I don't see what the fuss is about. It's not that different to how insurance companies determine premiums.

    • Moreover, this is being used for sentencing, not for determining one's innocence. The defendant's guilt has already been decided at this point. And on top of that, bounds are already established for what penalties can be levied for each variety of crime, so it can't exceed those bounds either. As you said, the judge is already a black box. All this does is apply some consistency between the black boxes we're using.

    • by havana9 ( 101033 )
      "All court proceedings must be motivated", the article 111 of the Italian constitution says. When a judge, makes a sentence has to write a document explaining why has chosen a sentence and why the drfendant is either guilty or innocent, even if there's a jury involved. So in Italy a closed sourced system used to mke sentences is uncostitutional. If a judge uses is stealthly, I think could risk a misconduit .
  • Perhaps the secret is that the algorithm is a black box method, and no-one actually knows exactly why it scores the way it does?
    Neural networks are all the rage now, and they are one of the absolutely worst when it comes to transparency and intelligibility.

    Had the company worked with rule-based classifiers or other transparent machine learning I'm sure they'd be happy to show just how smart they are, snippet by snippet. After all, we only want to avoid showing others when we're doing something simple or stu

    • by mhkohne ( 3854 )

      I doubt they'd be willing to show off even if they thought they were clever. They'll be afraid that someone will point out something stupid that they've done which calls the whole mess into question.

      The best thing to do with a naked emperor is to not let him out in public.

    • by sjames ( 1099 )

      That makes it especially problematic since they then can't prove that the system didn't infer race and then use that in hidden variables to determine the sentence.

  • Just think how much money the state will save on legal costs if convicts are unable to appeal their sentencing. Sure, there's the chance of giving people a few more years in prison than is really appropriate, but has there ever been a time when the public called for more rights for convicted criminals?

  • AFAIK software have no legal right.
    It should be the responsibility of the lawyer using the software. Closed or open source doesn't matter, what is important are the results and the legal entity backing them.

    If a lawyer uses voodoo magic to find culpability, why not, as long as the facts are right. If not, he better get a good explanation. "magic" won't cut it. Same for the software. The good thing with open source is that it is easier to explain results when challenged.

    I'd like opensource everywhere but I d

  • It is obvious that the answer should be no.
    Sentencing decisions should include the reasoning why the decision would be motivated.
    "Because The Software Says So" is no more valid than "Because I Say So".

    I think that if software is being used, not only should the software be Open Source, the sentencing document should include a mathematical explanation of the algorithm that the program uses together with the input parameters about the judged and various statistics that the decisions leans on.

  • Others have already shown why the answer to this question is No.

    I would like to point out why No is always the answer to questions like this.

    If the answer is Yes, the editor rephrases the question as a statement. I.E. no one writes a headline "Did Trump get elected?" Instead we write "Trump wins."

    We only use a question when the real headline would be boring, so we re-write it to sound more exciting and make it a question so we can't get sued. So if we know that Trump has not resigned, but we want peop

  • by IWantMoreSpamPlease ( 571972 ) on Sunday June 04, 2017 @07:52AM (#54545537) Homepage Journal

    Use compassion and moral guidance. Remember, a prison is supposed to be about rehab, not outright punishment. The Nordic countries do it right.

    • Remember, a prison is supposed to be about rehab, not outright punishment.

      Prisons have multiple functions: they protect society by physically separating criminals, they serve as punishment, they serve as deterrent, and they may also rehabilitate. If you say that their purpose ought to be only rehabilitation, well, you're probably largely on your own.

      The Nordic countries do it right.

      That's your opinion, not a fact. Many people are offended by Breivik's conditions of imprisonment [bbc.com] for example.

      Furthermore, it's

      • Re: (Score:3, Insightful)

        by Anonymous Coward

        > the US has a much more diverse population, and hence we have a much larger number of people who are difficult to rehabilitate.

        The elephant in the room is: the US has a revenge justice. Not alone in this, the element of revenge is, alas, quite widespread in our societies, sometimes stronger, sometimes weaker.

        In a modern society, revenge has no place. Protect the society, OK. Learn from mistakes (the "perpetrator", but also the society surrounding him/her), definitely. But revenge? We are at one level wi

  • by ooloorie ( 4394035 ) on Sunday June 04, 2017 @08:33AM (#54545657)

    This issue will grow more and more important, the law professor argues, since there's now proprietary analytics software that also predicts "the chances that any given person will be mentally ill, a bad employee, a failing student, a criminal, or a terrorist."

    Government needs to be able to explain its decisions about citizens to the public. Private organizations don't.

    So, proprietary analytical software that cannot justify its decisions is not acceptable for courts of any kind, for policing, for awarding government contracts, for public schools, etc.

    It is acceptable for businesses, medical providers, employee evaluations, private schools, etc.

  • Absolutely! (Score:4, Funny)

    by sinij ( 911942 ) on Sunday June 04, 2017 @08:35AM (#54545669)
    Absolutely it should. Conincidentally, I legally changed my name to 1;drop table.
  • Will the algorithm take into account the gender differences in sentencing?

    https://www.law.umich.edu/news... [umich.edu] For those too busy to read the citation, the research by Professor Sonja Starr indicates men receive prison sentences that average 63% longer than women convicted of the same crime. Women are also twice as likely to avoid incarceration altogether. The paper itself: https://papers.ssrn.com/sol3/p... [ssrn.com]

    So what we have here a bit of a minefield. If we are to continue this practice, the algorithm must t

    • by iamacat ( 583406 )

      Men commit far more crimes then women. If the premise of basing sentencing on probability of future crimes is accepted, gender (and race etc) disparity makes perfect sesnse. Even if the algorithm is not given gender and race inputs, its likely to reach same conclusions by considering other factors such as growing up in poverty.

      • by Cederic ( 9623 )

        Men commit far more crimes then women

        Men are prosecuted for crimes more than women, receive harsher sentences and are also more likely to have untreated mental issues that lead to behaviour considered criminal.

        So no, putting gender (or race) in doesn't make sense at all. In fact leaving gender out would probably help remove some of the sentencing discrepancy.

      • Men commit far more crimes then women. If the premise of basing sentencing on probability of future crimes is accepted, gender (and race etc) disparity makes perfect sesnse. Even if the algorithm is not given gender and race inputs, its likely to reach same conclusions by considering other factors such as growing up in poverty.

        Wow - good to see that the bogots have chimed in. That - iamcat - would be you.

        Any other groups you want individuals to be punished based on any of your pre-decided bigotry?

    • But if gender is not taken into account, and suddenly females get identical sentencing, you can bet there will be a lot of legal agitation from a different group.

      You seem to be assuming that the longer sentences are the fairer/better ones. If the "that guy looks big and scary" bias is removed and everyone gets the shorter sentence, it's good for all.

      • But if gender is not taken into account, and suddenly females get identical sentencing, you can bet there will be a lot of legal agitation from a different group.

        You seem to be assuming that the longer sentences are the fairer/better ones. If the "that guy looks big and scary" bias is removed and everyone gets the shorter sentence, it's good for all.

        Explain how I suggested anything other than there is a verifiable difference between between male and female sentencing. I said nothing about the length of punishment I desired. For the record, punishments in the US tend to be longer than I think is adequate.

        As for assumptions, you appear to be assuming what I am assuming. In this particular case, you are wrong.

  • While the open-sourcing (for critical purposes) of such an algorithm is all well and good in theory, the fact is it would then be gamed for advantage.

    If, for example, there is a statistically-significant correlation between a defendant showing up at a trial with a facial tattoo re-committing later, any intelligent defense attorney is going to either cover it with make up or get it removed.

    • Either the factors actually affect recidivism or they don't. If they do, then it's win-win, since the point of the criminal justice system is to reduce crime. If they don't, it's a bad algorithm, and it should be challenged.
      • by Cederic ( 9623 )

        Lawyer: Look, the alogorithm will give you a lower sentence if you get a haircut, find a job and stop getting drunk every night.

        Society: Stop gaming the system!

  • We are talking about Law, so pedantry and precision is the way to go[*] ......

    By definition an algorithm cannot be "open source" or "closed source". It might be proprietary ... but that is a different thing.

    Personally, I think that using some sort of Big Data / AI / Machine Learning thing to abdicate a Judge's responsibilities would seem to be the wrong way to go, particularily if you are using them to predict somone's future behaviour -- why have a Judge at all if you are going to do that?

    Furthermore, pre

  • Counterargument (Score:5, Informative)

    by bitkid ( 21572 ) * on Sunday June 04, 2017 @12:34PM (#54546847) Journal

    Frank Pasquale left out a couple of details in his opinion piece. First, these algorithms are only used in determining sentences, not to determine guilt. At that point in the trial guilt has been determined beyond a reasonable doubt. At the sentencing phase rules of evidence do not apply anymore and almost anything goes. That's why the prosecutor puts crying victims on the stand. There have been two supreme court cases (Malenchik vs Indiana and Loomis v. Wisconsin) that challenged the use of algorithms in sentencing and both upheld (in the later with some minor restrictions) that these algorithms can be used in sentencing. The conditions in general are that the algorithm has to be scientifically sound. That was the case in both cases that were challenged as there is existing peer-reviewed literature that examined the algorithms. Frank neglected to mention that. The secrecy of the algorithms is a consequence of patent and copyright law btw. The algorithms in these cases are a scoring function. Math is not (and should not) patentable or copyrightable. In this case the consequence is that the manufacturers only recourse is to keep it a trade secret. That could be solved better, but in my opinion people shouldn't get their hopes up that there's some exploitable loophole in the algorithm or something.

    We can debate whether assessments (actuarial prediction instruments) should be used in sentencing or criminal justice. I’m very much in favor as it does reduce bias and leads to reproducible results. It’s much easier to control for biases in decision making with statistical methods than it is to control or fix bias in humans. Does anybody believe that human judgement is less biased? You can read up on the work of Paul Meehl who spend his lifetime showing that even simple assessment tests outperform the judgment of trained clinicians. Part of the sentencing is taking into consideration how likely the perpetrator is to commit a new offense. Humans suck at making predictions and estimating probabilities. This is no different in criminal justice.

    Let me end this with pointing out some of the positive change that systems like this have brought: early release from incarceration. Low risk prisoners are more frequently released early (not just from overcrowded California prisons when ordered to do so by a federal judge), and then put on probation/parole. And work out well it did: http://time.com/4065359/califo... [time.com] The expected crime wave from federally mandated early release didn't materialize. In my opinion thanks to these prediction models.

    There are many things wrong with policing and criminal justice in the US, but the move to what’s generally referred to as “evidence based practices” (incl. actuarial prediction instruments) has been pretty positive. The great part is that both Dems and Reps are behind the idea of risk assessments so we might actually see some change for the better.

  • What matters are future results, and that the effectiveness of whatever algorithm is used is continually measured. There's also question of whether sentencing alone can effectively reduce recidivism. The Economist [economist.com] has an interesting article on making America's prisons work better.
  • Locking a person away because you think they will commit future crimes, probably.

    A justice system is about objectivity. It is not about punishing someone because you dislike them. Justice is blind, not a popularity contest.

    • by bitkid ( 21572 ) *

      A justice system is about objectivity. It is not about punishing someone because you dislike them. Justice is blind, not a popularity contest.

      What's more objective than using an algorithm for sentencing that is using data that has been shown to be predictive for recidivism? Indeed it's not about disliking somebody but about public safety and deterrence.

  • When a judge "decides" on a sentence, there can be many factors that influence the result. Not all of them are strictly relevant or objective and not all of them are easily explainable. Different judges would give different punishments for the same crime. Hell, the same judge can give different sentences for seemingly identical circumstances. When a program computes a sentence then the same data entered each time will lead to the same result.

    Whether it is an open-source program that gives a result or a pr

  • by davecb ( 6526 ) <davec-b@rogers.com> on Sunday June 04, 2017 @04:07PM (#54547785) Homepage Journal
    With a lot of caveats, of course, but basically an individual has the right to 'an explanation of the decision reached after [algorithmic] assessment'. Described in http://fusion.kinja.com/eu-cit... [kinja.com]
    • by bitkid ( 21572 ) *

      With a lot of caveats, of course, but basically an individual has the right to 'an explanation of the decision reached after [algorithmic] assessment'. Described in http://fusion.kinja.com/eu-cit... [kinja.com]

      Interesting. Thanks for sharing. The decisions of linear models (like the ones used for sentencing, credit scores etc.) can indeed be explained. The credit bureaus sell explanations of the score and tips on improving the score as an extra service, for example. That doesn't mean that people will like the explanation... If the EU will indeed require explanations for algorithmic decisions, then models will be limited to simple linear models.

      • by davecb ( 6526 )
        I encountered this when looking into machine learning: the EU wants "no magic", and poses IBM's AI lawer some problems (;-))

It is better to live rich than to die rich. -- Samuel Johnson

Working...