Catch up on stories from the past week (and beyond) at the Slashdot story archive

 



Forgot your password?
typodupeerror
×
The Courts Math Technology

Algorithm Predicts US Supreme Court Decisions 70% of Time 177

stephendavion writes A legal scholar says he and colleagues have developed an algorithm that can predict, with 70 percent accuracy, whether the US Supreme Court will uphold or reverse the lower-court decision before it. "Using only data available prior to the date of decision, our model correctly identifies 69.7 percent of the Court's overall affirm and reverse decisions and correctly forecasts 70.9% of the votes of individual justices across 7,700 cases and more than 68,000 justice votes," Josh Blackman, a South Texas College of Law scholar, wrote on his blog Tuesday.
This discussion has been archived. No new comments can be posted.

Algorithm Predicts US Supreme Court Decisions 70% of Time

Comments Filter:
  • biased algorith (Score:5, Insightful)

    by Dthief ( 1700318 ) on Thursday August 07, 2014 @04:38AM (#47621123)
    I (read: anyone) can make an algorithm that fits any previous data (even only using data that precedes the "prediction")......testing future predictability is the only way this means anything.
    • by mwvdlee ( 775178 )

      If only he could have made some predictions, travelled to the future to test the predictions, then travelled back and put the results in his blog post.
      Sadly, testing future predictability can only be done after the future has passed.

      • But once the future has passed, it's no longer future. So one can only assert to have tested the predictability formerly called future; also known as the Prince test.

      • Re: (Score:2, Insightful)

        by Anonymous Coward

        That's why you should always divide your data set into one subset for fitting/training of the algorithm, and another subset to verify its predictive ability.

        The algo doesn't know or care whether the data is actually from the future. That is irrelevant as long as it wasn't fitted on it.

        • But of course you tweak and change over time rather than having the first try work just perfectly and so that subset for verification ends up influencing the algorithm anyway.

      • by Dthief ( 1700318 )
        how about make the algorithm.......use it in a predictive manner after making it....THEN REPORT IT.....

        what they have done here is taken a data set and made algorithms until one of them matched well. If I have a model that predicts traffic patterns or weather patterns in the past, its only useful if it is then applied after the fact and is still comparably accurate to when it was developed.

    • Re:biased algorith (Score:4, Informative)

      by Chatterton ( 228704 ) on Thursday August 07, 2014 @04:47AM (#47621153) Homepage

      That why you train your algorithm on all the available cases but the last year ones. Then you can test it on that last year of cases. For the system the last year is the "future" on which you do your testing.

      • Re:biased algorith (Score:5, Informative)

        by Anonymous Coward on Thursday August 07, 2014 @05:11AM (#47621203)

        Yes, and then when the algorithm doesn't work you finetune it a bit and test again and suddenly you end up with an algorithm that has been trained on all data without actually training it against all data.

        One should be very skeptical against future predicting algorithms. Until they have been released in the wild for a while without the developer tampering with it it is pretty safe to guess that it more or less is another version of the Turk [wikipedia.org], even if its inventor doesn't realize it.

        The same principle can be applied to market research or climate studies. If the algorithm used is tampered with to produce more accurate results one can assume that it is useless.

        • by Dthief ( 1700318 )
          THIS...better worded than my original comment
        • I would assume that any person doing professional statistical research knows how to validate to a certain degree of trust.
          For example from:
          http://en.wikipedia.org/wiki/C... [wikipedia.org]

          "Repeated random sub-sampling validation

          This method randomly splits the dataset into training and validation data. For each such split, the model is fit to the training data, and predictive accuracy is assessed using the validation data. The results are then averaged over the splits."
          So you actually train against all data and validate aga

        • In this particular case, I'm not very impressed with a 70% prediction rate on a binary decision... you could get similar results by saying "Uphold" every time and ignoring all the data.

          What would be more impressive is if the algorithm could predict (with greater accuracy) how theoretical courts would perform with new justices assigned to the bench. Say a seat is opening up and there are several candidates for the position, can the algorithm tell you what the outcome of an upcoming case will be with the var

        • I beg to differ. While constructing a model there are often unknown relationships and parameters between variables for which you have to make assumptions. Like, for example, you suspect that two variables are related, but instead of digging in deeper and deeper in order to exactly resolve the relation you assume an e.g. linear relation, you fit the parameters to some data and move on. As long as you clearly present your methodology, I don't think there is anything wrong with this. The next guy can look clos

        • by sjames ( 1099 )

          However, time marches on. There will always be new data to test against.

          The more interesting question though is to look at what factors were involved in the successful algorithm. Ideally there won't be terms in there like gender or race of the justices or political affiliations. That, in turn offers a way to look (theoretically, of course) at how the current SCOTUS might have decided key cases in the past.

      • People develop predictive algorithms for all sorts of the things. The most obvious are trading algorithms for financial markets. Such an algorithm can be very accurate... until trends change in what you are predicting. Because the algorithm is built based on an analysis of the historical data it is generally going to be very successful at "prediction" when then run against that data.

        The utility of the algorithm doesn't become evident until it is tested against data which wasn't available when designing it a
    • by jellie ( 949898 )

      In this particular case, future predictability doesn't work. The sample size is way too small (as SCOTUS only hears ~80 cases/year), and the cases are not evenly distributed. The last couple years, for example, the court has become very conservative and happens to hear a significantly higher percentage of business-related cases. It's hard to predict anything from that.

      It would make more sense to divide the data into training and validation/cross-validation data sets like in a standard machine learning appro

    • I wouldn't be surprised if the primary predictive trait used is simply to check the biases of each judge and then assume they will vote along those biases. Assuming conservative judges will vote conservative and liberal judges will vote liberal should give you a pretty good score right off the bat.

      • by AthanasiusKircher ( 1333179 ) on Thursday August 07, 2014 @08:12AM (#47621913)

        I wouldn't be surprised if the primary predictive trait used is simply to check the biases of each judge and then assume they will vote along those biases. Assuming conservative judges will vote conservative and liberal judges will vote liberal should give you a pretty good score right off the bat.

        Only in a small minority of cases. Contrary to popular belief, most SCOTUS cases aren't highly politicized cases with a clear conservative/liberal divide. Most cases deal with rather technical issues of law which are much less susceptible to this sort of political analysis.

        The Roberts Court, for example, has averaged 40-50% unanimous rulings in recent years (last year about 2/3 of rulings were unanimous). So, your idea of "assume conservative vote conservative, liberal vote liberal" would tell you nothing about maybe half of the cases that have come before the court in recent years. (Historically, I believe about 1/3 or so of rulings tend to be unanimous.)

        And even with the closely divided cases, you have a problem. Of the 5-4 rulings (which in recent years have been only about 20-30% of the total rulings), about 1/4 to 1/3 of them don't divide up according to supposed "party lines."

        In sum, I don't know what factors this model ends up using, but "conservative vs. liberal" is way too simplistic to predict the vast majority of SCOTUS rulings. If you could factor in detailed perspectives on law (which often have little to do with the stereotyped political spectrum), you might have something... but that would require a lot more work, particularly over the 50 years of rulings TFA deals with.

        • Bear in mind, the model only gets it right 70% of the time, and a red-black roulette spin would get it right nearly 50% of the time.

          • Bear in mind, the model only gets it right 70% of the time, and a red-black roulette spin would get it right nearly 50% of the time.

            Yes, and if we were talking about a handful or even a few dozen outcomes, 70% accuracy wouldn't be significant. But we're talking about 68,000 individual decisions of justices. If your roulette spin came up red 70% of the time over 68,000 spins, you'd be darn certain it was rigged. Besides, focusing on this one statistic is relatively meaningless -- a model that gets 70% correct could be simplistic and stupid, or it could have a tremendous amount of insight... that one number says nothing.

            • Personally, I'd call an simplistic algorithm that gets 70% right brilliant, and one that has a tremendous amount of insight that also gets 70% correct overly complicated and prone to unpredictable failure.

        • So 40-50% are unanimous, and those should be easy to predict. For the remainder, predict party line, and you will get an additional 30-40% right. So an algorithm that gets only 70% right doesn't seem very impressive. Even simplistic guessing should do at least that well.

          • And you'd have a reasonable argument if we were only predicting affirmations vs. reversals. That's just a couple possible outcomes per case. But the model also predicts the votes of individual justices with over 70% accuracy... tens of thousands of them. Is it the best model ever? Probably not. But the results seem quite significant over such a large number of cases over 50 years... so it's something. You really think it's that easy to predict when a case will be unanimous vs. 8-1 or 7-2 or 6-3, and t
          • Just ran some numbers, since I was curious

            So 40-50% are unanimous, and those should be easy to predict.

            Only if you can predict which decisions will be unanimous with 100% accuracy.

            For the remainder, predict party line, and you will get an additional 30-40% right.

            Let's assume we can predict the 45% or so of unanimous decisions of the past few years with 100% accuracy (a dubious assumption), so we have the other 55% to deal with. Even if we assume an incredibly simple model where 4 justices are solidly on each "side" and only one justice is consistently a swing vote (empirically not true), we still have to deal with predicting the roughly 1/3 of cas

        • by Alomex ( 148003 )

          he Roberts Court, for example, has averaged 40-50% unanimous rulings in recent years

          If at all possible courts rule only on the parts they all or the vast majority agree on and skip parts they don't agree. For example, one judge might want to overturn the entire law, another just this specific application. Then the court unanimously rules to reverse the case and remains silent on the bigger issue.

          • The still need a reason to reverse the case. The SCOTUS or any court for that matter does not just go because we said so. They have reasons. Now, I agree that they may or may not overturn a law based on how much they agree with each other, but they still need a reason to invalidate a lower courts judgement.

      • I was going to say: media preference datapoint: NPR vs FOX, might be the strongest predictor.

    • by plopez ( 54068 )

      That is why you use split data sets. You calibrate on one half, or less, of historical data and then verify against data you did NOT calibrate against.

    • And sometimes an algorithm can't predict the future OR correctly duplicate the past. Just see global temperature models.
    • You mean like all the cosmological (is that the word?) algorithms and models that match how the universe was created?
    • by mysidia ( 191772 )

      I (read: anyone) can make an algorithm that fits any previous data

      Unless it was an honest test where the sets of cases used to build and train the algorithm were required to be random samples, AND the cases the prediction was tested against were also a random sample with no overlap with the cases used to build the algorithm (with no training of the algorithm based on the cases supposedly being used to validate it).

  • by Chrisq ( 894406 ) on Thursday August 07, 2014 @04:44AM (#47621137)
    Just identify the wrong decision and they are bond to pick it ;-)
  • by Anonymous Coward

    If the decisions have 50/50 distribution, then a random guess is right 50% of the time. For any other distribution it's more than that. Soooo 70% is at best a little bit better than random guess, at worst equal to it.

  • Useless (Score:5, Insightful)

    by Jiro ( 131519 ) on Thursday August 07, 2014 @04:52AM (#47621169)

    According to http://www.scotusblog.com/stat... [scotusblog.com] the Supreme Court recently affirmed 27% of lower court decisions and reversed 73%. This means that if you guess that the Supreme Court reverses the lower court every time, you'll be 73% accurate. 70% accuracy is ridiculously low if you can get 73% accuracy *without* taking into consideration the records of each justice or any other kind of details.

    • This should be +10 insightful.
      • Not really - The "algorithm" the grandparent has come up with can be written out as "The vote will be to reverse the ruling". Sure, you will get approximately 73% accuracy, assuming the distribution of the decisions remains the same. But it has zero utility as a predictive algorithm. Presumably, the algorithm that has been developed in TFA can predict both rulings to uphold and rulings to reverse with 70% accuracy. That's infinitely more useful than an algorithm that predicts rulings to reverse with 100%
    • It depends - there's a difference between saying 70% "in general" and "this one will be part of the 70%".

      Of course, since the percentages seem very close the practical implications would seem to be the same.

    • by dkf ( 304284 )

      According to http://www.scotusblog.com/stat... [scotusblog.com] the Supreme Court recently affirmed 27% of lower court decisions and reversed 73%. This means that if you guess that the Supreme Court reverses the lower court every time, you'll be 73% accurate. 70% accuracy is ridiculously low if you can get 73% accuracy *without* taking into consideration the records of each justice or any other kind of details.

      Of course, the usual reason why the case got to the Supremes in the first place is because there were two cases by different Appeals Circuits which conflicted.

    • Re:Useless (Score:5, Informative)

      by AthanasiusKircher ( 1333179 ) on Thursday August 07, 2014 @06:22AM (#47621395)

      70% accuracy is ridiculously low if you can get 73% accuracy *without* taking into consideration the records of each justice or any other kind of details.

      First, your link only deals with the past court term. TFA deals with predicting cases back to 1953. Is your 73% stat valid for the entire past half century?

      And even if it were, the algorithm is much more granular than that, predicting the way individual justices will vote. From TFA:

      69.7% of the Courtâ(TM)s overall affirm and reverse decisions and correctly forecasts 70.9% of the votes of individual justices across 7,700 cases and more than 68,000 justice votes. Also, before someone objects, please note that (contrary to popular belief) SCOTUS does not always vote 5-4 according to party lines. For instance, your own link notes that 2/3 of last year's opinions were UNANIMOUS. 5-4 decisions usually amount for only 25% of cases or so in recent years, and of those, usually a 1/3 or so don't divide up according to supposed "party line" votes.

      So, I agree with you that simply predicting reverse/affirm at 70% accuracy may be easy, but predicting 68000 individual justice votes with similar accuracy might be a significantly greater challenge.

      • Sorry -- accidentally hit submit early. Obviously the main quote from TFA is only one sentence... the rest is my commentary.
      • So, I agree with you that simply predicting reverse/affirm at 70% accuracy may be easy, but predicting 68000 individual justice votes with similar accuracy might be a significantly greater challenge.

        In fact, it looks like very much the same challenge: with most decisions being unanimous reversals, it seems only a small minority of those individual votes are votes to affirm the lower court decision. So, just as 'return "reverse";' is a 70+% accurate predictor of the overall court ruling in each case, the ver

        • In fact, it looks like very much the same challenge: with most decisions being unanimous reversals, it seems only a small minority of those individual votes are votes to affirm the lower court decision.

          Nope -- you just made the same error the GP did: extrapolating a false inference based on one year of data. It's true that last year had 2/3 unanimous rulings, but that was an outlier -- which I was mainly using to make a point about how the 5-4 rulings that make the news are not as common as we think.

          In reality, the Roberts court has averaged maybe 40-50% unanimous rulings, but this is an outlier historically too. Over the 50 years TFA deals with, the unanimous rate is more like 30-40%, I think, maybe

  • by mwvdlee ( 775178 ) on Thursday August 07, 2014 @04:55AM (#47621175) Homepage

    if defendant.bank_balance > plaintiff.bank_balance
          winner = defendant
    else
          winner = plaintiff

    I'd guess about 90% accurate.

    • by dywolf ( 2673597 )

      my algorithm is even better, and even more accurate. its simple: What is the worst possible outcome for the citizenry?

      • by swillden ( 191260 ) <shawn-ds@willden.org> on Thursday August 07, 2014 @09:34AM (#47622423) Journal

        my algorithm is even better, and even more accurate. its simple: What is the worst possible outcome for the citizenry?

        I don't know about the accuracy of your SCOTUS result-picking algorithm, but you and mwvdlee have a good algorithm to get modded up on slashdot: Just express deep cynicism about the system. Doesn't have to be true in the slightest.

        FWIW, I watch SCOTUS pretty closely, and I'd say their bad decisions are fairly rare. I'm unhappy with the outcome in a larger minority of cases, but it's not very common that upon reading the opinions and dissents that I find myself ultimately in disagreement with their conclusions. And in most cases I think they not only make the right legal call, but the right call for the citizenry (though that isn't, and shouldn't be, their primary focus).

        Of course, you and I may well disagree about some of the decisions.

        • by dywolf ( 2673597 )

          Cynicism? Yes. Warranted? Yes.
          To be fair, most of their decisions aren't earth shattering or even really newsworthy, so they dont get reported.
          But of those that are big deals...this current court is pretty atrocious. Particularly in terms of business, this court is one of the most pro-corporation-as-the-expense-of-citizens/consumers that has ever existed.

          Good decision: killing DOMA, upholding the ACA in general

          Bad decisions: allowing unlimited money in politics...twice (total ban on contribution limits will

          • The Court's first responsibility is to uphold the law -- not the law as they or anyone else wants it. This includes the Constitution, the "supreme law of the land" - they can't uphold a law that Congress has no authority to pass in the first place.

            From this viewpoint, let's take a look at those decisions:

            Bad decision: Calling the ACA a "tax". The ACA originated in the Senate, even though the Constitution requires that new taxes originate in the House. Furthermore, you can't compel people to buy something, a

            • by dywolf ( 2673597 )

              so money = speech, and if some just happen to have a louder voice because they have more money...oh well?
              Bugger that.

              and the rest of your comments are nonsense too: the ACA did originate in the house; if we can require people to carry car insurance so they dont become a burden and impose a burden on others then theres no reason not to do it for healthcare, especially when healthcare costs are 20% of our entire economy (course this whole "problem" goes away if we just instituted medicaire for all....plus it'

              • by dywolf ( 2673597 )

                and just so I can be especially clear: if it happens that the right decision in terms ofthe Law is NOT the right decision in terms of the public good, then the Law MUST be changed. The two concepts should, indeed they must, be aligned.

                Otherwise you become a country who worships the law even to the detriment of soctety itself.

              • You don't own a newspaper to deliver your opinion to the front steps of millions of people... Oh well?

                That doesn't mean we can go around neutering newspapers. Now, I never said "money = speech", but that doesn't make the First Amendment implications any less relevant. You cannot enforce a law that has the effect of chilling speech. Period full stop.

                Everything for Obamacare/PPACA, including the "penalty" tax and tax on medical devices, was introduced in the Senate. They could only pass the Senate version bec

            • by dywolf ( 2673597 )

              and just so I can be especially clear: if it happens that the right decision in terms ofthe Law is NOT the right decision in terms of the public good, then the Law MUST be changed. The two concepts should, indeed they must, be aligned in a nation of free people.

              Otherwise you become a country who worships the law even to the detriment of society itself.

    • I mean when will this ever end?
    • Or

      if defendant.lawyer_pay > plaintiff.lawyer_pay
      winner = defendant
      else
      winner = plaintiff
    • I'd guess about 90% accurate.

      lol if you ever actually want to bet money on that algorithm, let me know. I'll even give you 5 to 1 odds, instead of the 9 to 1 you suggest.

  • by Lumpy ( 12016 ) on Thursday August 07, 2014 @05:46AM (#47621287) Homepage

    Install software in the helmet, Set the judges loose on the city....

    I AM THE LAW!

  • A 70% prediction rate is not impressive. In the UK, where the weather seems pretty unpredictable, "it will be pretty much the same as yesterday" is right about 70% of the time. Weather forecasting and track individual storms, but It took a long time and a lot of research for the weather forecast success flat rate to get any better than this. The model may be important: the success rate probably isn't.
    • by jd ( 1658 )

      It's 70% average. For the Democratic judges, it's much lower. For the Republican judges, you could probably dispense with them and use the code as it stands. Since the algorithm falls short of true AI, this clearly implies a lot about how decisions are made and what with.

    • A 70% prediction rate is not impressive.

      Doesn't that rather depend on what you're predicting, and how good previous algorithms were?

      Isn't that a bit like complaining that "10mph is not impressive" while commenting on a story about the world's fastest snail?

      In the UK, where the weather seems pretty unpredictable, "it will be pretty much the same as yesterday" is right about 70% of the time.

      I can predict with 99.9% accuracy what the weather will be like in five minutes. Does that mean any prediction less than 99.9% accurate is not impressive?

  • The US Constitution is only about 4 pages, 4400 words (and the bulk of that is structural & procedural minutiae about the US government).
    The role of the USSC is simply resolving if a law does or does not conform to the US Constitution.

    Given those relatively limited boundaries, it shouldn't be that complex of an issue to predict algorithmically the results of a given judicial ruling, one would think. (The devil's in the details about parsing meaning and context.)

    Of course, I believe phrases like "the ri

    • by jd ( 1658 )

      Since you fail on the example you tried to parse, I suggest that although the theory is easy, personal prejudice always takes precedence over what is written.

    • Determining constitutionality is an important part of the Supreme Court's work, but it's hardly all of it.

      Did you know that Federal law varies over the country? The law as written is the same, of course, but it's interpreted differently. One of the Supreme Court's functions is to rule on a sample case when it gets too bad.

  • Per their own data:
    They reviewed 7700 cases.
    The court reversed 5077 of those cases.
    So the court reverses 66% of cases it sees. Which makes sense, that's what the court does.
    So I can get damn close to their results with my model which is: "The court will reverse 100% of the time"

    I don't see their model in there, and I don't really care to look that hard. But they said they used the same data previous models did. Most of that data are things like:
    Which court heard the origional case?
    Was the decision liberal o

    • More specifically, if the court is reasonably happy with a circuit court decision, there's no real need for the Supreme Court to intervene. If they Supremes disagree, they might well want to hear the case. I'd suspect that many of the upholdings were matters of different circuit courts making different interpretations, so the Supremes would grab a case and hear it to set consistent binding precedent across the country.

  • Despite all the (partially true) snark. Isn't this a good thing? Shouldn't the highest court of the land be producing rulings that are predictably consistent with previous rulings? Unless a case is truly novel, past performance should be a good predictor of future performance here, since case law is cumulative.
    • While I agree that we do need a logically coherent set of laws and rulings that doesn't seem to be what we get.
      • by geekoid ( 135745 )

        Laws are created by men to get a society to work and last. Society is not logically coherent, so why would you expect the laws to be?

  • by geekoid ( 135745 )

    is still withing chance because the error bars on the are huge.

  • by Opportunist ( 166417 ) on Thursday August 07, 2014 @10:10AM (#47622739)

    I'd actually have expected more. I mean, let's be reasonable here:

    A 50% accuracy can be achieved by the average unbiased coin. Now throw in the rulings that are easy to foresee because any other decision would be politically suicidal and you should easily arrive at more than 70%.

    • Now throw in the rulings that are easy to foresee because any other decision would be politically suicidal

      The supreme court doesn't have to worry about political 'suicide.' By design their position was made to be shielded from the problems of politics.

  • ...Is that in most cases the decision to 'give cert' for Supreme Court review of given lower-court cases vests with the Justices' law clerks. This may be as weird as that decision to give the Librarian of Congress veto power over unlocking our cellphones, but that's the way it is. How accurate can any model be at delving into the minds of law clerks?

  • The Supreme Court is dominated by a bunch of fanatic right wing corporate toadies.
    So, the decision comes down in favor of corporations (on economic issues) or social conservatives (on social issues).
    The Constitution has nothing to do with it.
    The Supreme Court is the ultimate cheerleader for our fascist state.

The 11 is for people with the pride of a 10 and the pocketbook of an 8. -- R.B. Greenberg [referring to PDPs?]

Working...