Predictive Policing Software Terrible At Predicting Crimes (wired.com) 62
An anonymous reader quotes a report from Wired: Crime predictions generated for the police department in Plainfield, New Jersey, rarely lined up with reported crimes, an analysis by The Markup has found, adding new context to the debate over the efficacy of crime prediction software. Geolitica, known as PredPol until a 2021 rebrand, produces software that ingests data from crime incident reports and produces daily predictions on where and when crimes are most likely to occur. We examined 23,631 predictions generated by Geolitica between February 25 and December 18, 2018, for the Plainfield Police Department (PD). Each prediction we analyzed from the company's algorithm indicated that one type of crime was likely to occur in a location not patrolled by Plainfield PD. In the end, the success rate was less than half a percent. Fewer than 100 of the predictions lined up with a crime in the predicted category, that was also later reported to police. Diving deeper, we looked at predictions specifically for robberies or aggravated assaults that were likely to occur in Plainfield and found a similarly low success rate: 0.6 percent. The pattern was even worse when we looked at burglary predictions, which had a success rate of 0.1 percent.
No Problem! (Score:2)
Re: (Score:2, Insightful)
Re: (Score:3)
the outlet might look wonky, but the alleged numbers from the police itself are pretty eloquent:
the success rate was less than half a percent. ... specifically for robberies or aggravated assaults that were likely to occur in Plainfield and found a similarly low success rate: 0.6 percent. The pattern was even worse when we looked at burglary predictions, which had a success rate of 0.1 percent.
less than 1% success rate has hardly anything to do with "confidence intervals" being too high, or "people not understanding ai". a drunk monkey would be 50% more accurate than that. try again.
Re: (Score:2)
a drunk monkey would be 50% more accurate than that.
If I asked you to pick some houses in Tokyo that were likely to be robbed tonight, I bet your success rate would be not small but zero. Guess we'd better choose the drunken monkey instead.
(Or perhaps you are just being obtuse about the statistics)
Re: (Score:2)
the outlet might look wonky, but the alleged numbers from the police itself are pretty eloquent:
the success rate was less than half a percent. ... specifically for robberies or aggravated assaults that were likely to occur in Plainfield and found a similarly low success rate: 0.6 percent. The pattern was even worse when we looked at burglary predictions, which had a success rate of 0.1 percent.
less than 1% success rate has hardly anything to do with "confidence intervals" being too high, or "people not understanding ai". a drunk monkey would be 50% more accurate than that. try again.
What is the definition of success? A significant parameter in that definition is the granularity of the prediction over space and time. Since both space and time are continuous variables, the expected correct prediction rate for an exact space and time is always zero, so both space and time are necessarily granularized. The granularity of space and time predictions is extremely important. Is the space prediction granularity one meter, one house, one block, or one precinct? Is the time prediction granul
Re: (Score:2)
Uh, everyone...perhaps look at the source of this "information". So a few things about predictions, when scientists make predictions they have some kind of confidence interval.
But these weren't predictions by scientists. They were predictions by a company out to make money from police departments.
Predictive AI bad at predictions (Score:5, Funny)
Re:Predictive AI bad at predictions (Score:5, Funny)
Well, not the AI.
Re: (Score:2)
That is a really good point.
--
"Life is very interesting..." - Drew Barrymore
Re: (Score:2)
Just give it more time. Back train it on itself. That'll work, we're certain!
Re: (Score:2)
However you could predict something is being planned if you had information for example that a "safe cracker" is meeting with a "driver" and a guy considered "hired muscle and this information would come from meta data of phone calls not necessarily from the content of the call which may be encrypted. Or through some good police work
Re: Predictive AI bad at predictions (Score:2)
Re: (Score:3)
The difficulty would be to predict exact events.
There is no reason to expect software to predict individual events when dealing with humans rather than billiard balls.
In addition to all the other problems with predicting human behavior, there is the Heisenberg effect: observing changes the behavior. If the software predicts a burglary at 123 Main St., and the cops stake it out, then the burglar is gonna see the cops sitting in front of the house in their van and go rob a different house instead. So that's gonna be marked down as a failed prediction.
Re: (Score:2)
As it should, because the prediction was completely worthless - a house still got robbed, so nothing was gained.
Re: (Score:2)
The real world isn't GTA, though. "Safe cracker" isn't something a person can be tagged with in a police database. If they know somebody is a safe cracker, why aren't they in prison?
No, it's more likely to be more heuristic about it. Like a strong guy, a welder, and somebody with a fast car regularly meet and seem to have too much money to spend. Which of course now is far less reliable and results in hassling a lot of people who just happen to have a skill or interest that's deemed suspicious.
Re: (Score:3)
If they know somebody is a safe cracker, why aren't they in prison?
The cops tend to know quite well who's a professional criminal. Actually proving they took a part in a particular hit, though, takes a lot of work, and limited tuits the police has are de/reprioritized for political reasons to a huge degree -- which doesn't align to what the society, and your safety in particular, would need.
Amateurs [Re:Predictive AI bad at predictions ] (Score:3)
If they know somebody is a safe cracker, why aren't they in prison?
The cops tend to know quite well who's a professional criminal.
Nope.
First, most criminals are amateurs. Second, no, cops don't know who are professional criminals. You're watch too many Hollywood movies where the police know the gangs but can't put them in jail until an unconventional cop who ignores civil rights and gets information by beating informers up until they squeal solves the case and kills the bad guy in a shoot-out.
Re: (Score:2)
But "knowing" and KNOWING (ie: one is an unofficial... the other is substantial arrestable proof) are two different things.
Also something that is different these days as well.. just like how companies have gone through a period of consolidation (bigger companies
Re: (Score:2)
I dont think thats what these pieces of software are actually trying to do. The concept SEEMS to be "a% economic circumstances, b% social circummstances, c% physical circumstances means d% chance of crimes in this area" , ie predicting the best places to put cops to most likely catch crimes. Essentially an automated criminologist.
The problem is , well other than the fact they seem to fail at even that (Turns out the difference between an AI criminologist and a human one is the human one lives in the neighbo
Which is weird (Score:2)
Re: (Score:2)
But you're not actually selling to the police. You're selling to their funding managers, which are mayors/etc, ie not actually police.
It's really easy to snow people with fancy datasets and charts that don't know wtf they're looking at or what they should be looking for.
Re: (Score:2)
I believe that generally the funding managers don't micromanage how funds are spent - so you *are* selling to the police.
Re: (Score:2)
Sometimes this is true, but in my (brief) experience within public service systems it more often isn't.
Many times major funding comes with strings attached - ie, must be used for a specific purpose - for example, this is how you end up with a school library upgrade of dozens of new computers when the school (and even the district) has no idea how they'll ever use those systems for any purpose outside of casual minesweeper during free periods. Within the police spending context, this easily turns into 'must
Re:Which is weird (Score:5, Insightful)
Zipf's law for instance will tell you the distribution of the letters in a given text extremely well, even for comparatively short passages of text, but it does not allow you to predict the 100th character in a text even with some certainity (only that with about 12.6 percent probability, it is an e, but that's as good as it gets). Crimes are even in the worst areas rare random events, and in the most cases, a chain of events does not lead to a crime.
exponential geographical spike [Re:Which is weird] (Score:2)
There's statistics papers showing crimes have an exponential geographical spike,
I can't even figure out what the phrase "exponential geographical spike" means.
The function Crime[location] is Crime = e^(location) ?
It's not supposed to be (Score:2)
Re: (Score:2)
It's just there to establish cause for over policing the neighborhoods of undesirables. Then you can arrest them and take away voting rights.
And make them into slaves, which is permitted by the 13th Amendment
Re: (Score:3)
If you're sending extra police to the locations of the predictions. Won't that stop some of the crimes and make the predictions "wrong".
Only if the predictions where followed, and even then not necessarily. Aggravated assault might be a fight breaking out within a family, and those fights can not be prevented by the police showing up, as they happen behind closed doors.
No, we are in the "Predictions are hard, especially those concerning the future" category. Crimes are comparatively rare and very random events even in crime ridden places, which makes predicting them individually a nearly impossible task. If you take a longer time period a
Re: (Score:3)
Ahh you're a conspiracy nutter.
there is ample evidence of vote eligibility, access or significance manipulation in the us in many ways, from arbitrary rights revocations and biased appeal systems to plain and simple "sloppy" censing or sophisticated jerrymandering. police sweeps are a common tool for a host of purposes which have also included spurious economic or political interests, like manipulating an election in advance. it's an actual thing: https://en.wikipedia.org/wiki/... [wikipedia.org]
Re: (Score:3)
If you're sending extra police to the locations of the predictions. Won't that stop some of the crimes and make the predictions "wrong".
a prediction that isn't testable is worthless. to evaluate the thing and see if it is actually fit for purpose you should simply do nothing about it, then compare results, get the numbers, announce your accuracy. it's easy. anything else might sound better or worse but is just hocus pocus bullshit that isn't really interested about accuracy at all.
Re: (Score:2)
A conspiracy nutter?
The then-president's chief of staff has publicly admitted that the drug war was dreamed up in large part to revoke voting rights from minorities and anti-war hippies in order to further their political ambitions.
I think we're well past the point of dismissing such accusations as all coming from "conspiracy nutters".
precrime = long cryo jail time with no trail (Score:2)
precrime = long cryo jail time with no trail
Re: (Score:2, Insightful)
It's just there to establish cause for over policing the neighborhoods of undesirables. Then you can arrest them and take away voting rights.
Er, you mean like arresting the leading opposition candidate?
Re: (Score:2)
Yes, that has to be included. And after all, sometimes the guy arrested actually is guilty...but that still affects the voting.
The question in such cases is always "should it be allowed?", and at minimum this should require that the law actually be followed. Sometimes that's not sufficient, but that should be a minimum.
The idea (Score:3)
Re: (Score:3)
But... experienced police officers are EXPENSIVE!
Re: (Score:2)
You couldn't pay me enough to be a Plainfield, NJ cop. I mean it's better than NYPD but only by a little. Having lived there (ok, North Plainfield...) it's not a great place to live.
Anyone old and experienced on the force probably retired three years ago, if they were smart.
Re: (Score:1)
I lived in Scotch Plains and wholeheartedly agree with you!
Re:The idea (Score:5, Interesting)
I've worked on this stuff. Mostly it was used correctly - not for prediction of future events, but for analysis of past ones to help identify prime suspects.
There should be no question that a decent analyst with a properly populated data set is going to be better at pulling out non-obvious correlations than a human. The database has no trouble taking criminal records, repeat offender parole enforcement, officer contact reports, and recent occurrences and telling you that Jane's probably hooking up with John, who put her in touch with Joe, and now Jane and Joe seem to be working together even though no officer has directly observed this yet.
Since most criminals are habitual ones, 'known to the police', this is a very good place to start when looking for suspects.
In terms of predictive abilities... simple heat maps contrasting recent criminal activity with police visibility (the cars and even individual cops are all constantly GPS-tracked and logged now, I've built the systems that do it for multiple agencies) can show you trends. Then you blend in time of day, day of week, day of month, and zoning info and it's not terribly surprising that the system can tell you which areas are likely to see an increase in crime. It won't tell you that 12 Main Street is going to be burgled tonight at 3:12 AM, but it'll tell you that it's prime crime time, cops haven't been visibly patrolling recently, and there have been occurrences in the area that are probably going to repeat soon because the apparent opportunity is looking pretty good.
It's still a good percentage 'voodoo' that maybe a good officer should be able to handle without a crime analyst and a database to point it out to them - but when you're talking a 24/7 service in a decent-sized city there is a lot of opportunity for these things to be missed as cops move around and rotate through shifts.
Are most criminal known to the police (Score:2)
What you're doing is more like "John committed a crime, so Jane is probably a criminal because Jane knows John".
That's just poverty. It's got a name. "Broken Windows Policing". And it's history is not great.
So does the software work? Yes and no. Is Jane a criminal? Well, she's got a baggie of p
Re: (Score:2)
It has been shown to not work, in theory and now in practice
What is worse is that if you send police to an area they don't normally patrol, and they don't know much about, except that crime has been predicted there, it is both likely they might put off criminals - or not ... and because they are there they WILL find crimes, which in the past has been taken to show that the prediction was "correct" ...
Wrong cognition tool (Score:2)
Predictive Policing Software Terrible At Predicting Crimes
That's what happens when you cheap out and use Postcogs instead of Precogs [fandom.com] ...
Garbage in, garbage out (Score:3)
Sounds like a case of not putting any useful data into the system.
And probably no real AI to analyse it anyway. Geolitica sounds like a bunch of buzzwords slapped together to win a lucrative police contract.
Maybe if a real AI (glorified predictive pattern recognition system) has access to the cell-phone network tracking, and social media traffic, if might be able to recognise patterns, and do better than a cop with a map of last years crime locations, and a calendar. I'm sure the NSA has such tech.
But this story is set in the US, where there is so little trust of authorities, that something like this could be difficult to set up, to say the least. I don't think people would be happy with even anonymised data used that way, so it is flying blind.
Fool and money (Score:2)
Re: (Score:3)
"They should simply fire the fools that spend money on prediction software like this."
My friend, we're talking about law enforcement. Just like when a cop gets caught beating the crap out of an innocent citizen, taxpayers get stuck with the bill, while the cop gets a few weeks of paid vacation, or a new job at the next town down the road. There is zero accountability when law enforcement wastes money on yet another tool that doesn't work at all, or costs a hundred times what it's worth.
Re: (Score:2)
Army wold grab it so fast you would only hear rumours about it existence
Douglas Lenat [wikipedia.org] recently passed away. Known for, among other things, developing a type of AI system called Eurisko [wikipedia.org]The details of his subsequent work may have been classified. But if you are working in the biz, other examples of DoD interest in stuff that actually works aren't that scarce.
Use the Law of the Excluded Middle, Luke! (Score:5, Funny)
Re: (Score:2)
But what do you do with this prediction?
Re:Use the Law of the Excluded Middle, Luke! (Score:4, Funny)
But what do you do with this prediction?
Put a horrendous user interface on it, ideally with some inexplicably poor hardware and software requirements to make installation and deployment difficult and expensive, charge a huge fee plus exorbitant ongoing maintenance, call it AI and sell it to the police.
Duh.
Of course it failed (Score:2)
Anyone even partially competent at writing prediction software gets lured away to work on the stock market, where even "only some usefulness prediction" is still a high enough bar to attract interest and can be valued because with non-linear returns it can afford to be wildly wrong on minor things repeatedly leading to sustainable losses but correct on one hugely valuable thing and turn a profit on the overall.
This context, no. Prediction software fucks up even once and you can expect a lawsuit against the
Re: (Score:2)
Prediction software fucks up even once and you can expect a lawsuit against the police department
Prediction software is used to allocate resources to the locations more likely to experience crime. How are you going to sue for a mis-allocation? And why wouldn't you see lawsuits because some fat-assed shift sergeant sent more officers to the neighborhoods where the POC live?
More shitty police technology (Score:2)
call it precrime unit (Score:1)
Duh (Score:3)
Let's take Houston for example:
10062 sq miles
435 murders per year
Probability per sq mi: 4% PER YEAR
Probability per sq mi: 0.01% PER DAY
26,223 Violent Crimes per year
Probability per sq mi: 260% PER YEAR
Probability per sq mi: 0.7% PER DAY
probably bogus (Score:2)
They correlated with crimes that are reported to police. I used to live around there. Most of the population of Plainfield is not going to report property crime to the police, so the "study" doesn't really tell us much. Of course, the software is probably bogus, too, but that's not proven by this "study."
Have we learnt nothing from... (Score:2)
...Minority Report?
Gotta start somewhere (Score:2)
AI vs human intelligence (Score:2)