The particular program, System Risk Indication (SyRI), uses government data to predict whether welfare recipients are “likely” to commit welfare fraud. In use for years, the Dutch government has asserted that it poses no undue threat to citizen privacy. However, after numerous privacy advocacy groups – including the United Nations – expressed concerns over the what amounts to an automated criminal investigation of welfare recipients in specific low-income neighborhoods, the Court declined to see things the state’s way. Read: We gave US police, ICE, and CBP AI without regulations in 2016: Now it’s 1984 Per a press release from the Hague District Court (translated via Google Translate): What’s important here is that the court didn’t use GDPR regulations to come to its conclusions. Instead it derived its conclusions from the European Convention on Human Rights, which calls for any technology used to potentially profile people to strike a fair balance between privacy and necessity. According to the court, the legislation does not meet the ‘fair balance’ that the ECHR requires in order to be able to speak about a sufficiently justified violation of private life. Regarding the deployment of SyRI, the legislation is insufficiently clear and verifiable. The legislation is unlawful because it violates higher law and is therefore non-binding. The Dutch government submits that it’s crucial to the state’s well-being that citizens maintain trust in the social security system and thus has defended its use of profiling software the same way governments in the US, UK, and countless other nations have: with the “human in the loop” loophole. By claiming there’s a “human in the loop,” the government avoids being accused of using completely autonomous systems to investigate and prosecute people. As Tech Crunch’s Natasha Lomas reports: By dismissing this notion and declaring such an invasion of privacy a human rights violation, the Hague District Court has set a precedent by which this loophole can finally be circumvented. The Dutch government refused to reveal to the court how SyRI worked, where it got its data, or how it determined risk citing concerns that criminals would learn how to exploit the system were it to divulge such information. Privacy advocates say this amounts to the government viewing every citizen as ‘not innocent’ and preemptively conducting criminal investigations without disclosing probable cause or evidence for such an investigation. Of even greater concern was the fact that the Dutch government only used SyRI in a handful of low-income neighborhoods, many of which are considered havens for poor immigrants. Human Rights Watch declared the ruling a global victory in a statement released today:
