Suggestions
Ben Winokur
Chief Of Staff at Passport
Working on Subsalt. @passporthq, @wustl (x2), @washuwash, @dukelaw, @instagift alum. @auburnfootball and @lfc. Sock enthusiast.
Highlights
Privacy is going to be used as an excuse to undermine AI transparency.
"We can't release our training data for evaluation/research purposes, because of privacy risks."
"Open-source and open-weights models are dangerous, because they risk exposing training data."
The providers of commercial large language models have every incentive to restrict access to models and training data for commercial reasons, including retaining their ability to build and monetize AI safety and transparency tooling on top of their proprietary models.
Security and privacy will be the public explanation for why they have to keep everything secret.
We'll be worse off if that's how this new technology develops, because that will limit the space of experimentation and innovation. And in the long-run, it will produce less resilient, less battle-tested, less-secure systems.
It's critically important that we think carefully about how to preserve BOTH privacy and transparency so that AI follows the pattern of the open Internet, which produced a cambrian explosion of new, valuable, societally-beneficial products and services.
Training data anonymization is one of those important steps on the path to more open and transparent AI.
In the recent FTC investigation, Avast claimed that they were entitled to sell and share browsing data because it was anonymized. The FTC found that the anonymization measures employed by Avast were wildly insufficient to actually prevent re-identification -- and even where contracts contained prohibitions on re-identification, the language allowed customers to join their first-party data to the data purchased from Avast.
Two important points for other companies:
- If your privacy policy states that you may share or sell aggregated or anonymized data, the FTC has now confirmed that "anonymized" has a concrete technical definition consistent with the de-identified and anonymized data exemptions in state and global data protection laws. If you're just removing PII, you're not anonymizing your data, and you're exposing yourself to compliance risk if you subsequently sell or share that data.
I'd look out for litigation here, as VERY few organizations that use that language in their privacy policies anonymize data to this standard before selling or sharing.
- It is going to be VERY hard to claim that data is anonymized if it can be joined with other data assets. Shared pseudonyms and IDs that allow parties to share and combine their data are going to be considered de facto personal information where each party holds identifiable information about users.
Even if both parties otherwise de-identify data prior to joining, the resulting data sets will often be identifiable to both parties. Anonymization post-join and pre-processing is going to be necessary.
