In the past months, there’s a been a issue in various instances where accounts would start uploading blatant CSAM to popular communities. First of all this traumatizes anyone who gets to see it before the admins get to it, including the admins who have to review to take it down. Second of all, even if the content is a link to an external site, lemmy sill caches the thumbnail and stores it in the local pict-rs, causing headaches for the admins who have to somehow clear that out. Finally, both image posts and problematic thumbnails are federated to other lemmy instances, and then likewise stored in their pict-rs, causing such content to be stored in their image storage.
This has caused multiple instances to take radical measures, from defederating liberaly, to stopping image uploads to even shutting down.
Today I’m happy to announce that I’ve spend multiple days developing a tool you can plug into your instance to stop this at the source: pictrs-safety
Using a new feature from pictr-rs 0.4.3 we can now cause pictrs to call an arbitary endpoint to validate the content of an image before uploading it. pictrs-safety builds that endpoint which uses an asynchronous approach to validate such images.
I had already developed fedi-safety which could be used to regularly go through your image storage and delete all potential CSAM. I have now extended fedi-safety to plug into pict-rs safety and scan images sent by pict-rs.
The end effect is that any images uploaded or federated into your instance will be scanned in advance and if fedi-safety thinks they’re potential CSAM, they will not be uploaded to your image storage at all!
This covers three important vectors for abuse:
- Malicious users cannot upload CSAM to for trolling communities. Even novel GenerativeAI CSAM.
- Users cannot upload CSAM images and never submit a post or comment (making them invisible to admins). The images will be automatically rejected during upload
- Deferated images and thumbnails of CSAM will be rejected by your pict-rs.
Now, that said, this tool is AI-driven and thus, not perfect. There will be false positives, especially around lewd images and images which contain children or child-topics (even if not lewd). This is the bargain we have to take to prevent the bigger problem above.
By my napkin calculations, false positive rates are below 1%, but certainly someone’s innocent meme will eventually be affected. If this happen, I request to just move on as currently we don’t have a way to whitelist specific images. Don’t try to resize or modify the images to pass the filter. It won’t help you.
For lemmy admins:
- pictrs-safety contains a docker-compose sample you can add to your lemmy’s docker-compose. You will need to your put the .env in the same folder, or adjust the provided variables. (All kudos to @Penguincoder@beehaw.org for the docker support).
- You need to adjust your pict-rs ENVIRONMENT as well. Check the readme.
- fedi-safety must run on a system with GPU. The reason for this is that lemmy provides just a 10-seconds grace period for each upload before it times out the upload regardless of the results. A CPU scan will not be fast enough. However my architecture allows the fedi-safety to run on a different place than pictrs-safety. I am currently running it from my desktop. In fact, if you have a lot of images to scan, you can connect multiple scanning workers to pictrs-safety!
- For those who don’t have access to a GPU, I am working on a NSFW-scanner which will use the AI-Horde directly instead and won’t require using fedi-safety at all. Stay tuned.
For other fediverse software admins
fedi-safety can already be used to scan your image storage for CSAM, so you can also protect yourself and your users, even on mastodon or firefish or whatever.
I will try to provide real-time scanning in the future for each software as well and PRs are welcome.
Divisions by zero
This tool is already active now on divisions by zero. It’s usage should be transparent to you, but do let me know if you notice anything wrong.
Support
If you appreciate the priority work that I’ve put in this tool, please consider supporting this and future development work on liberapay:
All my work is and will always be FOSS and available for all who need it most.
I think you’re off base here. The utopian socialists were arguing against the methods and outcomes of revolutionary socialism, this person is trying their best to explain that this particular tool has serious legal repercussions within the framework we all live under. Those are pretty different.
The reason I see the logic in their arguments is because there’s longstanding legal precedent for misuse of a tool or material because it’s better than nothing to not be a defense even if there are no other options available.
So if you built a car so big no type of shock absorber could handle it cornering at speed and you knew it, using some amazing whiz bang material for shocks isn’t a defense because even though it’s the best thing you knew it wouldn’t work.
Legally speaking, the right choice there is not to make an excessively dangerous vehicle if you don’t want to be held liable for negligence.
It’s also the argument throughout unsafe at any speed although the courts always seem to side with the automakers 🤔
Or if one were to get sued for hosting csam, using the latest whiz bang ai system for detection wouldn’t be a defense or even a point in your favor because you knew it wasn’t a reasonable use of the underlying technologies. You can’t say “judge, I was relying on the ai csam detector!” When the component parts of the ai csam detector have big “prototype, do not use in production” stickers all over them.
Ultimately while these tools might protect mods and users from having to view csam in the moderation process, that’s just one side of the struggle and on its other side they’re a paper shield at best and proof of negligence at worst.
No they’re not. They’re making up bullshit. The legal framework that social media sites where user generated content exists are expected to follow is “take reasonable measures within the resources of your organisation to prevent it” in almost every single country in the world. That’s certainly the tl;dr of how it works for the US and EU anyhow.
This is above and beyond what other major platforms are doing. If you went to imgur right now and slyly uploaded CSAM absolutely fuck all would happen until someone reports it. There is NO proactive approach to countering it at the point of upload. Not on youtube, not on imgur, not on facebook, not fucking anywhere. They all don’t do it because they all don’t have to do it, they all argue that what they’re doing is reasonable, and will cite some absurd percentage of user uploaded content to CSAM reports as their reasoning for it.
If we’re better than that on services without any source of profit-based income we’re absurdly above any level of “reasonable” that exists.
The mistake here is people acting like online social media organisations (which each of these lemmy instances absolutely is as an entity) are regulated in the same way as a random shmuck individual. They’re not. If they were then 4chan would have been shut down 20 years ago and Moot would’ve been imprisoned for life.
I uh actually agree with you almost entirely. Except at the end I’m like “and that’s why it won’t work as protection”.
Software hasn’t been treated like other fields of engineering and all operators have needed for protection from liability was the twin shields of “nothing I could do” and “I was doing nothing” to come out of any courthouse relatively unscathed.
That type of “aww shucks technocracy” is only possible if you do the bare minimum or nothing at all. Once an operator implements some kind of protection (yes, even one with warning labels all over it), both defenses are rendered unusable.
Now that you’ve done something you’re able to be held liable for the effects of what you’ve done and for knowing there was a problem.
The picture gets even murkier when we look at how things are going! Lawsuits against Tesla for their self driving deaths are making waves not because they impugn the dignity of Americas biggest car manufacturer by market cap but because every judge who sees one raises the biggest eyebrow possible at software engineering not being held to the same standard as any other type, both in a court of law and within its own process.
There’s a good chance that software PEs will become a thing (again?) as a result.
The long and short of it is that because the only reason monsters like moot are able to exist is their sly lethargy and looking at the legal storm rolling into software engineering, having something bolted onto the backend like this would be a bad idea.
I think automated tools like this can be put to use though if they were hosted separately and provided with an api that linked up nicely with some moderation queue standard and returned something like “entries 1,5 and 9 are likely csam” back to the moderator. It would at least save the mod from dealing with the material directly.
So I guess I agree but come to the opposite conclusion.
Nah. Hard disagree. The idea that a court will hold you liable for imperfect implementation of better protection within your resources over NO protection is still absolute nonsense.
There is nothing that will save the sites from having a human that needs to deal with the material directly, and anyone advocating for that is going to get sites in legal trouble. The main benefit here is preventing it from posting until a human of the original instance has verified it, which protects federated sites from being sent it and ensures that if it is let through everyone can defederate from the instance that allows CSAM. I am absolutely not advocating for the complete removal of human beings and see that itself as a legal threat. The reduction in humans having to see this material will come from the fact that having such a system will reduce people even bothering to attempt to post this material because it raises the difficulty of attacking the platform beyond any worthwhile risk.