nsfwspy / nsfwspy.net Goto Github PK

A .NET image and video classifier used to identify explicit/pornographic content written in C#.

License: MIT License

C# 71.22% HTML 4.16% SCSS 3.48% TypeScript 21.14%

classification classifier csharp detector dotnet dotnet-core dotnetcore image-classification inappropriate machine-learning mlnet nsfw nsfw-classifier nsfw-detection nsfw-recognition nude nudity nudity-detection porn pornography

nsfwspy.net's Introduction

Introduction

NsfwSpy is a nudity/pornography image and video classifier built for .NET Core 2.0 and later, with support for Windows, macOS and Linux, to aid in moderating user-generated content for various different application types, written in C#. The ML.NET model has been trained against the ResNet V250 neural net architecture with 646,000 images (109GB), from 4 different categories:

Label	Description	Files
Pornography	Images that depict sexual acts and nudity.	106,000
Sexy	Images of people in their underwear and men who are topless.	78,000
Hentai	Drawings or animations of sexual acts and nudity.	83,000
Neutral	Images that are not sexual in nature.	378,000

Other Projects

Looking for a JavaScript version of NsfwSpy? We have you covered - NsfwSpy.js 😎

Performance

NsfwSpy isn't perfect, but the accuracy should be good enough to detect approximately 96% of Nsfw images, those being images that are classed as pornography, sexy or hentai.

	Pornography	Sexy	Hentai	Neutral
Is Nsfw _{^{(pornography + sexy + hentai >= 0.5)}}	95.8%	97.0%	95.2%	3.7%
Correctly Predicted Label	85.7%	84.4%	91.9%	96.54%

Quick Start

Looking to quickly try out NsfwSpy? Check out our steps to use NsfwSpy.App.

This project is available as a NuGet package and can be installed with the following commands:

Package Manager

Install-Package NsfwSpy

.NET CLI

dotnet add package NsfwSpy

Classify an Image File

var nsfwSpy = new NsfwSpy();
var result = nsfwSpy.ClassifyImage(@"C:\Users\username\Documents\flower.jpg");

Classify a Web Image

var uri = new Uri("https://raw.githubusercontent.com/d00ML0rDz/NsfwSpy/main/NsfwSpy.Test/Assets/flower.jpg");
var nsfwSpy = new NsfwSpy();
var result = nsfwSpy.ClassifyImage(uri);

Classify an Image from a Byte Array

var fileBytes = File.ReadAllBytes(filePath);
var nsfwSpy = new NsfwSpy();
var result = nsfwSpy.ClassifyImage(fileBytes);

Classify Multiple Image Files

var files = Directory.GetFiles(@"C:\Users\username\Pictures");
var nsfwSpy = new NsfwSpy();
var results = nsfwSpy.ClassifyImages(files, (filePath, result) =>
{
    Console.WriteLine($"{filePath} - {result.PredictedLabel}");
});

Classify a Gif File

var nsfwSpy = new NsfwSpy();
var result = nsfwSpy.ClassifyGif(@"C:\Users\username\Documents\happy.gif");

Classify a Web Gif

var uri = new Uri("https://raw.githubusercontent.com/d00ML0rDz/NsfwSpy/main/NsfwSpy.Test/Assets/cool.gif");
var nsfwSpy = new NsfwSpy();
var result = nsfwSpy.ClassifyGif(uri);

Classify a Video File

var nsfwSpy = new NsfwSpy();
var result = nsfwSpy.ClassifyVideo(@"C:\Users\username\Documents\happy.mp4");

Classify a Web Video

var uri = new Uri("https://raw.githubusercontent.com/d00ML0rDz/NsfwSpy/main/NsfwSpy.Test/Assets/bikini.mp4");
var nsfwSpy = new NsfwSpy();
var result = nsfwSpy.ClassifyVideo(uri);

Dependency Injection

services.AddScoped<INsfwSpy, NsfwSpy>();

Classify Video Support

To be able to make use of the ClassifyVideo methods, FFmpeg needs to be installed and available in the command line via the 'ffmpeg' command.

Windows

Follow this guide to download FFmpeg, extract it to your C:\ drive and add the required environment path variable.

macOS

Install FFmpeg on macOS using Homebrew via the following command:

brew install ffmpeg

Ubuntu

Install FFmpeg on Ubuntu using the following command:

sudo apt install ffmpeg

GPU Support

To get GPU support working, please follow the prerequisite steps here to install CUDA v10.1 and CUDNN v7.6.4 for CUDA 10.1. Later versions do not work (as I tried with CUDA v11.4). The SciSharp.TensorFlow.Redist-Windows-GPU and SciSharp.TensorFlow.Redist-Linux-GPU packages are already included as part of the NsfwSpy package.

macOS Support

To get NsfwSpy working on macOS, the SciSharp.TensorFlow.Redist v2.3.1 NuGet package also needs to be installed. This not included by default as it interfers with supporting GPUs on Windows and Linux. You can do this with either of the following commands:

Package Manager

Install-Package SciSharp.TensorFlow.Redist -Version 2.3.1

.NET CLI

dotnet add package SciSharp.TensorFlow.Redist --version 2.3.1

Please note that Macs that use M1 chips currently do not support TensorFlow with ML.NET and cannot make use of NsfwSpy.

Contact Us

Interested to get involved in the project? Whether you fancy adding features, providing images to train NsfwSpy with or something else, feel free to contact us via email at [email protected] or find us on Twitter at @nsfw_spy.

Notes

Using NsfwSpy? Let us know! We're keen to hear how the technology is being used and improving the safety of applications.

Got a feature request or found something not quite right? Report it here on GitHub and we'll try to help as best as possible.

nsfwspy.net's People

Contributors

Stargazers

Watchers

Forkers

drdeviant imjxl labofuni thundercls vn-os mrcece vaginessa cunny edwardveb jawar1mx astar-li s4cdesigns blindcolorteen cminorchord meznoos italiardam r0cc0t stormwild alexanderokosten

nsfwspy.net's Issues

ClassifyVideo does not work and other issues

Created a dumb console app (.NET 6) with the following
static void Main(string[] args)
{

        Console.WriteLine("Hello, World!");
        var nsfwSpy = new NsfwSpy();
      
        var result = nsfwSpy.ClassifyVideo(@"D:\movies for checking\Edit.mp4");

       
        Console.ReadKey();
    }

It seems that the code execution never comes out of using (var collection = new MagickImageCollection(video, MagickFormat.Mp4))
No exceptions or anything, i even left it running overnight thinking it might be taking long (the Edit.mp4 is first 10 minute clip of God Father Part 2), I cut it only to 10 min thinking it might be the size/time of all the movie being at (3h:22m run time and 1.4GB).
Here is what my console looks like as the program is running.

I have a 3070 running, with all the libraries installed properly (I think)

Hello, World!
2023-11-29 09:49:57.528567: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudart64_101.dll
2023-11-29 09:49:57.727815: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN)to use the following CPU instructions in performance-critical operations:  AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-11-29 09:49:57.739943: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x2602ddd4620 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2023-11-29 09:49:57.740026: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2023-11-29 09:49:57.741758: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library nvcuda.dll
2023-11-29 09:49:57.758938: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: GeForce RTX 3070 computeCapability: 8.6
coreClock: 1.725GHz coreCount: 46 deviceMemorySize: 8.00GiB deviceMemoryBandwidth: 417.29GiB/s
2023-11-29 09:49:57.759034: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudart64_101.dll
2023-11-29 09:49:58.152471: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cublas64_10.dll
2023-11-29 09:49:58.200779: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cufft64_10.dll
2023-11-29 09:49:58.228869: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library curand64_10.dll
2023-11-29 09:49:58.447674: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cusolver64_10.dll
2023-11-29 09:49:58.677289: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cusparse64_10.dll
2023-11-29 09:49:58.928462: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudnn64_7.dll
>>>>>2023-11-29 09:49:58.928755:** I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0
2023-11-29 09:51:52.426382: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1257] Device interconnect StreamExecutor with strength 1 edge matrix:
>>>>>>2023-11-29 09:51:52.426450:** I tensorflow/core/common_runtime/gpu/gpu_device.cc:1263]      0
2023-11-29 09:51:52.426524: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1276] 0:   N
2023-11-29 09:51:52.426887: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1402] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 6591 MB memory) -> physical GPU (device: 0, name: GeForce RTX 3070, pci bus id: 0000:01:00.0, compute capability: 8.6)
2023-11-29 09:51:52.429524: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x2606633c450 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2023-11-29 09:51:52.429562: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): GeForce RTX 3070, Compute Capability 8.6

I think u should be upgrading to .NET 6 at least (preferably 8 as 2.0 is deprecated), also note the time in bold in the above, it took a while for it to get to var collection = new MagickImageCollection(video, MagickFormat.Mp4)

Seems like the SciSharp, Magick, and Microsoft.ML can also be upgraded to later versions

I also wanted to do a PR where I add a progress event to the video processing so one can know how its going, below is the code if u wanna incorporate.

        //Code Added
        public delegate void UpdateVideoProgressDelegate(int currentFrame, int totalFrames);
        public event UpdateVideoProgressDelegate UpdateVideoProgress;
        
        public NsfwSpyFramesResult ClassifyVideo(byte[] video, VideoOptions videoOptions = null)
        {
            if (videoOptions == null)
                videoOptions = new VideoOptions();

            if (videoOptions.ClassifyEveryNthFrame < 1)
                throw new Exception("VideoOptions.ClassifyEveryNthFrame must not be less than 1.");

            var results = new ConcurrentDictionary<int, NsfwSpyResult>();

            var watch = Stopwatch.StartNew();
            using (var collection = new MagickImageCollection(video, MagickFormat.Mp4))
            {
                Console.WriteLine($"Loading video took {watch.Elapsed} to load video into collection of images");
                collection.Coalesce();
                var frameCount = collection.Count;

                Parallel.For(0, frameCount, (i, state) =>
                {
                    if (i % videoOptions.ClassifyEveryNthFrame != 0)
                        return;

                    if (state.ShouldExitCurrentIteration)
                        return;

                    var frame = collection[i];
                    frame.Format = MagickFormat.Jpg;

                    var result = ClassifyImage(frame.ToByteArray());
                    results.GetOrAdd(i, result);

                    //Code Added
                    if (videoOptions.EmitProgress)
                    {
                        UpdateVideoProgress?.Invoke(i, frameCount);
                    }

                    // Stop classifying frames if Nsfw frame is found
                    if (result.IsNsfw && videoOptions.EarlyStopOnNsfw)
                        state.Break();
                });
            }

            var resultDictionary = results.OrderBy(r => r.Key).ToDictionary(r => r.Key, r => r.Value);
            var gifResult = new NsfwSpyFramesResult(resultDictionary);
            return gifResult;
        }

Also how long does it take to process a 10 min vid in general.

Predict Take a long time

Thanks for your great library but i use this library in web api project and it take a long time and at the end it cant predict anything. i run your test code and its not work.

Hentai when all labels are 0

Hey, there's a minor issue like this, when i try to classify some "tough for AI neutral images":

here's the image used for this call:

here's my piece of code using your NuGet:

public NsfwSpyResult Classify(IFormFile file)
    {
        var nsfwSpy = new NsfwSpy();
        using (var ms = new MemoryStream())
        {
            file.CopyTo(ms);
            var fileBytes = ms.ToArray();
            var result = nsfwSpy.ClassifyImage(fileBytes);
            
            return result;
        }
        
    }

Console application automatically crashes while classifying the image

Im running a discord bot on railway.app, The bot automatically disconnects and reconnects while classifying the image.

var nsfwImagedetector = new NsfwSpy(); var uri = new Uri(message.Attachments.First().Url); var result = nsfwImagedetector.ClassifyImage(uri); float resultFloat = result.Sexy + result.Hentai + result.Pornography; bool isImageNsfw = resultFloat > 0.992000 ? true : false; Console.WriteLine(resultFloat); Console.WriteLine(isImageNsfw); if (isImageNsfw) { var nsfwUpdate = Database.createUpdateSet("nsfw", true); await cardCollection.UpdateOneAsync(filter, nsfwUpdate); }

These are some errors i found while classifying image. although it gives result regardless of errors. Only issue is crashing the console application
`2023-01-07 13:07:39.898091: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcudart.so.10.1'; dlerror: libcudart.so.10.1: cannot open shared object file: No such file or directory
2023-01-07 13:07:39.921942: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
2023-01-07 13:07:40.576038: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN)to use the following CPU instructions in performance-critical operations: AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-01-07 13:07:40.728653: I tensorflow/core/platform/profile_utils/cpu_utils.cc:104] CPU Frequency: 2200000000 Hz

2023-01-07 13:07:40.733302: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7fbea0005c30 initialized for platform Host (this does not guarantee that XLA will be used). Devices:

2023-01-07 13:07:40.733356: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version

2023-01-07 13:07:40.851229: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory

2023-01-07 13:07:40.851288: W tensorflow/stream_executor/cuda/cuda_driver.cc:312] failed call to cuInit: UNKNOWN ERROR (303)

2023-01-07 13:07:40.851336: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (railway): /proc/driver/nvidia/version does not exist

what type of this project model?

I want to transfer it to tflite model , use it in android project,thanks

Have any plan about implement python version support?

NsfwSpy.App Incompatibility with Node.js v18.6.0+ / v16.17.0+

When attempting to build the NsfwSpy web app when using v18.6.0+ / v16.17.0+ of Node.js, it will throw out a "ERR_LOADER_CHAIN_INCOMPLETE" error and get stuck trying to run 'npm install' on loop. Node.js v16.16.0 works perfectly fine due to "ERR_LOADER_CHAIN_INCOMPLETE" not existing as an error in previous versions.

Incorrect nsfw detection

Apparently it detects YouTube links as nsfw? And pretty much any imgur or tenor link

Net Framework 4.6.2

Good thing, you could create compatibility with net framework 4.6.2

DI Scope in Web APIs

Hi there, I would like to host NsfwSpy inside a simple WebAPI controller to determine whether or not incoming images are NSFW. My question - if I register NsfwSpy in the DI system as Scoped, will new instances also consume resources? I.e. if one instance consumes ~600mb RAM, will 10 simultaneous incoming classification requests require ~6gb of RAM? Alternatively, can I safely register NsfwSpy as a DI Singleton without encountering threading issues or similar? What do you recommend as a best practice for this situation? Note - I am happy processing all incoming requests sequentially, my workload does not require parallel classification at this stage.

And thank you very much for providing this package!

Image classification fails on M1 chip Macs

Thanks for a great work. I just tried to add this package to my test C# API and failed to run it on MacOS Monterey.

After successfully adding to our project, when the code hit NSFW Spy, it exits immediately without any error. Any tips would be appreciated.

Installed SciSharp.TensorFlow.Redist
Injected dependency
Environment:
.NET SDK
Version: 6.0.105
Commit: 1c35735293

Runtime Environment:
OS Name: Mac OS X
OS Version: 12.4
OS Platform: Darwin
RID: osx.12-x64
Base Path: /usr/local/share/dotnet/x64/sdk/6.0.105/

Host (useful for support):
Version: 6.0.5
Commit: 70ae3df4a6

.NET SDKs installed:
3.1.419 [/usr/local/share/dotnet/x64/sdk]
5.0.408 [/usr/local/share/dotnet/x64/sdk]
6.0.105 [/usr/local/share/dotnet/x64/sdk]

Dataset

Thank you for this repo. Is there any possibility to get access to the dataset used for training the network?

new NsfwSpy() cause program to abort with no error message or exception

Trying to write a small quick command line program to scan a disk for NSFW images and report what it finds, all very simple stuff using .NET 6.

"Directory.GetFiles" to get a list of file names to check, then new up an NsfwSpy object, then a foreach to classify each found image in the list.

Problem is, when the program gets to the "var nsfwSpy = new NsfwSpy();" line, it pauses for a few seconds then just exits.

I've single stepped in as far as I can and it seems to be in the file NsfwSpy.cs at around line 28 where it creates a new MLContext with the model zip file.

The model is present, all the required DLL's are present, no errors, no exceptions or anything are given, just a silent exit and the program stops executing at that point.

Win10 64 bit, developing using JetBrains rider with .NET V6 and NsfwSpy added using NuGet (Version 3.4.3)

Any advice?

It is working great but classification of sexy images is poor. How can I improve?

I would like it to classify sexy / explicit images better

Are there any newer models?

Currently used model version is : 1.7.1+4cf622eb13c07947a38e6e3336221657007e635d

Here some incorrectly classified images. these are supposed to be sexy but they are not classified such and many more

for example this site much better at classification with best model : https://nsfwjs.com/

https://github.com/NsfwSpy/NsfwSpy.NET/assets/19240467/b410003d-1cbb-482f-ba00-744740aed93d

https://github.com/NsfwSpy/NsfwSpy.NET/assets/19240467/21653348-5ff8-4260-8f04-488ec19bc196

Training

Hi,

Thanks for sharing this, it's pretty useful.

Could you advise how I can add additional classification?
I would like to add and classify gambling related images.

Also I've tried to run the Train project but getting an error on following line;

var trainedModel = trainingPipeline.Fit(trainSet);

And the error message is;

System.ArgumentOutOfRangeException: 'Only one class found in the LabelAsKey column. To build a multiclass classification model, the number of classes needs to be 2 or greater Arg_ParamName_Name'

Many thanks

Classify multiple images , best way to remove lamda?

I've used NsfwSpy successfully on a few test files , I'm now trying to set it up where the classification is written to a Dictionary.

Based on the results I want to run a function that moves the file into a corresponding directory based on the classification.

The issue I'm having currently is because I don't want to mess with the nuget package, all the images in a directory and classified and then the Dictionary is ready and files are moved. But if there is an error during classification the program crashes and the sorting function isn't used.

Question is , what is the best way to break up the lamda function so that I can sort files based on the classification? It will make it quiet a bit faster too so I can sort each file as it's classified instead of writing the entire list , then reading the entire list to sort.

Looking up the MIME of a file fails on Linux

On my system, I have nVidia Drivers installed, but no nVidia GPU at the moment. It crashes when it finds the driver but not apparently some magic files.

2022-05-27 22:50:07.900705: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcudart.so.10.1'; dlerror: libcudart.so.10.1: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /nix/store/gka59hya7l7qp26s0rydcgq8hj0d7v7k-gcc-11.3.0-lib/lib:/nix/store/p9d5i8nhx5g77h6mjfs8rqfl2cq33jrr-libsecret-0.20.5/lib:/nix/store/9757vpby0rqzlbm2p02dm812h28djrg5-e2fsprogs-1.46.5/lib:/nix/store/l55cmvhfrqyxhdvgcfxgq0jpizcdp7ld-libnotify-0.7.12/lib:/nix/store/m7d2jrrwprb4n48ddccsyg2ffqr6yj0m-pipewire-0.3.51-jack/lib
2022-05-27 22:50:07.900735: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
2022-05-27 22:50:08.072930: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN)to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2022-05-27 22:50:08.091265: I tensorflow/core/platform/profile_utils/cpu_utils.cc:104] CPU Frequency: 3593110000 Hz
2022-05-27 22:50:08.091881: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f92340093e0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2022-05-27 22:50:08.091913: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2022-05-27 22:50:08.093389: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /nix/store/gka59hya7l7qp26s0rydcgq8hj0d7v7k-gcc-11.3.0-lib/lib:/nix/store/p9d5i8nhx5g77h6mjfs8rqfl2cq33jrr-libsecret-0.20.5/lib:/nix/store/9757vpby0rqzlbm2p02dm812h28djrg5-e2fsprogs-1.46.5/lib:/nix/store/l55cmvhfrqyxhdvgcfxgq0jpizcdp7ld-libnotify-0.7.12/lib:/nix/store/m7d2jrrwprb4n48ddccsyg2ffqr6yj0m-pipewire-0.3.51-jack/lib
2022-05-27 22:50:08.093408: W tensorflow/stream_executor/cuda/cuda_driver.cc:312] failed call to cuInit: UNKNOWN ERROR (303)
2022-05-27 22:50:08.093431: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (uGamingPC): /proc/driver/nvidia/version does not exist
Unhandled exception. HeyRed.Mime.MagicException: Could not find any valid magic files!
   at HeyRed.Mime.Magic..ctor(MagicOpenFlags flags, String dbPath)
   at HeyRed.Mime.MimeGuesser.GuessMimeType(Byte[] buffer)
   at HeyRed.Mime.MimeGuesser.GuessFileType(Byte[] buffer)
   at NsfwSpyNS.NsfwSpy.ClassifyImage(Byte[] imageData)
   at NsfwSpyNS.NsfwSpy.ClassifyImage(Uri uri, WebClient webClient)
   at IsItYiff.Program.Main(String[] args) in /home/krutonium/RiderProjects/IsItYiff/IsItYiff/Program.cs:line 11