This demo application does image classification from camera output and text classification in the same github repo.
To get device camera output it uses Android CameraX API.
All the logic that works with CameraX is separated to org.pytorch.demo.vision.AbstractCameraXActivity
class.
void setupCameraX() {
final PreviewConfig previewConfig = new PreviewConfig.Builder().build();
final Preview preview = new Preview(previewConfig);
preview.setOnPreviewOutputUpdateListener(output -> mTextureView.setSurfaceTexture(output.getSurfaceTexture()));
final ImageAnalysisConfig imageAnalysisConfig =
new ImageAnalysisConfig.Builder()
.setTargetResolution(new Size(224, 224))
.setCallbackHandler(mBackgroundHandler)
.setImageReaderMode(ImageAnalysis.ImageReaderMode.ACQUIRE_LATEST_IMAGE)
.build();
final ImageAnalysis imageAnalysis = new ImageAnalysis(imageAnalysisConfig);
imageAnalysis.setAnalyzer(
(image, rotationDegrees) -> {
analyzeImage(image, rotationDegrees);
});
CameraX.bindToLifecycle(this, preview, imageAnalysis);
}
void analyzeImage(android.media.Image, int rotationDegrees)
Where the analyzeImage
method process the camera output, android.media.Image
.
It uses the aforementioned TensorImageUtils.imageYUV420CenterCropToFloat32Tensor
method to convert android.media.Image
in YUV420
format to input tensor.
After getting predicted scores from the model it finds top K classes with the highest scores and shows on the UI.
Another example is natural language processing, based on an LSTM model, trained on a reddit comments dataset.
The logic happens in TextClassificattionActivity
.
Result class names are packaged inside the TorchScript model and initialized just after initial module initialization.
The module has a get_classes
method that returns List[str]
, which can be called using method Module.runMethod(methodName)
:
mModule = Module.load(moduleFileAbsoluteFilePath);
IValue getClassesOutput = mModule.runMethod("get_classes");
The returned IValue
can be converted to java array of IValue
using IValue.toList()
and processed to an array of strings using IValue.toStr()
:
IValue[] classesListIValue = getClassesOutput.toList();
String[] moduleClasses = new String[classesListIValue.length];
int i = 0;
for (IValue iv : classesListIValue) {
moduleClasses[i++] = iv.toStr();
}
Entered text is converted to java array of bytes with UTF-8
encoding. Tensor.fromBlobUnsigned
creates tensor of dtype=uint8
from that array of bytes.
byte[] bytes = text.getBytes(Charset.forName("UTF-8"));
final long[] shape = new long[]{1, bytes.length};
final Tensor inputTensor = Tensor.fromBlobUnsigned(bytes, shape);
Running inference of the model is similar to previous examples:
Tensor outputTensor = mModule.forward(IValue.from(inputTensor)).toTensor()
After that, the code processes the output, finding classes with the highest scores.