Git Product home page Git Product logo

dropshots's Introduction

📱 Dropshots

Dropshots is a library and Gradle plugin that makes on-device screenshot testing on Android easy.

Other screenshot testing libraries take screenshots in your instrumentation tests on device, then download those images to your host machine to compare them to reference images, failing at that step. This means that your screenshot assertions aren't run as part of the rest of your test suite, and can't easily be run from within your IDE along with the rest of your tests.

Dropshots makes this process easier by performing your screenshot assertions right in your test, alongside all of your other tests. It's Gradle plugin ensures that your version controlled reference images are available on your test device so that your test screenshots are able to be compared to those reference images right within your test.

Installation

Apply the plugin in your module's build.gradle file.

Using the plugins DSL:

// build.gradle(.kts)
plugins {
  id("com.android.application")
  // or id("com.android.library")
  id("com.dropbox.dropshots") version "0.4.2"
}

Note that the plugin is currently published to Maven Central, so you need to add it to the repositories list in settings.gradle.

// settings.gradle(.kts)
pluginsManagement {
  repositories {
    mavenCentral()
    gradlePluginPortal()
  }
}
Using legacy plugin application
buildscript {
  repositories {
    mavenCentral()
  }
  dependencies {
    classpath "com.dropbox.dropshots:dropshots-gradle-plugin:0.4.2"
  }
}

apply plugin: "com.android.application"
// or apply plugin: "com.android.library"
apply plugin: "com.dropbox.dropshots"

Usage

Once the Dropshots plugin is added to your project, some new tasks will be created to create, validate and manager your screenshot reference images. While you can use the tasks directly, they are also automatically injected into your project's task graph to run as part of your normal testing workflow.

Write tests

Dropshots screenshot tests are simply standard Android Instrumentation tests which use the runtime library to compare screenshots with reference images. Simply add the Dropshots rule to an instrumentation test, setup the view you'd like to test, and use the Dropshots.assertSnapshot functions to validate them.

class MyTest {
    @get:Rule
    val activityScenarioRule = ActivityScenarioRule(TestActivity::class.java)

    @get:Rule
    val dropshots = Dropshots()

    @Before
    fun setup() {
        // Setup your activity however you like
        activityScenarioRule.scenario.onActivity {
            it.supportFragmentManager.beginTransaction()
                .add(android.R.id.content, ScreenshotTestFragment())
                .commitNow()
        }
    }

    @Test
    fun testMatchesFullScreenshot() {
        activityScenarioRule.scenario.onActivity {
            // Assert full-screen snapshots
            dropshots.assertSnapshot("MatchesFullScreenshot")
        }
    }

    @Test
    fun testMatchesActivityScreenshot() {
        activityScenarioRule.scenario.onActivity {
            // Assert activity snapshots
            dropshots.assertSnapshot(it, "MatchesActivityScreenshot")
        }
    }

    @Test
    fun testMatchesViewScreenshot() {
        activityScenarioRule.scenario.onActivity {
            // or assert view snapshots.
            dropshots.assertSnapshot(
                it.findViewById<View>(android.R.id.content),
                name = "MatchesViewScreenshot",
                path = "views/fullscreen" // optional parameter to set path of stored screenshots
            )
        }
    }
}

With this test in place, any time the connectedAndroidTest task is run the screenshot of the Activity or View will be validated against the reference images stored in the repository. If any screenshots fail to match the reference images (within configurable thresholds), then an image will be written to the test report folder that shows the reference image, the actual image, and the diff of the two. By default, the test report folder is ${project.buildDir}/outputs/androidTest-results/connected.

The first time you create a screenshot test, however, there won't be any reference images, so you'll have to create them...

Updating reference images

Updating reference screenshots is as simple as running the tests with a dropshots.record property added to Gradle. This makes it easy to update screenshots in a single step, without requiring you to interact with the emulator or use esoteric adb commands.

Important: Ensure that you record screenshots on an emulator that's been configured in the same way as the emulators on which you'll validate the screenshots.

./gradlew :path:to:module:connectedAndroidTest -Pdropshots.record

After running this command, you'll see that all reference screenshots for the module will have been updated in the src/androidTest/screenshots directory. After that, running connected tests, either from the gradlew CLI or directly from the IDE, will validate the screenshots against the new reference images.

Custom Validation

By default Dropshots will fail assertions if the supplied ImageComparator returns any pixels that don't match the reference image. If that is too strict for your use case, then you can supply a custom ResultValidator to specify how comparison results should be validated.

The included CountValidator validates comparison results which contain no more than the specified number of pixel differences. The included ThresholdValidator validates comparison results which contain no more then the specific percentage of pixel differences, based on the entire image size.

License

Copyright (c) 2022 Dropbox, Inc.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

dropshots's People

Contributors

dependabot[bot] avatar devpalacio avatar jayshortway avatar josealcerreca avatar rharter avatar saket avatar sebastienrouif avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

dropshots's Issues

Gradle record screenshot task failing - but works

When I run the command ./gradlew :app:connectedAndroidTest -Pdropshots.record I get the following error:

* What went wrong:
A problem was found with the configuration of task ':app:update<flavor>DebugAndroidTestScreenshots' (type 'PullScreenshotsTask').
  - Gradle detected a problem with the following location: '.../src/androidTest/screenshots'.
    
    Reason: Task ':app:mergeBaseDebugAndroidTestAssets' uses this output of task ':app:update<flavor>DebugAndroidTestScreenshots' without declaring an explicit or implicit dependency. This can lead to incorrect results being produced, depending on what order the tasks are executed.
    
    Possible solutions:
      1. Declare task ':app:update<flavor>DebugAndroidTestScreenshots' as an input of ':app:mergeBaseDebugAndroidTestAssets'.
      2. Declare an explicit dependency on ':app:update<flavor>DebugAndroidTestScreenshots' from ':app:mergeBaseDebugAndroidTestAssets' using Task#dependsOn.
      3. Declare an explicit dependency on ':app:update<flavor>DebugAndroidTestScreenshots' from ':app:mergeBaseDebugAndroidTestAssets' using Task#mustRunAfter.
    

How ever, the screenshots are done as expected. Still have to manually delete the screenshots folder in order to have them generated while this error is happening.

[Question] Any plans for snapshot diff?

Hey,
I want to ask, if you have any plans for showing diff of screenshots if diff is there. It would be nice to create for example new folder /diff and store diffs there.

I am not sure if this is possible by your approach.

Storage permission fix for API 29

Hi,

Thanks for creating this useful tool! I really appreciate the fact that assertSnapshot() accepts a direct Bitmap as well, as that's more fitting for my use case than a View or Activity.

Issue
I had some issues getting this to work with my Android 10 (API 29) device. The runtime library is unable to write to the Downloads folder.

To my understanding, Android 10 is a bit of a special case in terms of scoped storage. It's the first version with scoped storage, but it doesn't have the direct file paths feature yet, introduced in Android 11, which re-enables the use of the Java File API.

Fix
So on Android 10, if an app wants to write to a media folder such as Downloads, it has 2 options:

Accepting PRs?
I have the second option implemented in a local checkout of Dropshots and it's working well. Would you like me to create a pull request? I'd be happy to!

Task '.record' not found in root project

When I execute the command: ./gradlew app:connectedAndroidTest -Pdropshots.record

The following error appears:

FAILURE: Build failed with an exception.

  • What went wrong:
    Task '.record' not found in root project 'and_label'.

any ideas?

Text signs are different at same emulators on different machines

Letters and digits are different at the tests on local machine and on CI machine. Machines has a different proc architecture. Local machine is M1 arch. CI machine is Intel arch.
Zoomed diff image shows red pixels around the symbols.
Is symbols generating different depends on proc architecture?
How could i have the same screenshots at local and CI machines?

I can use tolerance for screenshots, but different screens contains a different amount of symbols. More symbols needs a large tolerance of screenshot. So I can't select the unique tolerance for every screen.
Снимок экрана 2024-07-04 в 12 59 03

pullDefaultDebugAndroidTestScreenshots FAILED

I made a basic test

@RunWith(Parameterized::class)
class NotificationActivitySmokeTests(val activityUnderTest: Class<out Activity>) {
    @get:Rule
    val sandboxRule = LoggedInFakeUserTestRule(activityUnderTest, true)

    @get:Rule
    val dropshots = com.dropbox.dropshots.Dropshots()

    companion object {
        @JvmStatic
        @Parameterized.Parameters(name = "Notification Payload Batch: {index}")
        fun activitesToSmokeTest(): List<Class<*>> =
            listOf(
                EmailNotificationsSettingsActivity::class.java,
                PushNotificationsSettingsActivity::class.java,
                SmsNotificationsSettingsActivity::class.java,
            )
    }

    @Test
    fun simpleTest() {
        sandboxRule.getScenario().onActivity {
            dropshots.assertSnapshot(it, "MatchesActivityScreenshot")
//            assertThat(it).isInstanceOf(activityUnderTest)
        }
    }
}

I then ran ./gradlew myModule:CAT

I then got error

> Task :subsystem:tfa:notifications:sandboxes:app:pullDefaultDebugAndroidTestScreenshots FAILED
adb: error: failed to stat remote object '/storage/emulated/0/screenshots/com.twitter.sandbox.notifications.debug/.': No such file or directory

> Task :subsystem:tfa:notifications:sandboxes:app:clearDefaultDebugAndroidTestScreenshots FAILED
rm: /storage/emulated/0/screenshots/com.twitter.sandbox.notifications.debug: No such file or directory

I think its unexpected that tasks get added to cAT without some flag. Maybe have an explicit flag or don't fail if directory for deletion doesn't exist

Can't sync gradle on a fresh project with the plugin added (KotlinJvmAndroidCompilation with name 'debug' not found.)

👋

I wanted to try out this library, but as soon as I add plugin:

plugins {
    id("com.android.application")
    id 'org.jetbrains.kotlin.android'
    id("com.dropbox.dropshots") version "0.2.0"
}

(plus mavenCentral() in the repos, I already had it) the project can't sync gradle with an error:

A problem occurred configuring project ':app'.
> Failed to notify project evaluation listener.
   > org/gradle/configurationcache/extensions/CharSequenceExtensionsKt
   > KotlinJvmAndroidCompilation with name 'debug' not found.

Zrzut ekranu 2022-06-30 o 12 29 21

Tried and can't make it run in three projects:

  • my main work project
  • fresh sample project
  • fresh sample project at my colleague's

with the same result

Am I missing something? 🤔

Environment:
a) macbook with m1 pro chip, Android Studio Chipmunk | 2021.2.1 Patch 1
b) my colleague's macbook with intel

Thanks for new lightweight library, sounds like something that I'd need 👏

OutOfMemoryError when generating images on API 23

Emulator details:
Device: Pixel 7
System image: Marshmallow 23
RAM: 2536mb
VM heap: 1228mb

Stacktraces:

java.lang.OutOfMemoryError: Failed to allocate a 20736012 byte allocation with 4194304 free bytes and 16MB until OOM
at dalvik.system.VMRuntime.newNonMovableArray(Native Method)
at android.graphics.Bitmap.nativeCreate(Native Method)
at android.graphics.Bitmap.createBitmap(Bitmap.java:831)
at android.graphics.Bitmap.createBitmap(Bitmap.java:808)
at android.graphics.Bitmap.createBitmap(Bitmap.java:775)
at com.dropbox.dropshots.Dropshots.generateDiffImage(Dropshots.kt:206)
at com.dropbox.dropshots.Dropshots.writeThen(Dropshots.kt:170)
at com.dropbox.dropshots.Dropshots.assertSnapshot(Dropshots.kt:119)
at com.dropbox.dropshots.Dropshots.assertSnapshot(Dropshots.kt:88)
java.lang.OutOfMemoryError: Failed to allocate a 10368012 byte allocation with 4194304 free bytes and 8MB until OOM
at com.dropbox.differ.Mask.<init>(Mask.kt:6)
at com.dropbox.dropshots.Dropshots.assertSnapshot(Dropshots.kt:129)
at com.dropbox.dropshots.Dropshots.assertSnapshot(Dropshots.kt:88)

Failed test does not store output - or wrong error message

java.lang.AssertionError: "MatchesActivityScreenshot" failed to match reference image. 18 pixels differ (8.6805556E-4 %)
Output written to: /storage/emulated/0/Download/screenshots/com.google.samples.apps.nowinandroid.feature.topic.test/MatchesActivityScreenshot.png

However the screenshots directory is empty, This might be a feature and the only bug is in the error message.

Using emulator API 30 and 28.

depend on android.library?

My gradle fu is not great but was wondering if it is possible to not have to apply android.library/android.app plugin before dropshots. Maybe there is an api that can check if it is on path rather than requiring ordering?

Screenshots are pulled from the wrong directory

adb: error: failed to stat remote object '/storage/emulated/0/screenshots/com.google.samples.apps.nowinandroid.feature.topic/.': No such file or directory

Looking at the device file explorer, the screenshots are there but in a different directory:

/storage/emulated/0/screenshots/com.google.samples.apps.nowinandroid.feature.topic.test/MatchesActivityScreenshot.png

Missing Custom Validation

Hey,

there's a section in README about custom validation with objects like ResultValidator, CountValidator and ThresholdValidator. But it seems that the Dropshots API is missing those elements in my project (ver 0.3.0), could you please add a sample or small snippet on how to use that feature?

New release for file path?

Hi, i was checking the issues and PRs and noticed that on May it was implemented the feature to specify the directory for the screenshots. However there was no release so far that includes such feature.

Issue: #24
PR: #35

The readme already includes the example with path parameter.

When is the release planned this?

[Request] Add tolerance

Hey,
I didn't find any option to add tolerance to snapshot. Do you think that it would be possible to add?

We have some chart changing in time and we don't care if it is pixel perfect - for example here

java.lang.AssertionError: "ICanSeeGraph" failed to match reference image. 135 pixels differ (0.00366482 %)

It would be nice to have option for that.

Unable to create screenshot storage directory.

I'm still investigating this, but I bumped Dropshot from 0.1.1 to 0.2.0 and everything worked great. I was able to record and run.

Then I removed the WRITE_EXTERNAL_STORAGE permission from the module's manifest as it's no longer required and I started getting the error.

I added the permission again and I still see the same error. I tried cleaning, etc.

min api is 23

Me again :-) any reason we need api23, I was unable to use in a minapi 21 app.


> Task :subsystem:tfa:notifications:sandboxes:app:processDefaultDebugAndroidTestManifest FAILED
[androidx.test.espresso:espresso-core:3.2.0] /Users/mnakhimovich/.gradle/caches/transforms-3/baa638c28051cb16a7630d000c6f64db/transformed/espresso-core-3.2.0/AndroidManifest.xml Warning:
        Package name 'androidx.test.espresso' used in: androidx.test.espresso:espresso-core:3.2.0, androidx.test.espresso:espresso-idling-resource:3.2.0.
/Users/mnakhimovich/workspace/twitter-android/build/projects/subsystem/tfa/notifications/sandboxes/app/intermediates/tmp/manifest/androidTest/default/debug/tempFile1ProcessTestManifest5391222334560463959.xml:5:5-74 Error:
        uses-sdk:minSdkVersion 21 cannot be smaller than version 23 declared in library [com.dropbox.dropshots:dropshots:0.1.1] /Users/mnakhimovich/.gradle/caches/transforms-3/469c2a42c974d58408744a407cefc928/transformed/dropshots-0.1.1/AndroidManifest.xml as the library might be using APIs not available in 21
        Suggestion: use a compatible library with a minSdk of at most 21,
                or increase this project's minSdk version to at least 23,
                or use tools:overrideLibrary="com.dropbox.dropshots" to force usage (may lead to runtime failures)

Captured screenshots do not show elevation of elements

We are running into an issue running tests that do not seem to take element elevation into account when the shot is taken. We can visually see the elevation (example on a Compose Card element) on the running emulator, but not in the resulting screenshot.

We were wondering if anyone has run into this same issue and knows of a work around? Alternately, is there a possible issue with the emulator we setup (needs hardware acceleration or something)?

I would share screenshots of the screenshots, but it is an internal only build and I can't share at this time.

Allow to save screenshots in folders

On top of the file name it would be great to be able to store the screenshots into folders.
I currently end up with hundreds of view screenshots and some names get truncated because they are too long. Having folders would allow more nested structure and would avoid prefix in file names
Happy to contribute if you agree with the feature

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.