Comments (4)
Hello Jssmbee,
- The goal of statistical testing is to have some safe guard against spurious results, but guaranteeing anything here is an open research question. That means, yes, the stat testing isn't perfect, but it's at least we can do. The cause of this uncertainty stems from the fact that we don't yet to have full understanding about high dimensional representations, although a big part of community is working on making progress. I do think we must do this testing to at least guard ourselves against spurious results (to some extend). I'm always open to new, less expensive ways to test this - let us know if you have ideas.
Computational issues: The most expensive part of the computation from my experience is getting activations of concept and random pictures, which is completely parallizable. Once you get the activations, each random runs are also completely paralleliz-able - each random run only regards concept pictures and a set of random pictures. Each random run also only requires getting derivative and training a linear classifier, both of which are not computationally expensive (relatively speaking, we are not inverting any giant matrices..).
- I'm not sure what you mean. we never compute a CAV with same sets of images. To compute random CAVs (that will be used for stat testing), you need two sets (positive set, negative set). Those are: (random500_0, random500_1), (random500_0, random500_2), ...(random500_0, random500_20). These aren't duplicates of each other, since each sets of random folders are different. Please let me know if you are asking different question.
from tcav.
Hi @BeenKim ,
Thanks for the response.
-
I agree on what you said. We have also been looking into it and believe that the gradients for a single set, e.g. random500_0 for a single bottleneck and a single target has the same gradient across tests, so instead of computing gradients every time for random0 against all random sets, we can only compute it once and cache it for rest of the tests. Let me know if i am missing anything here.
-
Apologies for the confusion. Well what I meant was that the TCAV score for random500_1 vs random500_0 should be the same as 1 - (random500_0 vs random500_1). So don't really understand why we need to do both the computations instead of just doing one? as mentioned here : #24
from tcav.
-
I think you got that right! The current code isn't designed to do that (rather, we just parallelize for each (random 0, random i) tuple). In my experience, getting gradient is pretty cheap, but getting activations of target images and concept images is more expensive (probably some function of TF). So I recommend for your activation_generation function to first run over all images in parallel (both target and concept images) to the network, then cache the activations before running TCAV. I find that most effective in accelerating. Another tip you may already know: don't use pickle since they are super expensive to load. Use np array to save (np.save) - much cheaper.
-
It's true that TCAV score for (random500_1, random500_0) is same as 1 - (random500_0, random500_1). However, we don't calculate that twice. To calculate random CAV TCAV scores, we do this by fixing positive set (random500_1) and changing negative sets. So: (random500_1, random500_0), (random500_1, random500_2), (random500_1, random500_3), (random500_1, random500_4)....(random500_1, random500_100). None of these are the same CAVs because negative sets are different. I might be still missing something - please let me know!
from tcav.
Thank you for the clarification and Tips @BeenKim !
from tcav.
Related Issues (20)
- Not able to run all the bottlenecks together for any given model HOT 2
- requirement.txt file has a mistake HOT 3
- How to use TCAV with custom keras models(h5) HOT 2
- why the 'get_direction_dir_sign' function returns 'dot_prod<0' instead 'dot_prod>0' ?? HOT 1
- Setup for Run_TCAV.ipynb fails HOT 2
- TCAV with TensorFlow Object Detection API models HOT 4
- TCAV on tabular data HOT 3
- Random Images for TCAV for Diabetic Retinopathy application HOT 9
- Concept Images for TCAV for DR application
- Problem with Relative TCAV plots
- Question regarding discrete model prediction layer activation function and model loss function
- InvalidArgumentError: Requested return tensor 'softmax2_pre_activation:0' not found in graph def HOT 1
- result not reproduced by taking gradient wrt logit and flipping sign HOT 1
- Extend TCAV for Object detection
- TCAV for multiple instance learning
- Clarifying experiments in Section 4.1.2 Empirical Deep Dream
- InvalidArgumentError: Node 'v5/mul_1' is not unique
- Reproduce Fig.4. Zebra TCAV in googlenet from paper
- Downloading the tcav data using the download_and_make_datasets
- Columns and DataType Not Explicitly Set on line 69 of imagenet_and_broden_fetcher.py
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from tcav.