Comments (3)
aquibjaved
commented on July 20, 2024
1
I was also facing the same issue, so, once you've loaded the model. put the model to model.eval() before passing it to classify func and the result will be deterministic :)
from ditto.
psungu
commented on July 20, 2024
1
Even open the eval mode and fixed random seed, it gives different inferences on single and batch prediction for the same textual pair.
Example:
COL title VAL "Nike Metcon 2 - Black/White/Wolf Grey/Volt"@en " Nike Mens Shoes Regular Training Grey/Volt "@en COL title VAL "Nike Metcon DSX Flyknit - Wolf Grey/Volt/Wolf Grey/Black"@en-GB " Nike Mens Shoes Regular Training Grey/Black "@en-GB 0
COL title VAL "Nike Zoom Structure 19 Comp Pack Men's Running Shoes SP16 007"@en 007 · Nike Moderate Support Trai ah"@en COL title VAL "KANMEI GS"@en-es GS | Unisex Kids Running Shoes ASICS"@en-es 0
COL title VAL "Air Jordan 5 Low “Dunk From Above” White/Metallic Gold Star-Midnight Navy For Sale"@en-US Sale | New Jordans 2016"@en-US COL title VAL "Air Jordan 5 Low “Dunk From Above” White-Gold/Midnight Navy 2016 Sale"@en-US Sale | Cheap Jordans 2017"@en-US 1
COL title VAL "Nike Womens Air Zoom Pegasus 33 - Black/White-Anthracite-Cool Grey"@en-GB " Nike Grey Shoes 831356-001 "@en-GB COL title VAL "NIKE AIR ZOOM PEGASUS 33"@en 33 - MAN RUNNING SHOES Nike colour Black 831352 001 Athletic footwear apparel and sports equipment"@en 0
COL title VAL "Air Jordan 14 Retro Low “Laney” Varsity Royal/Varsity Maize-Black-White For Sale"@en-US Sale | Cheap Jordans 2017"@en-US COL title VAL "Cheap Air Jordan 4 Retro “Motorsports” White/Varsity Blue-Black Sale"@en-US Sale | Cheap Jordans 2017"@en-US 0
COL title VAL "Nike AIR Zoom Pegasus 32 Men's Training Shoes - Total Orange/Ghost Green/Black"@en "Running and Training"@en COL title VAL "Nike AIR Zoom Pegasus 32 Men's Training Shoes - Green/Blue Lagoon/Black"@en "Clearance Footwear"@en 0
Probabilities: tensor([0.9999, 0.0167, 0.9953, 0.9995, 0.2451, 0.9998])
Now, predict only first 2 pairs:
COL title VAL "Nike Metcon 2 - Black/White/Wolf Grey/Volt"@en " Nike Mens Shoes Regular Training Grey/Volt "@en COL title VAL "Nike Metcon DSX Flyknit - Wolf Grey/Volt/Wolf Grey/Black"@en-GB " Nike Mens Shoes Regular Training Grey/Black "@en-GB 0
COL title VAL "Nike Zoom Structure 19 Comp Pack Men's Running Shoes SP16 007"@en 007 · Nike Moderate Support Trai ah"@en COL title VAL "KANMEI GS"@en-es GS | Unisex Kids Running Shoes ASICS"@en-es 0
probabilities: tensor([0.9999, 0.0159])
Now, predict only the second pair:
COL title VAL "Nike Zoom Structure 19 Comp Pack Men's Running Shoes SP16 007"@en 007 · Nike Moderate Support Trai ah"@en COL title VAL "KANMEI GS"@en-es GS | Unisex Kids Running Shoes ASICS"@en-es 0
probability: tensor([0.0114])
I don't know how to fix this, and what is the problem here?
from ditto.
tideng777
commented on July 20, 2024
and I check the tensor after embedding which are all same. Something happen in evaluation.
from ditto.
Related Issues (20)
- Code implementation - Training HOT 9
- ImportError: cannot import name 'LongformerModel' from 'transformers' (transformers=2.8)
- How is your F1 score calculated, whether you use weight or macro or micro HOT 1
- Add --save_model flag to the training example
- Error when using --summarize with matcher.py HOT 1
- Whether are special tokens like [COL] [VAL] and attribute names added into the vocabulary?
- [Question] Can I use this package in a notebook environment?
- Inferencing HOT 1
- How (code) to serialize the inputs ? HOT 1
- ModuleNotFoundError: No module named 'click._bashcomplete' HOT 1
- ValueError: not enough values to unpack (expected 2, got 1) - Textual/Company HOT 5
- drop_col gives error? HOT 5
- Which f1 should we report? HOT 1
- evaluation method seems to assign a new f1 value as a best score without computing f1 value by best_th=0.5 HOT 1
- Adding custom tokens
- F1 Score for Structured/Beer on paper can't be reproduced
- The link for the Company.zip file seems to be invalid. HOT 1
- Summarization sometimes removes attribute names between [COL] and [VAL]
- training HOT 7
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from ditto.