Comments (12)
Is this for re-running problems that are related to specific test execution orders?
I bet that it is very difficult to get the same specific execution orders as it is the responsibility of operating systems scheduler to give CPU time to different processes.. So I'm against this feature, if you can't do it in a way that really guarantees same timings - and I think it is impossible to do it.
from pabot.
Yes, this aims at re-running using the same execution order. In my case, I'm going to use 4 processes and only the execution order of each of this process matters.
But I agree with your point of view, 'deterministic' is too promising, I plan to specify the execution order of each process and not the overall execution order with the same timings. Does it make sense for you ?
from pabot.
@lionelperrin if the deterministic option if for a given process, shouldn't it be a Robot Framework option rather than a pabot one?
from pabot.
@laurentbristiel Robotframework now has the ability to re-run in the same execution order, even in random mode (https://code.google.com/p/robotframework/issues/detail?id=1673)
For pabot, the difficulty comes from the test distribution among the subprocesses. This distribution is computed on the fly by the multiprocessing python library. In my case, I would like to compute the distribution of the tests in a first step, and distribute the test on the subprocesses in a deterministic manner.
from pabot.
Current logic is based on a worker pool - the thread that is first ready will get to execute the next suite.
Actually it will create a new process for each suite, but the number of concurrent processes is limited.
So currently the only thing that controls the execution order is the timings of executed suites - and in most cases they are changing a bit.
I have to say that for a parallel tool like pabot, the tool must expect that tests (or at least test suites) that are executed are independent of each other OR the dependencies between tests must be handled in the test cases. So there shouldn't be any reason for this kind of functionality - and I also think it would be very hard to implement it.
from pabot.
If the tests are failing because of an interference between them (two test cases somehow affect each other), re-running that problem with the same timings would be very convenient. I would be interested in first getting an idea how your going to implement this?
I think the solution would have to be somehow based on the recordings (output.xmls) to get the timings correct. Maybe a new tool altogether - not pabot?
from pabot.
I think you're right about the complexity of implementing something based on the recordings. I have the feeling that it would add too much complexity to pabot and it's not suitable to implement it in pabot.
A '--deterministic' option (or a not so promising option name) is quite easy to implement (see https://github.com/lionelperrin/pabot/commits/deterministic) and very useful in my use case.
Nevertheless, I can understand that this need is not shared by the community.
from pabot.
@lionelperrin: I checked the code. I'm just not understanding the benefit of this - so could you explain it to me?
For example in a situation where we have 3 suites (s1, s2, s3) distributed between two workers.
Currently they would be executed in this order:
s1 and s2 start at the same time
s3 starts to execute when first one of s1 and s2 is ready.
Total execution time: E1 = max(min(timeof(s1), timeof(s2)) + timeof(s3), max(timeof(s1), timeof(s2)))
In your case they would be executed in the following way:
s1 and s2 start at the same time.
s3 starts when s1 is finished.
Total execution time: E2 = max(timeof(s1) + timeof(s3), timeof(s2))
And the ugly truth is that E1 <= E2
So why would you want to do it?
from pabot.
@mkorpela: You're right about the execution times. With the option enabled, there is no load balancing anymore and the total time is longer than without.
If you have 10 suites and 2 workers, with the option enabled, you can garantee that
worker1 will process s1, s3, s5, s7, s9 while worker2 will process s2, s4, s6, s8, s10.
The added value is that if your worker is buggy and can't process s3 after s1, then you'll see it and have a way to reproduce it.
Without this option, in this case, some run might work (if s2 is shorter than s1), some other might fail.
from pabot.
First of all the deterministic option only gives one possible execution order - thus it will reduce the possibility to detect this kind of errors (IMHO bad thing - more detected bugs is better than less). This is very much discussed area of test automation (see for example http://www.satisfice.com/repeatable.shtml ).
Second point is about probability of this mechanism working as expected.
If we have N parallel executors the probability of two interfering suites ending up to the same is 1/N.
So in 1-1/N cases the mechanism will fail. N >= 2 so this number is alway greater than 50% - so most times the deterministic option will fail.
from pabot.
@lionelperrin I've created a small script to visualise the execution order - might be that something based on it could be used to really re-generate the same specific execution order.
https://gist.github.com/mkorpela/d4954e094db810aa7fb4
from pabot.
Closing this issue as there hasn't been any activity or arguments against my points at the end of discussion
from pabot.
Related Issues (20)
- Using --argumentfile causes tests to be run in a sequence X times per process even though --testlevelsplit is used HOT 1
- Pabot is not handling error cause by the --command option
- Allow the pabot results to be left without aggregating them.
- If pabot process gets killed due to exceeding the maximum timeout it's left out of the report
- Reading XML source 'test/pabot_results/<test_no>/output.xml' failed: ParseError: no element found: line 1, column 0
- Unable to read file io.TextIOWrapper - EXCEPTION RAISED DURING PABOT EXECUTION
- Unable to locate the resource file when the directory name has spaces upon using Pabotlib --resourcefile
- Rerun failed tests using pabot, errors out with exit code 252
- Reusing test suites
- Pabot relies on python generated executable when spawning instance of robot framework
- RF7 > `--name` option causes ALL suites to run
- Pabot ordering file groups do not work correctly when using --suite command line argument
- reports error when robot case section headers use other languages HOT 1
- [ ERROR ] Reading XML source '' failed: No such file or directory - After Python upgrade HOT 1
- Allow for --processes auto to do the same as not setting it
- Question: What is the maximum number of argumentfile can be used in pabot?
- The actual number of allocated processes is incorrect with --dryrun and --processes options HOT 1
- required a support adding new flag which will wait before trigger next suite
- Incorrect total elapsed time in merged output.xml with schema version 5
- Parameter --legacy-output does not change pabot output.xml to old schema
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pabot.