Per offline discussion, we will maintain which checkers are on by default in the RESTler driver. To prevent confusion due to mismatched defaults, the engine should have all checkers off by default.

The current implementation overwrites the user arguments - this is a bug.

After installing dotnet 3.1 sdk, rebuilding the program, and re-running the quick start script, I get this error:

$ python restler-quick-start.py --api_spec_path ../openapi.yaml --restler_drop_dir bin/
Log share was not specified.  Logs will not be uploaded.
Starting task Compile...

ERROR: Compiler failed. See logs in /home/aharper/restler_working_dir/Compile directory for more information. 

Task Compile failed.
Collecting logs...

ERROR: The RESTler grammar file path /home/aharper/restler_working_dir/Compile/grammar.py does not exist.

The log file:

Restler Information: 0 : Log share was not specified.  Logs will not be uploaded.
    DateTime=2020-11-17T15:53:06.3059771Z
Restler Information: 0 : Starting task Compile...
    DateTime=2020-11-17T15:53:06.3131157Z
Restler Information: 0 : Launching 'dotnet "/mnt/c/Users/aharper/workspace/api-v3/.restler/bin/restler/../compiler/Restler.CompilerExe.dll" "/home/aharper/restler_working_dir/Compile/config.json"'
    DateTime=2020-11-17T15:53:06.5334522Z
Restler Information: 0 : Process execution terminated in 00:00:00.7213000 with exit code 0x86: 'dotnet "/mnt/c/Users/aharper/workspace/api-v3/.restler/bin/restler/../compiler/Restler.CompilerExe.dll" "/home/aharper/restler_working_dir/Compile/config.json"'
    DateTime=2020-11-17T15:53:07.2638454Z
Restler Information: 0 : (7061) dotnet "/mnt/c/Users/aharper/workspace/api-v3/.restler/bin/restler/../compiler/Restler.CompilerExe.dll" "/home/aharper/restler_working_dir/Compile/config.json" exited with code: 134
    DateTime=2020-11-17T15:53:07.2706808Z
Restler Error: 0 : Compiler failed. See logs in /home/aharper/restler_working_dir/Compile directory for more information.
    DateTime=2020-11-17T15:53:07.2782879Z
Restler Information: 0 : Task Compile failed.
    DateTime=2020-11-17T15:53:07.2822032Z
Restler Information: 0 : Collecting logs...
    DateTime=2020-11-17T15:53:07.2879897Z

The path .restler/bin/restler/../compiler/Restler.CompilerExe.dll seems odd, but .restler/bin/compiler/Restler.CompilerExe.dll does exist, along with all the other compilation artifacts. So not sure what the problem is or what the next thing to check would be.

(side note: hard coding a working dir in my home folder isn't cool... it should either be configurable or relative to the process working dir).

For the following Swagger defined properties:

          "timeSpan": {
              "type": "string",
              "description": "A string which is interpreted as a TimeSpan",
              "format": "date-span",
              "nullable": true
            }

          "targetUrl": {
            "type": "string",
            "format": "uri",
            "nullable": true
          }

Currently, these are compiled to restler_fuzzable_string.

It should instead be restler_fuzzable_date_span and restler_fuzzable_uri.

Hi!
Is RESTler can handle python scripts as token generation source?
Its looks like it is, but i cant get it work.

What i did:

1. Create python script which returns:
   return ("{'appname':{}} + "\n" + "<header-tag-name>: <token>")
2. Make it executable
3. Run script from command line with its absolute path to enshure that it returns what expected
4. Provide abs. path to script via --token_refresh_command "/absolute/path/script.py" and set --token_refresh_interval 10 and run restler

What i expect:
No 401 errors in Test/logs and increasing coverage (almost all endpoints require auth)

What i have:
0 coverage, only 401 errors

Could you please help and provide some details how to deal with this problem?

Thank you in advance!

Hello,

The PayloadBodyChecker appears to have a number of undocumented options (looking at the source code). It would be helpful to have these options documented, including the recipe file.

Use-case:

In my testing, it appears that the default settings for the payload body checker result in trying a low number of payload combinations, often ever sending a single value for most of the body parameters.

While it is often infeasible to try every possible combination, my thought it would be good to try a few fuzzed values for each parameter.

It appears to be possible to have the checker try more combinations via the settings. For example, this recipe does try other values for all of the body parameters, but it is too far in the other direction (making ~12000 requests vs. 58 with the defaults) and it could still just be exhaustively checking up to a (much higher) limit.

Thus, having the settings documented would be helpful.

Thanks!

The fact that this is a setting in the compiler causes the following two usability issues:

1. The compiled grammar from the quick start cannot be used, just adding authentication (user must find this in the setting, and set it in addition to setting the --token_refresh_command)

2. The grammar can't be easily used for parity testing with vs. without authentication.

Proposal:

• Always generate the refreshable authentication token element in the grammar.
• If the --token_refresh_command is set, then invoke it and substitute the token
• If there is no token refresh command, the grammar element refreshable_authentication_token 'is ignored.

Always

Hello,

Is it possible to specify via annotations or a fuzzing dictionary what parameters to a request need to be unique?

Use cases

POST on /resources to create where the server honors the user-provided unique identifier in the request body. Example repository/branch (RESTler-fuzzer output).
Current behavior: A fuzzable value is provided, causing 409 errors when RESTler-fuzzer attempts to create several of the same resource.

PUT on /resources/{resourceId} to create where the server expects the resourceId in the URL to match an identifier param in the request body. Example repository/branch (RESTler-fuzzer output).
Current behavior: A unique ID is provided for the resourceId in the URL, but a fuzzable value is provided in the body, causing 400 errors.

Note: These are cases from an existing real-world (closed-source/non-public) REST API.

Desired Functionality

Specifically, what I'm looking for is (in a maintainable/declarative manner):

1. A way to specify which parameter names for a request (or globally) should be unique
2. And for those unique parameters, which ones within a request should match (have the same unique value)

From my limited testing, this appears to be possible and work as desired when modifying the grammar.py directly to use primitives.restler_custom_payload_uuid4_suffix in place of restler_fuzzable_string, including that passing the same suffix in several places in the request sends the same unique value.

However, modifying the grammar (json or py) seems hard to maintain.

Thank you for your time,
Joshua

What I've tried

1. Adding the param name to the fuzzing dictionary under restler_custom_payload
2. Adding the param name to the fuzzing dictionary under restler_custom_payload_uuid4_suffix
3. Adding inline swagger annotations specifying the producer_resource_name as:
   1. "restler_fuzzable_uuid4"
   2. "restler_custom_payload_uuid_suffix__id" with a corresponding entry in the fuzzing dictionary

I did note the docs mention unique ids in the Fuzzing Dictionary guide, however this appears to be in the context of specifying a suffix to a unique ID rather than specifying what parameters should be unique.

Currently, RESTler hard-codes quotes in the grammar around properties/parameters of string types. This constraint should be removed so it can pass values of any type for string parameters.

Current behavior for string property
Grammar.json:

    primitives.restler_static_string('"'),
    primitives.restler_fuzzable_string("fuzzstring"),
    primitives.restler_static_string("""",

fuzzing dictionary:
"restler_fuzzable_string" : ["a", "b", "c"]

After this improvement:
grammar.json:

    primitives.restler_fuzzable_string("fuzzstring"),

dictionary:

"restler_fuzzable_string" : ["a", "b", "c"] <-- the engine will add quotes around these values

"restler_fuzzable_raw_string": ["1", "2", "3.5", "null"] <--the engine will not add quotes around these values

Make sure the documentation is updated as part of this PR.

Dear developers

I'm aware that this checker is currently off by default, but nevertheless I played around with it and have a few questions about it. It's basically three questions in one issue, but they are related to each other so I'm including them in one issue.

To test, I'm using the following REST API:

POST /rest/admin/products/{username}
=> This creates a product provided by user {username}. This can only be done by a user with a valid JSON Web Token (JWT) and only if the username in the token corresponds to {username}. It returns, among other data, the productID of the newly created product.

GET /rest/admin/products/{username}/{productID}
=> Allows to get a product. Can only be done with valid JWT that contains the corresponding username. So just like the POST request above, this is part of the "administrative area" of the API (anything below /rest/admin) to maintain products.

DELETE /rest/admin/products/{username}/{productID}
=> Allows to delete a product. Can only be done with valid JWT that contains the corresponding username. Also part of the "administrative area".

GET /rest/products/{productID}
=> Allows any user (no authentication required) to get a product. So in contrast to the requests above, this is not part of the "administrative area" of the API used for maintaining product, but allows anonymous users to view product details.

There's a little python program that returns valid JWTs of two users, luke and snoopy. This is used in parameter --token_refresh_command.

#!/usr/bin/env python
import sys
sys.stdout.write("{'luke':{}, 'snoopy':{}}\n")
sys.stdout.write("Authorization: Bearer eyJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJNYXJrZXRwbGFjZSIsInN1YiI6Imx1a2UiLCJleHAiOjE5MDczNTM5NzJ9.jdoYfz_diOQa...\n")
sys.stdout.write("Authorization: Bearer eyJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJNYXJrZXRwbGFjZSIsInN1YiI6InNub29weSIsImV4cCI6MTkwNzM1NDAxMX0.XiOtGu...\n")

I've added two vulnerabilities:

• With a valid JWT, one can GET the products of another user, e.g., snoopy can successfully execute GET /rest/admin/products/luke/25 to get a product that belongs to luke.
• With a valid JWT, one can DELETE the products of another user, e.g., snoopy can successfully execute DELETE /rest/admin/products/luke/25 to delete a product that belongs to luke.

The results of fuzz-lean are as follows:

• GET /rest/admin/products/luke/25 as user snoopy is correctly recognised as a vulnerability.
• GET /rest/products/25 is wrongly flagged as a vulnerability. Intuitively, I wouldn't have expected this as the POST request created the product below /rest/admin/products/luke. But apparently, a vulnerability is flagged if the corresponding productID is used in any other request, no matter what the full pat is. Is this intentional?
• DELETE /rest/admin/products/luke/25 as user snoopy is not recognised as a vulnerability. Apparently, when testing this, restler first does a POST and a corresponding DELETE (so the product is removed), and then it does another DELETE (I assume with the second user). This results in a 4xx error, as the product is longer there. This seems to be a wrong verification approach and a possible bug. The correct approach would be to do POST with the JWT of user1 and then DELETE with the JWT of user2.

And a final question: Is there a way to link the username in a token (that is returned by the python script above in my case) with the username that is used in the path of the POST request? Right now, I have hardcoded this in Grammar.py which works (so it always uses username luke), but this is not really an elegant solution.

So to summarise: three questions/issues:

• Is allowed access to GET /rest/products/25 intentional?
• Is the missed vulnerability with DELETE /rest/admin/products/luke/25 due to a bug?
• Is there a good way to link the username in a JWT to parameters used in requests?

For reference, I've attached Grammar.py, the network log and the two reported vulnerabilities.

Thanks a lot!
Marc
grammar.py.zip
network.testing.140572845729536.1.txt
NameSpaceRuleChecker_20x_1.txt
NameSpaceRuleChecker_20x_2.txt

Hello,

I think there's a bug with this checker that leads to false positives. The following sequence triggers a ResourceHierarchyChecker vulnerability:

This request creates customer1:

2020-12-04 11:03:13.322: Sending: 'PUT /Marketplace_vuln_fixed-rest/rest/customers/customer1 HTTP/1.1\r\nAccept: application/json\r\nHost: 10.0.0.21:8080\r\nContent-Length: 0\r\nUser-Agent: restler/7.1.0\r\n\r\n'

2020-12-04 11:03:13.327: Received: 'HTTP/1.1 200 OK\r\nContent-Type: application/json\r\nContent-Length: 69\r\nX-Frame-Options: SAMEORIGIN\r\n\r\n{"customer":"customer1","password":"customer1","rolename":"customer"}'

This requests posts a message of customer1 and returns the messageID of the new message (12):

2020-12-04 11:03:13.329: Sending: 'POST /Marketplace_vuln_fixed-rest/rest/customers/customer1/messages HTTP/1.1\r\nAccept: application/json\r\nHost: 10.0.0.21:8080\r\nContent-Type: application/json\r\nContent-Length: 30\r\nUser-Agent: restler/7.1.0\r\n\r\n{\n "message":"customer1"}\r\n'

2020-12-04 11:03:13.341: Received: 'HTTP/1.1 200 OK\r\nContent-Type: application/json\r\nContent-Length: 61\r\nX-Frame-Options: SAMEORIGIN\r\n\r\n{"customer":"customer1","message":"customer1","messageID":12}'

This accesses message 12 of customer1, which is legitimate:

2020-12-04 11:03:13.343: Sending: 'GET /Marketplace_vuln_fixed-rest/rest/customers/customer1/messages/12 HTTP/1.1\r\nAccept: application/json\r\nHost: 10.0.0.21:8080\r\nContent-Length: 0\r\nUser-Agent: restler/7.1.0\r\n\r\n'

2020-12-04 11:03:13.351: Received: 'HTTP/1.1 200 OK\r\nContent-Type: application/json\r\nContent-Length: 61\r\nX-Frame-Options: SAMEORIGIN\r\n\r\n{"customer":"customer1","message":"customer1","messageID":12}'

However, this sequence triggers a ResourceHierarchyChecker vulnerability, which is a false positive.

Note that the fuzzer never uses another customer than customer1 in the final GET request (although I've added "customer2" to restler_fuzzable_string in dict.json).

Any help is appreciated, thanks a lot!

Marc

Currently examples checker does not get run because there are no examples in the grammar. Add examples and add a unit test that runs the examples checker.

Hello,

I am wondering what the "goal" is for RESTler's fuzzing functionality: In general terms, is RESTler is intended to replace, or supplement, other HTTP fuzzing tools?

My understanding is its the former (replace), but I wanted to confirm.

Thanks!

For context, here are some examples that seem like gaps in fuzzing:

1. Valid parameter combinations are not fuzzed for endpoints with examples
2. Valid (based on isRequired) and invalid subsets of query parameters are not fuzzed (all query parameters defined in the grammar are always sent when fuzzing)
3. Randomized parameter values (e.g. random bit flipping, picking values from the fuzzing dictionary randomly, picking request candidate combinations randomly)
4. Brute force parameter values
5. General HTTP fuzzing (e.g. incorrect/missing headers, malformed/wrong content types for request bodies, etc...)

As a note, my only hands-on experience with fuzzing is through RESTler, so it is possible my expectations for what an HTTP fuzzer should do (what is valuable) are incorrect.

P.S. I saw this question in the FAQ, but "web scanners" appears to be a much broader field than just fuzzing.

RESTler currently only parses json specifications. It should also support yaml.

{
"final_spec_coverage": "9 / 27",
"rendered_requests": "9 / 27",
"rendered_requests_valid_status": "9 / 9",
"num_fully_valid": 9,
"num_sequence_failures": 0,
"num_invalid_by_failed_resource_creations": 0,
"throughput": 3.9059513782074413,
"total_object_creations": 236,
"total_unique_test_cases": 78.5,
"total_sequences": 435,
"total_requests_sent": {
"gc": 0,
"main_driver": 104,
"LeakageRuleChecker": 0,
"ResourceHierarchyChecker": 0,
"UseAfterFreeChecker": 0,
"NameSpaceRuleChecker": 0,
"InvalidDynamicObjectChecker": 0,
"PayloadBodyChecker": 426,
"ExamplesChecker": 0
},
"bug_buckets": {
"main_driver_500": 1
}
}

2020-12-01 11:19:31.668: 'Exception Creating Socket: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1108)'

I'm trying to test an SSL-only API locally where the certificate is self-signed. I couldn't find an option to disable certificate validation.

Add --host option to restler-quick-start script and make it optional.

The project https://github.com/microsoft/rest-api-fuz-testing needs a docker image to build on-top of. Create an image built on alpine linux. This should be build pulling from a tag or commit and saved into a test azure repository or pushed public when passing all testing stages.

Ran RESTler on petstore3 (and petstore which is v2 of openapi)

https://github.com/swagger-api/swagger-petstore

Exception in thread Fuzzer:
Traceback (most recent call last):
File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner
self.run()
File "/RESTler/engine/engine/core/fuzzer.py", line 36, in run
self._num_total_sequences = driver.generate_sequences(
File "/RESTler/engine/engine/core/driver.py", line 579, in generate_sequences
return generate_sequences_directed_smoketest(fuzzing_requests, checkers)
File "/RESTler/engine/engine/core/driver.py", line 495, in generate_sequences_directed_smoketest
renderings, response_body, rendering_information = render_request(req, new_seq)
File "/RESTler/engine/engine/core/driver.py", line 395, in render_request
apply_checkers(checkers, renderings, global_lock)
File "/RESTler/engine/engine/core/driver.py", line 143, in apply_checkers
checker.apply(renderings, global_lock)
File "/RESTler/engine/checkers/payload_body_checker.py", line 186, in apply
self._run_feedback_fuzzing(last_request, body_schema_list)
File "/RESTler/engine/checkers/payload_body_checker.py", line 634, in _run_feedback_fuzzing
self._run_invalid_json_task(
File "/RESTler/engine/checkers/payload_body_checker.py", line 736, in _run_invalid_json_task
self._run_body_value_fuzzing(
File "/RESTler/engine/checkers/payload_body_checker.py", line 842, in _run_body_value_fuzzing
self._exec_request_with_new_body(request, body_blocks, tracker)
File "/RESTler/engine/checkers/payload_body_checker.py", line 1115, in _exec_request_with_new_body
for rendered_data, parser in new_request.render_iter(
AttributeError: 'NoneType' object has no attribute 'render_iter'

Message produced by RESTler:

Failed to play sequence from log:
substring not found

The results analyzer fails to parse logs with the following stack:

Unhandled exception. System.Exception: Unexpected response without prior request at ...
at Microsoft.FSharp.Core.PrintfModule.PrintFormatToStringThenFail@1639.Invoke(String message) in F:\workspace_work\1\s\src\fsharp\FSharp.Core\printf.fs:line 1639
at Microsoft.FSharp.Collections.Internal.IEnumerator.choose@171.System-Collections-IEnumerator-MoveNext() in F:\workspace_work\1\s\src\fsharp\FSharp.Core\seq.fs:line 179
at Microsoft.FSharp.Collections.Internal.IEnumerator.map@75.DoMoveNext(b& curr) in F:\workspace_work\1\s\src\fsharp\FSharp.Core\seq.fs:line 77
at Microsoft.FSharp.Collections.Internal.IEnumerator.MapEnumerator1.System-Collections-IEnumerator-MoveNext() in F:\workspace\_work\1\s\src\fsharp\FSharp.Core\seq.fs:line 64 at Microsoft.FSharp.Core.CompilerServices.RuntimeHelpers.takeInner@266[T,TResult](ConcatEnumerator2 x, Unit unitVar0) in F:\workspace_work\1\s\src\fsharp\FSharp.Core\seqcore.fs:line 268
at Microsoft.FSharp.Collections.Internal.IEnumerator.map@75.DoMoveNext(b& curr) in F:\workspace_work\1\s\src\fsharp\FSharp.Core\seq.fs:line 77
at Microsoft.FSharp.Collections.Internal.IEnumerator.MapEnumerator1.System-Collections-IEnumerator-MoveNext() in F:\workspace\_work\1\s\src\fsharp\FSharp.Core\seq.fs:line 64 at Microsoft.FSharp.Collections.Internal.IEnumerator.map@75.DoMoveNext(b& curr) in F:\workspace\_work\1\s\src\fsharp\FSharp.Core\seq.fs:line 77 at Microsoft.FSharp.Collections.Internal.IEnumerator.MapEnumerator1.System-Collections-IEnumerator-MoveNext() in F:\workspace_work\1\s\src\fsharp\FSharp.Core\seq.fs:line 64
at Microsoft.FSharp.Core.CompilerServices.RuntimeHelpers.takeInner@266[T,TResult](ConcatEnumerator2 x, Unit unitVar0) in F:\workspace\_work\1\s\src\fsharp\FSharp.Core\seqcore.fs:line 268 at Microsoft.FSharp.Collections.Internal.IEnumerator.next@193[T](FSharpFunc2 f, IEnumerator1 e, FSharpRef1 started, Unit unitVar0) in F:\workspace_work\1\s\src\fsharp\FSharp.Core\seq.fs:line 194
at Microsoft.FSharp.Collections.Internal.IEnumerator.filter@188.System-Collections-IEnumerator-MoveNext() in F:\workspace_work\1\s\src\fsharp\FSharp.Core\seq.fs:line 196
at Microsoft.FSharp.Collections.SeqModule.oneStepTo@987[T](IEnumerable1 source, List1 prefix, FSharpRef1 enumeratorR, Int32 i) in F:\workspace\_work\1\s\src\fsharp\FSharp.Core\seq.fs:line 996 at Microsoft.FSharp.Collections.SeqModule.action@4164-1[T](IEnumerable1 source, List1 prefix, FSharpRef1 enumeratorR, Int32 i, Unit unitVar0) in F:\workspace_work\1\s\src\fsharp\FSharp.Core\seq.fs:line 1012
at [email protected](Int32 i)
at Microsoft.FSharp.Collections.Internal.IEnumerator.unfold@205.DoMoveNext(b& curr) in F:\workspace_work\1\s\src\fsharp\FSharp.Core\seq.fs:line 207
at Microsoft.FSharp.Collections.Internal.IEnumerator.MapEnumerator1.System-Collections-IEnumerator-MoveNext() in F:\workspace\_work\1\s\src\fsharp\FSharp.Core\seq.fs:line 64 at Microsoft.FSharp.Collections.SeqModule.Iterate[T](FSharpFunc2 action, IEnumerable1 source) in F:\workspace\_work\1\s\src\fsharp\FSharp.Core\seq.fs:line 496 at Microsoft.FSharp.Collections.SeqModule.groupByValueType[T,TKey](FSharpFunc2 keyf, IEnumerable1 seq) in F:\workspace\_work\1\s\src\fsharp\FSharp.Core\seq.fs:line 1068 at [email protected](Unit unitVar0) in F:\workspace\_work\1\s\src\fsharp\FSharp.Core\seq.fs:line 1076 at [email protected](Unit unitVar0) in F:\workspace\_work\1\s\src\fsharp\FSharp.Core\seq.fs:line 471 at Restler.ResultsAnalyzer.Analyze.Buckets.getBuckets@98.GenerateNext(IEnumerable1& next) in C:\restler-fuzzer\src\ResultsAnalyzer\Analyze\Buckets.fs:line 98
at Microsoft.FSharp.Core.CompilerServices.GeneratedSequenceBase1.MoveNextImpl() in F:\workspace\_work\1\s\src\fsharp\FSharp.Core\seqcore.fs:line 371 at Microsoft.FSharp.Core.CompilerServices.GeneratedSequenceBase1.System-Collections-IEnumerator-MoveNext() in F:\workspace_work\1\s\src\fsharp\FSharp.Core\seqcore.fs:line 403
at Microsoft.FSharp.Collections.SeqModule.ToList[T](IEnumerable`1 source)
at Restler.ResultsAnalyzer.Analyze.Main.main(AnalyzeArgs args) in C:\restler-fuzzer\src\ResultsAnalyzer\Analyze\Main.fs:line 99
at Restler.ResultsAnalyzer.Program.main(String[] argv) in C:\restler-fuzzer\src\ResultsAnalyzer\Program.fs:line 223

The above error is hit accessing the API at:
https://date.nager.at/api/v2/PublicHolidays/2017/AT

Swagger file:
https://date.nager.at/swagger/v1.0/swagger.json

See attached grammar.

Compile.zip

The command line is:

Restler.exe test --grammar_file .\Compile\grammar.py --dictionary_file .\Compile\dict.json --settings .\Compile\engine_settings.json --target_ip 89.22.101.162 --target_port 443

Payload body checker can't handle bodies that don't start with '{'.

For instance, if a body is a list, which is still valid json, it will fail.

It fails first because it can't find the body (doesn't start with '{').
If the above is handled for '[' it will behave incorrectly because it assumes the body will always start with a key value pair, so the replacement bodies are created like this: "": [ "..." ] instead of simply [ "..." ]

A crash was identified when running the examples checker :

Exception in thread Fuzzer:
Traceback (most recent call last):
File "C:\Python38\lib\threading.py", line 932, in _bootstrap_inner
self.run()
File "C:\restler_bin\engine\engine\core\fuzzer.py", line 36, in run
self._num_total_sequences = driver.generate_sequences(
File "C:\restler_bin\engine\engine\core\driver.py", line 578, in generate_sequences
return generate_sequences_directed_smoketest(fuzzing_requests, checkers)
File "C:\restler_bin\engine\engine\core\driver.py", line 494, in generate_sequences_directed_smoketest
renderings, response_body, rendering_information = render_request(req, new_seq)
File "C:\restler_bin\engine\engine\core\driver.py", line 394, in render_request
apply_checkers(checkers, renderings, global_lock)
File "C:\restler_bin\engine\engine\core\driver.py", line 143, in apply_checkers
checker.apply(renderings, global_lock)
File "C:\restler_bin\engine\checkers\examples_checker.py", line 59, in apply
self._send_each_example(last_request)
File "C:\restler_bin\engine\checkers\examples_checker.py", line 88, in _send_each_example
_send_request(new_request)
File "C:\restler_bin\engine\checkers\examples_checker.py", line 68, in _send_request
response, _ = self._render_and_send_data(seq, request_to_send)
File "C:\restler_bin\engine\checkers\checker_base.py", line 113, in _render_and_send_data
rendered_data, parser = request.render_current(self._req_collection.candidate_values_pool)
File "C:\restler_bin\engine\engine\core\requests.py", line 609, in render_current
return next(self.render_iter(candidate_values_pool,
File "C:\restler_bin\engine\engine\core\requests.py", line 489, in render_iter
quoted = request_block[2]
IndexError: tuple index out of range

The quick start guide says I need an OpenAPI spec (which I have), and then shows a python command (not explicitly stated, but I have python). Running that command gives me this error:

$ python restler-quick-start.py --apiSpec ../openapi.yaml --restler_drop_dir .drop/
No executable found matching command "dotnet-/mnt/c/Users/aharper/workspace/api-v3/.restler/.drop/restler/Restler.dll"

This seems to imply there are additional steps to get going from scratch not described in the quick start guide. From looking at the readme, there is a "build" section which references dotnet, so I'm guessing that the build instructions are also a prerequisite not mentioned in the quick start. Is that right? Are there any other prerequisite steps to get started?

I recommend reviewing that doc from the perspective of a new user who isn't familiar with your project and has no additional context, and trying to follow your own directions and seeing if it works.

I was trying to compile this

I had the same problem as #61 so I deleted the "required" in that spec.

Then I got the following error.

Unhandled exception. System.AggregateException: One or more errors occurred. (Could not resolve the JSON path 'TS29571_CommonData.yaml#/components/schemas/LinksValueSchema' with the full JSON path '/home/vmuser/5GC_APIs/TS29571_CommonData.yaml#/components/schemas/LinksValueSchema'.)
 ---> System.InvalidOperationException: Could not resolve the JSON path 'TS29571_CommonData.yaml#/components/schemas/LinksValueSchema' with the full JSON path '/home/vmuser/5GC_APIs/TS29571_CommonData.yaml#/components/schemas/LinksValueSchema'.
 ---> System.InvalidOperationException: Could not resolve the path '#/components/schemas/LinksValueSchema'.
   at NJsonSchema.JsonReferenceResolver.ResolveDocumentReference(Object rootObject, String jsonPath)
   at NJsonSchema.JsonReferenceResolver.ResolveReferenceAsync(Object rootObject, String jsonPath, Boolean append)
   at NJsonSchema.JsonReferenceResolver.ResolveReferenceAsync(Object rootObject, String jsonPath)
   at NJsonSchema.JsonReferenceResolver.ResolveFileReferenceWithAlreadyResolvedCheckAsync(String fullJsonPath, String jsonPath, Boolean append)
   --- End of inner exception stack trace ---
   at NJsonSchema.JsonReferenceResolver.ResolveFileReferenceWithAlreadyResolvedCheckAsync(String fullJsonPath, String jsonPath, Boolean append)
   at NJsonSchema.JsonReferenceResolver.ResolveReferenceAsync(Object rootObject, String jsonPath, Boolean append)
   at NJsonSchema.JsonReferenceResolver.ResolveReferenceAsync(Object rootObject, String jsonPath)
   at NJsonSchema.JsonSchemaReferenceUtilities.JsonReferenceUpdater.VisitJsonReferenceAsync(IJsonReference reference, String path, String typeNameHint)
   at NJsonSchema.Visitors.AsyncJsonReferenceVisitorBase.VisitAsync(Object obj, String path, String typeNameHint, ISet`1 checkedObjects, Action`1 replacer)
   at NJsonSchema.Visitors.AsyncJsonReferenceVisitorBase.VisitAsync(Object obj, String path, String typeNameHint, ISet`1 checkedObjects, Action`1 replacer)
   at NJsonSchema.Visitors.AsyncJsonReferenceVisitorBase.VisitAsync(Object obj, String path, String typeNameHint, ISet`1 checkedObjects, Action`1 replacer)
   at NJsonSchema.Visitors.AsyncJsonReferenceVisitorBase.VisitAsync(Object obj, String path, String typeNameHint, ISet`1 checkedObjects, Action`1 replacer)
   at NJsonSchema.Visitors.AsyncJsonReferenceVisitorBase.VisitAsync(Object obj, String path, String typeNameHint, ISet`1 checkedObjects, Action`1 replacer)
   at NJsonSchema.Visitors.AsyncJsonReferenceVisitorBase.VisitAsync(Object obj, String path, String typeNameHint, ISet`1 checkedObjects, Action`1 replacer)
   at NJsonSchema.Visitors.AsyncJsonReferenceVisitorBase.VisitAsync(Object obj, String path, String typeNameHint, ISet`1 checkedObjects, Action`1 replacer)
   at NJsonSchema.Visitors.AsyncJsonReferenceVisitorBase.VisitAsync(Object obj, String path, String typeNameHint, ISet`1 checkedObjects, Action`1 replacer)
   at NJsonSchema.Visitors.AsyncJsonReferenceVisitorBase.VisitAsync(Object obj, String path, String typeNameHint, ISet`1 checkedObjects, Action`1 replacer)
   at NJsonSchema.Visitors.AsyncJsonReferenceVisitorBase.VisitAsync(Object obj, String path, String typeNameHint, ISet`1 checkedObjects, Action`1 replacer)
   at NJsonSchema.Visitors.AsyncJsonReferenceVisitorBase.VisitAsync(Object obj, String path, String typeNameHint, ISet`1 checkedObjects, Action`1 replacer)
   at NJsonSchema.Visitors.AsyncJsonReferenceVisitorBase.VisitAsync(Object obj)
   at NJsonSchema.JsonSchemaReferenceUtilities.JsonReferenceUpdater.VisitAsync(Object obj)
   at NJsonSchema.JsonSchemaReferenceUtilities.UpdateSchemaReferencesAsync(Object rootObject, JsonReferenceResolver referenceResolver, IContractResolver contractResolver)
   at NJsonSchema.Infrastructure.JsonSchemaSerialization.FromJsonAsync[T](String json, SchemaType schemaType, String documentPath, Func`2 referenceResolverFactory, IContractResolver contractResolver)
   at NSwag.OpenApiDocument.FromJsonAsync(String data, String documentPath, SchemaType expectedSchemaType, Func`2 referenceResolverFactory)
   at NSwag.OpenApiYamlDocument.FromYamlAsync(String data, String documentPath, SchemaType expectedSchemaType, Func`2 referenceResolverFactory)
   at NSwag.OpenApiYamlDocument.FromFileAsync(String filePath)
   --- End of inner exception stack trace ---
   at Microsoft.FSharp.Control.AsyncResult`1.Commit() in F:\workspace\_work\1\s\src\fsharp\FSharp.Core\async.fs:line 350
   at Microsoft.FSharp.Control.AsyncPrimitives.RunSynchronouslyInCurrentThread[a](CancellationToken cancellationToken, FSharpAsync`1 computation) in F:\workspace\_work\1\s\src\fsharp\FSharp.Core\async.fs:line 882
   at Microsoft.FSharp.Control.AsyncPrimitives.RunSynchronously[T](CancellationToken cancellationToken, FSharpAsync`1 computation, FSharpOption`1 timeout) in F:\workspace\_work\1\s\src\fsharp\FSharp.Core\async.fs:line 890
   at Microsoft.FSharp.Control.FSharpAsync.RunSynchronously[T](FSharpAsync`1 computation, FSharpOption`1 timeout, FSharpOption`1 cancellationToken) in F:\workspace\_work\1\s\src\fsharp\FSharp.Core\async.fs:line 1154
   at [email protected](String fp) in /home/vmuser/restler-fuzzer/src/compiler/Restler.Compiler/Workflow.fs:line 88
   at Microsoft.FSharp.Primitives.Basics.List.map[T,TResult](FSharpFunc`2 mapping, FSharpList`1 x) in F:\workspace\_work\1\s\src\fsharp\FSharp.Core\local.fs:line 248
   at Microsoft.FSharp.Collections.ListModule.Map[T,TResult](FSharpFunc`2 mapping, FSharpList`1 list) in F:\workspace\_work\1\s\src\fsharp\FSharp.Core\list.fs:line 75
   at Restler.Workflow.generateGrammarFromSwagger(String grammarOutputDirectoryPath, FSharpOption`1 swaggerDoc, Config config) in /home/vmuser/restler-fuzzer/src/compiler/Restler.Compiler/Workflow.fs:line 88
   at Restler.Workflow.generateRestlerGrammar(FSharpOption`1 swaggerDoc, Config config) in /home/vmuser/restler-fuzzer/src/compiler/Restler.Compiler/Workflow.fs:line 233
   at Program.main(String[] argv) in /home/vmuser/restler-fuzzer/src/compiler/Restler.CompilerExe/Program.fs:line 37

It seems that it cannot properly find $ref location. So I used prance to merge the files into one by prance compile TS29510_Nnrf_NFManagement.yaml out.yaml.(I also manually deleted "required" since it hasn't been fixed) The generated out.yaml can be found here. Then restler compile got this error:

Unhandled exception. System.AggregateException: One or more errors occurred. (Exception of type 'Restler.Compiler.UnsupportedType' was thrown.)
 ---> Restler.Compiler.UnsupportedType: Exception of type 'Restler.Compiler.UnsupportedType' was thrown.
   at [email protected](Tuple`2 tupledArg) in /home/vmuser/restler-fuzzer/src/compiler/Restler.Compiler/SwaggerVisitors.fs:line 321
   at Microsoft.FSharp.Collections.Internal.IEnumerator.choose@171.System-Collections-IEnumerator-MoveNext() in F:\workspace\_work\1\s\src\fsharp\FSharp.Core\seq.fs:line 179
   at Microsoft.FSharp.Core.CompilerServices.RuntimeHelpers.takeOuter@273[T,TResult](ConcatEnumerator`2 x, Unit unitVar0) in F:\workspace\_work\1\s\src\fsharp\FSharp.Core\seqcore.fs:line 274
   at Microsoft.FSharp.Core.CompilerServices.RuntimeHelpers.takeInner@266[T,TResult](ConcatEnumerator`2 x, Unit unitVar0) in F:\workspace\_work\1\s\src\fsharp\FSharp.Core\seqcore.fs:line 268
   at Microsoft.FSharp.Collections.SeqModule.IsEmpty[T](IEnumerable`1 source) in F:\workspace\_work\1\s\src\fsharp\FSharp.Core\seq.fs:line 702
   at Restler.Compiler.SwaggerVisitors.generateGrammarElementForSchema(JsonSchema schema, FSharpOption`1 exampleValue, FSharpList`1 parents) in /home/vmuser/restler-fuzzer/src/compiler/Restler.Compiler/SwaggerVisitors.fs:line 335
   at Restler.Compiler.SwaggerVisitors.processProperty@177(String tupledArg0, JsonSchemaProperty tupledArg1, FSharpOption`1 propertyValue, FSharpList`1 parents) in /home/vmuser/restler-fuzzer/src/compiler/Restler.Compiler/SwaggerVisitors.fs:line 221
   at Restler.Compiler.SwaggerVisitors.declaredPropertyParameters@290.Invoke(KeyValuePair`2 item) in /home/vmuser/restler-fuzzer/src/compiler/Restler.Compiler/SwaggerVisitors.fs:line 296
   at Microsoft.FSharp.Collections.Internal.IEnumerator.choose@171.System-Collections-IEnumerator-MoveNext() in F:\workspace\_work\1\s\src\fsharp\FSharp.Core\seq.fs:line 179
   at Microsoft.FSharp.Core.CompilerServices.RuntimeHelpers.takeInner@266[T,TResult](ConcatEnumerator`2 x, Unit unitVar0) in F:\workspace\_work\1\s\src\fsharp\FSharp.Core\seqcore.fs:line 268
   at Microsoft.FSharp.Collections.SeqModule.Iterate[T](FSharpFunc`2 action, IEnumerable`1 source) in F:\workspace\_work\1\s\src\fsharp\FSharp.Core\seq.fs:line 496
   at Restler.Grammar.Tree.iterCtx[a,LeafData,INodeData](FSharpFunc`2 fLeaf, FSharpFunc`2 fNode, FSharpFunc`2 fCtx, a ctx, Tree`2 tree) in /home/vmuser/restler-fuzzer/src/compiler/Restler.Compiler/Grammar.fs:line 56
   at Restler.Dependencies.getParameterDependencies(ParameterKind parameterKind, IEnumerable`1 globalAnnotations, Tuple`2[] requestData, RequestId requestId, String parameterName, Tree`2 parameterPayload, FSharpOption`1 namingConvention) in /home/vmuser/restler-fuzzer/src/compiler/Restler.Compiler/Dependencies.fs:line 741
   at [email protected](Tuple`2 p) in /home/vmuser/restler-fuzzer/src/compiler/Restler.Compiler/Dependencies.fs:line 788
   at Microsoft.FSharp.Collections.Internal.IEnumerator.map@75.DoMoveNext(b& curr) in F:\workspace\_work\1\s\src\fsharp\FSharp.Core\seq.fs:line 78
   at Microsoft.FSharp.Collections.Internal.IEnumerator.MapEnumerator`1.System-Collections-IEnumerator-MoveNext() in F:\workspace\_work\1\s\src\fsharp\FSharp.Core\seq.fs:line 64
   at Microsoft.FSharp.Core.CompilerServices.RuntimeHelpers.takeOuter@273[T,TResult](ConcatEnumerator`2 x, Unit unitVar0) in F:\workspace\_work\1\s\src\fsharp\FSharp.Core\seqcore.fs:line 274
   at Microsoft.FSharp.Core.CompilerServices.RuntimeHelpers.takeInner@266[T,TResult](ConcatEnumerator`2 x, Unit unitVar0) in F:\workspace\_work\1\s\src\fsharp\FSharp.Core\seqcore.fs:line 268
   at Microsoft.FSharp.Collections.SeqModule.DistinctBy@1098.GenerateNext(IEnumerable`1& next) in F:\workspace\_work\1\s\src\fsharp\FSharp.Core\seq.fs:line 1099
   at Microsoft.FSharp.Core.CompilerServices.GeneratedSequenceBase`1.MoveNextImpl() in F:\workspace\_work\1\s\src\fsharp\FSharp.Core\seqcore.fs:line 371
   at Microsoft.FSharp.Core.CompilerServices.GeneratedSequenceBase`1.System-Collections-IEnumerator-MoveNext() in F:\workspace\_work\1\s\src\fsharp\FSharp.Core\seqcore.fs:line 403
   at Microsoft.FSharp.Collections.SeqModule.oneStepTo@987[T](IEnumerable`1 source, List`1 prefix, FSharpRef`1 enumeratorR, Int32 i) in F:\workspace\_work\1\s\src\fsharp\FSharp.Core\seq.fs:line 996
   at Microsoft.FSharp.Collections.SeqModule.action@4164-1[T](IEnumerable`1 source, List`1 prefix, FSharpRef`1 enumeratorR, Int32 i, Unit unitVar0) in F:\workspace\_work\1\s\src\fsharp\FSharp.Core\seq.fs:line 1012
   at [email protected](Int32 i)
   at Microsoft.FSharp.Collections.Internal.IEnumerator.unfold@205.DoMoveNext(b& curr) in F:\workspace\_work\1\s\src\fsharp\FSharp.Core\seq.fs:line 207
   at Microsoft.FSharp.Collections.Internal.IEnumerator.MapEnumerator`1.System-Collections-IEnumerator-MoveNext() in F:\workspace\_work\1\s\src\fsharp\FSharp.Core\seq.fs:line 64
   at Microsoft.FSharp.Collections.SeqModule.Iterate[T](FSharpFunc`2 action, IEnumerable`1 source) in F:\workspace\_work\1\s\src\fsharp\FSharp.Core\seq.fs:line 496
   at [email protected](Tuple`2 tupledArg) in /home/vmuser/restler-fuzzer/src/compiler/Restler.Compiler/Dependencies.fs:line 872
   at Microsoft.FSharp.Collections.ArrayModule.Parallel.Map@1317-3.Invoke(Int32 i) in F:\workspace\_work\1\s\src\fsharp\FSharp.Core\array.fs:line 1318
   at System.Threading.Tasks.Parallel.<>c__DisplayClass19_0`1.<ForWorker>b__1(RangeWorker& currentWorker, Int32 timeout, Boolean& replicationDelegateYieldedBeforeCompletion)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw(Exception source)
   at System.Threading.Tasks.Parallel.<>c__DisplayClass19_0`1.<ForWorker>b__1(RangeWorker& currentWorker, Int32 timeout, Boolean& replicationDelegateYieldedBeforeCompletion)
   at System.Threading.Tasks.TaskReplicator.Replica`1.ExecuteAction(Boolean& yieldedBeforeCompletion)
   at System.Threading.Tasks.TaskReplicator.Replica.Execute()
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.TaskReplicator.Run[TState](ReplicatableUserAction`1 action, ParallelOptions options, Boolean stopOnFirstFailure)
   at System.Threading.Tasks.Parallel.ForWorker[TLocal](Int32 fromInclusive, Int32 toExclusive, ParallelOptions parallelOptions, Action`1 body, Action`2 bodyWithState, Func`4 bodyWithLocal, Func`1 localInit, Action`1 localFinally)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw(Exception source)
   at System.Threading.Tasks.Parallel.ThrowSingleCancellationExceptionOrOtherException(ICollection exceptions, CancellationToken cancelToken, Exception otherException)
   at System.Threading.Tasks.Parallel.ForWorker[TLocal](Int32 fromInclusive, Int32 toExclusive, ParallelOptions parallelOptions, Action`1 body, Action`2 bodyWithState, Func`4 bodyWithLocal, Func`1 localInit, Action`1 localFinally)
   at System.Threading.Tasks.Parallel.For(Int32 fromInclusive, Int32 toExclusive, Action`1 body)
   at Microsoft.FSharp.Collections.ArrayModule.Parallel.Map[T,TResult](FSharpFunc`2 mapping, T[] array) in F:\workspace\_work\1\s\src\fsharp\FSharp.Core\array.fs:line 1319
   at Restler.Dependencies.extractDependencies(Tuple`2[] requestData, IEnumerable`1 globalAnnotations, MutationsDictionary customDictionary, Boolean queryDependencies, Boolean bodyDependencies, Boolean allowGetProducers, Boolean dataFuzzing, FSharpMap`2 perResourceDictionaries, FSharpOption`1 namingConvention) in /home/vmuser/restler-fuzzer/src/compiler/Restler.Compiler/Dependencies.fs:line 833
   at Restler.Compiler.Main.generateRequestGrammar(FSharpList`1 swaggerDocs, MutationsDictionary dictionary, Config config, FSharpList`1 globalExternalAnnotations) in /home/vmuser/restler-fuzzer/src/compiler/Restler.Compiler/Compiler.fs:line 554
   at Restler.Workflow.generateGrammarFromSwagger(String grammarOutputDirectoryPath, FSharpOption`1 swaggerDoc, Config config) in /home/vmuser/restler-fuzzer/src/compiler/Restler.Compiler/Workflow.fs:line 151
   at Restler.Workflow.generateRestlerGrammar(FSharpOption`1 swaggerDoc, Config config) in /home/vmuser/restler-fuzzer/src/compiler/Restler.Compiler/Workflow.fs:line 233
   at Program.main(String[] argv) in /home/vmuser/restler-fuzzer/src/compiler/Restler.CompilerExe/Program.fs:line 37

Currently, it creates it in the home folder:

RESTLER_TEMP_DIR = Path(Path.home()).joinpath('restler_working_dir')

This is another instance of the bug where defaults override user-specified args.

These two need to be fixed:

UseBodyExamples
UseQueryExamples

If a bucket is market as ‘non-reproducible’, any subsequent bug matching this bucket should be re-tried as well, and we should count these numbers of ‘reproducibility attempts/retries’ with each bucket.

Currently the bugs are tagged with whether or not they were reproduced, but there is no count for the number of times the bug was seen and if the bug was already found once, it currently will not attempt to reproduce again.

There is currently a bug in dependency analysis, as follows:

PUT /stores/{storeId}/orders
{
...
"storeId": "123"
}

A producer-consumer dependency is inferred between "storeId" in the body and the
PUT /stores/{storeId}, and set to the same storeId as in the path. This is not intentional, and is
a bug that should be fixed - the current algorithm is intended only for path dependencies.

When starting a service under test and RESTler at the same time; RESTler fails with "connection refused" since service under test is still starting up.

Suggestion: Add a startup wait parameter to RESTler. Thus allowing RESTler to ignore refused connection until start up wait timeout expired

schema.Reference can be null even when schema.HasReference is true, because HasReference includes AllOf, OneOf, and AnyOf references

Unhandled exception. System.NullReferenceException: Object reference not set to an instance of an object.
at Restler.Compiler.SwaggerVisitors.getActualSchema@269(JsonSchema s) in D:\git\restler-fuzzer\src\compiler\Restler.Compiler\SwaggerVisitors.fs:line 270
at Restler.Compiler.SwaggerVisitors.generateGrammarElementForSchema(JsonSchema schema, FSharpOption1 exampleValue, FSharpList1 parents) in D:\git\restler-fuzzer\src\compiler\Restler.Compiler\SwaggerVisitors.fs:line 274
at [email protected](IEnumerable1& next) at Microsoft.FSharp.Core.CompilerServices.GeneratedSequenceBase1.MoveNextImpl() in F:\workspace_work\1\s\src\fsharp\FSharp.Core\seqcore.fs:line 371
at Microsoft.FSharp.Core.CompilerServices.GeneratedSequenceBase1.System-Collections-IEnumerator-MoveNext() in F:\workspace\_work\1\s\src\fsharp\FSharp.Core\seqcore.fs:line 403 at Microsoft.FSharp.Collections.SeqModule.TryHead[T](IEnumerable1 source) in F:\workspace_work\1\s\src\fsharp\FSharp.Core\seq.fs:line 1370
at [email protected](IEnumerable1& next) in D:\git\restler-fuzzer\src\compiler\Restler.Compiler\Compiler.fs:line 461 at Microsoft.FSharp.Core.CompilerServices.GeneratedSequenceBase1.MoveNextImpl() in F:\workspace_work\1\s\src\fsharp\FSharp.Core\seqcore.fs:line 371
at Microsoft.FSharp.Core.CompilerServices.GeneratedSequenceBase1.System-Collections-IEnumerator-MoveNext() in F:\workspace\_work\1\s\src\fsharp\FSharp.Core\seqcore.fs:line 403 at Microsoft.FSharp.Core.CompilerServices.RuntimeHelpers.takeInner@266[T,TResult](ConcatEnumerator2 x, Unit unitVar0) in F:\workspace_work\1\s\src\fsharp\FSharp.Core\seqcore.fs:line 268
at System.Collections.Generic.List1..ctor(IEnumerable1 collection)
at Microsoft.FSharp.Collections.SeqModule.ToArray[T](IEnumerable1 source) in F:\workspace\_work\1\s\src\fsharp\FSharp.Core\seq.fs:line 825 at Restler.Compiler.Main.generateRequestGrammar(FSharpList1 swaggerDocs, MutationsDictionary dictionary, Config config, FSharpList1 globalExternalAnnotations) in D:\git\restler-fuzzer\src\compiler\Restler.Compiler\Compiler.fs:line 523 at Restler.Workflow.generateGrammarFromSwagger(String grammarOutputDirectoryPath, FSharpOption1 swaggerDoc, Config config) in D:\git\restler-fuzzer\src\compiler\Restler.Compiler\Workflow.fs:line 151
at Restler.Workflow.generateRestlerGrammar(FSharpOption`1 swaggerDoc, Config config) in D:\git\restler-fuzzer\src\compiler\Restler.Compiler\Workflow.fs:line 230
at Program.main(String[] argv) in D:\git\restler-fuzzer\src\compiler\Restler.CompilerExe\Program.fs:line 36

C:\Program Files\dotnet\sdk\5.0.100\Sdks\Microsoft.NET.Sdk\targets\Microsoft.PackageDependencyResolution.targets(241,5): error NETSDK1005: Assets file 'D:\a\1\s\src\ResultsAnalyzer\obj\project.assets.json' doesn't have a target for 'netcoreapp3.1'. Ensure that restore has run and that you have included 'netcoreapp3.1' in the TargetFrameworks for your project. [D:\a\1\s\src\ResultsAnalyzer\ResultsAnalyzer.fsproj]
##[error]Error: The process 'C:\Program Files\dotnet\dotnet.exe' failed with exit code 1
"C:\Program Files\dotnet\dotnet.exe" publish D:\a\1\s\src\compiler\Restler.CompilerExe\Restler.CompilerExe.fsproj --no-restore -c Release /p:version=7.1.209
Microsoft (R) Build Engine version 16.8.0+126527ff1 for .NET
Copyright (C) Microsoft Corporation. All rights reserved.

C:\Program Files\dotnet\sdk\5.0.100\Sdks\Microsoft.NET.Sdk\targets\Microsoft.PackageDependencyResolution.targets(241,5): error NETSDK1005: Assets file 'D:\a\1\s\src\compiler\Restler.CompilerExe\obj\project.assets.json' doesn't have a target for 'netcoreapp3.1'. Ensure that restore has run and that you have included 'netcoreapp3.1' in the TargetFrameworks for your project. [D:\a\1\s\src\compiler\Restler.CompilerExe\Restler.CompilerExe.fsproj]
##[error]Error: The process 'C:\Program Files\dotnet\dotnet.exe' failed with exit code 1
"C:\Program Files\dotnet\dotnet.exe" publish D:\a\1\s\src\driver\Restler.Driver.fsproj --no-restore -c Release /p:version=7.1.209
Microsoft (R) Build Engine version 16.8.0+126527ff1 for .NET
Copyright (C) Microsoft Corporation. All rights reserved.

Currently, only query and body examples are supported. Path examples should also be supported.

The resulting grammar should enable fuzzing paths similarly to how the payload body checker fuzzes bodies, by separately analyzing the schema in grammar.json and extracting relevant values from the dictionary.

Currently the RESTler driver will output Task Succeeded at the end of a fuzzing/test run even if the run crashed and the stderr.txt file is not empty. This output should let the user know that there was an error, so they can investigate and not assume that the run completed successfully.

For every bug found by RESTler, define/compute a unique ID/hash, so that:

• we can determine how many unique bugs were found across many RESTler runs
• we can determine how many unique bugs were found in deployed RESTler jobs

Note: this ID/hash could be either generated by the RESTler engine and added to some logs, and/or it could be computed in a post-processing phase (so that we can also post-process past RESTlerLogs).

Some requests need to pass the same value in multiple places. For example, in this request,
PUT /resource/{resourceId} { ... "id" : "..." }
the values of resourceId and id need to be equal for it to be accepted.

This should be supported with an annotation, as follows:

x-restler-annotation {
  "endpoint": "...",
   "method": "...",
   "resource_names_equal": ["resourceId", "/bodyProperties/id"]
}

This should be supported in the engine as follows:

1. Any grammar element can have an "id" parameter that should be a unique ID for the individual request (NOT globally).
2. A new grammar element, restler_copied_string, will take a single parameter "id=", and will copy the element of the specified source string.

Example:

restler_custom_payload_uuid_suffix("appGatewayName", id=99)
...

Hello,

It appears that falsy PayloadBodyChecker settings are ignored when the default is truthy.

Example: Note how size_dep_budget is set to false in engine_settings.json, it is loaded and reported as True in PayloadBodyChecker.22992.txt.

Reproduction steps for that specific run are here: https://github.com/JoshuaEN/restler-fuzzer-testing/tree/problem/payload-body-settings-ignored#this-run-can-be-reproduced-by

I think this is because the loaded setting is ored with the default (e.g.). Thus, for values that default to true, it is impossible to set them to false.

Note: I also verified that the settings were actually being loaded by changing the defaults for _fuzz_valid and _fuzz_invalid (in payload_body_checker.py ) to False, in the configuration setting fuzz_valid to true, and then confirming that the log reported fuzzing valid True and fuzzing invalid False.

Thanks!

RESTler should be able to have examples be provided/maintained separately from the API specification, similar to annotations and the dictionary.

Currently, when 'DiscoverExamples' is specified, an 'examples' directory is created with discovered examples, and a corresponding examples.json is written. However, a missing piece is that only existing examples from the API specification can be discovered/modified.

The RESTler compiler should be able to take 'examples.json' as input, and use it as follows:

• ignored when DiscoverExamples is true - new examples.json is written
• if both the API specification and the example file provide examples, the example file examples take precedence (i.e. the default payload body will be the one from the external examples), and are merged with the specification examples (i.e., all values will be used by the payload body checker).

As part of this implementation, the format of the examples.json should be updated/improved.

See issue #74. This is due to an underlying issue in the NJsonSchema library parsing external references. This is already being worked around for json by preprocessing and inlining external refs. The same workaround needs to be implemented for yaml.

Unhandled exception. System.AggregateException: One or more errors occurred. (Could not resolve the JSON path 'TS29571_CommonData.yaml#/components/schemas/LinksValueSchema' with the full JSON path '/home/vmuser/5GC_APIs/TS29571_CommonData.yaml#/components/schemas/LinksValueSchema'.)
---> System.InvalidOperationException: Could not resolve the JSON path 'TS29571_CommonData.yaml#/components/schemas/LinksValueSchema' with the full JSON path '/home/vmuser/5GC_APIs/TS29571_CommonData.yaml#/components/schemas/LinksValueSchema'.
---> System.InvalidOperationException: Could not resolve the path '#/components/schemas/LinksValueSchema'.
at NJsonSchema.JsonReferenceResolver.ResolveDocumentReference(Object rootObject, String jsonPath)
at NJsonSchema.JsonReferenceResolver.ResolveReferenceAsync(Object rootObject, String jsonPath, Boolean append)
at NJsonSchema.JsonReferenceResolver.ResolveReferenceAsync(Object rootObject, String jsonPath)
at NJsonSchema.JsonReferenceResolver.ResolveFileReferenceWithAlreadyResolvedCheckAsync(String fullJsonPath, String jsonPath, Boolean append)
--- End of inner exception stack trace ---

Change InvalidDynamicObjectChecker's api-version test to instead say injected_query_string.

The user may have special status codes (e.g. similar to assertions) that they'd like bug buckets reported on regardless of the context.

Examples:

• User is running the RESTler smoke test, and wants all requests to succeed. They want to consume bugs in the same way as any other bug buckets. In this case, they would specify 40* status codes in addition to the defaults.

The current implementation overwrites the user arguments - this is a bug.

From user feedback:
If I have the Test directory open in vscode, I get an exception about trying to delete the Test directory when running restler so that's a little annoying. (running code Test).

The 'RESTlerLogs' directory is not needed. Only create one sub-directory in the specified root directory.

Hello,

I have managed to successfully test several of the checkers using an intentionally vulnerable test application I developed, but so far, I have had no luck with this checker.

The scenario is as follows:

• customer1 can POST messages to /customers/customer1/messages
• customer2 can do the same to /customers/customer2/messages

They can GET a message back (that has a specific messageID) using, e.g.:

• customer1 (who has posted message 1) from /customers/customer1/messages/1
• customer2 (who has posted message 2) from /customers/customer2/messages/2

I have implemented a bug that allows customer2 to access messages from customer1, e.g., GET /customers/customer1/messages/1 works. As I understand this checker, this should be detectable.

Compiling and testing works. The OpenAPI config is correct. restler test works correctly and the messageID returned by POST is associated with the corresponding GET request. The test application uses JWTs that have been configured correctly and restler uses the tokens correctly. I've also enable the checker using --enable_checkers namespacerule.

The program that returns the two JWTs is as follows (the first one is of customer1, the second of customer2):

#!/usr/bin/env python
import sys
sys.stdout.write("{'customer1':{}, 'customer2':{}}\n")
sys.stdout.write("Authorization: Bearer eyJhbGciOiJIUzI1...\n")
sys.stdout.write("Authorization: Bearer eyJhbGciOiJIUzI1....\n")

I'm unsure about the metadata that must be returned as the first line. I suppose that they should include the URL-part that identifies the customer-specific areas, but I cannot find any information about this (Authentication.md does only provide little information).

I added logging to the application that prints the JWT used in the requests from restler. They only use the first token, never the second, so it appear this checker is never invoked at all.

Any help would be appreciated, thanks!

Re-format the usage to print in a more compact way on the console.

Add the ability to disable logging in restler that could lead to leaked personal or sensitive information. I.e. request/response data sent to and received from the server.

I'm using this spec
https://github.com/jdegre/5GC_APIs/blob/master/TS29518_Namf_Communication.yaml

In grammar.py generated by compiler, there's option quoted=True. But capturing with wireshark shows that it's not quoted.

                        }
                    ,
                    "smsfId":"""),
    primitives.restler_fuzzable_uuid4("566048da-ed19-4cd3-8e0a-b7e0e1ec4d72", quoted=True),

Same thing happens with grouptag

primitives.restler_fuzzable_group("fuzzable_group_tag", ['3GPP_ACCESS','NON_3GPP_ACCESS']  ,quoted=True),