pmarkert / hyperpotamus Goto Github PK

I would like to improve the way that plugins are registered and linked. Right now, the cli takes a single path/directory and all javascript files in that directory are loaded as plugins. I don't remember if it already supports multiple directories, but perhaps we could use node's require syntax more directly and allow any installed plugin that can be required to be loaded. Also consider allowing environment variables to be configured for which plugins to register/require.

When you set a headers action, the key name has to be lowercase. It would be nice if the key name could be any case since http headers are case-insensitive. I was doing:

response:
  headers:
    Location: location
  print: "Location: <% location %>"

and it always printed 'undefined'. At first I thought I was doing something wrong, but it turned out the 'Location' key had to be 'location' instead.

I'd like to leave the base hyperpotamus engine as this module and then split-out all/most of the plugins to separate modules. I would want to have a few "collection packages" that bundle sensible groups of plugins together and then some of the heavier-weight (and especially un-safe) plugins could be in separate packages. This would allow/support a few benefits:

• New plugins can be added without having to bump the base hyperpotamus version.
• Updates to existing plugins can be versioned separately
• Smaller (lighter-weight) deployments can be tailored that limit what plugins are included
• Makes it easier for the community to add and support plugins
• Separation of safe/unsafe plugins from an installation standpoint.

Right now hyperpotamus plugins can only add (or replace) actions. Allow pluggable | pipes for interpolation processing and document how to create.

I've only implemented socks5/http. Would be trivial to add socks5/https, but hasn't come up as a specific need yet.

version should be able to check against an array of semver specs and if any of them pass, the version checks out. This is helpful because of things like locked versions (with -SPECIFIERS) must be exact.

I would like to have more consistent output/logging for errors. Right now, it's pretty chatty and sometimes difficult to figure out what went wrong.

Hey,
this tool is really awesome and powerful. Thank you for that!
In my script, I'm capturing an value from a first GET-request and reuse it in a subsequent POST-request. This works well. But the first request should also set a session cookie (the page does this) but this session cookie is not reused in the subsequent POST-request:

- request:
    url: http://example.com/login
  response:
    - jquery: "form[name='login'] input[name='username']"
      capture:
        username: [ "@value" ]
- request:
    url: http://example.com/login
    method: POST
    form:
      dst: https://duck.com
      popup: false
      username: <% username %>
      password: ****

I checked the headers in tcpdump but the Cookie: part is missing completely. How could I store these headers persistent?

Regards
Jonas

Right now, when calling a custom plugin/function asynchronously, if the callback is never invoked, the script does not handle this well. Would be nice to add some timeout capabilities.

Hey,
I would like to continue the execution of the script in case of an ENETUNREACH error. This occours when the requested target domain is not reachable.
In my example script, there is the possibility that a certain site sometimes is available at site1.example.org or site2.example.org. So I duplicated my script to target both endpoints. If one fail because the site is not reachable, it should continue execution and try to request the second site.
Best regards
Jonas

Support logging/trace output that more specifically shows the raw request/response details.

I believe this is related to an upstream issue mattcg/socks5-http-client#8. Waiting for resolution there.

I was trying to find a server that had an issue with a round robin DNS. Our changes had been rolled out to some nodes, but not others. I wrote a script that would keep making the request until it failed so we could pinpoint the problem, but it doesn't seem to be possible to get the IP address of the server contacted by the request in the response actions, so I had to resort to curl and bash.

For action elements, support a meta-information property that includes information such as the required plugin-version, plugin-url, descriptive text, etc. This could be used to override/force which plugin processes an action (to avoid conflicts). It can also include information/documentation to help users find/discover installable plugins if they are running a script from an external source.

Something like

- action_name: {actiony stuff here }
   on_success: ...
   on_failure: ...
   debugger: ...
   plugin:
     name: "plugin-name to override default binding/lookup"
     homepage: "Url for a page that contains information about the plugin"
     description: "Descriptive text about the plugin"
     version: "semver" or [ "semver1",..."semverN" ]  # Required semver spec that the plugin must match

I'm trying to write scripts that include other scripts that then include other scripts, but this causes the application to enter into an infinite loop. For example:

one.yaml

#!/usr/bin/env hyperpotamus

- print: This is file one

two.yaml

#!/usr/bin/env hyperpotamus

- !!inc/file ./one.yaml

- print: This is file two

three.yaml

#!/usr/bin/env hyperpotamus

- !!inc/file ./two.yaml

- print: This is file three

Executing either ./one.yaml or ./two.yaml work fine, but executing ./three.yaml causes the application to enter an infinite loop where it just prints This is file one over and over.

Hyperpotamus already supports emitting output text to multiple channels, although by default everything goes to the default channel. The hyperpotamus cli currently only supports the single (default) output channel, but add options to handle multiple channels to different locations. Any channels that are not specifically routed to a file would be displayed in stdout, but prefixed with the channel name.

The cli option --out <filename> already allows for the specification of a filename for the default channel, perhaps enhance this to support --out <channel>=<filename>.

Allow one or more maskable properties to be set that will not show up in error messages or dumps. This is useful for things like passwords and access_tokens that the user may want to prevent from being accidentally logged.

Any errors that are raised from hyperpotamus should include an error-code that can be cross-referenced to the wiki for documentation/tips/explanation.

I recently re-factored the set and defaults actions to be special cases of merge, however, upon further reflection, I'm not sure the enhanced behavior of lodash's merge functionality is always desired. I.e. if users are intending to replace an entire sub-object/tree, they are not expecting any existing properties in the session under that tree to be merged in. Let's leave set and defaults as more direct assignments and users can always use merge to combine things more inclusively.

When executing the function action, support referenced function definitions, either by pointing to a function defined in a session context property or by referencing another element elsewhere in the YAML script (I think this can already be done with YAML directives, but should be documented as a tip).

Currently, we have script.steps -> [ step ] and step.actions -> [ action ], so we have a 3-level hierarchy, script/step/action.

Part of the reason for this is so that top-level steps can have names and can be used a jump targets. If we convert steps objects from being a special case and instead think of them just like any other action (with a .actions property and nested actions), we can support a recursive structure. The main purpose of a step at that point would be to serve as a label/marker for a goto/jump activity. Right now, the only potential targets of a jump are the top-level step elements, so we would have to switch from tracking the current position in the script (and any jump targets) from a list to a tree/stack structure.

Right now, all hyperpotamus actions run on a single thread/CPU. For parallel/concurrent invocation (such as --concurrency with --loop and --csv options) spread out to consume multiple cores.

There are a few plugins that interact with the file-system for reading/writing. Anything that reads/writes to the file-system is already obviously marked as unsafe, but adding in some file-name sanitization seems like a wise choice to help protect systems.

Thanks for making and sharing this, @pmarkert, it looks fabulous.

I was about to embark on creating my own YAML-based scripting engine to generate modules/components for Site.js sites but you seem to have done nearly all the work for me (and then some) :)

I noticed, however, that lib/index.js is exposed as the main script but that the hyperpotamus.js script has quite a bit of important logic in it that I’d have to replicate if I tried to use Hyperpotamus as a library in my own CLI at the moment.

Would you be open to refactoring the project so that much of that functionality can be exposed either as a class or a function that can be instantiated by the Hyperpotamus CLI or by any other CLI app that uses Hyperpotamus as a library?

Thanks again for making this ;)

Currently the save action can save a file to disk, however, the entire response must be buffered in memory because of the way actions are processed. Support an optional request property that will cause the response to be saved directly to the specified location on disk. Needs to check whether safe_mode is enabled or not before allowing the feature. If the response is streamed to disk, it will obviously not be available for response actions to inspect, however, maybe a property can be set to indicate the location to where the file was saved. Actions can then print/display that property value if needed.

@pmarkert
Thanks a lot for your answer under pylot.
hyperpotamus is more convenient for me.
I read the doc, but still have three questions:

1. how to save the http response. I know there is action "save", but I tried some times, it failed:
   response:
   save:
   filename: "./result.txt"
   encoding: UTF-8
2. how to control the qps? like I want to run some test that qps = 300

3.how to send a lot requests at the same time when I try to do some performance test?
I need to do the test with different headers

Allow one server to be setup as a hyperpotamus master/controller server and other servers to be configured as slave servers. Then the master server dispatches processing requests to the remote servers for parallelization during load-testing scenarios.

If one of the fields being used by a csv action is a zero-length array, the loop is properly exited early, however, any partial parts of the line that had already been buffered are still emitted as a partial output line. There is a break statement in there to handle the case, but it also needs to set a temporary variable to skip the emit statement.

take a look at the following script:

- print: <% item1 %> <% item2 %> 

- name: promptForVariable
  callable:
    actions:
    - prompt:
       someVariable:
         message: Enter someVariable

and the following csv:

item1,item2
hello,world
hello,moon

when running the command hyperpotamus script.yaml --csv csv.csv, you get the expected output of

hello world
hello moon

However, when running hyperpotamus script.yaml --csv csv.csv --init promptForVariable, we get the following error:

hyperpotamus: Enter someVariable: test
[2020-03-05T10:35:32-05:00] [hyperpotamus.cli] [ERROR] - Error occurred while processing init script:
name: MissingKeyError
message: 'No matching value found in the session. Key: item1'
path: $.steps.0
help_link: 'https://github.com/pmarkert/hyperpotamus/wiki/errors#MissingKeyError'

This seems to work on version 0.36.5, but not on 0.37.2

Currently, before an action is processed, it is passed through the interpolation engine to replace any <%tokens%> . The interpolation engine does a recursive descent of all child properties, replacing any values that are tokenized. This is convenient for plugins when the entire state of the interpolation is available at the beginning of the action, however, for complex actions that allow for nested actions (such as "if") a user likely expects that multiple actions nested underneath the "if" or "else" branches, would be processed similarly to top-level actions. Unfortunately, if you have multiple actions under an if/else and subsequent actions depend upon interpolated values from the previous steps, the values have already been interpolated and replaced in the action tree before the if action even begins processing.

Here's an example:

- actions:
   - set:
       key: first value
   - if: true
     then:
      - set: 
          key: second value
      - emit: <% key %>

One might expect this script to emit "second value", however, the <% key %> token has already been replaced with the literal "first value" before the if action begins execution, and so "first value" is actually emitted instead.

Support a mechanism to include other .yaml files in hyperpotamus scripts. This could be used for a few things:

• Common sub-sections of code (examples)
  • a login flow that gets included in many other scripts)
  • work-item-api flow (i.e. common handling for asynchronous API responses with status/check-lookups)
  • Include a common set of defaults or properties (although this can also be done with --data)
  • Include common request_defaults, or default_actions elements.
  • Include common javascript functions

The include files will just be included directly into the parent script (as if they had been pre-processed and part of the master script inline), so there will be no special name-spacing, scoping or other allowances.

Allow context session properties to be explicitly removed with an unset action. The action should take a single path or an array of paths and delete any values from the context that match the path(s).

I have a script that looks like the following, but the auth_code variable is never set. I've tried several variations of the pattern field, but none of them seem to set a session variable.

- request:
    url: "<% login_form_target | resolve_url,`login_uri` %>"
    followRedirect: false
    method: POST
    form:
      userid: "<% username %>"
      password: "<% password %>"
  response:
    - status: 302
    - headers:
        location:
          pattern: '(:<auth_code>.)'
          # also
          #pattern: /(:<auth_code>.)/
          #pattern: '/(:<auth_code>.)/'
          #location: !!js/regex /(?<auth_code>.)/