Git Product home page Git Product logo

hdm's People

Contributors

alvagante avatar dan33l avatar dependabot[bot] avatar gguillotte avatar oneiros avatar phortx avatar rwaffen avatar tuxmea avatar wintermeyer avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar

hdm's Issues

add footer

I am still wondering whether we can have a real footer (at the bottom of the page).
The text I added is placed directly underneath the latest data.

YAML data types

  1. normal string, boolean, integer, float, hash, array
  2. pipe (e.g. file content)
  3. block syntax

Offer users a selector, which data type they would like to store.

Example for pipe:

long_pipe: |
  cluster.name: dev_cluster
  path.data: /var/lib/elasticsearch
  path.logs: /var/log/elasticsearch
  network.host: %{facts.fqdn}
  http.port: 9200
  discovery.seed_hosts: ["localhost"]
  cluster.initial_master_nodes: ["localhost"]
  action.destructive_requires_name: true

  xpack.security.enabled: true

Example for block:

long_block: >
    ENC[PKCS7,Y22exl+OvjDe+drmik2XEeD3VQtl1uZJXFFF2NnrMXDWx0csyqLB/2NOWefv
    NBTZfOlPvMlAesyr4bUY4I5XeVbVk38XKxeriH69EFAD4CahIZlC8lkE/uDh
    jJGQfh052eonkungHIcuGKY/5sEbbZl/qufjAtp/ufor15VBJtsXt17tXP4y
    l5ZP119Fwq8xiREGOL0lVvFYJz2hZc1ppPCNG5lwuLnTekXN/OazNYpf4CMd
    /HjZFXwcXRtTlzewJLc+/gox2IfByQRhsI/AgogRfYQKocZgFb/DOZoXR7wm
    IZGeunzwhqfmEtGiqpvJJQ5wVRdzJVpTnANBA5qxeA==]

allow usage of hiera lookup_options

Within a hiera hierarchy, one has the option to set the merge behavior using the key "lookup_options".

Example:
data/common.yaml

lookup_options:
  <key>:
    merge: <merge behavior>
  convert_to: <Data Type> # Usually "Sensitive"

The following merge behavior options are possible:

  • first - the default hiera behavior
  • unique - used on arrays
  • hash - merges all hash keys (on first level)
  • deep - similar to hash, but merges subkeys, too

The hiera key lookup_options should always be the very first in the list of keys (if set).

HDM should show the lookup_options providing a text message above the top hierarchy.

lookup option: merge: first, convert_to: <Data Type>

Note: lookup options can be set in every hiera hierarchy.
Updated test code is in mulitple_environmenrs.
Node sse8epsu.example42.training has lookup option for profile::auth::sshd_config_allowgroups set to 'first' and in common it is set to 'unique'.

Hiera uses deep merge on lookup_options.

e.g.

  1. Node specific:
lookup_options:
    <key1>:
     merge: deep
  <key2>:
    merge: first
    convert_to: 'Sensitive'
  1. common:
lookup_options:
    <key1>:
     merge: first

Allow filtering of hiera keys

After selecting a node, the list of all hiera keys is shown.
As this list might be very long: provide a filter field above the list of keys to liimit the keys shown.

read/write flag

At Puppetize PDX several attendees requested a read-only view on data.
Add a feature flag (set in hdm config file), which disables editing of values.

Cleanup test data

  • test/fixtures/files/puppet/hiera* - remove
  • expand environment test data (up to 1000 keys partly with long block entries or eyaml entries)
  • expand number of test nodes (500 nodes)

Check if test environments are used by travis CI tests!

Hiera lookup_options

Hiera has the option to set merge behavior via lookup_options. HDM should be able to parse these, too.

allow filtering of node names

In large environment one can have several thousands of nodes.
Instead of showing a pulldown menu, also allow to write node names and filter within the pulldown list.

Running on Passenger ist not working

I did:

  1. disable puppet http forwarding
  2. install passenger
  3. adopt selinux settings
  4. provide write access to user nobody to app tmp directory
  5. disabled default server in nginx.conf
  6. create nginx config file

starting nginx, accessing website causes error.

Steps for reproduction:

  1. Disabling http puppet redirect hiera data:
    puppet_enterprise::profile::console::proxy::http_redirect::enable_http_redirect: false

  2. selinux:

sealert  -a /var/log/audit/audit.log
/sbin/restorecon -v /opt/example42/hdm/public/favicon.ico
ausearch -c 'nginx' --raw | audit2allow -M hdm-nginx
semodule -i hdm-nginx.pp
  1. Nginx HDM config file:
server {
    listen 80 default_server;
    server_name hdm.demo.example42.training;

    # Tell Nginx and Passenger where your app's 'public' directory is
    root /opt/example42/hdm/public;

    # Turn on Passenger
    passenger_enabled on;
    passenger_ruby /opt/puppetlabs/puppet/bin/ruby;
    passenger_friendly_error_pages on;
    passenger_env_var HDM__CONFIG_DIR /etc/puppetlabs/code/environments;
    passenger_env_var HDM__PUPPET_DB__ENABLED true;
    passenger_env_var HDM__PUPPET_DB__SELF_SIGNED_CERT true;
    passenger_env_var HDM__PUPPET_DB__TOKEN 0e-ZhnlpGdpEM9_Yh1zQzHef4BzQdvbnZSOQZrVWCW7A;
    passenger_env_var HDM__PUPPET_DB__SERVER https://localhost:8081;
    passenger_env_var PATH /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/opt/puppetlabs/puppet/bin;
}

Error message from passenger:

Error: The application encountered the following error: exit (SystemExit)
    /opt/example42/hdm/vendor/ruby/2.5.0/gems/webpacker-4.0.2/lib/webpacker/railtie.rb:48:in `exit'
    /opt/example42/hdm/vendor/ruby/2.5.0/gems/webpacker-4.0.2/lib/webpacker/railtie.rb:48:in `block in <class:Engine>'
    /opt/example42/hdm/vendor/ruby/2.5.0/gems/railties-5.2.3/lib/rails/initializable.rb:32:in `instance_exec'
    /opt/example42/hdm/vendor/ruby/2.5.0/gems/railties-5.2.3/lib/rails/initializable.rb:32:in `run'
    /opt/example42/hdm/vendor/ruby/2.5.0/gems/railties-5.2.3/lib/rails/initializable.rb:61:in `block in run_initializers'
    /opt/puppetlabs/puppet/lib/ruby/2.5.0/tsort.rb:228:in `block in tsort_each'
    /opt/puppetlabs/puppet/lib/ruby/2.5.0/tsort.rb:350:in `block (2 levels) in each_strongly_connected_component'
    /opt/puppetlabs/puppet/lib/ruby/2.5.0/tsort.rb:431:in `each_strongly_connected_component_from'
    /opt/puppetlabs/puppet/lib/ruby/2.5.0/tsort.rb:349:in `block in each_strongly_connected_component'
    /opt/puppetlabs/puppet/lib/ruby/2.5.0/tsort.rb:347:in `each'
    /opt/puppetlabs/puppet/lib/ruby/2.5.0/tsort.rb:347:in `call'
    /opt/puppetlabs/puppet/lib/ruby/2.5.0/tsort.rb:347:in `each_strongly_connected_component'
    /opt/puppetlabs/puppet/lib/ruby/2.5.0/tsort.rb:226:in `tsort_each'
    /opt/puppetlabs/puppet/lib/ruby/2.5.0/tsort.rb:205:in `tsort_each'
    /opt/example42/hdm/vendor/ruby/2.5.0/gems/railties-5.2.3/lib/rails/initializable.rb:60:in `run_initializers'
    /opt/example42/hdm/vendor/ruby/2.5.0/gems/railties-5.2.3/lib/rails/application.rb:361:in `initialize!'
    /opt/example42/hdm/config/environment.rb:5:in `<top (required)>'
    config.ru:3:in `require_relative'
    config.ru:3:in `block in <main>'
    /opt/example42/hdm/vendor/ruby/2.5.0/gems/rack-2.0.8/lib/rack/builder.rb:55:in `instance_eval'
    /opt/example42/hdm/vendor/ruby/2.5.0/gems/rack-2.0.8/lib/rack/builder.rb:55:in `initialize'
    config.ru:1:in `new'
    config.ru:1:in `<main>'
    /usr/share/passenger/helper-scripts/rack-preloader.rb:101:in `eval'
    /usr/share/passenger/helper-scripts/rack-preloader.rb:101:in `preload_app'
    /usr/share/passenger/helper-scripts/rack-preloader.rb:189:in `block in <module:App>'
    /usr/share/ruby/vendor_ruby/phusion_passenger/loader_shared_helpers.rb:380:in `run_block_and_record_step_progress'
    /usr/share/passenger/helper-scripts/rack-preloader.rb:188:in `<module:App>'
    /usr/share/passenger/helper-scripts/rack-preloader.rb:30:in `<module:PhusionPassenger>'
    /usr/share/passenger/helper-scripts/rack-preloader.rb:29:in `<main>'

Manage eyaml data

  • show encrypted values as is. add a button on the data to show plain text. (decrypt)
  • if eyaml is configured: add an encrypt button to save encrypted data. (encrypt)

Prepared testing environment (eyaml).
Provide a config switch, if we want to decrypt values in hdm.conf.

EYaml is configured in hiera.yaml file, by setting the lookup_key to eyaml_lookup and specifying options for private and public key.

If the private key is readable by HDM: provide the option to "decrypt" the data.

Preparation for storing data:

  • allow users to encrypt data.

If a user decrypts a value, sets a new one and wants to save it in plain text: provide a warning, and the user must confirm to store the value.

Please consider that we might have usermanagement RBAC where we want to limit users to decrypt values.

Have a page where to look how values differ for a given key

Feature request: Create a page where to pass as parameter in the the name of an hiera key to analyse (for easy links wherever this key is shown) which shows what are the values for this key on all the nodes.
Ideally the list should be compressed to group nodes which have the same value and show information of the affected node (how many nodes have a given value) and possibly an option to expand them.

It would be nice, also, in the same page or another, to show where, for a given node, there's a duplicated and redundant entry which can be removed. For example where the same key with the same value is in common.yaml and in a node specific yaml, in this case the node specific entry is redundant and there should be a button to quickly remove it.

  • point: Design in a way that fits cases where the number of nodes could be in the order of a few thousand.

NoMethodError in KeysController#show if key is an integer

HDM running against a new PE 2019.2.2 installation, with issue #9 resolved by PR #10.

Steps to reproduce:

  1. Perform a clean install of PE 2019.2.2 and HDM on the same system.
  2. Tune puppet_enterprise::master::puppetserver::jruby_max_active_instances: 1 in $codedir/environments/production/data/common.yaml.
  3. Open HDM.
  4. Select production for "Choose the environment".
  5. Select the PE master node for "Choose the node".
  6. Select puppet_enterprise::master::puppetserver::jruby_max_active_instances as the key.

Expected behavior:

HDM displays its value (1 in the case of this lab system).

Observed behavior:

Error:

undefined method `gsub!' for nil:NilClass

Full function:

    def content_for_key(key)
      return nil unless content
      return nil unless content.has_key?(key)

      value = content[key]
      return "true" if value == true
      return "false" if value == false
      return value if value.is_a?(String)

      value_string = value.to_yaml
      value_string.gsub!("---\n", '').gsub!(/^$\n/, '')
      value_string
    end

Flagged line is:

      value_string.gsub!("---\n", '').gsub!(/^$\n/, '')

Rails.root:

/root/hdm

App trace:

app/models/hiera/read_file.rb:37:in `content_for_key'
app/models/hiera.rb:55:in `block (2 levels) in search_key'
app/models/hiera.rb:48:in `map'
app/models/hiera.rb:48:in `block in search_key'
app/models/hiera.rb:46:in `each_pair'
app/models/hiera.rb:46:in `search_key'
app/concepts/keys/operation/show.rb:28:in `node_data!'

Request parameters:

{"on"=>:member, "environment_id"=>"production", "node_id"=>"master.test.vlan", "id"=>"puppet_enterprise::master::puppetserver::jruby_max_active_instances"}

--

Web console output:

>> value

=> 1

>> value_string

=> "--- 1\n"

value_string is the result of value.to_yaml. If the value were a String instead of an Integer, return value if value.is_a?(String) would return before this.

The behavior of Integer.to_yaml:

>> 1.to_yaml

=> "--- 1\n"

>> 10.to_yaml

=> "--- 10\n"

>> "1".to_yaml

=> "--- '1'\n"

Since "--- 1" is apparently expected from 1.to_yaml, then:

>> value_string.gsub!("---\n", '').gsub!(/^$\n/, '')

NoMethodError: undefined method `gsub!' for nil:NilClass
	from /root/hdm/app/models/hiera/read_file.rb:37:in `content_for_key'

"---\n" matches nothing in "--- 1". gsub! returns nil if there's no match, so nil is also expected. We could convert the first result to a String/.to_s, but it feels like the integer value would still be lost unless the match itself were changed.

For reference:

>> content

=> {"puppet_enterprise::profile::amq::broker::heap_mb"=>"96", "puppet_enterprise::master::puppetserver::jruby_max_active_instances"=>1, "puppet_enterprise::master::puppetserver::reserved_code_cache"=>"96m", "puppet_enterprise::profile::master::java_args"=>{"Xmx"=>"384m", "Xms"=>"128m", "XX:MaxPermSize"=>"=96m", "XX:PermSize"=>"=64m", "XX:+UseG1GC"=>""}, "puppet_enterprise::profile::puppetdb::java_args"=>{"Xmx"=>"128m", "Xms"=>"64m", "XX:MaxPermSize"=>"=96m", "XX:PermSize"=>"=64m", "XX:+UseG1GC"=>""}, "puppet_enterprise::puppetdb::read_maximum_pool_size"=>4, "puppet_enterprise::puppetdb::write_maximum_pool_size"=>2, "puppet_enterprise::profile::console::java_args"=>{"Xmx"=>"128m", "Xms"=>"64m", "XX:MaxPermSize"=>"=96m", "XX:PermSize"=>"=64m", "XX:+UseG1GC"=>""}, "puppet_enterprise::trapperkeeper::database_settings::activity::maximum_pool_size"=>2, "puppet_enterprise::trapperkeeper::database_settings::classifier::maximum_pool_size"=>2, "puppet_enterprise::trapperkeeper::database_settings::rbac::maximum_pool_size"=>2, "puppet_enterprise::profile::console::delayed_job_workers"=>1, "puppet_enterprise::profile::database::shared_buffers"=>"4MB", "puppet_enterprise::profile::orchestrator::java_args"=>{"Xmx"=>"128m", "Xms"=>"128m", "XX:+UseG1GC"=>""}}

There are several integer values here, and they would be common in many PE installs. I'd expect any of them to fail.

Attempting to return an Integer value prevents the error when attempting to display the key's value when selecting it in HDM:

--- a/app/models/hiera/read_file.rb
+++ b/app/models/hiera/read_file.rb
@@ -31,7 +31,7 @@ class Hiera
       value = content[key]
       return "true" if value == true
       return "false" if value == false
-      return value if value.is_a?(String)
+      return value if value.is_a?(String) || value.is_a?(Integer)

But after making this change, then editing the selected value in HDM's interface (for instance from 1 to 2 in this example), HDM appends the new value to a new key in data/common.yaml with the same name but .json suffixed, instead of overwriting the existing key's value.

# grep -R max_active_instances $(puppet config print codedir)
...
/etc/puppetlabs/code/environments/production/data/common.yaml:puppet_enterprise::master::puppetserver::jruby_max_active_instances: 1
/etc/puppetlabs/code/environments/production/data/common.yaml:puppet_enterprise::master::puppetserver::jruby_max_active_instances.json: 2

HDM config file instead of environment variables

HDM is configurable by setting environmetn variables. We want HDM to be configurable via a config file.

/etc/hdm/hdm.conf (ini-File or YAML)

  • HDM Webserver Port
  • Use PuppetDB?
  • PuppetDB Access (URL and Token (if PE))
  • environment_path (Default puppet config print)
  • PuppetDB Self Signed Cert?
  • HDM Listen Address
  • HDM Read Only/Read-Write Flag

HDM file storage backend

HDM in read/write mode must be able to store the new data. Data path must be in hiera yaml file and in hdm config file.

  • Config in hdm.conf (hdm_data_backend = dir .... hdm_data_dir = /etc/hdm/data)
  • check: no .git dir may be in data dir

Change values result in error

Webinterface shows "Oops, something went wrong"
Changed data are written back to file.

Application throws the following error message:

NoMethodError (undefined method `persisted?' for nil:NilClass):
  
vendor/ruby/2.5.0/gems/actionpack-5.2.3/lib/action_dispatch/routing/polymorphic_routes.rb:266:in `handle_model'
vendor/ruby/2.5.0/gems/actionpack-5.2.3/lib/action_dispatch/routing/polymorphic_routes.rb:280:in `handle_model_call'
vendor/ruby/2.5.0/gems/actionpack-5.2.3/lib/action_dispatch/routing/url_for.rb:190:in `full_url_for'
vendor/ruby/2.5.0/gems/actionpack-5.2.3/lib/action_dispatch/routing/url_for.rb:169:in `url_for'
vendor/ruby/2.5.0/gems/actionpack-5.2.3/lib/action_controller/metal/redirecting.rb:110:in `_compute_redirect_to_location'
vendor/ruby/2.5.0/gems/actionpack-5.2.3/lib/action_controller/metal/redirecting.rb:63:in `redirect_to'
vendor/ruby/2.5.0/gems/actionpack-5.2.3/lib/action_controller/metal/flash.rb:58:in `redirect_to'
vendor/ruby/2.5.0/gems/actionpack-5.2.3/lib/action_controller/metal/instrumentation.rb:66:in `block in redirect_to'
vendor/ruby/2.5.0/gems/activesupport-5.2.3/lib/active_support/notifications.rb:168:in `block in instrument'
vendor/ruby/2.5.0/gems/activesupport-5.2.3/lib/active_support/notifications/instrumenter.rb:23:in `instrument'
vendor/ruby/2.5.0/gems/activesupport-5.2.3/lib/active_support/notifications.rb:168:in `instrument'
vendor/ruby/2.5.0/gems/actionpack-5.2.3/lib/action_controller/metal/instrumentation.rb:65:in `redirect_to'
vendor/ruby/2.5.0/gems/turbolinks-5.2.0/lib/turbolinks/redirection.rb:12:in `redirect_to'
vendor/ruby/2.5.0/gems/reativo-0.1.5/app/controllers/reativo/crud_controller.rb:101:in `block (2 levels) in update'
vendor/ruby/2.5.0/gems/actionpack-5.2.3/lib/action_controller/metal/mime_responds.rb:203:in `respond_to'
vendor/ruby/2.5.0/gems/reativo-0.1.5/app/controllers/reativo/crud_controller.rb:99:in `update'
vendor/ruby/2.5.0/gems/actionpack-5.2.3/lib/action_controller/metal/basic_implicit_render.rb:6:in `send_action'
vendor/ruby/2.5.0/gems/actionpack-5.2.3/lib/abstract_controller/base.rb:194:in `process_action'
vendor/ruby/2.5.0/gems/actionpack-5.2.3/lib/action_controller/metal/rendering.rb:30:in `process_action'
vendor/ruby/2.5.0/gems/actionpack-5.2.3/lib/abstract_controller/callbacks.rb:42:in `block in process_action'
vendor/ruby/2.5.0/gems/activesupport-5.2.3/lib/active_support/callbacks.rb:109:in `block in run_callbacks'
vendor/ruby/2.5.0/gems/react-rails-2.5.0/lib/react/rails/controller_lifecycle.rb:31:in `use_react_component_helper'
vendor/ruby/2.5.0/gems/activesupport-5.2.3/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
vendor/ruby/2.5.0/gems/activesupport-5.2.3/lib/active_support/callbacks.rb:136:in `run_callbacks'
vendor/ruby/2.5.0/gems/actionpack-5.2.3/lib/abstract_controller/callbacks.rb:41:in `process_action'
vendor/ruby/2.5.0/gems/actionpack-5.2.3/lib/action_controller/metal/rescue.rb:22:in `process_action'
vendor/ruby/2.5.0/gems/actionpack-5.2.3/lib/action_controller/metal/instrumentation.rb:34:in `block in process_action'
vendor/ruby/2.5.0/gems/activesupport-5.2.3/lib/active_support/notifications.rb:168:in `block in instrument'
vendor/ruby/2.5.0/gems/activesupport-5.2.3/lib/active_support/notifications/instrumenter.rb:23:in `instrument'
vendor/ruby/2.5.0/gems/activesupport-5.2.3/lib/active_support/notifications.rb:168:in `instrument'
vendor/ruby/2.5.0/gems/actionpack-5.2.3/lib/action_controller/metal/instrumentation.rb:32:in `process_action'
vendor/ruby/2.5.0/gems/actionpack-5.2.3/lib/action_controller/metal/params_wrapper.rb:256:in `process_action'
vendor/ruby/2.5.0/gems/activerecord-5.2.3/lib/active_record/railties/controller_runtime.rb:24:in `process_action'
vendor/ruby/2.5.0/gems/actionpack-5.2.3/lib/abstract_controller/base.rb:134:in `process'
vendor/ruby/2.5.0/gems/actionview-5.2.3/lib/action_view/rendering.rb:32:in `process'
vendor/ruby/2.5.0/gems/actionpack-5.2.3/lib/action_controller/metal.rb:191:in `dispatch'
vendor/ruby/2.5.0/gems/actionpack-5.2.3/lib/action_controller/metal.rb:252:in `dispatch'
vendor/ruby/2.5.0/gems/actionpack-5.2.3/lib/action_dispatch/routing/route_set.rb:52:in `dispatch'
vendor/ruby/2.5.0/gems/actionpack-5.2.3/lib/action_dispatch/routing/route_set.rb:34:in `serve'
vendor/ruby/2.5.0/gems/actionpack-5.2.3/lib/action_dispatch/journey/router.rb:52:in `block in serve'
vendor/ruby/2.5.0/gems/actionpack-5.2.3/lib/action_dispatch/journey/router.rb:35:in `each'
vendor/ruby/2.5.0/gems/actionpack-5.2.3/lib/action_dispatch/journey/router.rb:35:in `serve'
vendor/ruby/2.5.0/gems/actionpack-5.2.3/lib/action_dispatch/routing/route_set.rb:840:in `call'
vendor/ruby/2.5.0/gems/warden-1.2.8/lib/warden/manager.rb:36:in `block in call'
vendor/ruby/2.5.0/gems/warden-1.2.8/lib/warden/manager.rb:34:in `catch'
vendor/ruby/2.5.0/gems/warden-1.2.8/lib/warden/manager.rb:34:in `call'
vendor/ruby/2.5.0/gems/rack-2.0.8/lib/rack/tempfile_reaper.rb:15:in `call'
vendor/ruby/2.5.0/gems/rack-2.0.8/lib/rack/etag.rb:25:in `call'
vendor/ruby/2.5.0/gems/rack-2.0.8/lib/rack/conditional_get.rb:38:in `call'
vendor/ruby/2.5.0/gems/rack-2.0.8/lib/rack/head.rb:12:in `call'
vendor/ruby/2.5.0/gems/actionpack-5.2.3/lib/action_dispatch/http/content_security_policy.rb:18:in `call'
vendor/ruby/2.5.0/gems/rack-2.0.8/lib/rack/session/abstract/id.rb:259:in `context'
vendor/ruby/2.5.0/gems/rack-2.0.8/lib/rack/session/abstract/id.rb:253:in `call'
vendor/ruby/2.5.0/gems/actionpack-5.2.3/lib/action_dispatch/middleware/cookies.rb:670:in `call'
vendor/ruby/2.5.0/gems/activerecord-5.2.3/lib/active_record/migration.rb:559:in `call'
vendor/ruby/2.5.0/gems/actionpack-5.2.3/lib/action_dispatch/middleware/callbacks.rb:28:in `block in call'
vendor/ruby/2.5.0/gems/activesupport-5.2.3/lib/active_support/callbacks.rb:98:in `run_callbacks'
vendor/ruby/2.5.0/gems/actionpack-5.2.3/lib/action_dispatch/middleware/callbacks.rb:26:in `call'
vendor/ruby/2.5.0/gems/actionpack-5.2.3/lib/action_dispatch/middleware/executor.rb:14:in `call'
vendor/ruby/2.5.0/gems/actionpack-5.2.3/lib/action_dispatch/middleware/debug_exceptions.rb:61:in `call'
vendor/ruby/2.5.0/gems/web-console-3.7.0/lib/web_console/middleware.rb:135:in `call_app'
vendor/ruby/2.5.0/gems/web-console-3.7.0/lib/web_console/middleware.rb:22:in `block in call'
vendor/ruby/2.5.0/gems/web-console-3.7.0/lib/web_console/middleware.rb:20:in `catch'
vendor/ruby/2.5.0/gems/web-console-3.7.0/lib/web_console/middleware.rb:20:in `call'
vendor/ruby/2.5.0/gems/actionpack-5.2.3/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
vendor/ruby/2.5.0/gems/railties-5.2.3/lib/rails/rack/logger.rb:38:in `call_app'
vendor/ruby/2.5.0/gems/railties-5.2.3/lib/rails/rack/logger.rb:26:in `block in call'
vendor/ruby/2.5.0/gems/activesupport-5.2.3/lib/active_support/tagged_logging.rb:71:in `block in tagged'
vendor/ruby/2.5.0/gems/activesupport-5.2.3/lib/active_support/tagged_logging.rb:28:in `tagged'
vendor/ruby/2.5.0/gems/activesupport-5.2.3/lib/active_support/tagged_logging.rb:71:in `tagged'
vendor/ruby/2.5.0/gems/railties-5.2.3/lib/rails/rack/logger.rb:26:in `call'
vendor/ruby/2.5.0/gems/sprockets-rails-3.2.1/lib/sprockets/rails/quiet_assets.rb:13:in `call'
vendor/ruby/2.5.0/gems/actionpack-5.2.3/lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
vendor/ruby/2.5.0/gems/actionpack-5.2.3/lib/action_dispatch/middleware/request_id.rb:27:in `call'
vendor/ruby/2.5.0/gems/rack-2.0.8/lib/rack/method_override.rb:22:in `call'
vendor/ruby/2.5.0/gems/rack-2.0.8/lib/rack/runtime.rb:22:in `call'
vendor/ruby/2.5.0/gems/activesupport-5.2.3/lib/active_support/cache/strategy/local_cache_middleware.rb:29:in `call'
vendor/ruby/2.5.0/gems/actionpack-5.2.3/lib/action_dispatch/middleware/executor.rb:14:in `call'
vendor/ruby/2.5.0/gems/actionpack-5.2.3/lib/action_dispatch/middleware/static.rb:127:in `call'
vendor/ruby/2.5.0/gems/rack-2.0.8/lib/rack/sendfile.rb:111:in `call'
vendor/ruby/2.5.0/gems/webpacker-4.0.2/lib/webpacker/dev_server_proxy.rb:29:in `perform_request'
vendor/ruby/2.5.0/gems/rack-proxy-0.6.5/lib/rack/proxy.rb:57:in `call'
vendor/ruby/2.5.0/gems/railties-5.2.3/lib/rails/engine.rb:524:in `call'
vendor/ruby/2.5.0/gems/puma-3.12.2/lib/puma/configuration.rb:227:in `call'
vendor/ruby/2.5.0/gems/puma-3.12.2/lib/puma/server.rb:674:in `handle_request'
vendor/ruby/2.5.0/gems/puma-3.12.2/lib/puma/server.rb:476:in `process_client'
vendor/ruby/2.5.0/gems/puma-3.12.2/lib/puma/server.rb:334:in `block in run'
vendor/ruby/2.5.0/gems/puma-3.12.2/lib/puma/thread_pool.rb:135:in `block in spawn_thread'

User Access

At the moment there is no login in front of hdm. We want to be able to manage users (locally)

  • simple local user management (config file: /etc/hdm/users.yaml|txt)

Docker Support?

Do you think it is possible to run this project in docker?

Enhance test data

  1. Add more data, especially large data (long lines, blocks, eyaml content)
  2. Add many many many keys.
  3. Add new test nodes (with dot)

Data: test/fixtures/files/puppet/environments/
Nodes and Facts: test/fixtures/files/puppet/nodes/

Prepare for beta release

The following should not be part of the git repo.
This information must be added to a document describing the way how to package HDM.

Rails uses different environments. At the moment we use development.
For packaging, we need production environment.

  1. Compile JS/CSS:
  • bundle exec rails assets:precompile
  1. Ruby Gems:
  • bundle install --deployment
  1. Start check:
  • required ruby version?
  • bundler installed?
  • switch to hdm base dir
  • set rails environment
  • create database (db:create, db:migrate und db:seed) db:setup
  • start webserver
  1. Rails:
    bundle exec rails server -e

reading hiera settings

  1. a hiera.yaml file is fully optional.
    If no file is available, hiera uses internal defaults

  2. HDM expects hiera data to be in environmentpath/data
    But data can be set using hiera.yaml file:

---
version: 5
defaults:
  datadir: modules/hieradata/data

Defaults are in hiera ruby gem:
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/hiera/config.rb

class Hiera::Config
  class << self
    ##
    # load takes a string or hash as input, strings are treated as filenames
    # hashes are stored as data that would have been in the config file
    #
    # Unless specified it will only use YAML as backend with a
    # hierarchy of 'nodes/%{::trusted.certname}' and 'common', and with a
    # console logger.
    #
    # @return [Hash] representing the configuration.
    def load(source)
      @config = {:backends => ["yaml"],
                 :hierarchy => ["nodes/%{::trusted.certname}", "common"],
                 :merge_behavior => :native }

      if source.is_a?(String)
        if File.exist?(source)
          config = begin
                     yaml_load_file(source)
                   rescue TypeError => detail
                     case detail.message
                     when /no implicit conversion from nil to integer/
                       false

File config/credentials.yml.enc should not be in git repo?

To verify with a Rails expert:
Should file config/credentials.yml.enc be kept in the repo or removed and added to .gitignore?
It can be regenerated with rails:credentials:edit
Maybe describe in setup instructions or via the Puppet module.

NoMethodError in NodesController#show if defaults is nil

HDM running against a new PE 2019.2.2 installation.

Steps to reproduce:

  1. Perform a clean install of PE 2019.2.2 and HDM on the same system. No control repo is configured, and no user code is added.
  2. Open HDM.
  3. Select production for "Choose the environment".
  4. Select the PE master node for "Choose the node".

Expected behavior:

The default PE keys for that node appear.

Observed behavior:

Error:

undefined method `has_key?' for nil:NilClass

Extracted source reported:

    def default_yaml_data?
      defaults.has_key?("data_hash") && defaults["data_hash"] == "yaml_data"
    end

Rails.root:

/root/hdm

App trace:

app/models/hiera/config_file.rb:15:in `default_yaml_data?'
app/models/hiera/config_file.rb:25:in `block in initialize_environments'
app/models/hiera/config_file.rb:24:in `map'
app/models/hiera/config_file.rb:24:in `initialize_environments'
app/models/hiera/config_file.rb:7:in `initialize'
app/models/hiera.rb:85:in `new'
app/models/hiera.rb:85:in `config_file'
app/models/hiera.rb:14:in `paths'
app/models/hiera.rb:27:in `all_keys'
app/concepts/nodes/operation/show.rb:21:in `node_data!'

Request parameters:

{"on"=>:member, "environment_id"=>"production", "id"=>"master.test.vlan"}

--

defaults is defined in Hiera::ConfigFile as:

    def defaults
      content["defaults"]
    end

Ruby web console inspection of content after the error:

>> content

=> {"version"=>5, "defaults"=>nil, "hierarchy"=>[{"name"=>"Per-node data (yaml version)", "path"=>"nodes/%{::trusted.certname}.yaml"}, {"name"=>"Other YAML hierarchy levels", "paths"=>["common.yaml"]}]}

content is defined in app/models/hiera.rb:

    def content
      @content ||= YAML.load(
        File.read(Pathname.new(Settings.config_dir).join(environment, "hiera.yaml"))
      )
    end

It reads the environment's hiera.yaml:

      @content ||= YAML.load(
        File.read(Pathname.new(Settings.config_dir).join(environment, "hiera.yaml"))
      )

./code/environments/production/hiera.yaml with comments removed:

---
version: 5
defaults:
hierarchy:
  - name: "Per-node data (yaml version)"
    path: "nodes/%{::trusted.certname}.yaml"
  - name: "Other YAML hierarchy levels"
    paths:
      - "common.yaml"

defaults has no value, so nil is expected.

content is consumed in initialize_environments:

    def initialize_environments
      return @environments = [] unless content.has_key?("hierarchy")
      @environments = content['hierarchy'].map do |hierarchy|
        default_data_hash = default_yaml_data? ? 'yaml_data' : nil
        Hierarchy.new(hierarchy: hierarchy, default_data_hash: default_data_hash)
      end
    end

So if default_yaml_data is truthy, return 'yaml_data'. Otherwise, return nil.

The problem appears to be that def default_yaml_data? attempts to use has_key on a nil-value defaults and fails. def defaults results in nil if content["defaults"] is nil, as it is in the above hiera.yaml.

Therefore, def default_yaml_data? should at least check if defaults itself is truthy/exists before attempting to use methods on it.

    def default_yaml_data?
      defaults && defaults.has_key?("data_hash") && defaults["data_hash"] == "yaml_data"
    end

automate HDM installation using Puppet

Configure the following options:

  • config/hdm.yaml
    • PuppetDB URL
    • Access (Cert or PE Token)
    • read_only
    • config_dir (= puppet codedir setting)
  • Nginx Proxy (PE)
  • Nginx or Apache Proxy (for remote installation or open source installation) - add to documentation (with Puppet code example)
  • archive fetch and extract
  • hdm user and group
  • systemd unit file und servcie hdm (without daemon_reload if using systemd module)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.