example42 / hdm Goto Github PK

View Code? Open in Web Editor NEW

17.0 17.0 3.0 2.41 MB

Hiera Data Manager

Ruby 74.89% JavaScript 4.63% CSS 0.58% HTML 17.73% Dockerfile 0.48% SCSS 1.55% Shell 0.13%

hiera puppet

hdm's People

Contributors

Stargazers

Watchers

Forkers

dan33l isabella232 tuxmea

hdm's Issues

Cleanup test data

test/fixtures/files/puppet/hiera* - remove
expand environment test data (up to 1000 keys partly with long block entries or eyaml entries)
expand number of test nodes (500 nodes)

Check if test environments are used by travis CI tests!

NoMethodError in NodesController#show if defaults is nil

HDM running against a new PE 2019.2.2 installation.

Steps to reproduce:

Perform a clean install of PE 2019.2.2 and HDM on the same system. No control repo is configured, and no user code is added.
Open HDM.
Select production for "Choose the environment".
Select the PE master node for "Choose the node".

Expected behavior:

The default PE keys for that node appear.

Observed behavior:

Error:

undefined method `has_key?' for nil:NilClass

Extracted source reported:

    def default_yaml_data?
      defaults.has_key?("data_hash") && defaults["data_hash"] == "yaml_data"
    end

Rails.root:

/root/hdm

App trace:

app/models/hiera/config_file.rb:15:in `default_yaml_data?'
app/models/hiera/config_file.rb:25:in `block in initialize_environments'
app/models/hiera/config_file.rb:24:in `map'
app/models/hiera/config_file.rb:24:in `initialize_environments'
app/models/hiera/config_file.rb:7:in `initialize'
app/models/hiera.rb:85:in `new'
app/models/hiera.rb:85:in `config_file'
app/models/hiera.rb:14:in `paths'
app/models/hiera.rb:27:in `all_keys'
app/concepts/nodes/operation/show.rb:21:in `node_data!'

Request parameters:

{"on"=>:member, "environment_id"=>"production", "id"=>"master.test.vlan"}

defaults is defined in Hiera::ConfigFile as:

    def defaults
      content["defaults"]
    end

Ruby web console inspection of content after the error:

>> content

=> {"version"=>5, "defaults"=>nil, "hierarchy"=>[{"name"=>"Per-node data (yaml version)", "path"=>"nodes/%{::trusted.certname}.yaml"}, {"name"=>"Other YAML hierarchy levels", "paths"=>["common.yaml"]}]}

content is defined in app/models/hiera.rb:

    def content
      @content ||= YAML.load(
        File.read(Pathname.new(Settings.config_dir).join(environment, "hiera.yaml"))
      )
    end

It reads the environment's hiera.yaml:

      @content ||= YAML.load(
        File.read(Pathname.new(Settings.config_dir).join(environment, "hiera.yaml"))
      )

./code/environments/production/hiera.yaml with comments removed:

---
version: 5
defaults:
hierarchy:
  - name: "Per-node data (yaml version)"
    path: "nodes/%{::trusted.certname}.yaml"
  - name: "Other YAML hierarchy levels"
    paths:
      - "common.yaml"

defaults has no value, so nil is expected.

content is consumed in initialize_environments:

    def initialize_environments
      return @environments = [] unless content.has_key?("hierarchy")
      @environments = content['hierarchy'].map do |hierarchy|
        default_data_hash = default_yaml_data? ? 'yaml_data' : nil
        Hierarchy.new(hierarchy: hierarchy, default_data_hash: default_data_hash)
      end
    end

So if default_yaml_data is truthy, return 'yaml_data'. Otherwise, return nil.

The problem appears to be that def default_yaml_data? attempts to use has_key on a nil-value defaults and fails. def defaults results in nil if content["defaults"] is nil, as it is in the above hiera.yaml.

Therefore, def default_yaml_data? should at least check if defaults itself is truthy/exists before attempting to use methods on it.

    def default_yaml_data?
      defaults && defaults.has_key?("data_hash") && defaults["data_hash"] == "yaml_data"
    end

HDM config file instead of environment variables

HDM is configurable by setting environmetn variables. We want HDM to be configurable via a config file.

/etc/hdm/hdm.conf (ini-File or YAML)

HDM Webserver Port
Use PuppetDB?
PuppetDB Access (URL and Token (if PE))
environment_path (Default puppet config print)
PuppetDB Self Signed Cert?
HDM Listen Address
HDM Read Only/Read-Write Flag

nodejs 8 is EOL

You are using nodejs 8. It is EOL since 2020-01-01.
See https://nodejs.org/en/about/releases/

Adopt rails6 branch to layout vizualization of web site according to old HDM

Restore layout from master branch in rails6 branch

Change values result in error

Webinterface shows "Oops, something went wrong"
Changed data are written back to file.

Application throws the following error message:

NoMethodError (undefined method `persisted?' for nil:NilClass):
  
vendor/ruby/2.5.0/gems/actionpack-5.2.3/lib/action_dispatch/routing/polymorphic_routes.rb:266:in `handle_model'
vendor/ruby/2.5.0/gems/actionpack-5.2.3/lib/action_dispatch/routing/polymorphic_routes.rb:280:in `handle_model_call'
vendor/ruby/2.5.0/gems/actionpack-5.2.3/lib/action_dispatch/routing/url_for.rb:190:in `full_url_for'
vendor/ruby/2.5.0/gems/actionpack-5.2.3/lib/action_dispatch/routing/url_for.rb:169:in `url_for'
vendor/ruby/2.5.0/gems/actionpack-5.2.3/lib/action_controller/metal/redirecting.rb:110:in `_compute_redirect_to_location'
vendor/ruby/2.5.0/gems/actionpack-5.2.3/lib/action_controller/metal/redirecting.rb:63:in `redirect_to'
vendor/ruby/2.5.0/gems/actionpack-5.2.3/lib/action_controller/metal/flash.rb:58:in `redirect_to'
vendor/ruby/2.5.0/gems/actionpack-5.2.3/lib/action_controller/metal/instrumentation.rb:66:in `block in redirect_to'
vendor/ruby/2.5.0/gems/activesupport-5.2.3/lib/active_support/notifications.rb:168:in `block in instrument'
vendor/ruby/2.5.0/gems/activesupport-5.2.3/lib/active_support/notifications/instrumenter.rb:23:in `instrument'
vendor/ruby/2.5.0/gems/activesupport-5.2.3/lib/active_support/notifications.rb:168:in `instrument'
vendor/ruby/2.5.0/gems/actionpack-5.2.3/lib/action_controller/metal/instrumentation.rb:65:in `redirect_to'
vendor/ruby/2.5.0/gems/turbolinks-5.2.0/lib/turbolinks/redirection.rb:12:in `redirect_to'
vendor/ruby/2.5.0/gems/reativo-0.1.5/app/controllers/reativo/crud_controller.rb:101:in `block (2 levels) in update'
vendor/ruby/2.5.0/gems/actionpack-5.2.3/lib/action_controller/metal/mime_responds.rb:203:in `respond_to'
vendor/ruby/2.5.0/gems/reativo-0.1.5/app/controllers/reativo/crud_controller.rb:99:in `update'
vendor/ruby/2.5.0/gems/actionpack-5.2.3/lib/action_controller/metal/basic_implicit_render.rb:6:in `send_action'
vendor/ruby/2.5.0/gems/actionpack-5.2.3/lib/abstract_controller/base.rb:194:in `process_action'
vendor/ruby/2.5.0/gems/actionpack-5.2.3/lib/action_controller/metal/rendering.rb:30:in `process_action'
vendor/ruby/2.5.0/gems/actionpack-5.2.3/lib/abstract_controller/callbacks.rb:42:in `block in process_action'
vendor/ruby/2.5.0/gems/activesupport-5.2.3/lib/active_support/callbacks.rb:109:in `block in run_callbacks'
vendor/ruby/2.5.0/gems/react-rails-2.5.0/lib/react/rails/controller_lifecycle.rb:31:in `use_react_component_helper'
vendor/ruby/2.5.0/gems/activesupport-5.2.3/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
vendor/ruby/2.5.0/gems/activesupport-5.2.3/lib/active_support/callbacks.rb:136:in `run_callbacks'
vendor/ruby/2.5.0/gems/actionpack-5.2.3/lib/abstract_controller/callbacks.rb:41:in `process_action'
vendor/ruby/2.5.0/gems/actionpack-5.2.3/lib/action_controller/metal/rescue.rb:22:in `process_action'
vendor/ruby/2.5.0/gems/actionpack-5.2.3/lib/action_controller/metal/instrumentation.rb:34:in `block in process_action'
vendor/ruby/2.5.0/gems/activesupport-5.2.3/lib/active_support/notifications.rb:168:in `block in instrument'
vendor/ruby/2.5.0/gems/activesupport-5.2.3/lib/active_support/notifications/instrumenter.rb:23:in `instrument'
vendor/ruby/2.5.0/gems/activesupport-5.2.3/lib/active_support/notifications.rb:168:in `instrument'
vendor/ruby/2.5.0/gems/actionpack-5.2.3/lib/action_controller/metal/instrumentation.rb:32:in `process_action'
vendor/ruby/2.5.0/gems/actionpack-5.2.3/lib/action_controller/metal/params_wrapper.rb:256:in `process_action'
vendor/ruby/2.5.0/gems/activerecord-5.2.3/lib/active_record/railties/controller_runtime.rb:24:in `process_action'
vendor/ruby/2.5.0/gems/actionpack-5.2.3/lib/abstract_controller/base.rb:134:in `process'
vendor/ruby/2.5.0/gems/actionview-5.2.3/lib/action_view/rendering.rb:32:in `process'
vendor/ruby/2.5.0/gems/actionpack-5.2.3/lib/action_controller/metal.rb:191:in `dispatch'
vendor/ruby/2.5.0/gems/actionpack-5.2.3/lib/action_controller/metal.rb:252:in `dispatch'
vendor/ruby/2.5.0/gems/actionpack-5.2.3/lib/action_dispatch/routing/route_set.rb:52:in `dispatch'
vendor/ruby/2.5.0/gems/actionpack-5.2.3/lib/action_dispatch/routing/route_set.rb:34:in `serve'
vendor/ruby/2.5.0/gems/actionpack-5.2.3/lib/action_dispatch/journey/router.rb:52:in `block in serve'
vendor/ruby/2.5.0/gems/actionpack-5.2.3/lib/action_dispatch/journey/router.rb:35:in `each'
vendor/ruby/2.5.0/gems/actionpack-5.2.3/lib/action_dispatch/journey/router.rb:35:in `serve'
vendor/ruby/2.5.0/gems/actionpack-5.2.3/lib/action_dispatch/routing/route_set.rb:840:in `call'
vendor/ruby/2.5.0/gems/warden-1.2.8/lib/warden/manager.rb:36:in `block in call'
vendor/ruby/2.5.0/gems/warden-1.2.8/lib/warden/manager.rb:34:in `catch'
vendor/ruby/2.5.0/gems/warden-1.2.8/lib/warden/manager.rb:34:in `call'
vendor/ruby/2.5.0/gems/rack-2.0.8/lib/rack/tempfile_reaper.rb:15:in `call'
vendor/ruby/2.5.0/gems/rack-2.0.8/lib/rack/etag.rb:25:in `call'
vendor/ruby/2.5.0/gems/rack-2.0.8/lib/rack/conditional_get.rb:38:in `call'
vendor/ruby/2.5.0/gems/rack-2.0.8/lib/rack/head.rb:12:in `call'
vendor/ruby/2.5.0/gems/actionpack-5.2.3/lib/action_dispatch/http/content_security_policy.rb:18:in `call'
vendor/ruby/2.5.0/gems/rack-2.0.8/lib/rack/session/abstract/id.rb:259:in `context'
vendor/ruby/2.5.0/gems/rack-2.0.8/lib/rack/session/abstract/id.rb:253:in `call'
vendor/ruby/2.5.0/gems/actionpack-5.2.3/lib/action_dispatch/middleware/cookies.rb:670:in `call'
vendor/ruby/2.5.0/gems/activerecord-5.2.3/lib/active_record/migration.rb:559:in `call'
vendor/ruby/2.5.0/gems/actionpack-5.2.3/lib/action_dispatch/middleware/callbacks.rb:28:in `block in call'
vendor/ruby/2.5.0/gems/activesupport-5.2.3/lib/active_support/callbacks.rb:98:in `run_callbacks'
vendor/ruby/2.5.0/gems/actionpack-5.2.3/lib/action_dispatch/middleware/callbacks.rb:26:in `call'
vendor/ruby/2.5.0/gems/actionpack-5.2.3/lib/action_dispatch/middleware/executor.rb:14:in `call'
vendor/ruby/2.5.0/gems/actionpack-5.2.3/lib/action_dispatch/middleware/debug_exceptions.rb:61:in `call'
vendor/ruby/2.5.0/gems/web-console-3.7.0/lib/web_console/middleware.rb:135:in `call_app'
vendor/ruby/2.5.0/gems/web-console-3.7.0/lib/web_console/middleware.rb:22:in `block in call'
vendor/ruby/2.5.0/gems/web-console-3.7.0/lib/web_console/middleware.rb:20:in `catch'
vendor/ruby/2.5.0/gems/web-console-3.7.0/lib/web_console/middleware.rb:20:in `call'
vendor/ruby/2.5.0/gems/actionpack-5.2.3/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
vendor/ruby/2.5.0/gems/railties-5.2.3/lib/rails/rack/logger.rb:38:in `call_app'
vendor/ruby/2.5.0/gems/railties-5.2.3/lib/rails/rack/logger.rb:26:in `block in call'
vendor/ruby/2.5.0/gems/activesupport-5.2.3/lib/active_support/tagged_logging.rb:71:in `block in tagged'
vendor/ruby/2.5.0/gems/activesupport-5.2.3/lib/active_support/tagged_logging.rb:28:in `tagged'
vendor/ruby/2.5.0/gems/activesupport-5.2.3/lib/active_support/tagged_logging.rb:71:in `tagged'
vendor/ruby/2.5.0/gems/railties-5.2.3/lib/rails/rack/logger.rb:26:in `call'
vendor/ruby/2.5.0/gems/sprockets-rails-3.2.1/lib/sprockets/rails/quiet_assets.rb:13:in `call'
vendor/ruby/2.5.0/gems/actionpack-5.2.3/lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
vendor/ruby/2.5.0/gems/actionpack-5.2.3/lib/action_dispatch/middleware/request_id.rb:27:in `call'
vendor/ruby/2.5.0/gems/rack-2.0.8/lib/rack/method_override.rb:22:in `call'
vendor/ruby/2.5.0/gems/rack-2.0.8/lib/rack/runtime.rb:22:in `call'
vendor/ruby/2.5.0/gems/activesupport-5.2.3/lib/active_support/cache/strategy/local_cache_middleware.rb:29:in `call'
vendor/ruby/2.5.0/gems/actionpack-5.2.3/lib/action_dispatch/middleware/executor.rb:14:in `call'
vendor/ruby/2.5.0/gems/actionpack-5.2.3/lib/action_dispatch/middleware/static.rb:127:in `call'
vendor/ruby/2.5.0/gems/rack-2.0.8/lib/rack/sendfile.rb:111:in `call'
vendor/ruby/2.5.0/gems/webpacker-4.0.2/lib/webpacker/dev_server_proxy.rb:29:in `perform_request'
vendor/ruby/2.5.0/gems/rack-proxy-0.6.5/lib/rack/proxy.rb:57:in `call'
vendor/ruby/2.5.0/gems/railties-5.2.3/lib/rails/engine.rb:524:in `call'
vendor/ruby/2.5.0/gems/puma-3.12.2/lib/puma/configuration.rb:227:in `call'
vendor/ruby/2.5.0/gems/puma-3.12.2/lib/puma/server.rb:674:in `handle_request'
vendor/ruby/2.5.0/gems/puma-3.12.2/lib/puma/server.rb:476:in `process_client'
vendor/ruby/2.5.0/gems/puma-3.12.2/lib/puma/server.rb:334:in `block in run'
vendor/ruby/2.5.0/gems/puma-3.12.2/lib/puma/thread_pool.rb:135:in `block in spawn_thread'

add footer

I am still wondering whether we can have a real footer (at the bottom of the page).
The text I added is placed directly underneath the latest data.

Have a page where to look how values differ for a given key

Feature request: Create a page where to pass as parameter in the the name of an hiera key to analyse (for easy links wherever this key is shown) which shows what are the values for this key on all the nodes.
Ideally the list should be compressed to group nodes which have the same value and show information of the affected node (how many nodes have a given value) and possibly an option to expand them.

It would be nice, also, in the same page or another, to show where, for a given node, there's a duplicated and redundant entry which can be removed. For example where the same key with the same value is in common.yaml and in a node specific yaml, in this case the node specific entry is redundant and there should be a button to quickly remove it.

point: Design in a way that fits cases where the number of nodes could be in the order of a few thousand.

NoMethodError in KeysController#show if key is an integer

HDM running against a new PE 2019.2.2 installation, with issue #9 resolved by PR #10.

Steps to reproduce:

Perform a clean install of PE 2019.2.2 and HDM on the same system.
Tune puppet_enterprise::master::puppetserver::jruby_max_active_instances: 1 in $codedir/environments/production/data/common.yaml.
Open HDM.
Select production for "Choose the environment".
Select the PE master node for "Choose the node".
Select puppet_enterprise::master::puppetserver::jruby_max_active_instances as the key.

Expected behavior:

HDM displays its value (1 in the case of this lab system).

Observed behavior:

Error:

undefined method `gsub!' for nil:NilClass

Full function:

    def content_for_key(key)
      return nil unless content
      return nil unless content.has_key?(key)

      value = content[key]
      return "true" if value == true
      return "false" if value == false
      return value if value.is_a?(String)

      value_string = value.to_yaml
      value_string.gsub!("---\n", '').gsub!(/^$\n/, '')
      value_string
    end

Flagged line is:

      value_string.gsub!("---\n", '').gsub!(/^$\n/, '')

Rails.root:

/root/hdm

App trace:

app/models/hiera/read_file.rb:37:in `content_for_key'
app/models/hiera.rb:55:in `block (2 levels) in search_key'
app/models/hiera.rb:48:in `map'
app/models/hiera.rb:48:in `block in search_key'
app/models/hiera.rb:46:in `each_pair'
app/models/hiera.rb:46:in `search_key'
app/concepts/keys/operation/show.rb:28:in `node_data!'

Request parameters:

{"on"=>:member, "environment_id"=>"production", "node_id"=>"master.test.vlan", "id"=>"puppet_enterprise::master::puppetserver::jruby_max_active_instances"}

Web console output:

>> value

=> 1

>> value_string

=> "--- 1\n"

value_string is the result of value.to_yaml. If the value were a String instead of an Integer, return value if value.is_a?(String) would return before this.

The behavior of Integer.to_yaml:

>> 1.to_yaml

=> "--- 1\n"

>> 10.to_yaml

=> "--- 10\n"

>> "1".to_yaml

=> "--- '1'\n"

Since "--- 1" is apparently expected from 1.to_yaml, then:

>> value_string.gsub!("---\n", '').gsub!(/^$\n/, '')

NoMethodError: undefined method `gsub!' for nil:NilClass
	from /root/hdm/app/models/hiera/read_file.rb:37:in `content_for_key'

"---\n" matches nothing in "--- 1". gsub! returns nil if there's no match, so nil is also expected. We could convert the first result to a String/.to_s, but it feels like the integer value would still be lost unless the match itself were changed.

For reference:

>> content

=> {"puppet_enterprise::profile::amq::broker::heap_mb"=>"96", "puppet_enterprise::master::puppetserver::jruby_max_active_instances"=>1, "puppet_enterprise::master::puppetserver::reserved_code_cache"=>"96m", "puppet_enterprise::profile::master::java_args"=>{"Xmx"=>"384m", "Xms"=>"128m", "XX:MaxPermSize"=>"=96m", "XX:PermSize"=>"=64m", "XX:+UseG1GC"=>""}, "puppet_enterprise::profile::puppetdb::java_args"=>{"Xmx"=>"128m", "Xms"=>"64m", "XX:MaxPermSize"=>"=96m", "XX:PermSize"=>"=64m", "XX:+UseG1GC"=>""}, "puppet_enterprise::puppetdb::read_maximum_pool_size"=>4, "puppet_enterprise::puppetdb::write_maximum_pool_size"=>2, "puppet_enterprise::profile::console::java_args"=>{"Xmx"=>"128m", "Xms"=>"64m", "XX:MaxPermSize"=>"=96m", "XX:PermSize"=>"=64m", "XX:+UseG1GC"=>""}, "puppet_enterprise::trapperkeeper::database_settings::activity::maximum_pool_size"=>2, "puppet_enterprise::trapperkeeper::database_settings::classifier::maximum_pool_size"=>2, "puppet_enterprise::trapperkeeper::database_settings::rbac::maximum_pool_size"=>2, "puppet_enterprise::profile::console::delayed_job_workers"=>1, "puppet_enterprise::profile::database::shared_buffers"=>"4MB", "puppet_enterprise::profile::orchestrator::java_args"=>{"Xmx"=>"128m", "Xms"=>"128m", "XX:+UseG1GC"=>""}}

There are several integer values here, and they would be common in many PE installs. I'd expect any of them to fail.

Attempting to return an Integer value prevents the error when attempting to display the key's value when selecting it in HDM:

--- a/app/models/hiera/read_file.rb
+++ b/app/models/hiera/read_file.rb
@@ -31,7 +31,7 @@ class Hiera
       value = content[key]
       return "true" if value == true
       return "false" if value == false
-      return value if value.is_a?(String)
+      return value if value.is_a?(String) || value.is_a?(Integer)

But after making this change, then editing the selected value in HDM's interface (for instance from 1 to 2 in this example), HDM appends the new value to a new key in data/common.yaml with the same name but .json suffixed, instead of overwriting the existing key's value.

# grep -R max_active_instances $(puppet config print codedir)
...
/etc/puppetlabs/code/environments/production/data/common.yaml:puppet_enterprise::master::puppetserver::jruby_max_active_instances: 1
/etc/puppetlabs/code/environments/production/data/common.yaml:puppet_enterprise::master::puppetserver::jruby_max_active_instances.json: 2

read-only mode does not show data

When using read_only: true in config/hdm.yaml, no values are shown.

Running on Passenger ist not working

I did:

disable puppet http forwarding
install passenger
adopt selinux settings
provide write access to user nobody to app tmp directory
disabled default server in nginx.conf
create nginx config file

starting nginx, accessing website causes error.

Steps for reproduction:

Disabling http puppet redirect hiera data:
puppet_enterprise::profile::console::proxy::http_redirect::enable_http_redirect: false
selinux:

sealert  -a /var/log/audit/audit.log
/sbin/restorecon -v /opt/example42/hdm/public/favicon.ico
ausearch -c 'nginx' --raw | audit2allow -M hdm-nginx
semodule -i hdm-nginx.pp

Nginx HDM config file:

server {
    listen 80 default_server;
    server_name hdm.demo.example42.training;

    # Tell Nginx and Passenger where your app's 'public' directory is
    root /opt/example42/hdm/public;

    # Turn on Passenger
    passenger_enabled on;
    passenger_ruby /opt/puppetlabs/puppet/bin/ruby;
    passenger_friendly_error_pages on;
    passenger_env_var HDM__CONFIG_DIR /etc/puppetlabs/code/environments;
    passenger_env_var HDM__PUPPET_DB__ENABLED true;
    passenger_env_var HDM__PUPPET_DB__SELF_SIGNED_CERT true;
    passenger_env_var HDM__PUPPET_DB__TOKEN 0e-ZhnlpGdpEM9_Yh1zQzHef4BzQdvbnZSOQZrVWCW7A;
    passenger_env_var HDM__PUPPET_DB__SERVER https://localhost:8081;
    passenger_env_var PATH /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/opt/puppetlabs/puppet/bin;
}

Error message from passenger:

Error: The application encountered the following error: exit (SystemExit)
    /opt/example42/hdm/vendor/ruby/2.5.0/gems/webpacker-4.0.2/lib/webpacker/railtie.rb:48:in `exit'
    /opt/example42/hdm/vendor/ruby/2.5.0/gems/webpacker-4.0.2/lib/webpacker/railtie.rb:48:in `block in <class:Engine>'
    /opt/example42/hdm/vendor/ruby/2.5.0/gems/railties-5.2.3/lib/rails/initializable.rb:32:in `instance_exec'
    /opt/example42/hdm/vendor/ruby/2.5.0/gems/railties-5.2.3/lib/rails/initializable.rb:32:in `run'
    /opt/example42/hdm/vendor/ruby/2.5.0/gems/railties-5.2.3/lib/rails/initializable.rb:61:in `block in run_initializers'
    /opt/puppetlabs/puppet/lib/ruby/2.5.0/tsort.rb:228:in `block in tsort_each'
    /opt/puppetlabs/puppet/lib/ruby/2.5.0/tsort.rb:350:in `block (2 levels) in each_strongly_connected_component'
    /opt/puppetlabs/puppet/lib/ruby/2.5.0/tsort.rb:431:in `each_strongly_connected_component_from'
    /opt/puppetlabs/puppet/lib/ruby/2.5.0/tsort.rb:349:in `block in each_strongly_connected_component'
    /opt/puppetlabs/puppet/lib/ruby/2.5.0/tsort.rb:347:in `each'
    /opt/puppetlabs/puppet/lib/ruby/2.5.0/tsort.rb:347:in `call'
    /opt/puppetlabs/puppet/lib/ruby/2.5.0/tsort.rb:347:in `each_strongly_connected_component'
    /opt/puppetlabs/puppet/lib/ruby/2.5.0/tsort.rb:226:in `tsort_each'
    /opt/puppetlabs/puppet/lib/ruby/2.5.0/tsort.rb:205:in `tsort_each'
    /opt/example42/hdm/vendor/ruby/2.5.0/gems/railties-5.2.3/lib/rails/initializable.rb:60:in `run_initializers'
    /opt/example42/hdm/vendor/ruby/2.5.0/gems/railties-5.2.3/lib/rails/application.rb:361:in `initialize!'
    /opt/example42/hdm/config/environment.rb:5:in `<top (required)>'
    config.ru:3:in `require_relative'
    config.ru:3:in `block in <main>'
    /opt/example42/hdm/vendor/ruby/2.5.0/gems/rack-2.0.8/lib/rack/builder.rb:55:in `instance_eval'
    /opt/example42/hdm/vendor/ruby/2.5.0/gems/rack-2.0.8/lib/rack/builder.rb:55:in `initialize'
    config.ru:1:in `new'
    config.ru:1:in `<main>'
    /usr/share/passenger/helper-scripts/rack-preloader.rb:101:in `eval'
    /usr/share/passenger/helper-scripts/rack-preloader.rb:101:in `preload_app'
    /usr/share/passenger/helper-scripts/rack-preloader.rb:189:in `block in <module:App>'
    /usr/share/ruby/vendor_ruby/phusion_passenger/loader_shared_helpers.rb:380:in `run_block_and_record_step_progress'
    /usr/share/passenger/helper-scripts/rack-preloader.rb:188:in `<module:App>'
    /usr/share/passenger/helper-scripts/rack-preloader.rb:30:in `<module:PhusionPassenger>'
    /usr/share/passenger/helper-scripts/rack-preloader.rb:29:in `<main>'

Missing tests files and tests specify full path to individual dev location

Tests check for files with absolute path from individual location:

/Users/fernandes/src/hdm/puppet

Please add all missing test files.

Needed to add travis-ci.

allow filtering of node names

In large environment one can have several thousands of nodes.
Instead of showing a pulldown menu, also allow to write node names and filter within the pulldown list.

Prepare for beta release

The following should not be part of the git repo.
This information must be added to a document describing the way how to package HDM.

Rails uses different environments. At the moment we use development.
For packaging, we need production environment.

Compile JS/CSS:

bundle exec rails assets:precompile

Ruby Gems:

bundle install --deployment

Start check:

required ruby version?
bundler installed?
switch to hdm base dir
set rails environment
create database (db:create, db:migrate und db:seed) db:setup
start webserver

Rails:
bundle exec rails server -e

read/write flag

At Puppetize PDX several attendees requested a read-only view on data.
Add a feature flag (set in hdm config file), which disables editing of values.

Manage eyaml data

show encrypted values as is. add a button on the data to show plain text. (decrypt)
if eyaml is configured: add an encrypt button to save encrypted data. (encrypt)

Prepared testing environment (eyaml).
Provide a config switch, if we want to decrypt values in hdm.conf.

EYaml is configured in hiera.yaml file, by setting the lookup_key to eyaml_lookup and specifying options for private and public key.

If the private key is readable by HDM: provide the option to "decrypt" the data.

Preparation for storing data:

allow users to encrypt data.

If a user decrypts a value, sets a new one and wants to save it in plain text: provide a warning, and the user must confirm to store the value.

Please consider that we might have usermanagement RBAC where we want to limit users to decrypt values.

Visualize existance of hierarchies

available file and data present: bold
file available, no data inside: normal
file unavailable: italic

PuppetDB instead of FakePuppetDB

Provide a documentation on how to test against a real puppetdb using psick/vagrant setup

List of nodes should only list nodes from selected environment

When using puppetdb, all nodes from all environments are shown.

The list of nodes should only show the ones from the selected environment.

Enhance test data

Add more data, especially large data (long lines, blocks, eyaml content)
Add many many many keys.
Add new test nodes (with dot)

Data: test/fixtures/files/puppet/environments/
Nodes and Facts: test/fixtures/files/puppet/nodes/

Saving creates a new entry with ".json" appended to the key name

Example from common.yaml:

Prior editing:

hdm::boolean: true

Changing value to false creates a new entry:

hdm::boolean.json: false

The original key-value pair is still in common.yaml

Please deprecated/archive

@alvagante David Sandilands asked if this repo can be archived and maybe redirect to betadots hdm.
betadots/hdm#105

User Access

At the moment there is no login in front of hdm. We want to be able to manage users (locally)

simple local user management (config file: /etc/hdm/users.yaml|txt)

Hostnames with dot causes error

Prepare eyaml test environment

add hiera-eyaml gem depcendency
create a hiera-eyaml public/private key pair inside the test data structure

Removing a value does not work

When choosing to remove a value, hdm shows OK, but value does not get removed.

Allow users of HDM to specify default data sets

Default data sets are stored in a config yaml file.

e.g.

hdm_user_data.yaml
hash:
  key: value
  key2: value
array:
    - el1
    - el2
boolean: true
integer: 10
float: 1.1
string: 'a string'
``´

automate HDM installation using Puppet

Configure the following options:

config/hdm.yaml
- PuppetDB URL
- Access (Cert or PE Token)
- read_only
- config_dir (= puppet codedir setting)
Nginx Proxy (PE)
Nginx or Apache Proxy (for remote installation or open source installation) - add to documentation (with Puppet code example)
archive fetch and extract
hdm user and group
systemd unit file und servcie hdm (without daemon_reload if using systemd module)

Add description for development setup using psick vargant pe environment

let user configure a different file as hiera.yaml file

reading hiera settings

a hiera.yaml file is fully optional.
If no file is available, hiera uses internal defaults
HDM expects hiera data to be in environmentpath/data
But data can be set using hiera.yaml file:

---
version: 5
defaults:
  datadir: modules/hieradata/data

Defaults are in hiera ruby gem:
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/hiera/config.rb

class Hiera::Config
  class << self
    ##
    # load takes a string or hash as input, strings are treated as filenames
    # hashes are stored as data that would have been in the config file
    #
    # Unless specified it will only use YAML as backend with a
    # hierarchy of 'nodes/%{::trusted.certname}' and 'common', and with a
    # console logger.
    #
    # @return [Hash] representing the configuration.
    def load(source)
      @config = {:backends => ["yaml"],
                 :hierarchy => ["nodes/%{::trusted.certname}", "common"],
                 :merge_behavior => :native }

      if source.is_a?(String)
        if File.exist?(source)
          config = begin
                     yaml_load_file(source)
                   rescue TypeError => detail
                     case detail.message
                     when /no implicit conversion from nil to integer/
                       false

Allow filtering of hiera keys

After selecting a node, the list of all hiera keys is shown.
As this list might be very long: provide a filter field above the list of keys to liimit the keys shown.

hiera config with globs

Hiera lookup_options

Hiera has the option to set merge behavior via lookup_options. HDM should be able to parse these, too.

Set up HDM in PSICK PE vagrant environment

see DEVELOPMENT.md

Docker Support?

Do you think it is possible to run this project in docker?

make rails6 the default branch and rename

rename master to old.

Which new default branch name should we use?
main?

HDM does no longer show hiera data after latest merges

Adopt Web UI to exampel42 color scheme

At the moment HDM uses a default webtheme. We want to change layout to be compiant with example42 color scheme.

YAML data types

normal string, boolean, integer, float, hash, array
pipe (e.g. file content)
block syntax

Offer users a selector, which data type they would like to store.

Example for pipe:

long_pipe: |
  cluster.name: dev_cluster
  path.data: /var/lib/elasticsearch
  path.logs: /var/log/elasticsearch
  network.host: %{facts.fqdn}
  http.port: 9200
  discovery.seed_hosts: ["localhost"]
  cluster.initial_master_nodes: ["localhost"]
  action.destructive_requires_name: true

  xpack.security.enabled: true

Example for block:

long_block: >
    ENC[PKCS7,Y22exl+OvjDe+drmik2XEeD3VQtl1uZJXFFF2NnrMXDWx0csyqLB/2NOWefv
    NBTZfOlPvMlAesyr4bUY4I5XeVbVk38XKxeriH69EFAD4CahIZlC8lkE/uDh
    jJGQfh052eonkungHIcuGKY/5sEbbZl/qufjAtp/ufor15VBJtsXt17tXP4y
    l5ZP119Fwq8xiREGOL0lVvFYJz2hZc1ppPCNG5lwuLnTekXN/OazNYpf4CMd
    /HjZFXwcXRtTlzewJLc+/gox2IfByQRhsI/AgogRfYQKocZgFb/DOZoXR7wm
    IZGeunzwhqfmEtGiqpvJJQ5wVRdzJVpTnANBA5qxeA==]

Verify dependabot issues and upgrade proposals

https://github.com/example42/hdm/security/dependabot

Project needs more documentation

It would be nice to have a description of what this is and what it does. Maybe a screenshot, if it has an user interface.

File config/credentials.yml.enc should not be in git repo?

To verify with a Rails expert:
Should file config/credentials.yml.enc be kept in the repo or removed and added to .gitignore?
It can be regenerated with rails:credentials:edit
Maybe describe in setup instructions or via the Puppet module.

HDM file storage backend

HDM in read/write mode must be able to store the new data. Data path must be in hiera yaml file and in hdm config file.

Config in hdm.conf (hdm_data_backend = dir .... hdm_data_dir = /etc/hdm/data)
check: no .git dir may be in data dir

multiple hierarchies are not working as expected

Only a subset of the hierarchies is shown.
see fixtures multiple_hierarchies test data.

allow usage of hiera lookup_options

Within a hiera hierarchy, one has the option to set the merge behavior using the key "lookup_options".

Example:
data/common.yaml

lookup_options:
  <key>:
    merge: <merge behavior>
  convert_to: <Data Type> # Usually "Sensitive"

The following merge behavior options are possible:

first - the default hiera behavior
unique - used on arrays
hash - merges all hash keys (on first level)
deep - similar to hash, but merges subkeys, too

The hiera key lookup_options should always be the very first in the list of keys (if set).

HDM should show the lookup_options providing a text message above the top hierarchy.

lookup option: merge: first, convert_to: <Data Type>

Note: lookup options can be set in every hiera hierarchy.
Updated test code is in mulitple_environmenrs.
Node sse8epsu.example42.training has lookup option for profile::auth::sshd_config_allowgroups set to 'first' and in common it is set to 'unique'.

Hiera uses deep merge on lookup_options.

e.g.

Node specific:

lookup_options:
    <key1>:
     merge: deep
  <key2>:
    merge: first
    convert_to: 'Sensitive'

common:

lookup_options:
    <key1>:
     merge: first

HDM needs puppet directory in app root

Any reason why we still need this directory?
I suppose that this is a leftover from test data when building prototype.