Git Product home page Git Product logo

teracy-dev-core's People

Contributors

datphan avatar hieptranquoc avatar hoatle avatar hoavt avatar phuonglm avatar

Stargazers

 avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

teracy-dev-core's Issues

variables processor should support boolean value

config:

variables:
  my_extension_enabled: false

teracy-dev:
    - _id: "entry-my-extension"
      path:
        extension: project/teracy-dev-my-extension
      location:
        git: [email protected]:org/project.git
        branch: develop
      require_version: ">= 0.1.0-SNAPSHOT"
      enabled: "%{my_extension_enabled}"

Expected: it should work as normal as my_extension_enabled: "false"

Actual: error

/Users/hoatle/my-dev/extensions/teracy-dev-core/lib/teracy-dev-core/processors/variables.rb:15:in `match'
/Users/hoatle/my-dev/extensions/teracy-dev-core/lib/teracy-dev-core/processors/variables.rb:15:in `block in process'
/Users/hoatle/my-dev/extensions/teracy-dev-core/lib/teracy-dev-core/processors/variables.rb:14:in `each'
/Users/hoatle/my-dev/extensions/teracy-dev-core/lib/teracy-dev-core/processors/variables.rb:14:in `process'
/Users/hoatle/my-dev/lib/teracy-dev/processors/manager.rb:33:in `block in process'
/Users/hoatle/my-dev/lib/teracy-dev/processors/manager.rb:32:in `each'
/Users/hoatle/my-dev/lib/teracy-dev/processors/manager.rb:32:in `process'
/Users/hoatle/my-dev/lib/teracy-dev/loader.rb:134:in `process'
/Users/hoatle/my-dev/lib/teracy-dev/loader.rb:103:in `build_settings'
/Users/hoatle/my-dev/lib/teracy-dev/loader.rb:33:in `start'
/Users/hoatle/my-dev/lib/teracy-dev.rb:31:in `<module:TeracyDev>'
/Users/hoatle/my-dev/lib/teracy-dev.rb:10:in `<top (required)>'
/opt/vagrant/embedded/lib/ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
/opt/vagrant/embedded/lib/ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
/Users/hoatle/my-dev/Vagrantfile:21:in `<top (required)>'
/opt/vagrant/embedded/gems/2.1.5/gems/vagrant-2.1.5/lib/vagrant/config/loader.rb:239:in `load'
/opt/vagrant/embedded/gems/2.1.5/gems/vagrant-2.1.5/lib/vagrant/config/loader.rb:239:in `block in procs_for_path'
/opt/vagrant/embedded/gems/2.1.5/gems/vagrant-2.1.5/lib/vagrant/config.rb:53:in `block in capture_configures'
/opt/vagrant/embedded/gems/2.1.5/gems/vagrant-2.1.5/lib/vagrant/config.rb:48:in `synchronize'
/opt/vagrant/embedded/gems/2.1.5/gems/vagrant-2.1.5/lib/vagrant/config.rb:48:in `capture_configures'
/opt/vagrant/embedded/gems/2.1.5/gems/vagrant-2.1.5/lib/vagrant/config/loader.rb:237:in `procs_for_path'
/opt/vagrant/embedded/gems/2.1.5/gems/vagrant-2.1.5/lib/vagrant/config/loader.rb:223:in `procs_for_source'
/opt/vagrant/embedded/gems/2.1.5/gems/vagrant-2.1.5/lib/vagrant/config/loader.rb:67:in `block in set'
/opt/vagrant/embedded/gems/2.1.5/gems/vagrant-2.1.5/lib/vagrant/config/loader.rb:61:in `each'
/opt/vagrant/embedded/gems/2.1.5/gems/vagrant-2.1.5/lib/vagrant/config/loader.rb:61:in `set'
/opt/vagrant/embedded/gems/2.1.5/gems/vagrant-2.1.5/lib/vagrant/environment.rb:488:in `config_loader'
/opt/vagrant/embedded/gems/2.1.5/gems/vagrant-2.1.5/lib/vagrant/environment.rb:793:in `vagrantfile'
/opt/vagrant/embedded/gems/2.1.5/gems/vagrant-2.1.5/lib/vagrant/environment.rb:178:in `initialize'
/opt/vagrant/embedded/gems/2.1.5/gems/vagrant-2.1.5/bin/vagrant:144:in `new'
/opt/vagrant/embedded/gems/2.1.5/gems/vagrant-2.1.5/bin/vagrant:144:in `<main>'
Vagrant failed to initialize at a very early stage:

There was an error loading a Vagrantfile. The file being loaded
and the error message are shown below. This is usually caused by
a syntax error.

Path: /Users/hoatle/my-dev/Vagrantfile
Line number: 15
Message: TypeError: no implicit conversion of false into String

_id_deprecated, _op should be deleted from provisioner options

when configuring a provisioner, _id_deprecated, _op is set will cause the following similar error:

$ vagrant up
Bringing machine 'teracy-dev.local' up with 'virtualbox' provider...
There are errors in the configuration of this machine. Please fix
the following errors and try again:
chef solo provisioner:
* The following settings shouldn't exist: _id_deprecated

should create TeracyDevCore.register_provider(type, provider) to manage provider implementations

with APIs:

  • TeracyDevCore.register_provider(type, impl) which is the proxy to Providers::Manager.register(provider_type, provider_impl)

and delegate the configuration to the right providers

ProviderManager allows registering different provider implementations with the same provider_type

Which provider is registered first will be delegated first.

Providers must implement the configure(provider_settings, config) signature for the delegation job handling.

should delete "name" key from provisioner settings for options

the following settings did not work:

  provisioners:
  - _id: certs-ansible
    name: teracy-dev-certs
    type: ansible_local
    verbose: "-vv"
    weight: 8
    enabled: true
    playbook: extensions/teracy-dev-certs/provisioners/ansible/certs.yaml
    extra_vars:
      common_name: k8s.local
      alt_names:
      - accounts.k8s.local
      - login.k8s.local

expected: $ vagrant provision --provision-with teracy-dev-certs should work

actual:

There are errors in the configuration of this machine. Please fix
the following errors and try again:

ansible local provisioner:
* The following settings shouldn't exist: name

should have consistent and intuitive config for forwarded_port network config

currently supported config:

nodes:
  - _id: "core-0"
    vm:
      networks:
        - _id: entry-fwd-port
          type: forwarded_port
          forwarded_ports:
            - _id: entry-fwd-port-0
              guest: 80
              host: 8080
            - _id: entry-fwd-port-1
              guest: 81
              host: 8081

we should deprecate the above config and use this config instead:

nodes:
  - _id: "core-0"
    vm:
      networks:
        - _id: entry-fwd-port-0
          type: forwarded_port
          guest: 80
          host: 8080
        - _id: entry-fwd-port-1
          type: forwarded_port
          guest: 81
          host: 8081

should display friendly log messages for the not configured variable key(s)

Expect: Friendly message to guide users

Actual: scary log

/Users/hoatle/teracy-dev/workspace/teracy-dev-core/lib/teracy-dev-core/processors/variables.rb:38:in `%'
/Users/hoatle/teracy-dev/workspace/teracy-dev-core/lib/teracy-dev-core/processors/variables.rb:38:in `process'
/Users/hoatle/teracy-dev/lib/teracy-dev/processors/manager.rb:40:in `block in process'
/Users/hoatle/teracy-dev/lib/teracy-dev/processors/manager.rb:37:in `each'
/Users/hoatle/teracy-dev/lib/teracy-dev/processors/manager.rb:37:in `process'
/Users/hoatle/teracy-dev/lib/teracy-dev/loader.rb:173:in `process'
/Users/hoatle/teracy-dev/lib/teracy-dev/loader.rb:138:in `build_settings'
/Users/hoatle/teracy-dev/lib/teracy-dev/loader.rb:33:in `start'
/Users/hoatle/teracy-dev/lib/teracy-dev.rb:31:in `<module:TeracyDev>'
/Users/hoatle/teracy-dev/lib/teracy-dev.rb:10:in `<top (required)>'
/opt/vagrant/embedded/lib/ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
/opt/vagrant/embedded/lib/ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
/Users/hoatle/teracy-dev/Vagrantfile:21:in `<top (required)>'
/opt/vagrant/embedded/gems/2.2.0/gems/vagrant-2.2.0/lib/vagrant/config/loader.rb:239:in `load'
/opt/vagrant/embedded/gems/2.2.0/gems/vagrant-2.2.0/lib/vagrant/config/loader.rb:239:in `block in procs_for_path'
/opt/vagrant/embedded/gems/2.2.0/gems/vagrant-2.2.0/lib/vagrant/config.rb:53:in `block in capture_configures'
/opt/vagrant/embedded/gems/2.2.0/gems/vagrant-2.2.0/lib/vagrant/config.rb:48:in `synchronize'
/opt/vagrant/embedded/gems/2.2.0/gems/vagrant-2.2.0/lib/vagrant/config.rb:48:in `capture_configures'
/opt/vagrant/embedded/gems/2.2.0/gems/vagrant-2.2.0/lib/vagrant/config/loader.rb:237:in `procs_for_path'
/opt/vagrant/embedded/gems/2.2.0/gems/vagrant-2.2.0/lib/vagrant/config/loader.rb:223:in `procs_for_source'
/opt/vagrant/embedded/gems/2.2.0/gems/vagrant-2.2.0/lib/vagrant/config/loader.rb:67:in `block in set'
/opt/vagrant/embedded/gems/2.2.0/gems/vagrant-2.2.0/lib/vagrant/config/loader.rb:61:in `each'
/opt/vagrant/embedded/gems/2.2.0/gems/vagrant-2.2.0/lib/vagrant/config/loader.rb:61:in `set'
/opt/vagrant/embedded/gems/2.2.0/gems/vagrant-2.2.0/lib/vagrant/environment.rb:488:in `config_loader'
/opt/vagrant/embedded/gems/2.2.0/gems/vagrant-2.2.0/lib/vagrant/environment.rb:793:in `vagrantfile'
/opt/vagrant/embedded/gems/2.2.0/gems/vagrant-2.2.0/lib/vagrant/environment.rb:178:in `initialize'
/opt/vagrant/embedded/gems/2.2.0/gems/vagrant-2.2.0/bin/vagrant:144:in `new'
/opt/vagrant/embedded/gems/2.2.0/gems/vagrant-2.2.0/bin/vagrant:144:in `<main>'
Vagrant failed to initialize at a very early stage:

There was an error loading a Vagrantfile. The file being loaded
and the error message are shown below. This is usually caused by
a syntax error.

Path: /Users/hoatle/teracy-dev/Vagrantfile
Line number: 38
Message: KeyError: key{node_name_prefix} not found

should add extension path variables dynamically

so that extension configs should have the right path, for example, chef cookbooks_path for https://github.com/teracyhq-incubator/teracy-dev-v05-compat/pull/2/files

We should have <extension-name>-path variable so that, only enabled extensions will have the associated <extension-name>-path, the path should not include slash character (/) at the end.

      cookbooks_path:
        - teracy-dev-v05-compat/vendor-cookbooks
        - teracy-dev-v05-compat/main-cookbooks

should become:

      cookbooks_path:
        - "%{teracy-dev-v05-compat-path}/vendor-cookbooks"
        - "%{teracy-dev-v05-compat-path}/main-cookbooks"

By supporting this, teracyhq-incubator/teracy-dev-v05-compat#2 will not have to create a new configurator but using yaml config only.

should create ExtensionResources processor to set memory, cpus, etc dynamically

  • add optional resources for each extension config and create an ExtensionResources processor (the last one in the pipeline to run to calculate and set VM's memory dynamically by overriding each node provider's config)

  • users can explicitly set the exact memory, cpus on the provider config (assumed empty by default on the default.providers or node.providers, node.providers has higher precedence)

  • resources.{minimum, requests}.{memory, cpus} is optional

example config format without user explict memory/cpus setting:

teracy-dev:
  extensions:
    - _id: "entry-iorad-common"
      path:
        extension: teracy-dev-iorad-common
      location:
        git:
          remote:
            origin: [email protected]:iorad/teracy-dev-iorad-common.git
          branch: v0.4.0
      require_version: ">= 0.4.0"
      enabled: true
      resources:
        minimum:
          memory: 1024
          cpus: 1
        requests:
          memory: 1024
          cpus: 1


    - _id: "entry-iorad-app"
      path:
        lookup: workspace
        extension: iorad/teracy-dev-iorad-app/master
        # extension: iorad/teracy-dev-iorad-app/master # the master stable branch
        # extension: iorad/teracy-dev-iorad-app/develop # the develop unstable branch
      location:
        git:
          remote:
            origin: [email protected]:iorad/iorad.git
          branch: develop
      require_version: ">= 0.1.0"
      enabled: false
      resources:
        minimum:
          memory: 3072
          cpus: 0
        requests:
          memory: 3072
          cpus: 3

    - _id: "entry-iorad-extension"
      path:
        lookup: workspace
        extension: iorad-extension/teracy-dev-iorad-extension/master
        # extension: iorad-extension/teracy-dev-iorad-extension/master # the master stable branch
        # extension: iorad-extension/teracy-dev-iorad-extension/develop # the develop unstable branch
      location:
        git:
          remote:
            origin: [email protected]:iorad/iorad-extension.git
          branch: develop
      require_version: ">= 0.1.0"
      enabled: false
      resources:
        minimum:
          memory: 0
          cpus: 0
        requests:
          memory: 1024
          cpus: 0

    - _id: "entry-iorad-video-export"
      path:
        lookup: workspace
        extension: video-export/teracy-dev-iorad-video-export/master
        # extension: video-export/teracy-dev-iorad-video-export/master # the master stable branch
        # extension: video-export/teracy-dev-iorad-video-export/develop # the develop unstable branch
      location:
        git:
          remote:
            origin: [email protected]:iorad/video-export.git
          branch: master
      require_version: ">= 0.1.0"
      enabled: false
      resources:
        minimum:
          memory: 0
          cpus: 0
        requests:
          memory: 1024
          cpus: 0

default:
  providers:
    - _id: "0" # override
      memory: 0 # zero or empty means no set
      cpus: "0" # zero or empty means no set

nodes:
  - _id: "0"
    providers:
      - _id: "0" # override the default by each node
        memory: ""
        cpus: ""

Algorithm:

  • total minimum memory = total minimum memory of enabled extensions

  • total requested memoy = total requested memory of the enabled extensions

  • total minimum cpus = total minimum cpus of enabled extensions

  • total requested cpus = total requested cpus of enabled extensions

  • available memory = calculated available memory for the VMs to run

  • available cpus = calculated available cpus for the VMs to run

  • explit user-set memory/cpus via default.providers or node.providers (query provider by type to set values correctly: multiple provider support if applicable)

  • total minimum must be less than or equal to total requested

  • total_available_memory:
    For Mac:

    • max memory can use is 75% * (physic memory - wired memory).
      For Linux:
    • max memory can use is 100% * (free memory + cache + buff )
      for windows:
    • Please use FreePhysicalMemory instead of TotalVisibleMemorySize.

    We know that Linux VM use a lot of free memory for cache, buff. Allocate too much memory for VMs will make host machine out of memory and dev will not able to use their machine

  • for each node, we must allocate available_memory basing on the number of nodes, available_cpus is equal to the total_available_cpus respectively:

    • available_memory = total_available_memory / the number of nodes
    • available_cpus = total_available_cpus

    for example:

    total_available_memory: 6000, number of nodes: 2 => each node has floor(6000/2) = 3000 available_memory

    total_available_cpus: 4 => available_cpus for each node: 4

if explicit_user_set_memory exists
  if total_minimum_memory <= explicit_user_set_memory <= available_memory:
    set VM's memory to explit_user_set_memory
  else
    abort with error message
else
  if available_memory >= total_requested_memory:
    set VM's memory to total_requested_memory
  else if available_memory >= total_minimum_memory:
    warn
    set VM's memory to available_memory
  else
    abort with error message
if explicit_user_set_cpus exists
  if total_minimum_cpus <= explicit_user_set_cpus <= available_cpus:
    set VM's cpus to explit_user_set_cpus
  else
    abort with error message
else
  if available_cpus >= total_requested_cpus:
    set VM's cpus to total_requested_cpus
  else if available_cpus >= total_minimum_cpus:
    warn
    set VM's cpus to available_cpus
  else
    abort with error message

should add "weight" for provisioners to change the orders

similar to teracyhq/dev#497
ref: https://www.vagrantup.com/docs/provisioning/basic_usage.html#multiple-provisioners

need to work on this: https://github.com/teracyhq-incubator/teracy-dev-core/blob/develop/lib/teracy-dev-core/config/provisioners.rb

If no "weight" is defined, it's 5 by default.

for example:

nodes:
  - _id: "0"
    provisioners:
      - _id: '0'
        version: 13.1.31
        type: chef_solo
        log_level: info
        enabled: true
      - _id: '1'
        version: 13.1.31
        type: chef_solo
        log_level: info
        enabled: true
        weight: 6
      - _id: '2'
        version: 13.1.31
        type: chef_solo
        log_level: info
        enabled: true
        weight: 6
      - _id: '3'
        version: 13.1.31
        type: chef_solo
        log_level: info
        enabled: true
        weight: 3
      - _id: '4'
        version: 13.1.31
        type: chef_solo
        log_level: info
        enabled: true
        weight: 4

=> the expected running order (_id) is: 1, 2, 0, 4, 3

VBoxManage not found on windows

After fetching the latest code of teracy-dev, vagrant up is not working

```Administrator@8EIPUXSN50GYWXU /cygdrive/f/teracy/teracy-dev
$ vagrant up
[TeracyDev::Location::GitSynch][INFO]: cd F:/teracy/teracy-dev/lib/../extensions && git clone https://github.com/teracyhq-incubator/teracy-dev-core.git
F:/teracy/teracy-dev/lib/teracy-dev/location/git_synch.rb:59:in `block in sync'
Cloning into 'teracy-dev-core'...
remote: Counting objects: 171, done.
remote: Compressing objects: 100% (95/95), done.
Receiving objectsremote: Total 171 (delta 74), reused 165 (delta 69), pack-reused 0
Receiving objects: 100% (171/171), 28.47 KiB | 170.00 KiB/s, done.
Resolving deltas: 100% (74/74), done.
[TeracyDev::Location::GitSynch][INFO]: cd F:/teracy/teracy-dev/lib/../extensions/teracy-dev-core && git checkout v0.2.0
F:/teracy/teracy-dev/lib/teracy-dev/location/git_synch.rb:64:in `block in sync'
Note: checking out 'v0.2.0'.
You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.
If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:
  git checkout -b <new-branch-name>
HEAD is now at ded0fc7 Merge pull request #14 from hoatle/tasks/#13-v0.2.0-release
F:/teracy/teracy-dev/extensions/teracy-dev-core/lib/teracy-dev-core/config/virtualbox_provider.rb:17:in ``': No such file or directory - VBoxManage --version (Errno::ENOENT)
        from F:/teracy/teracy-dev/extensions/teracy-dev-core/lib/teracy-dev-core/config/virtualbox_provider.rb:17:in `block in configure_node'
        from F:/teracy/teracy-dev/extensions/teracy-dev-core/lib/teracy-dev-core/config/virtualbox_provider.rb:11:in `each'
        from F:/teracy/teracy-dev/extensions/teracy-dev-core/lib/teracy-dev-core/config/virtualbox_provider.rb:11:in `configure_node'
        from F:/teracy/teracy-dev/lib/teracy-dev/config/configurator.rb:17:in `configure'
        from F:/teracy/teracy-dev/lib/teracy-dev/config/manager.rb:33:in `block in configure'
        from F:/teracy/teracy-dev/lib/teracy-dev/config/manager.rb:32:in `each'
        from F:/teracy/teracy-dev/lib/teracy-dev/config/manager.rb:32:in `configure'
        from F:/teracy/teracy-dev/lib/teracy-dev/loader.rb:135:in `configure'
        from F:/teracy/teracy-dev/lib/teracy-dev/loader.rb:148:in `block (3 levels) in configure_vagrant'
        from C:/HashiCorp/Vagrant/embedded/gems/2.1.0/gems/vagrant-2.1.0/lib/vagrant/config/v2/loader.rb:37:in `load'
        from C:/HashiCorp/Vagrant/embedded/gems/2.1.0/gems/vagrant-2.1.0/lib/vagrant/config/loader.rb:126:in `block (2 levels) in load'
        from C:/HashiCorp/Vagrant/embedded/gems/2.1.0/gems/vagrant-2.1.0/lib/vagrant/config/loader.rb:119:in `each'
        from C:/HashiCorp/Vagrant/embedded/gems/2.1.0/gems/vagrant-2.1.0/lib/vagrant/config/loader.rb:119:in `block in load'
        from C:/HashiCorp/Vagrant/embedded/gems/2.1.0/gems/vagrant-2.1.0/lib/vagrant/config/loader.rb:116:in `each'
        from C:/HashiCorp/Vagrant/embedded/gems/2.1.0/gems/vagrant-2.1.0/lib/vagrant/config/loader.rb:116:in `load'
        from C:/HashiCorp/Vagrant/embedded/gems/2.1.0/gems/vagrant-2.1.0/lib/vagrant/vagrantfile.rb:166:in `machine_config'
        from C:/HashiCorp/Vagrant/embedded/gems/2.1.0/gems/vagrant-2.1.0/lib/vagrant/environment.rb:322:in `default_provider'
        from C:/HashiCorp/Vagrant/embedded/gems/2.1.0/gems/vagrant-2.1.0/lib/vagrant/plugin/v2/command.rb:177:in `block in with_target_vms'
        from C:/HashiCorp/Vagrant/embedded/gems/2.1.0/gems/vagrant-2.1.0/lib/vagrant/plugin/v2/command.rb:204:in `block in with_target_vms'
        from C:/HashiCorp/Vagrant/embedded/gems/2.1.0/gems/vagrant-2.1.0/lib/vagrant/plugin/v2/command.rb:186:in `each'
        from C:/HashiCorp/Vagrant/embedded/gems/2.1.0/gems/vagrant-2.1.0/lib/vagrant/plugin/v2/command.rb:186:in `with_target_vms'
        from C:/HashiCorp/Vagrant/embedded/gems/2.1.0/gems/vagrant-2.1.0/plugins/commands/up/command.rb:87:in `execute'
        from C:/HashiCorp/Vagrant/embedded/gems/2.1.0/gems/vagrant-2.1.0/lib/vagrant/cli.rb:54:in `execute'
        from C:/HashiCorp/Vagrant/embedded/gems/2.1.0/gems/vagrant-2.1.0/lib/vagrant/environment.rb:275:in `cli'
        from C:/HashiCorp/Vagrant/embedded/gems/2.1.0/gems/vagrant-2.1.0/bin/vagrant:156:in `<main>'
Administrator@8EIPUXSN50GYWXU /cygdrive/f/teracy/teracy-dev

Try the command "VBoxManage" ==> the command is not found

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.