teracyhq-incubator / teracy-dev-core Goto Github PK
View Code? Open in Web Editor NEWteracy-dev-core extension
License: MIT License
teracy-dev-core extension
License: MIT License
from the config.yaml, deprecate the old ones and use the new ones
providers:
- _id: "0" => _id: "core-virtualbox"
nodes:
config:
variables:
my_extension_enabled: false
teracy-dev:
- _id: "entry-my-extension"
path:
extension: project/teracy-dev-my-extension
location:
git: [email protected]:org/project.git
branch: develop
require_version: ">= 0.1.0-SNAPSHOT"
enabled: "%{my_extension_enabled}"
Expected: it should work as normal as my_extension_enabled: "false"
Actual: error
/Users/hoatle/my-dev/extensions/teracy-dev-core/lib/teracy-dev-core/processors/variables.rb:15:in `match'
/Users/hoatle/my-dev/extensions/teracy-dev-core/lib/teracy-dev-core/processors/variables.rb:15:in `block in process'
/Users/hoatle/my-dev/extensions/teracy-dev-core/lib/teracy-dev-core/processors/variables.rb:14:in `each'
/Users/hoatle/my-dev/extensions/teracy-dev-core/lib/teracy-dev-core/processors/variables.rb:14:in `process'
/Users/hoatle/my-dev/lib/teracy-dev/processors/manager.rb:33:in `block in process'
/Users/hoatle/my-dev/lib/teracy-dev/processors/manager.rb:32:in `each'
/Users/hoatle/my-dev/lib/teracy-dev/processors/manager.rb:32:in `process'
/Users/hoatle/my-dev/lib/teracy-dev/loader.rb:134:in `process'
/Users/hoatle/my-dev/lib/teracy-dev/loader.rb:103:in `build_settings'
/Users/hoatle/my-dev/lib/teracy-dev/loader.rb:33:in `start'
/Users/hoatle/my-dev/lib/teracy-dev.rb:31:in `<module:TeracyDev>'
/Users/hoatle/my-dev/lib/teracy-dev.rb:10:in `<top (required)>'
/opt/vagrant/embedded/lib/ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
/opt/vagrant/embedded/lib/ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
/Users/hoatle/my-dev/Vagrantfile:21:in `<top (required)>'
/opt/vagrant/embedded/gems/2.1.5/gems/vagrant-2.1.5/lib/vagrant/config/loader.rb:239:in `load'
/opt/vagrant/embedded/gems/2.1.5/gems/vagrant-2.1.5/lib/vagrant/config/loader.rb:239:in `block in procs_for_path'
/opt/vagrant/embedded/gems/2.1.5/gems/vagrant-2.1.5/lib/vagrant/config.rb:53:in `block in capture_configures'
/opt/vagrant/embedded/gems/2.1.5/gems/vagrant-2.1.5/lib/vagrant/config.rb:48:in `synchronize'
/opt/vagrant/embedded/gems/2.1.5/gems/vagrant-2.1.5/lib/vagrant/config.rb:48:in `capture_configures'
/opt/vagrant/embedded/gems/2.1.5/gems/vagrant-2.1.5/lib/vagrant/config/loader.rb:237:in `procs_for_path'
/opt/vagrant/embedded/gems/2.1.5/gems/vagrant-2.1.5/lib/vagrant/config/loader.rb:223:in `procs_for_source'
/opt/vagrant/embedded/gems/2.1.5/gems/vagrant-2.1.5/lib/vagrant/config/loader.rb:67:in `block in set'
/opt/vagrant/embedded/gems/2.1.5/gems/vagrant-2.1.5/lib/vagrant/config/loader.rb:61:in `each'
/opt/vagrant/embedded/gems/2.1.5/gems/vagrant-2.1.5/lib/vagrant/config/loader.rb:61:in `set'
/opt/vagrant/embedded/gems/2.1.5/gems/vagrant-2.1.5/lib/vagrant/environment.rb:488:in `config_loader'
/opt/vagrant/embedded/gems/2.1.5/gems/vagrant-2.1.5/lib/vagrant/environment.rb:793:in `vagrantfile'
/opt/vagrant/embedded/gems/2.1.5/gems/vagrant-2.1.5/lib/vagrant/environment.rb:178:in `initialize'
/opt/vagrant/embedded/gems/2.1.5/gems/vagrant-2.1.5/bin/vagrant:144:in `new'
/opt/vagrant/embedded/gems/2.1.5/gems/vagrant-2.1.5/bin/vagrant:144:in `<main>'
Vagrant failed to initialize at a very early stage:
There was an error loading a Vagrantfile. The file being loaded
and the error message are shown below. This is usually caused by
a syntax error.
Path: /Users/hoatle/my-dev/Vagrantfile
Line number: 15
Message: TypeError: no implicit conversion of false into String
see: https://github.com/hashicorp/vagrant/blob/v2.1.2/plugins/kernel_v2/config/vm.rb#L249
This is required for teracyhq-incubator/teracy-dev-essential#3
The network id should be this format: "#{node_id}-#{network_type}-#{network_id}"
for virtualbox provider as the default config
when configuring a provisioner, _id_deprecated
, _op
is set will cause the following similar error:
$ vagrant up
Bringing machine 'teracy-dev.local' up with 'virtualbox' provider...
There are errors in the configuration of this machine. Please fix
the following errors and try again:
chef solo provisioner:
* The following settings shouldn't exist: _id_deprecated
v0.6.0-a4 - the latest release at present - should be used
v0.3.0 is ready to be shipped
except for the generated keys from the dynamic_paths, they can be unused.
Other keys are expected to be used, otherwise, a warning message should be helpful.
with APIs:
and delegate the configuration to the right providers
ProviderManager allows registering different provider implementations with the same provider_type
Which provider is registered first will be delegated first.
Providers must implement the configure(provider_settings, config)
signature for the delegation job handling.
should update:
config_default.yaml
to config.yaml
v0.1.0 is ready to be shipped
providers:
- _id: "0"
enabled: false
type: virtualbox
require_version: ">= 5.2"
gui: false
memory: 1024
description: "%{node_name_prefix} #{Time.now.getutc.to_i}"
Ignore setting when enabled: false
From @hoatle on December 28, 2018 4:6
I guess we could prevent orphan VMs with an extension and then we'll add that extension into default extensions config.
Copied from original issue: teracyhq/dev#587
it's expected to raise the target version to v0.6 (stable final version) so that we can use new features.
so teracy-dev-core v0.4.0 will work with teracy-dev v0.6.0-a5 or later but not v0.7.0 or later.
the following settings did not work:
provisioners:
- _id: certs-ansible
name: teracy-dev-certs
type: ansible_local
verbose: "-vv"
weight: 8
enabled: true
playbook: extensions/teracy-dev-certs/provisioners/ansible/certs.yaml
extra_vars:
common_name: k8s.local
alt_names:
- accounts.k8s.local
- login.k8s.local
expected: $ vagrant provision --provision-with teracy-dev-certs
should work
actual:
There are errors in the configuration of this machine. Please fix
the following errors and try again:
ansible local provisioner:
* The following settings shouldn't exist: name
v0.2.0 is ready to be shipped
currently supported config:
nodes:
- _id: "core-0"
vm:
networks:
- _id: entry-fwd-port
type: forwarded_port
forwarded_ports:
- _id: entry-fwd-port-0
guest: 80
host: 8080
- _id: entry-fwd-port-1
guest: 81
host: 8081
we should deprecate the above config and use this config instead:
nodes:
- _id: "core-0"
vm:
networks:
- _id: entry-fwd-port-0
type: forwarded_port
guest: 80
host: 8080
- _id: entry-fwd-port-1
type: forwarded_port
guest: 81
host: 8081
Expect: Friendly message to guide users
Actual: scary log
/Users/hoatle/teracy-dev/workspace/teracy-dev-core/lib/teracy-dev-core/processors/variables.rb:38:in `%'
/Users/hoatle/teracy-dev/workspace/teracy-dev-core/lib/teracy-dev-core/processors/variables.rb:38:in `process'
/Users/hoatle/teracy-dev/lib/teracy-dev/processors/manager.rb:40:in `block in process'
/Users/hoatle/teracy-dev/lib/teracy-dev/processors/manager.rb:37:in `each'
/Users/hoatle/teracy-dev/lib/teracy-dev/processors/manager.rb:37:in `process'
/Users/hoatle/teracy-dev/lib/teracy-dev/loader.rb:173:in `process'
/Users/hoatle/teracy-dev/lib/teracy-dev/loader.rb:138:in `build_settings'
/Users/hoatle/teracy-dev/lib/teracy-dev/loader.rb:33:in `start'
/Users/hoatle/teracy-dev/lib/teracy-dev.rb:31:in `<module:TeracyDev>'
/Users/hoatle/teracy-dev/lib/teracy-dev.rb:10:in `<top (required)>'
/opt/vagrant/embedded/lib/ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
/opt/vagrant/embedded/lib/ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
/Users/hoatle/teracy-dev/Vagrantfile:21:in `<top (required)>'
/opt/vagrant/embedded/gems/2.2.0/gems/vagrant-2.2.0/lib/vagrant/config/loader.rb:239:in `load'
/opt/vagrant/embedded/gems/2.2.0/gems/vagrant-2.2.0/lib/vagrant/config/loader.rb:239:in `block in procs_for_path'
/opt/vagrant/embedded/gems/2.2.0/gems/vagrant-2.2.0/lib/vagrant/config.rb:53:in `block in capture_configures'
/opt/vagrant/embedded/gems/2.2.0/gems/vagrant-2.2.0/lib/vagrant/config.rb:48:in `synchronize'
/opt/vagrant/embedded/gems/2.2.0/gems/vagrant-2.2.0/lib/vagrant/config.rb:48:in `capture_configures'
/opt/vagrant/embedded/gems/2.2.0/gems/vagrant-2.2.0/lib/vagrant/config/loader.rb:237:in `procs_for_path'
/opt/vagrant/embedded/gems/2.2.0/gems/vagrant-2.2.0/lib/vagrant/config/loader.rb:223:in `procs_for_source'
/opt/vagrant/embedded/gems/2.2.0/gems/vagrant-2.2.0/lib/vagrant/config/loader.rb:67:in `block in set'
/opt/vagrant/embedded/gems/2.2.0/gems/vagrant-2.2.0/lib/vagrant/config/loader.rb:61:in `each'
/opt/vagrant/embedded/gems/2.2.0/gems/vagrant-2.2.0/lib/vagrant/config/loader.rb:61:in `set'
/opt/vagrant/embedded/gems/2.2.0/gems/vagrant-2.2.0/lib/vagrant/environment.rb:488:in `config_loader'
/opt/vagrant/embedded/gems/2.2.0/gems/vagrant-2.2.0/lib/vagrant/environment.rb:793:in `vagrantfile'
/opt/vagrant/embedded/gems/2.2.0/gems/vagrant-2.2.0/lib/vagrant/environment.rb:178:in `initialize'
/opt/vagrant/embedded/gems/2.2.0/gems/vagrant-2.2.0/bin/vagrant:144:in `new'
/opt/vagrant/embedded/gems/2.2.0/gems/vagrant-2.2.0/bin/vagrant:144:in `<main>'
Vagrant failed to initialize at a very early stage:
There was an error loading a Vagrantfile. The file being loaded
and the error message are shown below. This is usually caused by
a syntax error.
Path: /Users/hoatle/teracy-dev/Vagrantfile
Line number: 38
Message: KeyError: key{node_name_prefix} not found
so that extension configs should have the right path, for example, chef cookbooks_path for https://github.com/teracyhq-incubator/teracy-dev-v05-compat/pull/2/files
We should have <extension-name>-path variable so that, only enabled extensions will have the associated <extension-name>-path, the path should not include slash character (/) at the end.
cookbooks_path:
- teracy-dev-v05-compat/vendor-cookbooks
- teracy-dev-v05-compat/main-cookbooks
should become:
cookbooks_path:
- "%{teracy-dev-v05-compat-path}/vendor-cookbooks"
- "%{teracy-dev-v05-compat-path}/main-cookbooks"
By supporting this, teracyhq-incubator/teracy-dev-v05-compat#2 will not have to create a new configurator but using yaml config only.
see: https://github.com/hashicorp/vagrant/blob/v2.1.2/plugins/kernel_v2/config/vm.rb#L249
so we need to deprecate "mode" (add warn log if mode is used: "'mode' is deprecated, use 'type' instead for networks")
expected:
networks and sync_folders should be available for configuring nodes
actual:
networks and sync_folders was empty
Add docs for reference
when it's ready to be shipped
add optional resources for each extension config and create an ExtensionResources processor (the last one in the pipeline to run to calculate and set VM's memory dynamically by overriding each node provider's config)
users can explicitly set the exact memory, cpus on the provider config (assumed empty by default on the default.providers or node.providers, node.providers has higher precedence)
resources.{minimum, requests}.{memory, cpus} is optional
example config format without user explict memory/cpus setting:
teracy-dev:
extensions:
- _id: "entry-iorad-common"
path:
extension: teracy-dev-iorad-common
location:
git:
remote:
origin: [email protected]:iorad/teracy-dev-iorad-common.git
branch: v0.4.0
require_version: ">= 0.4.0"
enabled: true
resources:
minimum:
memory: 1024
cpus: 1
requests:
memory: 1024
cpus: 1
- _id: "entry-iorad-app"
path:
lookup: workspace
extension: iorad/teracy-dev-iorad-app/master
# extension: iorad/teracy-dev-iorad-app/master # the master stable branch
# extension: iorad/teracy-dev-iorad-app/develop # the develop unstable branch
location:
git:
remote:
origin: [email protected]:iorad/iorad.git
branch: develop
require_version: ">= 0.1.0"
enabled: false
resources:
minimum:
memory: 3072
cpus: 0
requests:
memory: 3072
cpus: 3
- _id: "entry-iorad-extension"
path:
lookup: workspace
extension: iorad-extension/teracy-dev-iorad-extension/master
# extension: iorad-extension/teracy-dev-iorad-extension/master # the master stable branch
# extension: iorad-extension/teracy-dev-iorad-extension/develop # the develop unstable branch
location:
git:
remote:
origin: [email protected]:iorad/iorad-extension.git
branch: develop
require_version: ">= 0.1.0"
enabled: false
resources:
minimum:
memory: 0
cpus: 0
requests:
memory: 1024
cpus: 0
- _id: "entry-iorad-video-export"
path:
lookup: workspace
extension: video-export/teracy-dev-iorad-video-export/master
# extension: video-export/teracy-dev-iorad-video-export/master # the master stable branch
# extension: video-export/teracy-dev-iorad-video-export/develop # the develop unstable branch
location:
git:
remote:
origin: [email protected]:iorad/video-export.git
branch: master
require_version: ">= 0.1.0"
enabled: false
resources:
minimum:
memory: 0
cpus: 0
requests:
memory: 1024
cpus: 0
default:
providers:
- _id: "0" # override
memory: 0 # zero or empty means no set
cpus: "0" # zero or empty means no set
nodes:
- _id: "0"
providers:
- _id: "0" # override the default by each node
memory: ""
cpus: ""
Algorithm:
total minimum memory = total minimum memory of enabled extensions
total requested memoy = total requested memory of the enabled extensions
total minimum cpus = total minimum cpus of enabled extensions
total requested cpus = total requested cpus of enabled extensions
available memory = calculated available memory for the VMs to run
available cpus = calculated available cpus for the VMs to run
explit user-set memory/cpus via default.providers or node.providers (query provider by type to set values correctly: multiple provider support if applicable)
total minimum must be less than or equal to total requested
total_available_memory:
For Mac:
We know that Linux VM use a lot of free memory for cache, buff. Allocate too much memory for VMs will make host machine out of memory and dev will not able to use their machine
for each node, we must allocate available_memory basing on the number of nodes, available_cpus is equal to the total_available_cpus respectively:
for example:
total_available_memory: 6000, number of nodes: 2 => each node has floor(6000/2) = 3000 available_memory
total_available_cpus: 4 => available_cpus for each node: 4
if explicit_user_set_memory exists
if total_minimum_memory <= explicit_user_set_memory <= available_memory:
set VM's memory to explit_user_set_memory
else
abort with error message
else
if available_memory >= total_requested_memory:
set VM's memory to total_requested_memory
else if available_memory >= total_minimum_memory:
warn
set VM's memory to available_memory
else
abort with error message
if explicit_user_set_cpus exists
if total_minimum_cpus <= explicit_user_set_cpus <= available_cpus:
set VM's cpus to explit_user_set_cpus
else
abort with error message
else
if available_cpus >= total_requested_cpus:
set VM's cpus to total_requested_cpus
else if available_cpus >= total_minimum_cpus:
warn
set VM's cpus to available_cpus
else
abort with error message
similar to teracyhq/dev#497
ref: https://www.vagrantup.com/docs/provisioning/basic_usage.html#multiple-provisioners
need to work on this: https://github.com/teracyhq-incubator/teracy-dev-core/blob/develop/lib/teracy-dev-core/config/provisioners.rb
If no "weight" is defined, it's 5 by default.
for example:
nodes:
- _id: "0"
provisioners:
- _id: '0'
version: 13.1.31
type: chef_solo
log_level: info
enabled: true
- _id: '1'
version: 13.1.31
type: chef_solo
log_level: info
enabled: true
weight: 6
- _id: '2'
version: 13.1.31
type: chef_solo
log_level: info
enabled: true
weight: 6
- _id: '3'
version: 13.1.31
type: chef_solo
log_level: info
enabled: true
weight: 3
- _id: '4'
version: 13.1.31
type: chef_solo
log_level: info
enabled: true
weight: 4
=> the expected running order (_id) is: 1, 2, 0, 4, 3
so that variables processor should run very last
depends on: teracyhq/dev#497
After fetching the latest code of teracy-dev, vagrant up is not working
```Administrator@8EIPUXSN50GYWXU /cygdrive/f/teracy/teracy-dev
$ vagrant up
[TeracyDev::Location::GitSynch][INFO]: cd F:/teracy/teracy-dev/lib/../extensions && git clone https://github.com/teracyhq-incubator/teracy-dev-core.git
F:/teracy/teracy-dev/lib/teracy-dev/location/git_synch.rb:59:in `block in sync'
Cloning into 'teracy-dev-core'...
remote: Counting objects: 171, done.
remote: Compressing objects: 100% (95/95), done.
Receiving objectsremote: Total 171 (delta 74), reused 165 (delta 69), pack-reused 0
Receiving objects: 100% (171/171), 28.47 KiB | 170.00 KiB/s, done.
Resolving deltas: 100% (74/74), done.
[TeracyDev::Location::GitSynch][INFO]: cd F:/teracy/teracy-dev/lib/../extensions/teracy-dev-core && git checkout v0.2.0
F:/teracy/teracy-dev/lib/teracy-dev/location/git_synch.rb:64:in `block in sync'
Note: checking out 'v0.2.0'.
You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.
If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:
git checkout -b <new-branch-name>
HEAD is now at ded0fc7 Merge pull request #14 from hoatle/tasks/#13-v0.2.0-release
F:/teracy/teracy-dev/extensions/teracy-dev-core/lib/teracy-dev-core/config/virtualbox_provider.rb:17:in ``': No such file or directory - VBoxManage --version (Errno::ENOENT)
from F:/teracy/teracy-dev/extensions/teracy-dev-core/lib/teracy-dev-core/config/virtualbox_provider.rb:17:in `block in configure_node'
from F:/teracy/teracy-dev/extensions/teracy-dev-core/lib/teracy-dev-core/config/virtualbox_provider.rb:11:in `each'
from F:/teracy/teracy-dev/extensions/teracy-dev-core/lib/teracy-dev-core/config/virtualbox_provider.rb:11:in `configure_node'
from F:/teracy/teracy-dev/lib/teracy-dev/config/configurator.rb:17:in `configure'
from F:/teracy/teracy-dev/lib/teracy-dev/config/manager.rb:33:in `block in configure'
from F:/teracy/teracy-dev/lib/teracy-dev/config/manager.rb:32:in `each'
from F:/teracy/teracy-dev/lib/teracy-dev/config/manager.rb:32:in `configure'
from F:/teracy/teracy-dev/lib/teracy-dev/loader.rb:135:in `configure'
from F:/teracy/teracy-dev/lib/teracy-dev/loader.rb:148:in `block (3 levels) in configure_vagrant'
from C:/HashiCorp/Vagrant/embedded/gems/2.1.0/gems/vagrant-2.1.0/lib/vagrant/config/v2/loader.rb:37:in `load'
from C:/HashiCorp/Vagrant/embedded/gems/2.1.0/gems/vagrant-2.1.0/lib/vagrant/config/loader.rb:126:in `block (2 levels) in load'
from C:/HashiCorp/Vagrant/embedded/gems/2.1.0/gems/vagrant-2.1.0/lib/vagrant/config/loader.rb:119:in `each'
from C:/HashiCorp/Vagrant/embedded/gems/2.1.0/gems/vagrant-2.1.0/lib/vagrant/config/loader.rb:119:in `block in load'
from C:/HashiCorp/Vagrant/embedded/gems/2.1.0/gems/vagrant-2.1.0/lib/vagrant/config/loader.rb:116:in `each'
from C:/HashiCorp/Vagrant/embedded/gems/2.1.0/gems/vagrant-2.1.0/lib/vagrant/config/loader.rb:116:in `load'
from C:/HashiCorp/Vagrant/embedded/gems/2.1.0/gems/vagrant-2.1.0/lib/vagrant/vagrantfile.rb:166:in `machine_config'
from C:/HashiCorp/Vagrant/embedded/gems/2.1.0/gems/vagrant-2.1.0/lib/vagrant/environment.rb:322:in `default_provider'
from C:/HashiCorp/Vagrant/embedded/gems/2.1.0/gems/vagrant-2.1.0/lib/vagrant/plugin/v2/command.rb:177:in `block in with_target_vms'
from C:/HashiCorp/Vagrant/embedded/gems/2.1.0/gems/vagrant-2.1.0/lib/vagrant/plugin/v2/command.rb:204:in `block in with_target_vms'
from C:/HashiCorp/Vagrant/embedded/gems/2.1.0/gems/vagrant-2.1.0/lib/vagrant/plugin/v2/command.rb:186:in `each'
from C:/HashiCorp/Vagrant/embedded/gems/2.1.0/gems/vagrant-2.1.0/lib/vagrant/plugin/v2/command.rb:186:in `with_target_vms'
from C:/HashiCorp/Vagrant/embedded/gems/2.1.0/gems/vagrant-2.1.0/plugins/commands/up/command.rb:87:in `execute'
from C:/HashiCorp/Vagrant/embedded/gems/2.1.0/gems/vagrant-2.1.0/lib/vagrant/cli.rb:54:in `execute'
from C:/HashiCorp/Vagrant/embedded/gems/2.1.0/gems/vagrant-2.1.0/lib/vagrant/environment.rb:275:in `cli'
from C:/HashiCorp/Vagrant/embedded/gems/2.1.0/gems/vagrant-2.1.0/bin/vagrant:156:in `<main>'
Administrator@8EIPUXSN50GYWXU /cygdrive/f/teracy/teracy-dev
Try the command "VBoxManage" ==> the command is not found
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.