lazywolves / elasticpyproxy Goto Github PK

Add relevant doc for changes made in issue #4

As of now EP2 is capable of managing only one HAProxy backend.
The current procedure is:

• mention the name of the backend in the config in haproxy section
• this backend must be present in the template
• there is one orchestrator section (which is only aws) as of now.

However this methodology is limiting us to only one backend. This issue is about allowing EP2 manage multple backends rather than just one.

A sample new conf file with this feature implemented might look like this:

[haproxy]
haproxy_config_file = /etc/haproxy/haproxy.cfg
template_file = /home/deep/elasticpyproxy/etc/haproxy.config.template
backend_port = 6003
haproxy_binary = /usr/sbin/haproxy
start_by = systemd
haproxy_socket_file = /var/run/haproxy/haproxy.sock
pid_file = /run/haproxy.pid
backend_names = haproxynode1,haproxynodes2
update_type = update_by_runtime
node_slots = 5
service_name = haproxy
lock_dir = /home/deep/elasticpyproxy/etc
sleep_before_next_run = 5
log_file = /var/log/ep2/ep2.log

[haproxynode1]
orchestrator = aws
asg_name = 
region_name =

[haproxynode2]
orchestrator = aws
asg_name = 
region_name =

[AWSCREDS]
aws_access_key_id =
aws_secret_access_key =

Instead of backend_name we now have a param called backend_names which takes a list of comma separated backend names.

For each backend name we have a section, which mentions from where the backend should be fetched and what is the orchestrator (only aws as of now).

A new section is also introduced for aws creds to keep it completely decoupled. However each section might have its own aws creds too. In that case the sectional aws creds will overwrite the global aws creds. In that way we might be able to create backends pertaining to different orchestrator accounts.

All these stuff will require quite some amount of code change and documentation updation.

Post #4 the config file now accepts a list of comma separated socket file when EP2/HAProxy is configured in multi process mode.
Since it now accepts a list of sockets, the reload by binary method should only pick the first socket file to continue with the reload. It is assumed that the first socket file represents the socket with expos fd-listener.

Doc has to be updated accordingly.

As of now there are plenty of lint errors in EP2. These errors needs to be fixed.

Steps

• Install dev dependencies using sudo pip3 install -r requirements-dev.txt
• open .flake8 file present at the root of the project.
• remove one rule from the ignore list.
• run flake8
• Solve the errors reported by flake8
• Open an issue for the rule you are fixing (one rule per issue)
• Refer this issue there
• Send a PR. In the PR too, refer this issue as parent.
• Comment that you are working on it.

The param asg_name now supports multiple ASGs separated by comma.
So now it should be plural and not singular.
However, updating the param name means the dictionary lookups also needs to be updated and
corresponding variable names should also be updated for better nomenclature.

All tests should be moved to docker environment.

Mostly this wont require any changes from python testing side, but this essentially would require a
docker fille, running a container using which will run the tests.
This would keep all the changes made to HAProxy and EP2 while tesing isolated from the HAProxy
running on our host.

Depends on #11

When Haproxy is run multi process mode, there will be multiple sockets, each bound to one process.
When udating by runtime, the socket commands should be sent to all the sockets.

The objective of this issue is to Dockerise EP2.
Once EP2 is dockerised and we know its working fine, going forward we can move all our tests to docker.
That way we wont be disturbing the existing HAProxy setup for testing. Also we can integrate with
Travis dockerised testing when new code is pushed.

Currently, number of tests are very less, EP2 requires more tests.

Steps:

• For adding tests for a particular module, open an issue.
• In the issue, refer this issue as parent.
• Add the tests
• Refer this issue in the PR
• Send PR.

To find out modules which require more tests, following steps can be performed:

• Run sudo pip3 install -r requirements.txt
• Run ```pytest --cov=./
• Check the report and find out modules with 0 or very less coverage.
• Write tests for those modules.

When EP2 tries to fetch the back end nodes using boto3 api call and if for some reason no back ends are available, EP2 throws an exception. The exception mostly indicates that logger is a NoneType object.

Instead of the above, the exception should be logged in the log file.

Documentations for contributors needs to be improved.

Presently only way to run EP2 mentioned in the doc is to run sudo python3 setup.py install
Running this command again and again while developing can prove to be tiring.

EP2 can be also run as a module using python3 -m src.driver.driver -f [config_file]
This needs to be mentioned in the doc.
So for development purpose, one can follow the following steps to run Saberx and test their
changes:

• sudo pip3 install -r requirements-dev.txt
• hack, hack, hack...
• python3 -m src.driver.driver -f [config_file]

Also there needs to be doc for running tests, lint test and coverage.

Tests can be performed by simple running pytest from the root directory.
Lint test can performed by simply running flake8 from the root directory
Coverage can be determined by running pytest --cov=./ from the root directory

Lastly, while developing and adding new features or fixing bugs, one might want to keep it
separate from the global installation. This can be achieved by using any isolation tool, like
virtualenv. We also need doc on how to setup EP2 from within virtualenv.

All these things needs to be documented in CONTRIBUTING.md.

Presently the asg_name accepts only one asg name and EP2 monitors only that ASG.

Expected:
asg_name should accept a list of asg names and monitor all of them. So essentially, all the nodes
in all the monitored ASGs should get added to HAProxy.

The doc needs to be update in order to add the changes done in issue #3

In short, the issue was about adding the capability to monitor changes in multiple ASGs for the
same backend.
For example, now for asg_name parameter we can define multiple ASGs separated by comma:
asg_name: asg1,asg2,asg3

However this feature is yet not mentioned in the doc.
The PR for this issue is this