openbaton / ems Goto Github PK

I am trying to deploy the iperf network service on a server that seems to be properly running both openbaton and openstack. The problem I have encountered is that when deploying such network service from OpenBaton dashboard (or command line) both network service record and virtual network function records get stuck in the ERROR state "org.openbaton.common.vnfm_sdk.exception.VnfmSdkException: Timeout waiting for EMS: iperf-server-9598388".
It seemed to me that the problem was related to EMS inside VNF (Virtual Network Function) so I sshed into the virtual machine running the VNF iperf-server and I tried to manually install EMS using your instructions to better understand what the problem was. All commands run just fine but "pip install openbaton-ems" whose log is the following:

Downloading/unpacking openbaton-ems
Getting page https://pypi.python.org/simple/openbaton-ems/
URLs to search for versions for openbaton-ems:

• https://pypi.python.org/simple/openbaton-ems/
  Analyzing links from page https://pypi.org/simple/openbaton-ems/
  Found link https://files.pythonhosted.org/packages/61/46/65220fe85b6914a3764080a63b79a0ad43603971eae2198ba071bccaf362/openbaton-ems-1.0.3.tar.gz#sha256=b574a7a9fe06cb86427d8b1bea67c3849c24454bbdf8d698d1ad9a025f969c33 (from https://pypi.org/simple/openbaton-ems/), version: 1.0.3
  Found link https://files.pythonhosted.org/packages/86/f8/97cf73a527bd6f91cdaca3333624d4a772e01cda51789de93288915c06d5/openbaton-ems-1.0.8.tar.gz#sha256=353f0057929a21c675d846df1d12d1dbaf647440a4439ecbf6a89e79774696b3 (from https://pypi.org/simple/openbaton-ems/), version: 1.0.8
  Found link https://files.pythonhosted.org/packages/8b/28/809b3f408d10a1fddc7ca6a02ad9f62de751f4dbfd7cf780a6bcf45854e4/openbaton-ems-1.0.9.tar.gz#sha256=a63946a5a3f4ed805b92e8d0aaaeeb6c0ea3269acc0bfd627c27ee5150d6ca3a (from https://pypi.org/simple/openbaton-ems/), version: 1.0.9
  Using version 1.0.9 (newest of versions: 1.0.9, 1.0.8, 1.0.3)
  Downloading openbaton-ems-1.0.9.tar.gz
  Downloading from URL https://files.pythonhosted.org/packages/8b/28/809b3f408d10a1fddc7ca6a02ad9f62de751f4dbfd7cf780a6bcf45854e4/openbaton-ems-1.0.9.tar.gz#sha256=a63946a5a3f4ed805b92e8d0aaaeeb6c0ea3269acc0bfd627c27ee5150d6ca3a (from https://pypi.org/simple/openbaton-ems/)
  Running setup.py (path:/tmp/pip_build_root/openbaton-ems/setup.py) egg_info for package openbaton-ems
  /usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'entry_points'
  warnings.warn(msg)
  /usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'install_requires'
  warnings.warn(msg)
  usage: -c [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
  or: -c --help [cmd1 cmd2 ...]
  or: -c --help-commands
  or: -c cmd --help

  error: invalid command 'egg_info'
  Complete output from command python setup.py egg_info:
  /usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'entry_points'

warnings.warn(msg)

/usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'install_requires'

warnings.warn(msg)

usage: -c [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]

or: -c --help [cmd1 cmd2 ...]

or: -c --help-commands

or: -c cmd --help

error: invalid command 'egg_info'

Cleaning up...
Removing temporary dir /tmp/pip_build_root...
Command python setup.py egg_info failed with error code 1 in /tmp/pip_build_root/openbaton-ems
Exception information:
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/pip/basecommand.py", line 122, in main
status = self.run(options, args)
File "/usr/lib/python2.7/dist-packages/pip/commands/install.py", line 278, in run
requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundle=self.bundle)
File "/usr/lib/python2.7/dist-packages/pip/req.py", line 1230, in prepare_files
req_to_install.run_egg_info()
File "/usr/lib/python2.7/dist-packages/pip/req.py", line 326, in run_egg_info
command_desc='python setup.py egg_info')
File "/usr/lib/python2.7/dist-packages/pip/util.py", line 715, in call_subprocess
% (command_desc, proc.returncode, cwd))
InstallationError: Command python setup.py egg_info failed with error code 1 in /tmp/pip_build_root/openbaton-ems

I am not sure about it but it seems that this problem prevents ems from installing and it causes the VNFM (Virtual Network Function Manager) to timeout while waiting for VNF's EMS to reply. I tried to manually install other version of EMS but still the same problem came up. How can I solve it? Is the problem only related to EMS or it regards also the generic VNFM? Are the warnings about entry_points and install_requires related to the failure? Any help would be very much appreciated.
NB: It may be worth mentioning that when I sshed to VM running iperf-server VNF I run also the command "sudo apt-get install -y python-pip". This command actually installed pip so it wasn't run automatically during deployment as you say in your guide.
Thank you very much
Ferretti Valerio

Sometimes the ems stop running without clear reason: this is the last log lines:

INFO:receiver:Received message:
INFO:receiver:{"action":"EXECUTE","payload":"scscf_install.sh","env":{"port":"6060","mgmt":"192.168.133.175","name":"scscf","hostname":"scscf-7544977","mgmt_floatingIp":"192.168.161.160","diameter_p":"3870"}}
INFO:receiver:Received execute command
INFO:receiver:Executing: /opt/openbaton/scripts/scscf_install.sh with env {u'name': u'scscf', u'hostname': u'scscf-7544977', u'diameter_p': u'3870', u'mgmt': u'192.168.133.175', u'mgmt_floatingIp': u'192.168.161.160', u'port': u'6060'}
INFO:receiver:Executed: ERR:  OUT: scscf : Using custom script path /opt/openbaton/scripts
scscf : Installing packages
scscf : Finished installing packages
scscf : Checking out source-code
scscf : Setting Service Upstart scscf on
scscf : compiling OpenIMSCore

INFO:receiver:answer is: {"status": 0, "output": "scscf : Using custom script path /opt/openbaton/scripts\nscscf : Installing packages\nscscf : Finished installing packages\nscscf : Checking out source-code\nscscf : Setting Service Upstart scscf on\nscscf : compiling OpenIMSCore\n", "err": ""}
INFO:__main__:Answer sent
ERROR:pika.adapters.base_connection:Read empty data, calling disconnect
WARNING:pika.adapters.base_connection:Socket closed when connection was open
WARNING:pika.connection:Disconnected from RabbitMQ at 10.147.66.131:5672 (0): Not specified
CRITICAL:pika.adapters.blocking_connection:Connection close detected

connection was not close by the Generic, or at least it was not stopped.
It needs further investigations.

here:

https://github.com/openbaton/ems/blob/develop/receiver.py#L112-L115

in order to support python 3 we need to run python3 instead of python. also in the second case /bin/bash does not work even if the first line is correctly setup

it it get a timeout it does not try again. Sometimes it could be useful to try some more then one time.

Hello.

We would like to run the iPerf Integration Test, but we are behind an HTTP Proxy. The Generic VNFM's default "user-data.sh" script, sent via cloud-init to the VM, tries to "wget" the EMS repository, but it fails to proceed because of http proxy.

We have created a "custom" /etc/openbaton/openbaton-vnfm-generic-user-data.sh file, corresponding to the one we found on https://github.com/openbaton/generic-vnfm/blob/master/src/main/resources/user-data.sh, in which we have added "export http_proxy=XXX" and "export https_proxy=XXX" lines, and the first problem was solved: the user-data.sh script managed to "wget" the EMS repository and to install EMS.
The script is also creating a /etc/profile.d/myproxy.sh file containing the same http(s) proxy export lines.

Now, we are facing a problem where the EMS itself is doing a "git clone"... Unfortunately, the EMS service is started by pid=1, which was running BEFORE our /etc/profile.d/myproxy.sh file was created. So, the http(s) proxy environment variables are NOT defined, so the EMS fails to do its "git clone".

We are also anticipating that we'll also face another problem if we manage to handle that one: the EMS will try to contact OpenBaton's Generic VNFM, but these calls MUST NOT go through the HTTP Proxy... So, we'll also have to define a "no_proxy" environment variable containing the IP of OpenBaton's Generic VNFM - is there a way to have this IP somehow in the user-data.sh script (something like the #hostname which seems to be replaced at some point)?

And for iPerf Integration Test, the client will try to contact the server - if the request is done through HTTP, we'll also have to add the server's IP address in the "no_proxy" environment variable. That IP address is dynamically created and provided through a configuration parameter to the client, but is there a way to have access to it in our user-data.sh script (and if yes, how)?

Thanks for your attention,
Daniel

PS: a workaround for that specific use case is to have user-data.sh configuring GIT to use the proxy with "git config --global --add http.proxy " and "git config --global --add https.proxy " and not creating the /etc/profile.d/myproxy.sh file, but for other use cases where we need to have the proxy environment variables defined, it would still apply...

Edit: we "solved" the problem by:

• defining http_proxy and https_proxy variables in the script iself (as explained above)
• NOT creating the /etc/profile.d/myproxy.sh file, despite what was mentioned previously. Otherwise, we would need to also create a "no_proxy" environment variable, with variable IPs...
• creating a /root/.gitconfig file containing the Proxy lines
• creating a /etc/apt/apt.conf.d/proxy file containing the Proxy lines

But for me, it looks more like a workaround than a real solution...

According to the changelog of pika (https://github.com/pika/pika/blob/345c9626ff3e79691241ecafe1573d3cf1ae0e5d/CHANGELOG.rst) the heartbeat_interval parameter of pika.ConnectionParameters was removed on 2019-03-26 (it was deprecated on 2017-07-29). Because ems.py uses the heatbeat_interval parameter in its call to pika.ConnectionParameters, an exception "unexpected kwargs: {'heartbeat_interval': 120}" is generated when it tries to connect. This exception is silently ignored but causes the connection to the RabbitMQ server to fail on every retry. Consequently, ems suddenly stopped working in deployments created after 2019-03-26.