piku / piku Goto Github PK

View Code? Open in Web Editor NEW

2.6K 28.0 88.0 3.17 MB

The tiniest PaaS you've ever seen. Piku allows you to do git push deployments to your own servers.

Home Page: http://piku.github.io

License: MIT License

Python 95.10% Shell 4.63% Makefile 0.27%

arm raspberry-pi paas python nodejs raspbian ubuntu heroku

piku's Introduction

piku, inspired by dokku, allows you do git push deployments to your own servers, no matter how small they are.

Demo

Documentation: Install | Using | Procfile | ENV | Examples | Roadmap | Contributing | LinuxConf Talk | Fast Web App Tutorial | Discussion Forum

Install

TL;DR:

curl https://piku.github.io/get | sh

There are also other installation methods available, including cloud-init and manual installation.

Project Activity

piku is considered STABLE. It is actively maintained, but "actively" here means the feature set is pretty much done, so it is only updated when new language runtimes are added or reproducible bugs crop up.

It currently requires Python 3.7 or above, since even though 3.8+ is now the baseline Python 3 version in Ubuntu LTS 20.04 and Debian 11 has already moved on to 3.9, there are no substantial differences between those versions.

Motivation

We wanted an Heroku/CloudFoundry-like way to deploy stuff on a few ARM boards, but since dokku didn't work on ARM at the time and even docker can be overkill sometimes, a simpler solution was needed.

piku is currently able to deploy, manage and independently scale multiple applications per host on both ARM and Intel architectures, and works on any cloud provider (as well as bare metal) that can run Python, nginx and uwsgi.

Workflow

piku supports a Heroku-like workflow:

Create a git SSH remote pointing to your piku server with the app name as repo name: git remote add piku piku@yourserver:appname.
Push your code: git push piku master (or if you want to push a different branch than the current one use git push piku release-branch-name).
piku determines the runtime and installs the dependencies for your app (building whatever's required).
- For Python, it segregates each app's dependencies into a virtualenv.
- For Go, it defines a separate GOPATH for each app.
- For Node, it installs whatever is in package.json into node_modules.
- For Java, it builds your app depending on either pom.xml or build.gradle file.
- For Clojure, it can use either leiningen or the Clojure CLI and a deps.edn file.
- For Ruby, it does bundle install of your gems in an isolated folder.
It then looks at a Procfile and starts the relevant workers using uwsgi as a generic process manager.
You can optionally also specify a release worker which is run once when the app is deployed.
You can then remotely change application settings (config:set) or scale up/down worker processes (ps:scale).
You can also bake application and nginx settings into an ENV file. You can also deploy a gh-pages style static site using a static worker type, with the root path as the argument, and run a release task to do some processing on the server after git push.

Virtual Hosts and SSL

piku has full virtual host support - i.e., you can host multiple apps on the same VPS and use DNS aliases to access them via different hostnames.

piku will also set up either a private certificate or obtain one via Let's Encrypt to enable SSL.

If you are on a LAN and are accessing piku from macOS/iOS/Linux clients, you can try using piku/avahi-aliases to announce different hosts for the same IP address via Avahi/mDNS/Bonjour.

Caching and Static Paths

Besides static sites, piku also supports directly mapping specific URL prefixes to filesystem paths (to serve static assets) or caching back-end responses (to remove load from applications).

These features are configured by setting appropriate values in the ENV file.

Supported Platforms

piku is intended to work in any POSIX-like environment where you have Python, nginx, uwsgi and SSH: it has been deployed on Linux, FreeBSD, Cygwin and the Windows Subsystem for Linux.

As a baseline, it began its development on an original 256MB Rasbperry Pi Model B, and still runs reliably on it.

But its main use is as a micro-PaaS to run applications on cloud servers with both Intel and ARM CPUs, with Debian and Ubuntu Linux as target platforms.

Supported Runtimes

piku currently supports apps written in Python, Node, Clojure, Java and a few other languages (like Go) in the works.

But as a general rule, if it can be invoked from a shell, it can be run inside piku.

Core values

Run on low end devices.
Accessible to hobbyists and K-12 schools.
~1500 lines readable code.
Functional code style.
Few (single?) dependencies
12 factor app.
Simplify user experience.
Cover 80% of common use cases.
Sensible defaults for all features.
Leverage distro packages in Raspbian/Debian/Ubuntu (Alpine and RHEL support is WIP)
Leverage standard tooling (git, ssh, uwsgi, nginx).
Preserve backwards compatibility where possible

piku's People

Contributors

Stargazers

Watchers

Forkers

madsmadsen kernelpanicaus marchon zoosky luizribeiro cloudxtreme tf198 pnftech smcl chr15m devseq onlymejosh sousaworld matrixjnr ggornes jsenin dwightmulcahy longjohncoder csmart spoxaka vrehm zeta1999 todokku kennest xet7 kakamband xydinesh archiephan78 rheehot ni101 rssnyder nelfin simhaonline hirajanwin johntellsall fujohnwang 0xbadcoffe abhirkeesara bardswork demagnevalcegelec varianone ekmixon ediboko1980 namsral baharsah angordeyev alperkal noway56 mlainez timhuebener verakai fhchina kinostl liamdgray clusterfudge pedronet4k mindkhichdi gregkrsak kianmeng ewalk153 5l1v3r1 webclinic017 cloudnepal axaryk lgtm-migrator icodein geekwolverine amirulandalib kaseyhinton mariusa pfw umarahmad125 toburger jfmatth sgnconnects jaberwiki al-rikabi gmh5225 ego brunoscaglione vck yoshitakanaraoka jczzle t2be jpdro jpedro rohanrusta21

piku's Issues

detecting a Dockerfile and building an image?

Hi great work on this. Is it possible to push an arbitrary dockerized app to Piku and skip any auto detection on the source code by simply detecting a Dockerfile and building an image out of that?

Originally posted by @IRobL in #11 (comment)

Pushing update to example app does not seem to change running version

I've pushed the example in examples/python/ to my piku instance and it works great. The only change I made was to add the variable NGINX_SERVER_NAME so that I know where to see the result.

However, when I modified main.py and changed the output tables to add some random text I noticed that the text does not show up in the page after I commit and push to piku/master again. Looking at the logs it does not seem like uwsgi is restarting the server. What should I look for to diagnose this, or is it expected behaviour?

piku-bootstrap sub-command to install the piku script

piku-bootstrap install-helper ~/bin/ should make a symlink to the script.

Deployment broken for all workers except jwsgi

Currently pushing any worker type the following is thrown:

remote: Warning: found both 'wsgi' and 'web' workers, disabling 'web'

Probably we need some basic tests to catch this kind of stuff.

NGINX_STATIC_MAPPINGS is being written into LIVE_ENV

NGINX_STATIC_MAPPINGS is being written into LIVE_ENV. The problem with that is that if you use NGINX_STATIC_PATHS when deploying with static files, you get an invalid LIVE_ENV that looks like this:

NGINX_STATIC_PATHS=/static/:conf/staticfiles/
NGINX_STATIC_MAPPINGS=
  location /static/ {
      sendfile on;
      sendfile_max_chunk 1m;
      tcp_nopush on;
      directio 8m;
      aio threads;
      alias /home/piku/.piku/apps/djangosandbox/conf/staticfiles/;
  }

APP=djangosandbox

This causes commands such as piku run to fail:

% piku run djangosandbox ./manage.py
Error: malformed setting '  location /static/ {
', ignoring file.
Traceback (most recent call last):
  File "./manage.py", line 8, in <module>
    from django.core.management import execute_from_command_line                                                                                                                                                                              ImportError: No module named 'django'

SCALING file

From reading piku.py the SCALING file is created by piku.py in the ENV_ROOT directory.

I wonder if this would be useful to expose this into the APP_ROOT so that it could be created alongside the ENV file so that the process scaling could be created at the beginning instead of having to bump the processes with the ps:scale command. I know this is a one time deal since the ENV_ROOT directory is persistent between deployments, just might be useful to have this as something that could be specified before deployment time.

FriendlyARM install did not start piku.py on ssh login.

I did everything to set up my Raspbery compatible device with piku. All went fine till last testing part.

When I log in to box it did not give me output you show at RASPBERRY_PI_QUICKSTART.md

After this I found that repository is not created at piku machine after push. I actually cannot push at all.
(response was there is no app/repository)

PS. Also there was problem with .ssh directory rights, but I did solve it.

Binaries on linux(Ubuntu)

It is probably good if you use the virtualenv from Ubuntu repository during installation.
I also noticed that piku's check_requirements(binaries) does not recognize binaries that are exported on the .bashrc file.

So install whatever you need using package manager.

$sudo apt-get install virtualenv maven nodejs etc.

"social preview" can use new logo

@rcarmo if you go into settings you can add the new logo to the "social preview":

This means when github.com/rcarmo/piku gets shared on Twitter or other social media the logo will show up. Small thing but I think it helps with marketing.

Document Procfile

The Procfile documentation is spread out in various places. Would be good to bring it together into a single place like the ENV file.

Python simple.http server in `web` worker mode tries to start twice

The log is full of this:

web.1 | [uwsgi-daemons] respawning "./redirector.py" (uid: 1000 gid: 33)
web.1 | Starting redirector HTTPServer.
web.1 | Server params: https://exceltohtml.net/ 52401
web.1 | Traceback (most recent call last):
web.1 |   File "./redirector.py", line 24, in <module>
web.1 |     HTTPServer(("127.0.0.1", int(port)), Redirect).serve_forever()
web.1 |   File "/usr/lib/python3.5/socketserver.py", line 440, in __init__
web.1 |     self.server_bind()
web.1 |   File "/usr/lib/python3.5/http/server.py", line 138, in server_bind
web.1 |     socketserver.TCPServer.server_bind(self)
web.1 |   File "/usr/lib/python3.5/socketserver.py", line 454, in server_bind
web.1 |     self.socket.bind(self.server_address)
web.1 | OSError: [Errno 98] Address already in use
web.1 | [uwsgi-daemons] throttling "./redirector.py" for 31 seconds

Replicate this bug using this repository.

installation does not validate git it's installed

Installation does not validate if git client it's installed, does not exit or notify installation error.

root@piku:~# ./piku-bootstrap
Looks like this is your first time running piku-bootstrap.
This script will self-install dependencies into /root/.piku-bootstrap now.
Hit enter to continue or ctrl-C to abort.


 #> Virtualenv setup not found. Installing it into /root/.piku-bootstrap.
 #> Downloading & installing Virtualenv.
 #> Setting up the virtual environment.
 #> Piku repo not found. Installing it into /root/.piku-bootstrap/piku.
**./piku-bootstrap: 39: ./piku-bootstrap: git: not found**
 #> ansible-playbook binary not found. Installing it into /root/.piku-bootstrap.

Usage:
....

RFC: GitHub org and repo splits

This is more of a discussion thread - the former owner of @piku has agreed to forego his username so that we can build a GitHub org under that name, and I was thinking of splitting up this repository into two or three things in different repos:

A (badly needed) documentation site (probably managed with something like docsifyj
The core repo, with just the main Python script and required config files
everything else (provisioning scripts, the Docker image I’ve been dabbling with, maybe even the scripts to build an SD card image I haven’t published)

Thoughts? I haven’t set up the org properly yet (just created it before breakfast to hold on to the name), but would like to get feedback before starting to break repo URLs and suchlike :)

Support for Lua/WSAPI

I'm considering hacking this in sometime in the near future, probably enforcing a threading model. I could do it inside nginx too, but I might need some TCP port servers and not just HTTP.

NGINX_COMMON is being written into LIVE_ENV

Ugh, I've introduced a truly facepalming bug in #77. 🤦‍♂️

A bunch of vars are being leaked into LIVE_ENV at write time.

My LIVE_ENV looks like this at the moment:

APP=pypostgres
NGINX_IPV6_ADDRESS=[::]
PATH=/home/piku/.piku/envs/pypostgres/bin:/home/piku/.piku/envs/pypostgres/bin:/usr/lib/git-core:/usr/lib/git-core:/usr/lib/git-core:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games
NGINX_CUSTOM_CLAUSES=
VIRTUAL_ENV=/home/piku/.piku/envs/pypostgres
NGINX_BLOCK_GIT=location ~ /\.git { deny all; }
NGINX_COMMON=
  listen              [::]:443 ssl http2;
  listen              0.0.0.0:443 ssl http2;
  ssl                 on;
  ssl_certificate     /home/piku/.piku/nginx/pypostgres.crt;
  ssl_certificate_key /home/piku/.piku/nginx/pypostgres.key;
  server_name         pypostgres.piku.mccormickit.com;

  # These are not required under systemd - enable for debugging only
  # access_log        /home/piku/.piku/logs/pypostgres/access.log;
  # error_log         /home/piku/.piku/logs/pypostgres/error.log;
  
  # Enable gzip compression
  gzip on;
  gzip_proxied any;
  gzip_types text/plain text/xml text/css application/x-javascript text/javascript application/xml+rss application/atom+xml;
  gzip_comp_level 7;
  gzip_min_length 2048;
  gzip_vary on;
  gzip_disable "MSIE [1-6]\.(?!.*SV1)";
  
  # set a custom header for requests
  add_header X-Deployed-By Piku;

  

  
  location /static {
      sendfile on;
      sendfile_max_chunk 1m;
      tcp_nopush on;
      directio 8m;
      aio threads;
      alias /home/piku/.piku/apps/pypostgres/static;
  }


  location ~ /\.git { deny all; }

  location    / {
    
    uwsgi_pass pypostgres;
    uwsgi_param QUERY_STRING $query_string;
    uwsgi_param REQUEST_METHOD $request_method;
    uwsgi_param CONTENT_TYPE $content_type;
    uwsgi_param CONTENT_LENGTH $content_length;
    uwsgi_param REQUEST_URI $request_uri;
    uwsgi_param PATH_INFO $document_uri;
    uwsgi_param DOCUMENT_ROOT $document_root;
    uwsgi_param SERVER_PROTOCOL $server_protocol;
    uwsgi_param REMOTE_ADDR $remote_addr;
    uwsgi_param REMOTE_PORT $remote_port;
    uwsgi_param SERVER_ADDR $server_addr;
    uwsgi_param SERVER_PORT $server_port;
    uwsgi_param SERVER_NAME $server_name;

    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
    proxy_set_header Host $http_host;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header X-Forwarded-For $remote_addr;
    proxy_set_header X-Forwarded-Port $server_port;
    proxy_set_header X-Request-Start $msec;
    
 }
HOME=/home/piku
NGINX_IPV4_ADDRESS=0.0.0.0
NGINX_ACL=

Uwsgi deploys are broken

This is due to a typo in my previous PR: https://github.com/piku/piku/pull/97/files#diff-bc9247ae52bec05571146b5895110a89R714

Sending a patch now.

Exposing the .git folder to the web is a security risk

Putting this here so I have something to PR against (preparing a patch now).

"no request plugin is loaded" warning when deploying simple app

I'm seeing this WARNING when starting a test app to piku...

web.1    | *** Operational MODE: no-workers ***
web.1    | !!!!!!!!!!!!!! WARNING !!!!!!!!!!!!!!
web.1    | no request plugin is loaded, you will not be able to manage requests.
web.1    | you may need to install the package for your language of choice, or simply load it with --plugin.
web.1    | !!!!!!!!!!! END OF WARNING !!!!!!!!!!
web.1    | spawned uWSGI master process (pid: 3974)
web.1    | [uwsgi-daemons] spawning "python main.py" (uid: 1001 gid: 33)
web.1    | 14-Jun 08:58:38-INFO: started main.py
web.1    |  * Serving Flask app "main" (lazy loading)
web.1    |  * Environment: production
web.1    |    WARNING: This is a development server. Do not use it in a production deployment.
web.1    |    Use a production WSGI server instead.
web.1    |  * Debug mode: on
web.1    | 14-Jun 08:58:39-INFO:  * Running on http://127.0.0.1:9191/ (Press CTRL+C to quit)
web.1    | 14-Jun 08:58:39-INFO:  * Restarting with stat
web.1    | 14-Jun 08:58:41-INFO: started main.py
web.1    | 14-Jun 08:58:42-WARNING:  * Debugger is active!
web.1    | 14-Jun 08:58:42-INFO:  * Debugger PIN: 921-136-616

my ProcFile:

web: python main.py
worker: python worker.py

I've also tried wsgi: main:app with the same results.

my main.py is a very simple Flask app.

import flask
import requests

app = flask.Flask(__name__)

@app.route("/")
def hello():
    logging.info('root restapi endpoint executing')
    return "<b>Not implemented yet...</b>"

if __name__ == '__main__':
    app.run(host='127.0.0.1', port=9191, debug=True)

very simple requirements.txt file:

requests
flask

when I open piku.local:9191 I get

This site can’t be reached 
piku.local refused to connect.

any suggestions?

Split off sample projects into separate repositories

Define name spacing (sample-language-app)
Create new repositories for each sample project
Assign contributors to each repository

Not available piku-bootstrap install-cli

At README.md installation refers to a way to install piku cli using piku-bootstrap but its only available at commit https://github.com/piku/piku/blob/60da61456ed3f44c5d6691ed7a55ecbe0dfff382/piku-bootstrap

Readme reference:

To make life easier you can also install the piku helper CLI. Install it into your path e.g. ~/bin to run it from anywhere.

./piku-bootstrap install-cli ~/bin

node deploy: modifying package.json tries to re-install node with nvm

@rcarmo this bug can be assigned to me to fix.

remote: -----> Installing node version '10.16.0' using nodeenv
remote:  * Environment already exists: /home/piku/.piku/envs/excelhtmltable
remote:  * Install node (10.16.0)..cp: cannot create regular file '/home/piku/.piku/envs/excelhtmltable/bin/node': Text file busy
remote: cp: cannot create regular file '/home/piku/.piku/envs/excelhtmltable/bin/node': Text file busy
remote: Traceback (most recent call last):
remote:   File "/usr/bin/nodeenv", line 9, in <module>
remote:     load_entry_point('nodeenv==0.13.4', 'console_scripts', 'nodeenv')()
remote:   File "/usr/lib/python3/dist-packages/nodeenv.py", line 890, in main
remote:     create_environment(env_dir, opt)
remote:   File "/usr/lib/python3/dist-packages/nodeenv.py", line 731, in create_environment
remote:     install_node(env_dir, src_dir, opt)
remote:   File "/usr/lib/python3/dist-packages/nodeenv.py", line 605, in install_node
remote:     copy_node_from_prebuilt(env_dir, src_dir)
remote:   File "/usr/lib/python3/dist-packages/nodeenv.py", line 522, in copy_node_from_prebuilt
remote:     callit(['cp', '-a', src_dir + '/%s-v*/*' % prefix, env_dir], True, env_dir)
remote:   File "/usr/lib/python3/dist-packages/nodeenv.py", line 461, in callit
remote:     % (cmd_desc, proc.returncode))
remote: OSError: Command cp -a /home/piku/.piku/env...ltable/src/node-v*/* /home/piku/.piku/envs/excelhtmltable failed with error code 1

This does not actually break anything because it harmlessly tries to install over the existing binary, but it would be good to put in a check to skip the install if the correct node version is already installed.

Undefined name 'deploy_java'

Undefined name have the potential to raise NameError at runtime.

flake8 testing of https://github.com/rcarmo/piku on Python 3.6.3

$ flake8 . --count --select=E901,E999,F821,F822,F823 --show-source --statistics

./piku.py:264:17: F821 undefined name 'deploy_java'
                deploy_java(app, deltas)
                ^
1     F821 undefined name 'deploy_java'
1

Allow pushing of non-master branches

Hello, thanks for this fantastic project @rcarmo! I just successfully deployed my first piku instance. 🎉

I'm about to issue a PR for something else but I wanted to write this down here while I can as it surprised me. I tried to push a non-master branch to my piku instance and nothing happened whereas on other PaaS this is not a problem and the app deploys as expected.

`clock` type worker?

I'm needing to schedule a couple of weekly tasks to run, is using APScheduler a recommended way (https://devcenter.heroku.com/articles/clock-processes-python) or is there an easier way?

Looking at the code it looks like web and wsgi are the only ones that get processed and the rest are created as the tag name.

sorry, fairly new to Heroku process types

Support for Java

I have tried adding support for Java but once I deploy a Java Application without an isolated environment an error File or Directory not found. Adding a virtualenv path piku automatically spins a python virtual environment and detects a Python App instead a Java App.
I know the directory must exist for the Live config and scaling files.
I did a dummy creation of the app's directory under .piku/envs(app) to enable Java deployments.

Check the demo below

I will create the PR soon, but make sure you have uswgi integration plugins. I am using jwsgi and jvm plugins.

install python3.6

recently started using a f-strings in a project and pushed it to piku and noticed that is was giving syntax errors. python3.5.3 installed.

would it causes issues to install 3.6?

Linting and Tests

So #94 drove home the point that we need automated linting and tests, and I'm thinking about doing that - either in CI or via some other GitHub app.
Suggestions?

Detect App Language First.

I suggest that checking requirements should be done after an app's language has been detected. It makes it sound more like a PaaS than a know machine that knows what to check requirements for.
Just a logical suggestion.

Like instead of:

remote: -----> Checking requirements: ['java', 'lein', 'clojure']
remote: ['/usr/bin/java', '/usr/bin/lein', '/usr/local/bin/clojure']
remote: -----> Clojure app detected.
remote: -----> Building Clojure Application

to:

remote: -----> Clojure app detected.
remote: -----> Checking requirements: ['java', 'lein', 'clojure']
remote: ['/usr/bin/java', '/usr/bin/lein', '/usr/local/bin/clojure']
remote: -----> Building Clojure Application

Support for static-only sites like GH pages

It would be great to be able to do git push deployments of static sites (e.g. frontend only), just like GitHub pages.

This could be implemented by installing the Nginx config in the case where a) wsgi and web workers are not defined, and b) NGINX_STATIC_PATHS is defined. In this case the root level proxy stanza should be excluded from the config as there would be no listening server to proxy through to.

Pick static site generator for piku.github.io

Following up on #90, I'm partial to picking out a simple static file generator to maintain all the docs:

Standard Jekyll and gh-pages
docsify (which is cute and simple and lightweight but is lousy for SEO)
Something else (suggestions?)

Versioning?

Ok, so there have been changes since I installed this on my raspberry pi. Is it time to start tagging releases or is it still too early?

Use pyenv to allow arbitrary Python versions

@rcarmo what would you think of a PR which supported arbitrary versions of Python via pyenv?

I was thinking that if PYTHON_VERSION contains a detailed version number like 2.7.15 then pyenv could be invoked to do the work of installing the custom python version and sandboxing the app in there. This means users can be specific about exactly which version of Python they want to run their app under as some in the 3.x series have quite different features.

I could install pyenv during bootstrap if piku-bootstrap gets merged, or otherwise it could be installed at this point if not already installed. Here are the installation instructions.

A similar thing could be done for nodejs via nvm down the track too.

ansible-playbook failed to install silently

ansible-playbook was not able to be installed at my box due missing gcc ( aarch64-linux-gnu-gcc )

(virtualenv) root@piku:~# pip install -q "ansible==2.7.10" 
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
  ERROR: Command errored out with exit status 1:
   command: /root/.piku-bootstrap/virtualenv/bin/python /root/.piku-bootstrap/virtualenv/local/lib/python2.7/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-Ip7tPt/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- 'setuptools>=40.6.0' wheel 'cffi>=1.8,!=1.11.3; platform_python_implementation != '"'"'PyPy'"'"''
       cwd: None
  Complete output (124 lines):
  DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
  Collecting setuptools>=40.6.0
    Using cached https://files.pythonhosted.org/packages/6a/9a/50fadfd53ec909e4399b67c74cc7f4e883488035cfcdb90b685758fa8b34/setuptools-41.4.0-py2.py3-none-any.whl
  Collecting wheel
    Using cached https://files.pythonhosted.org/packages/00/83/b4a77d044e78ad1a45610eb88f745be2fd2c6d658f9798a15e384b7d57c9/wheel-0.33.6-py2.py3-none-any.whl
  Collecting cffi!=1.11.3,>=1.8
    Using cached https://files.pythonhosted.org/packages/0d/aa/c5ac2f337d9a10ee95d160d47beb8d9400e1b2a46bb94990a0409fe6d133/cffi-1.13.1.tar.gz
  Processing ./.cache/pip/wheels/f2/9a/90/de94f8556265ddc9d9c8b271b0f63e57b26fb1d67a45564511/pycparser-2.19-py2.py3-none-any.whl
  Building wheels for collected packages: cffi
    Building wheel for cffi (setup.py): started
    Building wheel for cffi (setup.py): finished with status 'error'
    ERROR: Command errored out with exit status 1:
     command: /root/.piku-bootstrap/virtualenv/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-x9EONj/cffi/setup.py'"'"'; __file__='"'"'/tmp/pip-install-x9EONj/cffi/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-cd16r2 --python-tag cp27
         cwd: /tmp/pip-install-x9EONj/cffi/
    Complete output (48 lines):
    unable to execute 'aarch64-linux-gnu-gcc': No such file or directory
    unable to execute 'aarch64-linux-gnu-gcc': No such file or directory
  
        No working compiler found, or bogus compiler options passed to
        the compiler from Python's standard "distutils" module.  See
        the error messages above.  Likely, the problem is not related
        to CFFI but generic to the setup.py of any Python package that
        tries to compile C code.  (Hints: on OS/X 10.8, for errors about
        -mno-fused-madd see http://stackoverflow.com/questions/22313407/
        Otherwise, see https://wiki.python.org/moin/CompLangPython or
        the IRC channel #python on irc.freenode.net.)
  
        Trying to continue anyway.  If you are trying to install CFFI from
        a build done in a different context, you can ignore this warning.
  
    running bdist_wheel
    running build
    running build_py
    creating build
    creating build/lib.linux-aarch64-2.7
    creating build/lib.linux-aarch64-2.7/cffi
    copying cffi/api.py -> build/lib.linux-aarch64-2.7/cffi
    copying cffi/__init__.py -> build/lib.linux-aarch64-2.7/cffi
    copying cffi/lock.py -> build/lib.linux-aarch64-2.7/cffi
    copying cffi/commontypes.py -> build/lib.linux-aarch64-2.7/cffi
    copying cffi/model.py -> build/lib.linux-aarch64-2.7/cffi
    copying cffi/vengine_cpy.py -> build/lib.linux-aarch64-2.7/cffi
    copying cffi/verifier.py -> build/lib.linux-aarch64-2.7/cffi
    copying cffi/recompiler.py -> build/lib.linux-aarch64-2.7/cffi
    copying cffi/vengine_gen.py -> build/lib.linux-aarch64-2.7/cffi
    copying cffi/error.py -> build/lib.linux-aarch64-2.7/cffi
    copying cffi/ffiplatform.py -> build/lib.linux-aarch64-2.7/cffi
    copying cffi/setuptools_ext.py -> build/lib.linux-aarch64-2.7/cffi
    copying cffi/cffi_opcode.py -> build/lib.linux-aarch64-2.7/cffi
    copying cffi/backend_ctypes.py -> build/lib.linux-aarch64-2.7/cffi
    copying cffi/cparser.py -> build/lib.linux-aarch64-2.7/cffi
    copying cffi/pkgconfig.py -> build/lib.linux-aarch64-2.7/cffi
    copying cffi/_cffi_include.h -> build/lib.linux-aarch64-2.7/cffi
    copying cffi/parse_c_type.h -> build/lib.linux-aarch64-2.7/cffi
    copying cffi/_embedding.h -> build/lib.linux-aarch64-2.7/cffi
    copying cffi/_cffi_errors.h -> build/lib.linux-aarch64-2.7/cffi
    running build_ext
    building '_cffi_backend' extension
    creating build/temp.linux-aarch64-2.7
    creating build/temp.linux-aarch64-2.7/c
    aarch64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-md9kAN/python2.7-2.7.16=. -fstack-protector-strong -Wformat -Werror=format-security -fPIC -I/usr/include/ffi -I/usr/include/libffi -I/usr/include/python2.7 -c c/_cffi_backend.c -o build/temp.linux-aarch64-2.7/c/_cffi_backend.o
    unable to execute 'aarch64-linux-gnu-gcc': No such file or directory
    error: command 'aarch64-linux-gnu-gcc' failed with exit status 1
    ----------------------------------------
    ERROR: Failed building wheel for cffi
    Running setup.py clean for cffi
  Failed to build cffi
  Installing collected packages: setuptools, wheel, pycparser, cffi
      Running setup.py install for cffi: started
      Running setup.py install for cffi: finished with status 'error'
      ERROR: Command errored out with exit status 1:
       command: /root/.piku-bootstrap/virtualenv/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-x9EONj/cffi/setup.py'"'"'; __file__='"'"'/tmp/pip-install-x9EONj/cffi/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-4VveU0/install-record.txt --single-version-externally-managed --prefix /tmp/pip-build-env-Ip7tPt/overlay --compile --install-headers /root/.piku-bootstrap/virtualenv/include/site/python2.7/cffi
           cwd: /tmp/pip-install-x9EONj/cffi/
      Complete output (48 lines):
      unable to execute 'aarch64-linux-gnu-gcc': No such file or directory
      unable to execute 'aarch64-linux-gnu-gcc': No such file or directory
  
          No working compiler found, or bogus compiler options passed to
          the compiler from Python's standard "distutils" module.  See
          the error messages above.  Likely, the problem is not related
          to CFFI but generic to the setup.py of any Python package that
          tries to compile C code.  (Hints: on OS/X 10.8, for errors about
          -mno-fused-madd see http://stackoverflow.com/questions/22313407/
          Otherwise, see https://wiki.python.org/moin/CompLangPython or
          the IRC channel #python on irc.freenode.net.)
  
          Trying to continue anyway.  If you are trying to install CFFI from
          a build done in a different context, you can ignore this warning.
  
      running install
      running build
      running build_py
      creating build
      creating build/lib.linux-aarch64-2.7
      creating build/lib.linux-aarch64-2.7/cffi
      copying cffi/api.py -> build/lib.linux-aarch64-2.7/cffi
      copying cffi/__init__.py -> build/lib.linux-aarch64-2.7/cffi
      copying cffi/lock.py -> build/lib.linux-aarch64-2.7/cffi
      copying cffi/commontypes.py -> build/lib.linux-aarch64-2.7/cffi
      copying cffi/model.py -> build/lib.linux-aarch64-2.7/cffi
      copying cffi/vengine_cpy.py -> build/lib.linux-aarch64-2.7/cffi
      copying cffi/verifier.py -> build/lib.linux-aarch64-2.7/cffi
      copying cffi/recompiler.py -> build/lib.linux-aarch64-2.7/cffi
      copying cffi/vengine_gen.py -> build/lib.linux-aarch64-2.7/cffi
      copying cffi/error.py -> build/lib.linux-aarch64-2.7/cffi
      copying cffi/ffiplatform.py -> build/lib.linux-aarch64-2.7/cffi
      copying cffi/setuptools_ext.py -> build/lib.linux-aarch64-2.7/cffi
      copying cffi/cffi_opcode.py -> build/lib.linux-aarch64-2.7/cffi
      copying cffi/backend_ctypes.py -> build/lib.linux-aarch64-2.7/cffi
      copying cffi/cparser.py -> build/lib.linux-aarch64-2.7/cffi
      copying cffi/pkgconfig.py -> build/lib.linux-aarch64-2.7/cffi
      copying cffi/_cffi_include.h -> build/lib.linux-aarch64-2.7/cffi
      copying cffi/parse_c_type.h -> build/lib.linux-aarch64-2.7/cffi
      copying cffi/_embedding.h -> build/lib.linux-aarch64-2.7/cffi
      copying cffi/_cffi_errors.h -> build/lib.linux-aarch64-2.7/cffi
      running build_ext
      building '_cffi_backend' extension
      creating build/temp.linux-aarch64-2.7
      creating build/temp.linux-aarch64-2.7/c
      aarch64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-md9kAN/python2.7-2.7.16=. -fstack-protector-strong -Wformat -Werror=format-security -fPIC -I/usr/include/ffi -I/usr/include/libffi -I/usr/include/python2.7 -c c/_cffi_backend.c -o build/temp.linux-aarch64-2.7/c/_cffi_backend.o
      unable to execute 'aarch64-linux-gnu-gcc': No such file or directory
      error: command 'aarch64-linux-gnu-gcc' failed with exit status 1
      ----------------------------------------
  ERROR: Command errored out with exit status 1: /root/.piku-bootstrap/virtualenv/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-x9EONj/cffi/setup.py'"'"'; __file__='"'"'/tmp/pip-install-x9EONj/cffi/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-4VveU0/install-record.txt --single-version-externally-managed --prefix /tmp/pip-build-env-Ip7tPt/overlay --compile --install-headers /root/.piku-bootstrap/virtualenv/include/site/python2.7/cffi Check the logs for full command output.
  ----------------------------------------
ERROR: Command errored out with exit status 1: /root/.piku-bootstrap/virtualenv/bin/python /root/.piku-bootstrap/virtualenv/local/lib/python2.7/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-Ip7tPt/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- 'setuptools>=40.6.0' wheel 'cffi>=1.8,!=1.11.3; platform_python_implementation != '"'"'PyPy'"'"'' Check the logs for full command output.

"probably another instance of uWSGI is running on the same address (0.0.0.0:9080)"

this is a fresh installed piku on a raspberry pi. I've tried changing the port but still get this message? Any clues/help?

I'm seeing this error (every couple of minutes):

wsgi.1   | *** Starting uWSGI 2.0.15 (32bit) on [Thu Nov 14 13:57:34 2019] ***
wsgi.1   | compiled with version: 8.3.0 on 14 July 2019 14:52:34
wsgi.1   | os: Linux-4.19.75-v7+ #1270 SMP Tue Sep 24 18:45:11 BST 2019
wsgi.1   | nodename: pikuserver
wsgi.1   | machine: armv7l
wsgi.1   | clock source: unix
wsgi.1   | pcre jit disabled
wsgi.1   | detected number of CPU cores: 4
wsgi.1   | current working directory: /home/piku/.piku/uwsgi-enabled
wsgi.1   | detected binary path: /usr/local/bin/uwsgi
wsgi.1   | chdir() to /home/piku/.piku/apps/myapp
wsgi.1   | your processes number limit is 7733
wsgi.1   | your memory page size is 4096 bytes
wsgi.1   | detected max file descriptor number: 1024
wsgi.1   | lock engine: pthread robust mutexes
wsgi.1   | thunder lock: disabled (you can enable it with --thunder-lock)
wsgi.1   | uWSGI http bound on 0.0.0.0:9080 fd 6
wsgi.1   | probably another instance of uWSGI is running on the same address (0.0.0.0:9080).
wsgi.1   | bind(): Address already in use [core/socket.c line 769]

the ProcFile is this:

wsgi: main:app
worker: python3 myapp.py

main.py is a minimal flask app.

import flask

app = flask.Flask(__name__)

@app.route("/")
def hello():
    return "Hello!"

if __name__ == '__main__':
    HTTP_PORT = int(os.environ.get('PORT', 8000))pro
    BIND_ADDRESS = os.environ.get('BIND_ADDRESS', '127.0.0.1')
    DEBUG = 'true' == os.environ.get('DEBUG', 'false').lower()
    logging.info('Bind Address: {}:{}' % (BIND_ADDRESS, HTTP_PORT))
    logging.info('Debug set to {}' % DEBUG)
    app.run(host=BIND_ADDRESS, port=HTTP_PORT, debug=DEBUG)

the ENV file is this

# bind to the local machine only
BIND_ADDRESS=0.0.0.0
PORT=9080

git remote show piku throws a traceback

$ git remote show piku
Usage: piku.py [OPTIONS] COMMAND [ARGS]...
Try "piku.py --help" for help.

Error: No such command "git-upload-pack".
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

Would be good to pass some of the basic git commands through.

NGINX_STATIC_PATHS appears to be broken

Not 100% sure about this but I am seeing the following during git push:

remote: Error unexpected '{' in field name in static path spec: should be /url1:path1[,/url2:path2], ignoring.

When I have a the following setting:

NGINX_STATIC_PATHS=/static:static

It appears to be due to the brace characters not being handled correctly by .format()

INSTALL ERROR: ImportError: cannot import name which

fresh copy of Jessie 07-05 for Raspberry Pi... when I do python piku.py setup it gives me an import error.

piku@raspberrypi:~ $ python piku.py setup
Traceback (most recent call last):
  File "piku.py", line 16, in <module>
    from shutil import copyfile, rmtree, which
ImportError: cannot import name which

When there is no Procfile in repo, it gives error

I pushed NodeJS app to repo and get error:

remote:   File "/home/piku/piku.py", line 929, in git_hook
remote:     do_deploy(app)
remote:   File "/home/piku/piku.py", line 234, in do_deploy
remote:     if len(workers):
remote: TypeError: object of type 'NoneType' has no len()

Probably should be same with Go app, which has no Procfile. Imho in this case workers is None:

workers = parse_procfile(procfile)
if len(workers):

ENV file variable expansion is not performed correctly

When shipping an ENV file with an application,

FOO=$HOME/bar

...is not correctly expanded. Need to ensure it is done correctly for critical system variables like HOME, PATH, etc.

Web Interface

Can you implement a web interface for analytics?

Node.js support

First of all, very nice idea! Thanks for that.

I'm wondering if and how I can help to get Node.js support to piku.

deploying example gives "Invalid Procfile" on raspberry Pi

Ok... for grins I tried to pushing the python_example to my Piku setup on a raspberrypi.

I get an odd error remote: Error: Invalid Procfile for app 'python_example'....

Where can I look to get more info on what is failing?

dwight : ~/piku/examples/python (master)
🍺 $ git push piku master
Counting objects: 5, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (4/4), done.
Writing objects: 100% (5/5), 431 bytes | 215.00 KiB/s, done.
Total 5 (delta 2), reused 0 (delta 0)
remote: -----> Deploying app 'python_example'
remote: Your branch is up-to-date with 'origin/master'.
remote: Error: Invalid Procfile for app 'python_example'.
To piku.local:python_example
   672e0fe..dabd96b  master -> master

the Procfile:

🍺 $ cat Procfile
wsgi: main:app
worker: python worker.py

no logs that would help...

🍺 $ ssh [email protected] logs python_example
No logs found for app 'python_example'.

unable to get python example to work...

I was able to successfully publish the example...

dwight : ~/python-copy (master)
🍺 $ git push piku master
Counting objects: 15, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (12/12), done.
Writing objects: 100% (15/15), 10.80 KiB | 2.16 MiB/s, done.
Total 15 (delta 0), reused 0 (delta 0)
remote: -----> Creating app 'python_example'
remote: -----> Deploying app 'python_example'
remote: Your branch is up-to-date with 'origin/master'.
remote: -----> Python app detected.
remote: -----> Creating virtualenv for 'python_example'
remote: Already using interpreter /usr/bin/python3
remote: Using base prefix '/usr'
remote: New python executable in python_example/bin/python3
remote: Also creating executable in python_example/bin/python
remote: Installing setuptools, pip...done.
remote: -----> Running pip for 'python_example'
remote: Downloading/unpacking bottle (from -r /home/piku/.piku/apps/python_example/requirements.txt (line 1))
remote:   Running setup.py (path:/tmp/pip-build-39t24ff4/bottle/setup.py) egg_info for package bottle
remote:
remote:     warning: no files found matching 'LICENSE.txt'
remote: Installing collected packages: bottle
remote:   Running setup.py install for bottle
remote:     changing mode of build/scripts-3.4/bottle.py from 644 to 755
remote:
remote:     warning: no files found matching 'LICENSE.txt'
remote:     changing mode of /home/piku/.piku/envs/python_example/bin/bottle.py to 755
remote: Successfully installed bottle
remote: Cleaning up...
remote: -----> Spawning 'python_example:wsgi.1'
remote: -----> Setting HTTP to listen on 127.0.0.1:9080
remote: -----> Spawning 'python_example:worker.1'
To 192.168.1.18:python_example
 * [new branch]      master -> master

but when I try to curl the running example I get the following...

dwight : ~/python-copy (master)
🍺 $ curl 192.168.1.18:9080
curl: (7) Failed to connect to 192.168.1.18 port 9080: Connection refused

YAML as dependency

I've just noticed that #81 introduced a dependency on yaml, which means that the docs need to account for python3-yaml (the OS package) being installed.

git-hooks shouldn't be in the help description.

so doing a piku --help gives you all the commands you can send piku. I'm not sure it is appropriate to include the internal git-hooks since that is plumbing used to make piku work.

...
  git-hook          INTERNAL: Post-receive git hook
  git-receive-pack  INTERNAL: Handle git pushes for an app
  git-upload-pack   INTERNAL: Handle git upload pack for an app
...

IMHO

Support for nginx websocket proxying via environment variable

Similarly to setting up static routes it should be possible to set up websocket proxying via an environment variable.

https://www.nginx.com/blog/websocket-nginx/

upstream websocket {
    server 192.168.100.10:8010;
}

location /wsurl/ {
    proxy_pass http://websocket;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "Upgrade";
}

I think it's possible these days to have an app serve both HTTP and WS with the same server & port in which case the URL can default to / and the port & IP will be the same as the default http config (so the setting could be on/off for websockets).

@rcarmo I frequently need this for projects, so I will probably take a shot at implementation at some point in the future if nobody else beats me to it.

`ps:scale` missing user documentation

Notice this in the code but no minimal mention of it in the docs. (i.e. "scale up/down worker processes (ps:scale) at will.")

@piku.command("ps:scale")
@argument('app')
@argument('settings', nargs=-1)

the piku.py help shows (somewhat incorrectly?):
ps:scale Show application configuration

It should be showing that this takes arg(s) that specify the number to scale each type to.

This was what I was able to do to scale up the worker type processes (I have piku as an alias)
piku ps:scale test worker=2

ruby on rails

Hello ,
I congratulate you on your work.
But I have a question to which I have no answer: can we use ruby on rails?

thank you
Jbd

"Failed to install Ansible" error using bootstrap script

On a fresh install of Buster I'm getting this...

pi@pikuserver:~ $ curl https://piku.github.io/get | sh
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 215 100 215 0 0 1990 0 --:--:-- --:--:-- --:--:-- 1990

Downloading piku-bootstrap here and running it.

Looks like this is your first time running piku-bootstrap.
This script will self-install dependencies into /home/pi/.piku-bootstrap now.
Hit enter to continue or ctrl-C to abort.

#> Virtualenv setup not found. Installing it into /home/pi/.piku-bootstrap.
#> Downloading & installing Virtualenv.
#> Setting up the virtual environment.
#> Piku repo not found. Installing it into /home/pi/.piku-bootstrap/piku.
Cloning into '/home/pi/.piku-bootstrap/piku'...
remote: Enumerating objects: 12, done.
remote: Counting objects: 100% (12/12), done.
remote: Compressing objects: 100% (12/12), done.
remote: Total 1418 (delta 3), reused 0 (delta 0), pack-reused 1406
Receiving objects: 100% (1418/1418), 862.07 KiB | 2.36 MiB/s, done.
Resolving deltas: 100% (822/822), done.
#> ansible-playbook binary not found. Installing it into /home/pi/.piku-bootstrap.
Failed to install Ansible.
See /home/pi/.piku-bootstrap/install.log for details (you may be missing a C compiler).

The install log shows this interesting part at the end...

  arm-linux-gnueabihf-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-9NJ3qw/python2.7-2.7.16=. -fstack-protector-strong -Wformat -Werror=format-security -fPIC -DUSE__THREAD -DHAVE_SYNC_SYNCHRONIZE -I/usr/include/ffi -I/usr/include/libffi -I/usr/include/python2.7 -c c/_cffi_backend.c -o build/temp.linux-armv7l-2.7/c/_cffi_backend.o
  c/_cffi_backend.c:2:10: fatal error: Python.h: No such file or directory
   #include <Python.h>
            ^~~~~~~~~~
  compilation terminated.
  error: command 'arm-linux-gnueabihf-gcc' failed with exit status 1
  ----------------------------------------
ERROR: Command errored out with exit status 1: /home/pi/.piku-bootstrap/virtualenv/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-ye1mun/cffi/setup.py'"'"'; file='"'"'/tmp/pip-install-ye1mun/cffi/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' install --record /tmp/pip-record-zvEFd6/install-record.txt --single-version-externally-managed --prefix /tmp/pip-build-env-18etu7/overlay --compile --install-headers /home/pi/.piku-bootstrap/virtualenv/include/site/python2.7/cffi Check the logs for full command output.

ERROR: Command errored out with exit status 1: /home/pi/.piku-bootstrap/virtualenv/bin/python /home/pi/.piku-bootstrap/virtualenv/local/lib/python2.7/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-18etu7/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- 'setuptools>=40.6.0' wheel 'cffi>=1.8,!=1.11.3; platform_python_implementation != '"'"'PyPy'"'"'' Check the logs for full command output.

Support basic PHP-cgi installs

Just going to put this out there. Don't worry, I don't expect anybody to work on this except me. ;)

https://github.com/wilmoore/php-version