cyclestreets / cyclestreets-setup Goto Github PK
View Code? Open in Web Editor NEWDeployment scripts for installing CycleStreets website and related components
Deployment scripts for installing CycleStreets website and related components
I believe we reference suhosin (the PHP hardening patch) in the .htaccess file of the site.
We must therefore ensure suhosin is explicitly installed in the setup repo.
I think the package is php5-suhosin but haven't done any further research, e.g. checking whether this is actually installed.
Prompts for:
Authentication realm: http://svn.cyclestreets.net:80 CycleStreets SVN access
Password for 'cyclestreets':
Some way into the installation. Can this be added as a config option?
The Munin part of the installation seems very inconsistent across hosts.
MySQL 5.7 adds far better OpenGIS spatial support.
MySQL 5.7.5 adds native spatial GeoJSON support:
https://dev.mysql.com/doc/refman/5.7/en/spatial-geojson-functions.html
Currently the repo expects 5.6:
https://github.com/cyclestreets/cyclestreets-setup/blob/master/utility/installBaseWebserver.sh#L105-L106
Upgrading to 5.7.x on Ubuntu 14.04 is possible but requires installation from MySQL's package repo rather than being a native Ubuntu package:
https://www.digitalocean.com/community/tutorials/how-to-install-mysql-on-ubuntu-14-04
Ubuntu 16.04 LTS (Xenial)'s native package is MySQL 5.7:
http://packages.ubuntu.com/xenial/mysql-server-5.7
Obviously performance and tuning requirements would need to be assessed before a move to 5.7 but no major changes would seem likely.
Despite being listed as PHP_INI_ALL the following directives are not effectively assigned to values applied from the Apache configuration:
Directive | Changeable |
---|---|
session.gc_maxlifetime | PHP_INI_ALL |
session.cookie_lifetime | PHP_INI_ALL |
zend.assertions | PHP_INI_ALL with restrictions |
The assigned values are respected when included in the php.ini
system, as has been tried for example in 9998f11.
The zend.assertions
directive controls how PHP is compiled which might be why it has to be set in the php.ini system, and the qualification with restrictions
is a hint to that.
Despite looking at this many times over the last decade I have never gotten to the bottom of why the other two settings cannot be set reliably from the Apache configuration. E.g. when set in the either a virtualhost
configuration or the one of the higher level conf-enabled
files to 86400, phpinfo()
lists them as:
Directive | Local value | Master value |
---|---|---|
session.cookie_lifetime | 86400 | 1440 |
session.gc_maxlifetime | 86400 | 1440 |
(1440 = 24 minutes, 86400 = 24 hours)
but the setting in force appears to be the master value as about 24 minutes later the site requests a login. (This wait also makes it a pain to test / debug.)
I think it might be something to do with the server wide nature of these settings - e.g. the garbage collection lifetime presumably applies to the whole of php as perhaps hinted at by the note attached to this documentation for the session.gc_maxlifetime directive.
Until this matter can be resolved then changing these values from their defaults requires edits in the php.ini system files.
Munin setup is not routine.
It should be added for all machines via a distinct script, included in all installers.
It should pick up a set of IPv4/IPv6 addresses in the main .config.sh file.
There should be no manually-saved configurations.
Occasionally a route could fail for whatever reason, e.g. caused by some kind of timeout.
It would be useful therefore to have a Munin chart that scans the Apache logs for failures (5xx) in the last 5 minutes.
Presumably this could reuse this log file reader.
https://github.com/cyclestreets/cyclestreets-setup/blob/master/utility/readLogFile.py
Currently the installation of the journeylinger munin chart makes a link into the setup (installer) repo.
Instead this should copy in /live-deployment/cs-munin-journeylinger.sh
, doing placeholder replacement.
This will then unblock the clearing up of the journeysLog
parameter in the settings.
We aren't running application so binlogging should be disabled by default, as part of the install.
Aim to get rid of all cyclestreets.cnf writing - currently setup by import and live deployment.
The main thing they set is:
query-cache
utf8
The rest of the application should do what they are currently trying to do. But the utf8 setting should remain as really that should be system wide.
Even if a chart on the server node -specific page is in orange/red, that doesn't also show on the main page as such.
We have:
echo "journey_top90linger.warning 0:600"
echo "journey_top90linger.critical 0:700"
so I'm not really sure what is wrong.
There's a StackExchange question which is so far unanswered on this same topic:
https://unix.stackexchange.com/questions/635513/
An import currently sets the response to have an error status of: Routing service unavailable
.
This should be changed to: Data update in progress
, which would enable load balancing or other checking systems to detect a known state.
If using a central configuration repo, it would be useful if the start of the setup process created the symlinks into it at the start. It would mkdir each directory with parents then create the symlinks.
This is something along the lines of
cp -sR /orig/. /dest
as detailed at http://stackoverflow.com/a/1250782/180733 (and also see http://unix.stackexchange.com/a/180987) but that's not quite right.
Sometimes servers are temporarily offline during upgrades and installations of new routing edtions.
When these known planned maintenance events occur the SMS should avoid sending notifications.
Add detection of the 503 (Service unavailable) header in the tests code.
The failure can then be ignored for specific (end-point) servers but still be sent when the original target was a load balancer.
MySQL 8 doesn't work in Munin on Ubuntu 20.04:
https://github.com/cyclestreets/cyclestreets-setup/blob/master/live-deployment/deploy.sh#L53-L80
mysql_ | no | no [DBI connect('mysql;mysql_read_default_file=/etc/mysql/debian.cnf;mysql_connect_timeout=5','root',...) failed: Access denied for user 'root'@'localhost' (using password: YES)]
Unlike SVN, it seems you cannot clone into an existing directory.
However, the .mySuperUserCredentials.cnf
gets written into there (why?) earlier in the script.
Either we need to amend git clone
to clone into an existing directory somehow, or the MySQL file should be written after the git stage.
wkhtmltopdf --print-media-type /tmp/zib6EPozh.html /tmp/tmp55f9dN.pdf
The switch --print-media-type, is not support using unpatched qt, and will be ignored.
QXcbConnection: Could not connect to display
The map_leg_detail
table describes the edges in the routing graph. Some of these are CSV fields (e.g. distances and elevations) which compress well.
The following technique (based on this) has been manually applied to the current routing edition, reducing the table size from 13G to just under 6G. After the compression procedure the table works in the same way as previously. The reduced data size should yield a modest performance boost, although no benchmarking has been performed.
This procedure could be added to the main build process, but is a bit fiddly as involves working as root user on the command line, possibly while the mysql service is stopped.
# [email protected]:#
cd /var/lib/mysql/routing190109
# File sizes before compression (data for routing190101 is 13G)
ls -lh map_leg_detail.*
# Check status
myisamchk -dvv map_leg_detail
# Pack (takes five minutes)
date
myisampack map_leg_detail
date
# Check compressed tables (takes three minutes)
date
myisamchk -rq map_leg_detail
date
# File sizes after compression
ls -lh map_leg_detail.*
# Data file is 5.9G (before was 13G)
# Flush tables
mysql -uroot -e "flush tables;"
The utf8.conf
file is setup by
it was originally created to ensure default-character-set=utf8mb4
but has also become a bit of a dumping ground for other settings.
This needs reviewing, renaming and tidying.
This article just published recommends that web devs in future should not permitting framing of sites:
https://www.theregister.com/2021/03/08/post_spectre_programming/
using the header called X-Frame-Options
.
There's no reason to allow this, now that the white label sites have been split out of the codebase.
This should be added to various Apache configs.
These two scripts /dev-deployment/deploy.sh
and /install-dev/run.sh
separately and collectively install the dev server.
These need to merged together.
They don't appear to overlap.
The schema scripts build sample versions of the following databases:
cyclestreets
sample database contains a minimum set of data to run a new working installationroutingYYMMDD
contains minimal routing data for CambridgeThey result in files of several MB which are currently stored as part of the main repository.
This task is to move them out of the code repository and home them somewhere like downloads.cyclestreets.net
.
This would allow them to be rebuilt daily and hence keep them up to date, smoothing the installation process.
Hello,
wkhtmltopdf has moved the downloads provider from SourceForge to gna.org. As mentioned in the linked message, all earlier downloads have been removed from SourceForge and you are strongly advised to migrate to the new URLs. The following files in your repository will need to be updated:
It is recommended to either subscribe to the mailing list or Twitter feed to receive updates on new releases and further changes.
Thanks,
wkhtmltopdf maintainer
The SMS testing system has become stretched - this script needs a more fundamental rewrite.
It should have a defined list of server/API calls to test, not a single list with exceptions. It should also have support for testing specific areas.
The script needs to be amended to have the username and password in the config file, in a similar way that we used to have with SVN.
We should update
collation-server = utf8mb4_unicode_ci
to
collation-server = utf8mb4_0900_ai_ci
https://github.com/cyclestreets/cyclestreets-setup/blob/master/utility/installBaseWebserver.sh#L211
The backup check script at backup-deployment/check-backup.sh
reports any failed units.
It would be helpful if it, or another script doing the backup itself, reported lack of disk space on the target.
I ran the script, and it stopped at this point:
Failed to start cycleroutingd.service: Unit cycleroutingd.service not found
An alert often comes by text at 4.15am when the routing file is reloaded but this is not an error.
As exempting a specific time in cron isn't really possible, it would be useful to have a time range when an instantiation is set to stop.
zcsglobal.conf at
https://github.com/cyclestreets/cyclestreets-setup/blob/master/install-website/zcsglobal.conf
contains two things:
Global server-wide configuration, that is aimed at locking-down a server more securely, e.g. disabling the AcceptPathInfo misfeature;
Items specific to the journey planner site.
Because this is installed as part of the journey planner setup, and not as part of a general machine setup, setup routines such as for microsites are not getting (1).
We should split this into to two configurations.
We should also take the opportunity also to remove the cryptic 'zcsglobal' name, which was named to means 'cyclestreets global, with z to ensure it is last'.
Reproducible example: https://www.cyclestreets.net/journey/79719873/
As shown there and in the image below there are quite a few turns in each route option. From a user perspective the easiest option is to get straight onto the A201 which takes you straight there on reasonably quiet / protected infrastructure. The dotted red line represents the 'route 1' option from Google that is best from the perspective of minimising turns and maximising continuity.
Not sure this is the right place for people to submit feedback in an open forum on routing but hope this is useful feedback nonetheless.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.