I believe we reference suhosin (the PHP hardening patch) in the .htaccess file of the site.

We must therefore ensure suhosin is explicitly installed in the setup repo.

I think the package is php5-suhosin but haven't done any further research, e.g. checking whether this is actually installed.

Prompts for:

Authentication realm: http://svn.cyclestreets.net:80 CycleStreets SVN access
Password for 'cyclestreets':

Some way into the installation. Can this be added as a config option?

The Munin part of the installation seems very inconsistent across hosts.

MySQL 5.7 adds far better OpenGIS spatial support.

MySQL 5.7.5 adds native spatial GeoJSON support:
https://dev.mysql.com/doc/refman/5.7/en/spatial-geojson-functions.html

Currently the repo expects 5.6:
https://github.com/cyclestreets/cyclestreets-setup/blob/master/utility/installBaseWebserver.sh#L105-L106

Upgrading to 5.7.x on Ubuntu 14.04 is possible but requires installation from MySQL's package repo rather than being a native Ubuntu package:
https://www.digitalocean.com/community/tutorials/how-to-install-mysql-on-ubuntu-14-04

Ubuntu 16.04 LTS (Xenial)'s native package is MySQL 5.7:
http://packages.ubuntu.com/xenial/mysql-server-5.7

Obviously performance and tuning requirements would need to be assessed before a move to 5.7 but no major changes would seem likely.

Difficult directives

Despite being listed as PHP_INI_ALL the following directives are not effectively assigned to values applied from the Apache configuration:

The assigned values are respected when included in the php.ini system, as has been tried for example in 9998f11.

The zend.assertions directive controls how PHP is compiled which might be why it has to be set in the php.ini system, and the qualification with restrictions is a hint to that.

phpinfo()

Despite looking at this many times over the last decade I have never gotten to the bottom of why the other two settings cannot be set reliably from the Apache configuration. E.g. when set in the either a virtualhost configuration or the one of the higher level conf-enabled files to 86400, phpinfo() lists them as:

(1440 = 24 minutes, 86400 = 24 hours)

but the setting in force appears to be the master value as about 24 minutes later the site requests a login. (This wait also makes it a pain to test / debug.)

I think it might be something to do with the server wide nature of these settings - e.g. the garbage collection lifetime presumably applies to the whole of php as perhaps hinted at by the note attached to this documentation for the session.gc_maxlifetime directive.

Consequence

Until this matter can be resolved then changing these values from their defaults requires edits in the php.ini system files.

Munin setup is not routine.

It should be added for all machines via a distinct script, included in all installers.

It should pick up a set of IPv4/IPv6 addresses in the main .config.sh file.

There should be no manually-saved configurations.

Occasionally a route could fail for whatever reason, e.g. caused by some kind of timeout.

It would be useful therefore to have a Munin chart that scans the Apache logs for failures (5xx) in the last 5 minutes.

Presumably this could reuse this log file reader.
https://github.com/cyclestreets/cyclestreets-setup/blob/master/utility/readLogFile.py

Currently the installation of the journeylinger munin chart makes a link into the setup (installer) repo.

Instead this should copy in /live-deployment/cs-munin-journeylinger.sh, doing placeholder replacement.

This will then unblock the clearing up of the journeysLog parameter in the settings.

We aren't running application so binlogging should be disabled by default, as part of the install.

Aim to get rid of all cyclestreets.cnf writing - currently setup by import and live deployment.

The main thing they set is:

query-cache
utf8

The rest of the application should do what they are currently trying to do. But the utf8 setting should remain as really that should be system wide.

Even if a chart on the server node -specific page is in orange/red, that doesn't also show on the main page as such.

We have:

echo "journey_top90linger.warning 0:600"
echo "journey_top90linger.critical 0:700"

so I'm not really sure what is wrong.

There's a StackExchange question which is so far unanswered on this same topic:
https://unix.stackexchange.com/questions/635513/

An import currently sets the response to have an error status of: Routing service unavailable.

This should be changed to: Data update in progress, which would enable load balancing or other checking systems to detect a known state.

If using a central configuration repo, it would be useful if the start of the setup process created the symlinks into it at the start. It would mkdir each directory with parents then create the symlinks.

This is something along the lines of

cp -sR /orig/. /dest

as detailed at http://stackoverflow.com/a/1250782/180733 (and also see http://unix.stackexchange.com/a/180987) but that's not quite right.

Problem

Sometimes servers are temporarily offline during upgrades and installations of new routing edtions.

When these known planned maintenance events occur the SMS should avoid sending notifications.

Proposal

Add detection of the 503 (Service unavailable) header in the tests code.
The failure can then be ignored for specific (end-point) servers but still be sent when the original target was a load balancer.

MySQL 8 doesn't work in Munin on Ubuntu 20.04:

https://github.com/cyclestreets/cyclestreets-setup/blob/master/live-deployment/deploy.sh#L53-L80

mysql_ | no | no [DBI connect('mysql;mysql_read_default_file=/etc/mysql/debian.cnf;mysql_connect_timeout=5','root',...) failed: Access denied for user 'root'@'localhost' (using password: YES)]

Unlike SVN, it seems you cannot clone into an existing directory.

However, the .mySuperUserCredentials.cnf gets written into there (why?) earlier in the script.

Either we need to amend git clone to clone into an existing directory somehow, or the MySQL file should be written after the git stage.

The map_leg_detail table describes the edges in the routing graph. Some of these are CSV fields (e.g. distances and elevations) which compress well.

The following technique (based on this) has been manually applied to the current routing edition, reducing the table size from 13G to just under 6G. After the compression procedure the table works in the same way as previously. The reduced data size should yield a modest performance boost, although no benchmarking has been performed.

This procedure could be added to the main build process, but is a bit fiddly as involves working as root user on the command line, possibly while the mysql service is stopped.

# [email protected]:#
cd /var/lib/mysql/routing190109

# File sizes before compression (data for routing190101 is 13G)
ls -lh map_leg_detail.*

# Check status
myisamchk -dvv map_leg_detail

# Pack (takes five minutes)
date
myisampack map_leg_detail
date

# Check compressed tables (takes three minutes)
date
myisamchk -rq map_leg_detail
date

# File sizes after compression
ls -lh map_leg_detail.*
# Data file is 5.9G (before was 13G)

# Flush tables
mysql -uroot -e "flush tables;"

The utf8.conf file is setup by

it was originally created to ensure default-character-set=utf8mb4 but has also become a bit of a dumping ground for other settings.

This needs reviewing, renaming and tidying.

This article just published recommends that web devs in future should not permitting framing of sites:
https://www.theregister.com/2021/03/08/post_spectre_programming/
using the header called X-Frame-Options.

There's no reason to allow this, now that the white label sites have been split out of the codebase.

This should be added to various Apache configs.

These two scripts /dev-deployment/deploy.sh and /install-dev/run.sh separately and collectively install the dev server.

These need to merged together.

They don't appear to overlap.

The schema scripts build sample versions of the following databases:

1. cyclestreets sample database contains a minimum set of data to run a new working installation
2. routingYYMMDD contains minimal routing data for Cambridge

They result in files of several MB which are currently stored as part of the main repository.

This task is to move them out of the code repository and home them somewhere like downloads.cyclestreets.net.

This would allow them to be rebuilt daily and hence keep them up to date, smoothing the installation process.

Hello,

wkhtmltopdf has moved the downloads provider from SourceForge to gna.org. As mentioned in the linked message, all earlier downloads have been removed from SourceForge and you are strongly advised to migrate to the new URLs. The following files in your repository will need to be updated:

• install-website/run.sh

It is recommended to either subscribe to the mailing list or Twitter feed to receive updates on new releases and further changes.

Thanks,
wkhtmltopdf maintainer

The SMS testing system has become stretched - this script needs a more fundamental rewrite.

It should have a defined list of server/API calls to test, not a single list with exceptions. It should also have support for testing specific areas.

The script needs to be amended to have the username and password in the config file, in a similar way that we used to have with SVN.

We should update
collation-server = utf8mb4_unicode_ci
to
collation-server = utf8mb4_0900_ai_ci

https://github.com/cyclestreets/cyclestreets-setup/blob/master/utility/installBaseWebserver.sh#L211

The backup check script at backup-deployment/check-backup.sh reports any failed units.

It would be helpful if it, or another script doing the backup itself, reported lack of disk space on the target.

I ran the script, and it stopped at this point:

Failed to start cycleroutingd.service: Unit cycleroutingd.service not found

An alert often comes by text at 4.15am when the routing file is reloaded but this is not an error.

As exempting a specific time in cron isn't really possible, it would be useful to have a time range when an instantiation is set to stop.

zcsglobal.conf at
https://github.com/cyclestreets/cyclestreets-setup/blob/master/install-website/zcsglobal.conf

contains two things:

1. Global server-wide configuration, that is aimed at locking-down a server more securely, e.g. disabling the AcceptPathInfo misfeature;

2. Items specific to the journey planner site.

Because this is installed as part of the journey planner setup, and not as part of a general machine setup, setup routines such as for microsites are not getting (1).

We should split this into to two configurations.

We should also take the opportunity also to remove the cryptic 'zcsglobal' name, which was named to means 'cyclestreets global, with z to ensure it is last'.

Reproducible example: https://www.cyclestreets.net/journey/79719873/

As shown there and in the image below there are quite a few turns in each route option. From a user perspective the easiest option is to get straight onto the A201 which takes you straight there on reasonably quiet / protected infrastructure. The dotted red line represents the 'route 1' option from Google that is best from the perspective of minimising turns and maximising continuity.

Source: https://www.google.com/maps/dir/Kings+Cross,+London/SE11+4JP,+London/@51.5118702,-0.1440914,13z/data=!3m1!4b1!4m14!4m13!1m5!1m1!1s0x48761b10c73c4dcd:0x616f11fed05d6bb9!2m2!1d-0.1245845!2d51.5347488!1m5!1m1!1s0x4876049745fd7d11:0x8a34ecfec0841e5c!2m2!1d-0.1065995!2d51.4893899!3e1

Not sure this is the right place for people to submit feedback in an open forum on routing but hope this is useful feedback nonetheless.