After updating odk central to 0.4 and following the readme, I managed to run odk-central using docker-compose up -d, but somehow odk-central cannot be accessed in browsers with "This site can’t be reached" error. Is there any step in updating that I missed?

edit : i try to start docker-compose and here's the snippet of the log I suspect the culprit :

nginx_1     | 2019/02/17 11:03:44 [warn] 20#20: duplicate MIME type "text/html" in /etc/nginx/conf.d/odk.conf:22
nginx_1     | nginx: [warn] duplicate MIME type "text/html" in /etc/nginx/conf.d/odk.conf:22
nginx_1     | 2019/02/17 11:03:44 [emerg] 20#20: a duplicate default server for 0.0.0.0:80 in /etc/nginx/sites-enabled/default:22
nginx_1     | nginx: [emerg] a duplicate default server for 0.0.0.0:80 in /etc/nginx/sites-enabled/default:22

https://forum.opendatakit.org/t/odk-central-backup-to-google-drive-failed/19091/7

root@test:~/central# docker-compose build
postgres uses an image, skipping
Building mail
Step 1/2 : FROM namshi/smtp
 ---> 524b2f546192
Step 2/2 : RUN apt-get update && apt-get dist-upgrade -y
 ---> Using cache
 ---> 4b374ea2d639
Successfully built 4b374ea2d639
Successfully tagged central_mail:latest
Building service
Step 1/11 : FROM node:carbon
 ---> 55b43107a63d
Step 2/11 : WORKDIR /usr/odk
 ---> Using cache
 ---> dd4ac806ee2f
Step 3/11 : RUN echo "deb http://apt.postgresql.org/pub/repos/apt/ jessie-pgdg main" > /etc/apt/sources.list.d/pgdg.list;   wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc |   apt-key add -;   apt-get update; apt-get install -y cron gettext postgresql-client-9.6
 ---> Running in 8462e003ed6c
Warning: apt-key output should not be parsed (stdout is not a terminal)
key 7FCC7D46ACCC4CF8:
2 signatures not checked due to missing keys
OK
Get:1 http://security.debian.org/debian-security stretch/updates InRelease [94.3 kB]
Get:2 http://apt.postgresql.org/pub/repos/apt jessie-pgdg InRelease [56.5 kB]
Ign:3 http://deb.debian.org/debian stretch InRelease
Get:4 http://deb.debian.org/debian stretch-updates InRelease [91.0 kB]
Get:5 http://deb.debian.org/debian stretch Release [118 kB]
Get:6 http://deb.debian.org/debian stretch Release.gpg [2434 B]
Get:7 http://security.debian.org/debian-security stretch/updates/main amd64 Packages [467 kB]
Get:8 http://apt.postgresql.org/pub/repos/apt jessie-pgdg/main amd64 Packages [109 kB]
Get:9 http://deb.debian.org/debian stretch-updates/main amd64 Packages [5152 B]
Get:10 http://deb.debian.org/debian stretch/main amd64 Packages [7089 kB]
Fetched 8034 kB in 2s (3030 kB/s)
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 postgresql-client-9.6 : Depends: libpq5 (>= 9.6.11) but 9.6.10-0+deb9u1 is to be installed
E: Unable to correct problems, you have held broken packages.
ERROR: Service 'service' failed to build: The command '/bin/sh -c echo "deb http://apt.postgresql.org/pub/repos/apt/ jessie-pgdg main" > /etc/apt/sources.list.d/pgdg.list;   wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc |   apt-key add -;   apt-get update; apt-get install -y cron gettext postgresql-client-9.6' returned a non-zero code: 100
root@test:~/central# uname -a
Linux test 4.15.0-36-generic #39-Ubuntu SMP Mon Sep 24 16:19:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
root@test:~/central#

Related?

• https://forum.opendatakit.org/t/odk-central-v0-3-beta/16053/8
• #37

in the instructions for upgrade, we need to force the nginx/certbot base image to update, because they only ever tag latest and so when they change things around we don't know what we're actually working against.

as suggested here, people may not assume that there is explicit OData documentation on our end, and stumble around the excel/powerbi/tableau/etc docs instead.

I'm trying to think through whether it's possible/desirable to reduce how often Frontend needs to send a GET request immediately after a POST.

For example, when a user visits the app users page, Frontend requests the list of app users for the project (with extended metadata). If the user then creates a new app user, the response for that POST returns data for the new app user (without extended metadata). Yet Frontend doesn't use the POST response and simply refreshes the list of app users. It does so for two reasons:

• By refreshing the list of app users, Frontend doesn't need to know where in the list the new app user should be inserted. That means that outside of testing, Frontend does not need to know anything about how lists are sorted: that logic is centralized in Backend.
• Frontend uses extended metadata to render the app users table, but the POST response does not include extended metadata.
  • One idea: could the POST request indicate to Backend that it would like the response to include extended metadata? Or does logic around extended metadata really only belong in the GET endpoints?
  • Another idea: Frontend could modify the response, adding the extended metadata itself. Extended metadata for app users includes two things: lastUsed and more details about createdBy. But for a newly created app user, Frontend already knows both of those things: lastUsed is null, and createdBy is the user currently logged in. However, one downside of this approach is that if/when we add other extended metadata fields for app users, we may need to add those fields to Frontend as well.

It'd be great for Frontend not to send an extra request if it doesn't need to, but I'm also wary of premature optimization and of adding complexity to Frontend. @clint-tseng, do you have thoughts about whether it makes sense to try to eliminate this sort of request, and if so, what a good approach might be to do so?

For fully (offline) selfhosted setups it might be very usefull to be able to connect to a Ethercalc instance.
Ethercalc also supports some pretty cool web-form features, so a lot of nice functionality similar to Enketo could be quickly proto-typed with it.

I created an entry for possible support of odata here (although it might be that one of the already integrated libs already supports it):
audreyt/ethercalc#642

Since it is also a nodejs app, maybe someone here would be interested in looking into this connection?

Thanks a lot!

Edit: Selfhosting is really easy with the official docker image: https://hub.docker.com/r/audreyt/ethercalc/

Inspired by https://forum.opendatakit.org/t/odk-central-v0-5-beta/19647/2

I think we should link to some freer OData tools that let users analyze their data. The first step in the process is to verify that our feed works with those tools.

A few days ago, Debian announced the removal of non-LTS Jessie from their mirrors:

https://lists.debian.org/debian-devel-announce/2019/03/msg00006.html

This now causes problems for namshi/smtp, which is trying to access jessie-updates. I realise that this is an upstream issue that should probably be fixed on namshi/smtp, and I've opened an issue there (namshi/docker-smtp#50), but it might be worth trying to figure out a workaround?

This release improves functionality around Users in general, and in particular as they relate to Projects. Web Users will no longer necessarily be sitewide Administrators; instead, they may be granted permission to manage particular Projects. Comprehensive release criteria and story status may be found here.

Comments and feedback may be left directly on the document, or here in this ticket.

I noticed that like previous problem in images (.png not supported for images when previewing or downloading in user interface), the audio also had similar problem when trying to upload .mp3 format (as example). Is there any limitation on what extension for audio and video format? thank you.

Central uses the term "form name," but I'm noticing the term "form title" for the same thing in other places within ODK. I think we've heard that users are sometimes confused about the difference between the form ID and the form name/title, so I think it could be useful to standardize terminology. If "form title" is more common throughout ODK, perhaps we should start using that in Central?

Tried some test form with geopoints and image, all seems works well (create user, upload form, get blank form, upload submissions) until I try to retrieve images in submitted form which shown error {"message":"Could not find the resource you were looking for.","code":404.1} .
Downloading the form also only get the xls file in zip format, but no images present, is there any step in installing or configuring the server that I miss?

Is there any way of accessing the data of repeated groups through the OData feed?

I can see the main form under /Submissions, but so far I couldn't discover any link to groups in both the metadata or the form data.

Any pointers would be greatly appreciated!

I am trying to understand (sorry not a docker expert) how I can make this work with my Traefik.io based host (also using portainer so docker compose isn't preferred).

Looking at the docker-compose file the postgres database is easy to replicate, and I think the actual ODK-central service should also work. But from then on I am a bit lost...

However what is this "mail" service needed for (seems to be an smtp server?) and how can I avoid running an instance of Nginx?

Traefik.io has me pretty much covered for all the reverse-proxy/https needs, and NodeJS usually comes with a build-in webserver, no? So which port is exposed in the "service" container that I could point Traefik.io to?

Thanks for the help!

This release exposes administrative tracking logs on management actions performed on Central servers, which should help with security requirements, troubleshooting, training, and tracking down issues with the backups process.

It also begins work on form submission encryption, supporting both the current XML-defined workflow where users manage their own encryption keypairs on their local machine, as well as a new process whereby the server generates and manages keypairs for the user, gated by a user-controlled passphrase.

Comprehensive release criteria and story status may be found here.

Comments and feedback may be left directly on the document, or here in this ticket.

Creating an issue so we don't lose track.

As seen by me in dev env with self-signed certs and by Matthew MacGregor (ref) with a custom local ssl cert.

This release reorganizes Central around the concept of Projects. Comprehensive release criteria and story status may be found here.

Comments and feedback may be left directly on the document, or here in this ticket.

Setup

ODK Central 0.5 on rancher/K8s/Azure.

Our config contains our own mail server with settings that are confirmed working in other applications running out of the same K8s cluster.

Issue

• Logged in to ODK Central
• Users > Create Web User > enter email and name > Create
• Error message "Something went wrong: the server returned an invalid error."
• Error log on server:

IP - - [27/Jun/2019:03:59:20 +0000] "POST /v1/users HTTP/1.1" 500 348 "https://odkcentral.dbca.wa.gov.au/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:67.0) Gecko/20100101 Firefox/67.0"

Related

• Creating a web user through CLI as per docs works - so I assume the backend has at least one working pathway to create users. This step doesn't involve sending an email invite.
• Editing the user profile through the Central frontend works. This step doesn't involve sending an email invite.

Which steps could I take next to narrow down the problem?

Per this comment: https://forum.opendatakit.org/t/odk-central-v0-4-setup-duplicate-default-server-error/17942/5 .

Put it somewhere nice and in an HTML comment (good for support requests). Although, it's also at /version.txt

• DO now have two types of droplets and it could be confusing.
  We should specify that a Standard droplet, not an Optimized droplet is needed.

• Step 12 requires make to be installed, but make is not installed. We will need to add a apt-get install make.

• Step 13 makes it sound like if you run systemctl start docker-compose@effective-spork and see that the system is active, things are working. Based on what I can see, it can take a few minutes for the container to boot and run all the things

We have already discussed this before, and now that Briefcase supports Central, we thought better to document this as an issue and start discussing options.

The scenario:

• A user has gone through many versions of a form and has submissions belonging to all versions
• The user wants to use Briefcase to push the form and its submissions to a Central server
  • Briefcase stores the blank form belonging to the last existing version of it
  • Briefcase has submissions belonging to all versions that have existed
• The user hasn't sent anything to Central yet.

The problem:

• Briefcase has no record of previous versions of the form
• Briefcase is able to push the blank form
• Briefcase is able to push the submissions matching the version of the blank form
• Other submissions are rejected

When folks are coming directly to this repository, they won't be able to get a context of what's going on in this repo. I think it will be beneficial to pull some information from the docs repo and put it here to give some information and context of what's, what are the plan and the trajectory.

And possibly submitter or other metadata-level attributes. See original forum post here.

If I want to use custom ssl like from cloudflare, what is the steps to put the certificate into the server? thanks.

root@central-sandbox:~/central# docker-compose ps
       Name                     Command               State                    Ports
------------------------------------------------------------------------------------------------------
central_mail_1       /bin/entrypoint.sh exim -b ...   Up      25/tcp
central_nginx_1      /bin/bash /scripts/odk-set ...   Up      0.0.0.0:443->443/tcp, 0.0.0.0:80->80/tcp
central_postgres_1   docker-entrypoint.sh postgres    Up      5432/tcp
central_service_1    ./wait-for-it.sh postgres: ...   Up      8383/tcp

It just says Up, it doesn't say Up (healthy)

I upgrade 0.4.0 and its can't run, bug is fix on 0.4.0-beta.1
but can't find app user create setting.

Roughly, this would look something like this:

1. get one of the xlsform => xform converters running as a webservice in a container and integrate it into the docker-compose setup (preferably it also works off of a local host/port directly so you don't have to use docker for local development; see how postgres and email are wired up for different examples of this)
2. have the /v1/projects/:id/forms POST endpoint detect excel files and forward the file to the service for transformation into xml before resuming the rest of the form creation flow
3. update the frontend to accept excel files
4. forward any errors output by the xlsform convertor to the user interface, probably with help from @matthew-white and some UI thought by @issa-tseng

Is there a way to view submitted images using API or token so user don't have to login as administrator first? case example is integrating odk central with other apps for viewing submitted forms inside that apps.

Currently I managed to access the postgre db by enabling port 5432 in yaml to outside (it's not ideal though but I just want to make proof of concept), but viewing images requires you to have login sessions in odk central.

We are currently on 4.84_2.

Steps that I've done to install in Debian 8 :

1. Install Docker (18.09) and Docker CE (version 1.23.2)
2. use non root user with access to sudo and docker without sudo
3. git clone the repo and use the command to update the dependencies
4. edit the .env files with appropriate settings

When executing docker-compose build on my vps there's a warning like this :
Warning: apt-key output should not be parsed (stdout is not a terminal) key 7FCC7D46ACCC4CF8: 2 signatures not checked due to missing keys
followed by errors that stopped the script :
E: Unable to correct problems, you have held broken packages. ERROR: Service 'service' failed to build: The command '/bin/sh -c echo "deb http://apt.postgresql.org/pub/repos/apt/ jessie-pgdg main" > /etc/apt/sources.list.d/pgdg.list; wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | apt-key add -; apt-get update; apt-get install -y cron gettext postgresql-client-9.6' returned a non-zero code: 100
is there something that I missed?

I've tried following the readme given, by copying the systemctl files using cp files/[email protected] /etc/systemd/system and when I try to run
systemctl start docker-compose@central it says : (using non root user)

Failed to get D-Bus connection: No such file or directory

when I try using sudo, there's no error but when I check the status it said :

● [email protected] - central via docker-compose
   Loaded: loaded (/etc/systemd/system/[email protected]; disabled)
   Active: failed (Result: start-limit) since Sun 2019-02-17 16:57:44 WIB; 3s ago
  Process: 4456 ExecStop=/usr/local/bin/docker-compose stop -v (code=exited, status=200/CHDIR)
  Process: 4455 ExecStart=/usr/local/bin/docker-compose up (code=exited, status=200/CHDIR)
 Main PID: 4455 (code=exited, status=200/CHDIR)

Feb 17 16:57:43 <my domain name> systemd[1]: [email protected]: control process exited, code=exited status=200
Feb 17 16:57:43 <my domain name> systemd[1]: Unit [email protected] entered failed state.
Feb 17 16:57:44 <my domain name> systemd[1]: [email protected] holdoff time over, scheduling restart.
Feb 17 16:57:44 <my domain name> systemd[1]: Stopping central via docker-compose...
Feb 17 16:57:44 <my domain name> systemd[1]: Starting central via docker-compose...
Feb 17 16:57:44 <my domain name> systemd[1]: [email protected] start request repeated too quickly, refusing to start.
Feb 17 16:57:44 <my domain name> systemd[1]: Failed to start central via docker-compose.
Feb 17 16:57:44 <my domain name> systemd[1]: Unit [email protected] entered failed state.

docker-compose exec node should be docker-compose exec service node

Request here.

It shown errors that previous form already exist (although I've deleted it previously in odk central web) :

I only managed to reuploaded after changing the form_id, converted it to xml and re-upload it again.

as noted here, browser-based clients hoping to read information from Central won't be able to do so unless the server permits CORS requests.

at the very least, OData seems like a prime candidate for CORS allowance.

but we need to think through the security ramifications if we're going to allow this, and decide whether it should be on by default or toggled via some kind of option somehow.

should we decide to do this at the nginx level, the above-linked thread has some useful configuration.

The next several versions of ODK Central will implement a Projects concept, and reorganize all the objects within Central to fit within the Projects framework.

An overview of the planned changes may be found in this document.

Here is a drawing (not to be taken too literally) demonstrating the general direction in which we are planning to go:

And here is a second drawing illustrating how we can get from our current state to the proposal state in somewhat reasonable increments:

Comments and feedback may be left in the proposal document, or directly here on this issue ticket.

Not breaking, just FYI, when building ODK Central with docker-compose (1.22.0) on Ubuntu 16.04 LTS, I'm getting warnings from node about vulnerabilities.

Building frontend:

 ---> 0f11a4f500a8
Step 6/11 : RUN npm install --production
 ---> Running in fe5afd8ab9ac
npm WARN notice [SECURITY] base64url has the following vulnerability: 1 moderate. Go here for more details: https://nodesecurity.io/advisories?search=base64url&version=2.0.0 - Run `npm i npm@latest -g` to upgrade your npm version, and then `npm audit` to get more info.
npm WARN notice [SECURITY] debug has the following vulnerability: 1 low. Go here for more details: https://nodesecurity.io/advisories?search=debug&version=2.6.8

Building nginx:

Step 4/16 : RUN files/prebuild/build-frontend.sh
 ---> Running in aa5ad77feb33
npm WARN notice [SECURITY] tunnel-agent has the following vulnerability: 1 moderate. Go here for more details: https://nodesecurity.io/advisories?search=tunnel-agent&version=0.4.3 
npm WARN notice [SECURITY] stringstream has the following vulnerability: 1 moderate. Go here for more details: https://nodesecurity.io/advisories?search=stringstream&version=0.0.5
npm WARN notice [SECURITY] randomatic has the following vulnerability: 1 low. Go here for more details: https://nodesecurity.io/advisories?search=randomatic&version=1.1.7
npm WARN notice [SECURITY] parsejson has the following vulnerability: 1 high. Go here for more details: https://nodesecurity.io/advisories?search=parsejson&version=0.0.3
npm WARN notice [SECURITY] ws has the following vulnerability: 1 high. Go here for more details: https://nodesecurity.io/advisories?search=ws&version=1.1.2
npm WARN notice [SECURITY] macaddress has the following vulnerability: 1 critical. Go here for more details: https://nodesecurity.io/advisories?search=macaddress&version=0.2.8
npm WARN notice [SECURITY] debug has the following vulnerability: 1 low. Go here for more details: https://nodesecurity.io/advisories?search=debug&version=2.3.3
npm WARN notice [SECURITY] lodash has the following vulnerability: 1 low. Go here for more details: https://nodesecurity.io/advisories?search=lodash&version=3.10.1
npm WARN notice [SECURITY] debug has the following vulnerability: 1 low. Go here for more details: https://nodesecurity.io/advisories?search=debug&version=2.2.0
npm WARN notice [SECURITY] lodash has the following vulnerability: 1 low. Go here for more details: https://nodesecurity.io/advisories?search=lodash&version=3.10.1
npm WARN notice [SECURITY] hoek has the following vulnerability: 1 moderate. Go here for more details: https://nodesecurity.io/advisories?search=hoek&version=2.16.3

EXIM security has always been questionable but now it's causing a lot of operational headache, and the upstream package we rely on is slow to respond.

OpenBSD is pretty much the gold standard in reliable secure engineering in my book, and they wrote OpenSMTPD. There is a Docker image for it here. If it drops in okay, and we can figure out how to set it up for DKIM etc, then we should probably switch over.

Hi, sorry to pile this onto a young project... Absolutely great to see a security-first focus and automation in this package!

However, I'm trying to integrate ODK Central into an existing (Docker-based) infrastructure with an existing nginx reverse proxy, which handles SSL for us. The built-in nginx and its SSL handling did not work at all with our own layer of nginx goodness.

Is there a simple way to deploy ODK Central through docker-compose without the baked in nginx and its SSL certificates? Would it be as simple as modifying docker-compose.yml, dropping the nginx service, and exposing 8383 on the service service?

http://sandbox.central.opendatakit.org does not redirect HTTP to HTTPS. It should, and I think we might want to make this the default behavior on all installs.

I tried following the instructions and got as far as docker-compose up but it's failing at this step:

nginx_1     | writing a new nginx configuration file..
nginx_1     | 2019/07/31 19:57:29 [emerg] 40#40: BIO_new_file("/etc/selfsign|letsencrypt|customssl/live/local|your.domain.com/fullchain.pem") failed (SSL: error:02001002:system library:fopen:No such file or directory:fopen('/etc/selfsign|letsencrypt|customssl/live/local|your.domain.com/fullchain.pem','r') error:2006D080:BIO routines:BIO_new_file:no such file)
nginx_1     | nginx: [emerg] BIO_new_file("/etc/selfsign|letsencrypt|customssl/live/local|your.domain.com/fullchain.pem") failed (SSL: error:02001002:system library:fopen:No such file or directory:fopen('/etc/selfsign|letsencrypt|customssl/live/local|your.domain.com/fullchain.pem','r') error:2006D080:BIO routines:BIO_new_file:no such file)
central_nginx_1 exited with code 1

Tried closing and running again. Then it gets as far as the below and hangs:

nginx_1     | writing a new nginx configuration file..
nginx_1     | 2019/07/31 20:02:18 [emerg] 8#8: BIO_new_file("/etc/selfsign|letsencrypt|customssl/live/local|your.domain.com/fullchain.pem") failed (SSL: error:02001002:system library:fopen:No such file or directory:fopen('/etc/selfsign|letsencrypt|customssl/live/local|your.domain.com/fullchain.pem','r') error:2006D080:BIO routines:BIO_new_file:no such file)
nginx_1     | nginx: [emerg] BIO_new_file("/etc/selfsign|letsencrypt|customssl/live/local|your.domain.com/fullchain.pem") failed (SSL: error:02001002:system library:fopen:No such file or directory:fopen('/etc/selfsign|letsencrypt|customssl/live/local|your.domain.com/fullchain.pem','r') error:2006D080:BIO routines:BIO_new_file:no such file)
central_nginx_1 exited with code 1
service_1   | running migrations..
service_1   | starting cron..
service_1   | starting server.

Am I doing something wrong?

consider implementing GeoJSON export as an option after this functionality is implemented in Aggregate and Briefcase?

see getodk/roadmap#26