quru / qis Goto Github PK
View Code? Open in Web Editor NEWDynamic image server for web and print
Home Page: https://quruimageserver.com
License: GNU Affero General Public License v3.0
Dynamic image server for web and print
Home Page: https://quruimageserver.com
License: GNU Affero General Public License v3.0
There is a scenario whereby an API call to /api/v1/details/
or /api/v1/list/
can return a history
field in the image object(s). For example:
(qis) qurumatt:qis matt$ curl 'http://localhost:5000/api/v1/details/?src=Mixed/matt-new.tif'
...
"height": 374,
"history": [
{
"action": 1,
"action_info": "File detected: Mixed/matt-new.tif",
"action_time": "2018-10-16T15:51:49.579022Z",
"id": 246,
"image_id": 146,
"user": null,
"user_id": null
}
],
"id": 146,
...
Whereas if you make the same call a second time (and subsequently), there is correctly no history
field:
(qis) qurumatt:qis matt$ curl 'http://localhost:5000/api/v1/details/?src=Mixed/matt-new.tif'
...
"height": 374,
"id": 146,
...
This bug is due to the caching of image objects in the SQLAlchemy session. For image files that the application finds on disk for the first time, a new audit trail is saved against the image and the image object is returned from the SQLAlchemy session cache. For images that already existed, the audit trail remains unloaded because it is lazy-loaded and is neither accessed nor changed.
This needs fixing to remove the history
field (a) because it makes the return values inconsistent and (b) because it could in theory lead to user details being publicly visible in the same way as #13.
The bug affects APIs that deal with new files on disk (or the discovery of them). This is details
, list
and upload
, though purely by chance the bug does not arise for upload
because the internal implementation uses a separate SQLAlchemy session.
details
list
upload
(in case of internal implementation changes in future)With the mask it is possible to make an image into a circle (have a white mask with a transparent circle in the middle and overlay it over the image) however the circular image remains on a coloured background. Would it be possible to convert the colour of the background to transparent?
To find the colour in a simple mask:
convert mask.gif -format %c -colorspace LAB -colors 5 histogram:info:- | sort -n -r | grep -v "#00000000"
will show the 5 most dominant colours in the image.
convert image.png -fuzz 20% -transparent "#ff00ffff" transparentimage.png
will make all #ff00ffff pixels transparent.
Note that glflib is possibly quicker for getting the colour (need giflib-tools)
gifclrmp mask.gif
| grep -r "#00000000"`
Note that 2 colour gif palettes for the mask make this much easier. The fuzz option (ie needed for anti-aliasing etc) might need playing with.
It is possible that someone might want to make a colour in the original image transparent rather than in a mask...
Commit 624a468 added a limiter to itsdangerous of being less than v1. This is being upgraded continuously. Has a recent upgrade fixed the issue so that it can be used without large scale code changes?
We could easily have contrast adjustment using
convert testimage.jpg -level 0%,75% lighttestimage.jpg
convert testimage.jpg -level 25%,100% darktestimage.jpg
See http://www.imagemagick.org/Usage/color_mods/#level for more details.
The API /api/v1/portfolios/[portfolio id]/images/[image id]/position/
does correctly set the value of order_num
in the database, and does return the images in the correct order in the JSON, but the value of order_num
in the JSON still contains the old value for the image that was moved.
I suspect this is a case of SQLAlchemy session caching again, we need to refresh the portfolio-image object in the session. The code already has a line for this at data_manager.py line 1582 but it doesn't seem to be working.
Looking at updating the API docs for #12 it has become obvious that the docs don't properly describe the difference between "parameters" that are GET query string parameters and those that are actually POST/PUT data fields. The difference only becomes apparent when looking at the examples. We should define these separately in the docs and describe the difference in the intro.
Ideally where today we have 1 or 2 URLs and GET/POST/PUT/DELETE methods all described in one section, the docs should describe every URL and method separately. But this would be a complete rewrite of the document and I don't have time to do that right now.
The folder list admin function returns deleted records (need to check the file admin for this too). This doesn't seem like a sensible default, so change it to filter by status<>0
, controlled by a new optional boolean parameter for whether to return soft-deleted data (or perhaps an optional status
filter parameter would be better?)
Technically this is a breaking change but I don't think it was ever intended to work this way and should have filtered out deleted records from day 1. Nothing much can be done with the deleted records as the associated physical files are gone, so I don't expect this change to break any existing code.
The README.md file provides instructions on how to generate a SECRET_KEY as:
python -c 'import os; print(os.urandom(16))'
With python2 installed this becomes unicode characters. To ensure that it works simply run:
python3 -c 'import os; print(os.urandom(16))'
The API provides GET /api/v1/portfolios/[portfolio id]/
but no way of retrieving a portfolio from the unique human ID. It should be easy enough to add a URL for GET /api/v1/portfolios/?human_id=foo
that does this.
I am following the installation guide and have a problem with httpd starting python program. OS is Centos 7, locale is en_US.UTF-8
Here is what I found in httpd error log:
Current thread 0x00007fb1adcf6880 (most recent call first):
[Tue Oct 22 18:01:32.200714 2019] [core:notice] [pid 1951] AH00052: child pid 2170 exit signal Aborted (6)
[Tue Oct 22 18:01:32.201642 2019] [core:notice] [pid 1951] AH00052: child pid 2171 exit signal Aborted (6)
Fatal Python error: Py_Initialize: Unable to get the locale encoding
Fatal Python error: Py_Initialize: Unable to get the locale encoding
ImportErrorImportError: : No module named 'encodings'No module named 'encodings'
I found suggestions to fix python path variables, but I don't have them and manually invoking python commands with UTF-8 encodings works well. Also I tried to install latest python3 from centos repository and compile mod_wsgi from pip, but got the same error.
Commit fa583f4 was to fix groups of warnings in the logs like the one below. But the change made was only appropriate for groups of tiles requested very close together in time. These logs show a gap of nearly 40 seconds while still triggering the old warning, which suggests that _get_base_image()
was failing to identify the base image for the tile even after it had been generated. The new code will fix this in _get_tile_base_image()
but does not address the underlying problem that _get_base_image()
should have found the base image in the first place. Therefore this needs more testing using the image parameters shown below.
2018-11-01 23:10:44,256 qis_16219 WARNING Tile base generation already performed for 24807277-9-1.jpg IMG:33111,W1160,H1740,Inone,Q75,Krgb,X,D72,E9:16
2018-11-01 23:10:53,958 qis_16215 WARNING Tile base generation already performed for 24807277-9-1.jpg IMG:33111,W1160,H1740,Inone,Q75,Krgb,X,D72,E11:16
2018-11-01 23:10:59,392 qis_16215 WARNING Tile base generation already performed for 24807277-9-1.jpg IMG:33111,W1160,H1740,Inone,Q75,Krgb,X,D72,E13:16
2018-11-01 23:11:04,100 qis_16215 WARNING Tile base generation already performed for 24807277-9-1.jpg IMG:33111,W1160,H1740,Inone,Q75,Krgb,X,D72,E14:16
2018-11-01 23:11:08,483 qis_16215 WARNING Tile base generation already performed for 24807277-9-1.jpg IMG:33111,W1160,H1740,Inone,Q75,Krgb,X,D72,E15:16
2018-11-01 23:11:12,907 qis_16215 WARNING Tile base generation already performed for 24807277-9-1.jpg IMG:33111,W1160,H1740,Inone,Q75,Krgb,X,D72,E16:16
2018-11-01 23:11:13,575 qis_16219 WARNING Tile base generation already performed for 24807277-9-1.jpg IMG:33111,W1160,H1740,Inone,Q75,Krgb,X,D72,E1:16
2018-11-01 23:11:19,291 qis_16215 WARNING Tile base generation already performed for 24807277-9-1.jpg IMG:33111,W1160,H1740,Inone,Q75,Krgb,X,D72,E2:16
2018-11-01 23:11:23,667 qis_16215 WARNING Tile base generation already performed for 24807277-9-1.jpg IMG:33111,W1160,H1740,Inone,Q75,Krgb,X,D72,E3:16
2018-11-01 23:11:23,849 qis_16219 WARNING Tile base generation already performed for 24807277-9-1.jpg IMG:33111,W1160,H1740,Inone,Q75,Krgb,X,D72,E4:16
And another group spread out over nearly 2 minutes:
2018-11-01 22:47:47,890 qis_22804 WARNING Tile base generation already performed for 24807215-10-1.jpg IMG:29843,W3192,H2256,Q75,Krgb,X,D72,E45:64
2018-11-01 22:47:52,282 qis_22803 WARNING Tile base generation already performed for 24807215-10-1.jpg IMG:29843,W3192,H2256,Q75,Krgb,X,D72,E52:64
2018-11-01 22:47:52,883 qis_22805 WARNING Tile base generation already performed for 24807215-10-1.jpg IMG:29843,W3192,H2256,Q75,Krgb,X,D72,E53:64
2018-11-01 22:47:57,465 qis_22803 WARNING Tile base generation already performed for 24807215-10-1.jpg IMG:29843,W3192,H2256,Q75,Krgb,X,D72,E35:64
2018-11-01 22:47:57,880 qis_22805 WARNING Tile base generation already performed for 24807215-10-1.jpg IMG:29843,W3192,H2256,Q75,Krgb,X,D72,E43:64
2018-11-01 22:48:02,934 qis_22803 WARNING Tile base generation already performed for 24807215-10-1.jpg IMG:29843,W3192,H2256,Q75,Krgb,X,D72,E51:64
2018-11-01 22:48:05,110 qis_22805 WARNING Tile base generation already performed for 24807215-10-1.jpg IMG:29843,W3192,H2256,Q75,Krgb,X,D72,E59:64
2018-11-01 22:48:32,599 qis_22805 WARNING Tile base generation already performed for 24807215-10-1.jpg IMG:29843,W1732,H1224,Q75,Krgb,X,D72,E11:16
2018-11-01 22:48:32,988 qis_22804 WARNING Tile base generation already performed for 24807215-10-1.jpg IMG:29843,W6328,H4472,Q75,Krgb,X,D72,E38:64
2018-11-01 22:48:34,575 qis_22804 WARNING Tile base generation already performed for 24807215-10-1.jpg IMG:29843,W3192,H2256,Q75,Krgb,X,D72,E31:64
2018-11-01 22:48:49,628 qis_22804 WARNING Tile base generation already performed for 24807215-10-1.jpg IMG:29843,W1732,H1224,Q75,Krgb,X,D72,E12:16
2018-11-01 22:48:50,238 qis_22805 WARNING Tile base generation already performed for 24807215-10-1.jpg IMG:29843,W1732,H1224,Q75,Krgb,X,D72,E14:16
2018-11-01 22:48:53,542 qis_22804 WARNING Tile base generation already performed for 24807215-10-1.jpg IMG:29843,W6328,H4472,Q75,Krgb,X,D72,E46:64
2018-11-01 22:49:29,973 qis_22803 WARNING Tile base generation already performed for 24807215-10-1.jpg IMG:29843,W6328,H4472,Q75,Krgb,X,D72,E28:64
2018-11-01 22:49:30,690 qis_22804 WARNING Tile base generation already performed for 24807215-10-1.jpg IMG:29843,W6328,H4472,Q75,Krgb,X,D72,E29:64
I'm copying this issue here from Quru's internal bug tracker as it documents some interesting behaviour.
Seen in live today when it ruined a PDF report:
qis_access_log:
109.111.129.21 [08/Dec/2017:03:50:46 +0000] "GET /image?src=Images/live/2016-06/17/9226877-587-1.jpg.tif&top=0&left=0&right=1&bottom=1&fill=&angle=0&tmp=ConditionReport HTTP/1.1" 500 620 - 132606 - "Prince/10 (www.princexml.com)" "-"
No error in qis.log indicates that the error was not in the Python code. But qis.log does indicate that the httpd processes were being restarted at that time. Checking the cron log:
cron:
Dec 8 03:47:01 images7 run-parts(/etc/cron.daily)[8987]: starting logrotate
Dec 8 03:49:54 images7 run-parts(/etc/cron.daily)[11603]: finished logrotate
So yes there was a graceful restart in progress. Checking the Apache error log:
error_log:
[Fri Dec 08 03:50:46 2017] [error] [client 109.111.129.21] Premature end of script headers: runserver.wsgi
[Fri Dec 08 03:50:47 2017] [notice] child pid 11703 exit signal Segmentation fault (11)
We can see that the httpd process crashed at the same time and for the same client IP. Bingo.
It's going to be hard to find out if this is an issue with mod_wsgi on startup or with something the QIS code is doing e.g. when it forks off the auxiliary processes on the first request. The latter would seem more likely and can be tested seeing as this problem is reproducable. We may also need to try using the latest mod_wsgi instead of the old versions shipped in the standard o/s packages.
Looks like there is a missing configuration in the docker image?
Logs:
$ docker ps | grep as
59de577458a1 quru/qis-as "/run-qis.sh" 3 hours ago Up 3 hours (unhealthy) 0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp qis-via-docker_qis_as_1
$ docker logs 59de577458a1
Performing one-time initialization
Waiting 30 seconds for the Postgres database to be created
AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 172.***.***.3. Set the 'ServerName' directive globally to suppress this message
$ docker -v
Docker version 19.03.2, build 6a30dfc
$ curl --insecure -i https://localhost
HTTP/1.1 500 Internal Server Error
Date: Wed, 02 Oct 2019 11:50:37 GMT
Server: Apache/2.4.18 (Ubuntu)
Content-Length: 607
Connection: close
Content-Type: text/html; charset=iso-8859-1
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>500 Internal Server Error</title>
</head><body>
<h1>Internal Server Error</h1>
<p>The server encountered an internal error or
misconfiguration and was unable to complete
your request.</p>
<p>Please contact the server administrator at
[no address given] to inform them of the time this error occurred,
and the actions you performed just before this error.</p>
<p>More information about this error may be available
in the server error log.</p>
<hr>
<address>Apache/2.4.18 (Ubuntu) Server at localhost Port 443</address>
</body></html>
At present the Docker files and docker-compose script don't make it easy to install TLS certs or to override the QIS settings. We're also using an env var for the database password from the days before docker-compose had secrets support. I propose:
** This also allows multiple AS containers to use the same certs and session secret for load balancing
Hi,
in our project we use QIS as our image server, which the users upload images via REST from our React SPA. Currently the app authenticates via a single user which acts as a service account. I am wondering though, if it is possible to connect QIS to our IDM via OpenID Connect. [1]
Ultimately the user would have ownership about his images and we could implement access control bases on permissions managed via the IDM. The user's profile picture for example could be private for example, whereas images the of something like a blog post [2] would be visible to. any logged in user.
Bests,
Martin
[1] We use Keycloak as IDM.
[2] just an example
Hi, How it is possible to delete images via the rest api after they re uploaded by api/v1/upload
endpoint ?
Moving here from Quru's issue tracker to record the feature history:
QIS needs a publicly-accessible demo/playground page so that people can explore the imaging features - similar to what the publisher offers plus gallery / zoom viewing and so on. Whether it is enabled should be behind a system setting. Hopefully the URL-generating logic in the publisher page can be re-used.
The Portfolios API "helpfully" includes serialized user objects in its output for the owner
and history user
fields. Unfortunately portfolios can be public facing, and therefore this can leak the username and email address of anyone that has created or interacted with a portfolio. We need to remove these objects, just leaving the owner_id
and history user_id
fields in place. Anyone that needs to look up the user information will have to do so with a separate call to ensure they have user admin permission.
The Tasks API also does this but is technically not affected because it should only return the user object of the current user or else only to super users. I think it's best if we delete it from here too though. The current user already knows who they are after all, so it doesn't add much in the common case.
owner
objectuser
objectuser
objecturl, download, supported
fields are missing from file admin operations)I don't think that any of these changes will break compatibility (you only get HTML errors for calls that are broken anyway).
For images that have GPS fields in the EXIF data we display a Google map of its location when viewing the image in the admin area. This map is now degraded.
From https://cloud.google.com/maps-platform/user-guide/
In June 2016 we announced that we would stop supporting keyless usage, meaning any request that doesn’t include an API key or Client ID. This went into effect on June 11 2018, and keyless access is no longer supported. Keyless calls to the Maps JavaScript API and Street View API will return low-resolution maps watermarked with “for development purposes only.”
It is now mandatory to use a billing account and a Static Maps API key. This comes with $200 free monthly credits so it should still be free for most use cases, but we are going to have to get users of the image server to apply for and plug in their own API key. That or we change the map to run from a different service.
This one needs a quick review by @qururoland
The switch to Python 3 and/or the switch from pyldap to python-ldap has caused a regression when creating a new user account via an LDAP login. From the logs:
2018-08-09 12:09:44,514 qis_30863 DEBUG Authenticating user 'trevor'
2018-08-09 12:09:44,517 qis_30863 DEBUG Checking LDAP server for unknown user 'trevor'
2018-08-09 12:09:44,545 qis_30863 DEBUG Identified user 'trevor' on LDAP server, authentication OK, creating new user account
2018-08-09 12:09:44,546 qis_30863 DEBUG User details: {'gidNumber': [b'1234'], 'cn': [b'Trevor Foo'], 'uid': [b'trevor'], 'givenName': [b'Trevor'], 'uidNumber': [b'1234'], 'sn': [b'Foo'], 'dn': ['uid=trevor,cn=users,cn=accounts,dc=foo,dc=bar,dc=baz']}
2018-08-09 12:09:44,546 qis_30863 ERROR Error performing login: a bytes-like object is required, not 'str'
The code is trying to process that dictionary as strings whereas the entries now contain bytes (apart from the dn which is a str 🙄). I think python-ldap has a setting for whether it returns strings or bytes so we need to look at that. Alternatively we just need to str()
those bytes values in the code.
Some error conditions leak too much information back to the user, e.g. full operating system paths:
{ "status": 500,
"message": "Internal error ([Errno 13] Permission denied: '/opt/qis/images/published/2018-10-02/night.jpg')" }
In theory it is possible for worse things to leak, e.g. database connection details when the database is down. We should run error messages through a filtering routine before returning them. This can be initially based on the settings in base_settings.py
and any other obvious things to filter out.
It would be great if we can spin up a demo environment with this:
Needs testing to see whether it is possible to run a web service within it and get/supply the required host name for URL access. Also if so, whether there is any way of deploying HTTPS.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.