QGIS Enhancement 75: Add concept of "true north" to composer north arrows

Date 2016/10/10

Author Nyall Dawson (@nyalldawson)

Contact [email protected]

maintainer Nyall Dawson

Version QGIS 3.0, with backports to 2.x series

Summary

This QEP relates to the decade old feature request/bug https://hub.qgis.org/issues/192

Currently QGIS north arrows only consider grid north, and have no means of handling true (and magnetic) north.

This work is designed to be backported to all maintained branches.

Proposed Solution

UX

Currently, North Arrows are created using print composer image items and the only option is to sync the picture rotation with a map item's grid north.

A new combobox will be added below the existing "Sync with map" option, with options "Sync to grid north" and "Sync to true north". The default behaviour will stay at "sync to grid north".

Calculation

To calculate true north, the follow procedure will be used:

1. Take the center point of the linked map item (P1)
2. Project this point into EPSG:4326 (WGS84) (P1`)
3. Add a small offset to the projected point's latitude, ie move the point a small amount up it's current meridian (P2`)
4. Project this shifted point back to the map's CRS (P2)

The bearing between P1 and P2 will be used as direction to True North.

Backwards Compatibility

It is intended to backport these changes to 2.14, 2.18 and master_2. This will introduce new strings which will need to be translated.

Possible follow-ups

The work could be extended by adding an "angle offset" option which is enabled when "sync with map" is checked. This offset could be used to sync an arrow with magnetic north, by first calculating the grid convergence for a desired location/time.

Issue Tracking ID(s)

https://hub.qgis.org/issues/192

Votes

(required)

QGIS Enhancement 29: QGIS 3.0

Date 2015/10/21
Author Nyall Dawson
Contact nyall dot dawson at gmail dot com
Last Edited 2015/10/21
Status Draft
Version QGIS 3.0

1. Summary

Following the LTR 2.14 release, the subsequent release should be QGIS 3.0, and include:

• an API break
• Qt5 only build support
• Python 3.0 support

2. Background

3. Proposed Changes

3.1 API Break
.............

QGIS API has remained stable throughout the 2.0 series. While it is acknowledged
that this is desirable for plugin and script maintainers, QGIS 3.0 will allow
API breaks. This is required to clean up the codebase and make deeper changes
and bug fixes which are currently impossible to do without breaking API. It
also allows for removal of old compatibility code (currently QGIS master has over
300 methods marked as deprecated), simplifying future development and
maintenance efforts.

3.2 Qt 5 Requirement
....................

Currently QGIS supports building on both Qt4 and Qt5 libraries, with the default
build being Qt4. Qt4 has been end-of-lifed by it's developers and is no longer
maintained. This raises significant risks for QGIS if we do not move to Qt5 builds
as quickly as possible. For instance, Qt4 cannot be built on the latest OSX
release (10.11). Third party patches have been made which partly resolve this
issue, but the side effects are unknown. It is highly likely that a future
OSX release, Windows update or linux packaging change will prevent use of
Qt4 on one of our supported platforms and result in QGIS being unavailable
for that platform.

Additionally, numerous bugs present in the current QGIS release which are
caused by issues in Qt have been resolved in Qt5. Moving to Qt5 is the only way
to fix these issues.

There's also significant performance and feature benefits made possible by
the switch to the newer platform.

3.2.1 Issues with moving to Qt5
...............................

Qt5 has dropped support for the QWebView/QWebPage classes. The replacement
class, QtWebEngine, is immature and lacks several important features which
are currently used by QGIS. The largest impact this will have will be on
the composer "HTML" item. This item is currently rendered using a QWebPage
painting directly on to a QPainter surface, resulting in text rendering
as text objects and vectors. This is NOT currently possible with the
replacement classes, and it is unclear whether this will be brought back
by a future Qt release. It is also impossible to render QtWebEngine contents
using a transparent background. Unfortunately, there is no workaround for these
limitations and the feature will have to be modified so that HTML content is either:

• rendered as an upscaled (read: pixelated) image
• rendered using the subset of HTML and CSS supported by QTextDocument (see
  http://doc.qt.io/qt-5/richtext-html-subset.html )

3.3 Python 3.0
..............

Work is already underway and a PR to master is inplace to allow QGIS to be built using Python 3.0 and PyQt5.

4.0 Implications for 2.14 LTR release

The QGIS 3.0 release will follow the LTR 2.14 release, which will be the final release in the 2.0 series. In order to support organisations who will be stuck using the 2.0 series for the immediate future (eg due to plugin requirements or other script support requirements) only low-risk, small impact changes will be allowed in QGIS 2.14. This is intended to allow us to deliver the most-stable version possible as the final 2.0 series release.

Large changes should be held-off until master is un-frozen for 3.0 development.

.# Affected Files

..................

Almost everything!

. Further Considerations/Improvements

Support for plugin developers in the form of guides for porting to QGIS 3.0
will need to be developed.

. Voting History

(required)

QGIS Enhancement 38: Remove irregular 'star' symbol from simple marker

Date 2016-04-06
Author Nyall Dawson (@nyalldawson)
Contact nyall dot dawson at gmail dot com
Status Approved
Funding N/A
Version QGIS 2.16

Summary

The simple marker point symbology option includes a growing number of useful preset symbol shapes. One of these ("star" - see below) is a very ugly, distorted star symbol. It is extremely unlikely that users will be utilising this distorted symbol in their maps, so this QEP covers removal of the symbol.

At some stage a standard star symbol was also added, with the confusing name of 'regular_star'. There are now two star symbols in the simple marker symbology. This QEP proposes that the distorted 'star' symbol be removed, and transparently replaced with the better 'regular_star' symbol for existing projects.

Proposed Solution

The current distorted 'star' symbol will be removed, and the regular star symbol renamed as 'star' to take its place. If a project or data defined property references the 'regular_star' symbol, this will be treated as an alias for 'star'.

Affected classes

• QgsSimpleMarkerSymbolLayerV2: QgsSimpleMarkerSymbolLayerV2::prepareShape will be modified to remove the distorted star polygon and return the standard star if the symbol name is either 'star' or 'regular_star'.
• QgsSimpleMarkerSymbolLayerV2Widget: description text for the data defined symbol name will remove the 'regular_star' option. Note that this value will still function, it will just be treated as an alias for 'star'.

Votes

(required)

QGIS Enhancement 33: MetaSearch Publishing Enhancements

Date 2015/03/26
Author Tom Kralidis
Contact tomkralidis at gmail dot com
Last Edited 2015/03/26
Status Draft
Version QGIS 2.10

Summary

This QEP aims to enhance MetaSearch to provide metadata publishing to CSW providing the following benefits:

• easy, standards-based metadata publishing mechanism of QGIS layer metadata at the core (no duplications or QGIS plugins)
• interchangeable CSW service to publish to (user configured)
• enhanced discovery (better metadata is more easily discovered)

Overview

QIGS by default provides a Layer Properties Metadata tab which allows a user to enter various values (and select from various select boxes) under various sections like Description, Attribution, MetadataUrl, LegendUrl. The Properties section is has both editable (title, etc.) and read only attributes (autogenerated from the data such as bbox, etc.). These values are stored in the QGIS project file within the <maplayer> element.

MetaSearch is a QGIS plugin to interact with metadata catalogue services, supporting the OGC Catalogue Service for the Web (CSW) standard. MetaSearch provides an easy and intuitive approach and user-friendly interface to searching metadata catalogues within QGIS.

It is proposed to improve MetaSearch to add layer metadata publishing to the plugin seamlessly, efficiently, and in an authoritative manner.

This QEP will support an overall workflow per below:

manage Layer Metadata (QGIS core) -> publish/update/delete Layer Metadata (MetaSearch) to CSW

Proposed [Technical] Solution | Change

MetaSearch will add a 'Publishing' tab, which allows the user to:

• select from the list of CSW endpoints to publish to
• select a metadata format to publish (Dublin Core, ISO)
• click 'Publish', which does a CSW-T Insert or Update to the server. An 'Update' button will overwrite the entire metadata record on the server. A 'Delete' button will allow for deleting records from the CSW-T

Implementation Details

• modify settings tab to allow for authentication against CSW-T
• add publishing tab with 'Publish' 'Update' and 'Delete' buttons
  • metadata values will be derived from Layer Properties Metadata tab values
  • select box to choose format ('Dublin Core', 'ISO'). Metadata XML will be automagically generated against an XML template, filling in values from the Layer Properties Metadata tab

Affected Files
...............

• https://github.com/qgis/QGIS/blob/master/python/plugins/MetaSearch/ui/maindialog.ui
• https://github.com/qgis/QGIS/blob/master/python/plugins/MetaSearch/dialogs/maindialog.py

Test Coverage

(required for technical solutions/changes if applicable)

Performance Implications

No performance implications are expected at this time given the lightweight nature of this enhancement to MetaSearch.

Further Considerations/Improvements

This enhancement can be extended by adding the ability for users to upload user-defined XML templates, or select from more metadata standards/profiles.

Restrictions

Backwards Compatibility

Given this enhancement there will be no impact on backwards compatibility. There will be a UI design/change which will not break functionality.

Documentation

Documentation will be enhanced accordingly in https://github.com/qgis/QGIS-Documentation/blob/master/source/docs/user_manual/plugins/plugins_metasearch.rst

Issue Tracking ID(s)

(required)

References

(optional)

Miscellaneous

(optional)

Voting History

(required)

QGIS Enhancement 27 (was 20): Auxiliary storage

Date October 2015
Author Hugo Mercier
Contact hugo dot mercier at oslandia dot com
Last Edited May 2016
Status
Version

Summary

This proposal is about an evolution of QGIS aiming at storing auxiliary data in a layer. Such auxiliary data are data used mostly for the needs of QGIS (symbology) and have no real interest in being stored with the native raw geospatial data.

The need arises from the restrictions existing in the manual placement of labels.
Manual placement of labels are possible in QGIS by setting some labeling properties (X and Y position, and rotation angle optionaly) as being "data-defined", meaning that values come from a column (or an expression).
But setting this up on an existing layer requires either to add new columns to the source layer, while it is not always possible or desirable. Creating another table and using joins to it is possible but the synchronisation between the two tables is not possible, leading to a solution not satisfactory.

We propose here a more integrated solution where the creation of auxiliary columns or layers are brought more transparently to the user.

Requirements

The proposed solution have the following requirements :

• It should allow to transparently add columns of any type to a layer, without adding them to the source data.
  Beside usual types (string, real, integer), we may also think of geometry types. We only consider types that can be converted to atomic types (string for WKT or WKB)
• The user should be able to add these columns and to edit values even if the layer is in read-only mode. Auxiliary data may be updated by the user without updating source data, or for read only source data.
• These values should not be stored with the source data, and should be carried by the project, i.e.
  when the project file is moved or copied, the auxiliary data should still be there.
• Adding a new row to the layer also adds a new row for each auxiliary field (with a null or default value)
• New GUI elements should ease the creation of auxiliary fields
  • on each "data-defined" property
  • possibly on groups of "data-defined": for manual label placement or label connectors
• It should be possible to save auxiliary data as regular data into a new vector layer.

Proposed solution

We propose to extend the functionnalities of « joins » between vector layers by bringing write support and synchronisation.
Shortcuts in the main applications will allow to easily create these kind of linked layer with write support to answer the need of auxiliary fields in a layer.

Core change : Project file format

The current project file format (XML) is too limited to carry an efficient storage of such auxiliary data. It should somehow be evolved to include a binary indexable format. SQLite is chosen as a first candidate for this data format. But it might evolve if performance issues are found.

Given the current state of the core code where lots of classes assume the project file is an XML file (everything that is saved to the project file has a writeXML method), it seems hard to completely refactor the project file format.

We propose to use a Zip file as the root of the new project file format. A Zip file exposes a filesystem-like structure.
The original XML-based project file will then be embedded in the Zip file. And additional files could de added.
It only adds a step after actually saving the XML tree: including it in a Zip file. Symetrically during the loading, the first step would be to extract the XML file and pass it to the existing readXML methods.

Packaged layers: In order to be able to embed layers in a project file, file-based providers will be changed to accept files contained in a zip file (through a url syntax like zip://zipfile.zip/dir/file for instance)

Backwards Compatibility: Project files from previous versions will still be available for opening. When saving back these project files, they will be saved in the new file format by default.
An option would be available to save a project in the old file format for compatibility. In this case, auxiliary fields could not be packaged in the project file, and it will be proposed to the user to extract them in the project file directory.

Core change : Joined fields

Joins will be extended with write support. A joined field will be editable in the primary layer.

Two new options available during the creation of the join will be added :

• delete cascade option : when enabled, when a feature from the primary layer is deleted, the corresponding feature of the linked layer will be deleted as well
• upsert on edit option : when enabled each editing of the joined field from the primary layer will update the field of the corresponding field in the linked layer or insert a new feature in the linked layer if needed

Joins will also be extended to allow layers without explicit unique identifier field to be joined. In this case, the internal identifier of the feature will be used as a unique id. This feature will only be available for primary layers in read only mode.

During the creation of a join with write support, the user will be warned if a layer is joined more than once, in order to prevent concurrent write access by different layers which could lead to undesired behaviours.

Edit modes: Auxiliary fields should be editable even if the source layer is not. And edition of a joined field should be possible if the linked layer is in edit mode. QgsVectorLayer and QgsVectorLayerEditBuffer will be modified to implement these needs.

GUI Changes

Data-defined properties are the most probable place where auxiliary fields will be used.

In the menu that can be found for each data-defined property, a new entry will be added to allow the creation, in a click, of an auxiliary field for that property and link it as the source of the data-defined property.

If no auxiliary field exists yet, the user could choose to create an auxiliary layer for a given layer. This will create a new sqlite layer joined to the source layer, stored as a file attachment to the zip file project.

Similar GUI shortcuts could then be added in order to automate the creation of a group of data-defined properties.

The creation of such auxiliary fields will also be transparent for the user: the manual labeling tools will be available and if no fields already exist for data-defined properties, they will be automatically created through auxiliary fields.

QGIS Enhancement 32: Add QGIS server access control interface for python plugin

:Date: 2015/05/27
:Author: Stéphane Brunner
:Contact: stephane dot brunner at camptocamp dot com
:Last Edited: 2015/05/27
:Status: Draft | Adopted (YYYY/MM/DD) | Completed (YYYY/MM/DD) |
Superseded by :ref:QEP #[.#] <qep#[.#]> (YYYY/MM/DD) |
Abandoned (YYYY/MM/DD) | Withdrawn (YYYY/MM/DD)
:Version: QGIS X.X

.. note::

See :ref:`QEP 1 <qep1>` for description of QEP process.

. Summary

Add the possibility to create python plugin for access control
to all the services of QGIS server,

The plugin will implements the following interface:

• layerFilter Return an additional filter, used in WMS/GetMap,
  WMS/GetFeatureInfo, WFS/GetFeature to filter the features.
• layerPermissions Change the rights on the layer per user
  (known by the plugin). The concerned rights are: publish, insert, update,
  delete. Mostly used in WFS/Transaction, and the publish in all requests.
• authorizedLayerAttributes Be able to show some attributes only for a
  subset of user Used in: WMS/GetFeatureInfo, WFS/GetFeature.
• allowToEdit Be able to disallow the edition of a particular feature,
  in our case base on the Geometry Used in: WFS/Transaction.

Our need is to be able to do the following kind of restrictions::

• Geometric restriction that depends on the connected user and the layer.
• Attributes restriction that depends on the connected user and the layer.

All the knowledge about which user is connected and on what he is allowed to
access is the responsability of the plugin.

layerFilter and allowToEdit fill nearly the same use case,
but layerFilter is optimised to create requests and concerns the read access,
and allowToEdit is optimised to be used on an existing feature object
and concerns the write access.

. Implementation Details

See: qgis/QGIS#2056.

I create a new kind of server plugin and I don't extend the current server filter
for the following reason::

• We already many issue by fintring things in a generated xml file.
• The plugin implementation will be relay complex.
• The FEATURE_COUNT won't correctly work with hooks.

.# Python Bindings

...................

See summary.

.# Affected Files

..................

See: https://github.com/qgis/QGIS/pull/2056/files.

. Test Coverage

I don't know how we can do unit test for this. I think one possibility is to
run the server on apache, and test the query results.

. Performance Implications

Should be really low if there no plugin that implements it.
Also depends on the plugin implementation.

. Backwards Compatibility

New interface, then no incidence.

. Documentation

We can create restriction access python plugin.

The class to implements is qgis.server.qgsaccesscontrolplugin.

The optional methods to implement are:

.# layerFilter

...................

Return a filter for the map that will be used in the following requests::

• WMS/GetMap
• WMS/GetFeatureInfo
• WFS/GetFeature

.# layerPermissions

........................

Restrict the permissions on the layer. The concerned rights are:
publish, insert, update and delete.

The publish permission is used in all the request that concern a layer
including the GetCapabilities.

The others permissions are be used in the WFS/Transaction.

.# authorizedLayerAttributes

.................................

Restrict the published feature attributes.

Used in the following requests::

• WMS/GetFeatureInfo
• WFS/GetFeature

.# allowToEdit

Disallow the edition of a specific kind of feature.

Used in the following requests::

• WFS/Transaction

.# Python lugin example

.. code:: python

from qgis.server import QgsServerAccessControlFilter

def serverClassFactory(serverIface):
    serverIface.registerSecurity( ExampleAC(serverIface) )

class ExampleAc(QgsServerAccessControlFilter):

    # Return an additional expression filter
    def getLayerFilter(self, layer):
        # access only to element with ID 1
        return "1 = $id"

    # Return the layer rights
    def getLayerRights(self, layer):
        # access only to layer country
        rights = QgsServerSecurity.LayerRights()
        rights.publish = layer.name() != "Country"
        return rights

    # Return the authorised layer attributes
    def getAuthorizedLayerAttributes(self, layer, attributes):
        # access only to the attribute named "name"
        return [attrib for attrib in attributes if attrib == "name"]

    # Are we authorise to modify the following geometry
    def allowToEdit(self, layer, feature):
        # allow to edit only the feature with id 1
        return feature.id == 1

. Issue Tracking ID(s)

#2056

. Voting History

(required)

QGIS Enhancement: Attached documents

Date 2016/10/24

Author Matthias Kuhn (@m-kuhn)

Contact [email protected]

maintainer @m-kuhn

Version QGIS 3.0

Summary

Documents like text, images, videos or any type of file can currently be linked with an external resource widget into QGIS projects. For domain specific modules, it is often required to attach multiple documents to an entity. QGIS offers 1:N relations which handles this kind of requirement.

At the moment, this requires one document table per feature table. This sometimes (almost) duplicates the number of tables inside a database. To keep the models more simple, this QEP proposes to offer the possibility to maintain all documents in a single document table (or more generally, any type of attached fields and not only documents).

The management of this will introduce a new kind of dynamic relation where the referenced table is not fixed but determined by an expression at runtime.

Proposed Solution

Dynamic relations

A dynamic relation consists of the same properties like an ordinary relation except for the referenced layer which is calculated based on a field of the referencing layer. In its most simple form, the referencing layer will just insert the layer id of the referenced layer into this field.

By using an expression this can be adjusted to match other properties of the the referenced layer like the table name, schema, uri, layer id, ...

An example configuration could look like this:

• referencing_layer: documents_layer_id_20161030928471
• referencing_field: fk_feature_id
• referenced_field: id
• referenced_layer
  • referenced_layer_field: table_name
  • referenced_layer_expression: CONCAT( "schema", '.', "table_name" )

Layer metafeatures

To be able to compute expressions on layers, layers need to be exposed as features. To do this, the QgsLayerRegistry will receive a new method getFeatures() that returns a QgsFeatureIterator which will iterate over layers.

Each "feature" has attributes like

• id
• uri
• schema
• table_name
• file_name
• ...

Embedding the document layer

A document layer will be available as relation editor (n-side) for all other layers in a project by default if the “referenced field" is available on the layer.

When using the drag and drop designer, it is available the same way like ordinary relations are now already.

In an embedded document layer, only features where both conditions (referencing_field = referenced_field and referenced_layer_expression( parent form layer ) = referenced_layer_field ) match will be shown.

When adding a new entry, both fields (referencing_field and referenced_layer_field) will be prefilled.

The document layer form

On the document layer, the relation reference widget does not receive any integration.
This may be part of future work where the shown form changes with the feature.

Performance Implications

None foreseen.

Backwards Compatibility

• Existing relations will continue to work as before.
• Loops over relations in a project will have to be updated to deal appropriately with the fact of dynamic layers. They will either have to skip dynamic relations or deal with features from different layers when looping over related features.

API Changes

The QgsRelation API will be updated:

    /**
     * Will return the referenced layer. For dynamic relations it is required to pass the feature
     * or the return value will be a nullptr.
     */
    QgsVectorLayer* referencedLayer( const QgsFeature& feature = QgsFeature() );

    /**
     * In case of dynamic layers the resulting feature or request depends on the layer
     * derived from attrs. The layer can be retrieved with the method referencedLayer( feature )
     */
    xxx getReferencedFeature[xxx]( const QgsAttributes& attrs )

    /**
     * For dynamic relations, a layer will be required to successfully build the filter to get
     * features from the document layer.
     */
    xxx getRelatedFeatures[xxx]( const QgsFeature& feature, QgsVectorLayer* layer = nullptr );

Aggregate functions

Aggregate functions might not fully support dynamic layers for now. Support might be added where trivial.

Votes

(required)

QGIS Enhancement 42: Compulsory Unit Tests for Core Changes

Date 2014/10/05
Author Nyall Dawson
Contact [email protected]
Last Edited 2014/10/05
Status Draft
Version QGIS 2.7+

Summary

This proposal is aimed at providing more stable releases and preventing regressions in QGIS releases by expanding
the coverage of unit tests and adopting a hard line approach to compulsory unit tests.

Background

The QGIS coding guidelines (ref) state that since "November 2007 we require all new features going into master to be
accompanied with a unit test" (https://github.com/qgis/QGIS/blob/master/CODING#L738). However, this requirement is not
being enforced and the vast majority of core changes are not accompanied by unit tests. Lack of unit tests impact
on our ability to deliver stable, regression free releases and potentially harms our reputation as an enterprise-ready GIS.

Proposed Solution

From QGIS >=2.7, all commits which modify code in "core" are required to be accompanied by sufficient unit tests to ensure
that they will not suffer regressions in future QGIS releases. Commits which are not accompanied by unit tests will not be accepted
into master. Code changes for GUI and app are immune from this requirement due to the complexity in writing interactive
tests suites.

Exemption for specific commits may be granted, but only after discussion on the QGIS developer list.

Generally, unit tests in either c++ or python are acceptable (or both). However, methods which involve transfer of ownership
of objects must include python tests to ensure that this transfer is handled correctly.

To enforece this requirement, commits which modify core and which are not accompanied by unit tests (or have been granted
exemption via the Dev list) will be reverted. Pull requests without sufficient unit tests will not be considered for merging.

Implementation Details

If accepted, the existing requirement for unit tests in the CODING file will be updated to reflect the new requirements. The
new requirements will be advertised to existing developers via an announcement on the QGIS-Dev mailing list.

Voting History

(required)

QGIS Enhancement 73: Data definable label geometry

Date 2016/09/13

Author Nyall Dawson (@nyalldawson)

Contact [email protected]

maintainer Nyall Dawson (@nyalldawson)

Version QGIS >= 3.0

Summary

While current versions of QGIS include a very powerful and flexible labelling engine one use case which is not addressed is the ability to override a feature’s geometry for use within the label engine. Currently, the label engine always uses a feature’s original geometry when generating placement options for label candidates. In many cases this is not desirable, such as:

• when a feature’s geometry is complex and should be generalised or smoothed to allow optimal label placement
• when a manually created geometry should be used instead of the feature’s actual geometry for labelling, e.g. to create curved line based labels inside polygon water bodies
• altering label text (or appearance) for each label drawn from a multi-part input geometry (e.g. labelling each node in a line/polygon with its x/y value)

This proposal partially relates to the issues reported on the QGIS tracker at http://hub.qgis.org/issues/1780 and http://hub.qgis.org/issues/3482

QGIS 2.14 introduced the ability to use a “geometry generator” style while rendering feature symbols, allowing for QGIS’ flexible expression engine to be used to specify geometric transforms (such as smoothing, translating, buffering, etc) to apply to a feature’s geometry prior to the symbol rendering for that feature. Geometry generators only apply to a feature’s symbology and do not affect labelling.

Proposed Solution

The proposed solution to remedy this issue is to implement an expression based “label geometry generator”. Label generators will adopt a similar approach as the symbol geometry generators, and use geometry based expressions to modify the feature’s original geometry prior to labelling. Using the expression engine for geometry modification has proved to be a very flexible approach, and new geometry based functions are continually being added to the engine to increase its power.

The expression based approach will also allow for future expansion (not included in the scope of this
proposal) by allowing access to “secondary” geometry columns in supported data providers (such as PostGIS and spatialite). This would permit manual drawing of specific “labelling geometries” to be used solely for the label engine for specific features.

UI Changes

Label geometry generators will be exposed to users through new options in the label settings GUI. First, the label settings will be re-arranged to move some geometry-specific options from the “Rendering” tab to the “Placement” tab. Specifically, these are:

• the “Label every part of multi-part features” option
• the “Only draw labels which fit completely within feature” option (polygon geometries only)
• the “Suppress labelling of features smaller then” option
• the “Merge connected lines to avoid duplicate labels” option (line geometries only)

A new “modify label geometry” widget will be added to the placement tab. Like the symbology geometry generator configuration widget, this will allow users to specify the expected geometry type for the expression and enter a geometry expression (together with a button to show the full expression builder dialog).

Label geometry generators will only be applied when the corresponding check box is ticked. When ticked, the placement options in the tab will be updated to match the set “Geometry Type”. For instance, setting the type to “Polygon/Multipolygon” will show the placement options for polygons such as perimeter based placements, and setting the type to “Point/Multipoint” would show the point placement options (e.g. offset from point).

Note: there will be no support for mixed geometry type expressions (e.g., returning a centroid point for certain features and a polygons for others). If this setup is desirable then the supported approach would be to use rule-based labelling with two separate rules for the point and polygon based labels.

Supported expressions

Expressions entered in the geometry expression widget will accept the standard syntax, functions and features supported in the QGIS expression engine. Accordingly, some sample expressions which could be used for labelling include:

• buffer($geometry, -5 ): buffer a polygon geometry inward by 5 map units, i.e. prevent labels from being placed within 5 map units of the polygon’s edge
• smooth($geometry): smooths a linestring geometry for more gently curved line labels
• line_interpolate_point($geometry, 3.75): places point labels at a distance of 3.75 map units along a linestring geometry
• nodes_to_points($geometry): places labels at every node in a linestring/polygon geometry
• intersection( $geometry, geom_from_wkt('POLYGON((100 100, 200 100, 200 200, 100 200, 100 100))')): clips a polygon geometry to a rectangular bounds, only allowing labels within this clipped region

The label geometry generator expression will apply before any other label placement option, i.e. the geometry is first calculated and then used for tests such as the “suppress labelling of features smaller than” and “only draw labels which fit completely within feature”.

When using geometry generators for symbology, QGIS allows use of the @geometry_part_count and @geometry_part_num expression variables for retrieving the total number of parts in the current multipart geometry and for retrieving the geometry current part number respectively. These variables can be used along with the geometry_n() function to retrieve properties of the current part of the geometry being rendered. These same expression variables will be made available for use with labelling geometry generators and for all labelling expressions and data defined overrides. By using these variables and functions it will be possible to achieve many advanced labelling approaches, such as labelling each node in a geometry with its corresponding x/y/z or m value, i.e., the combination of:

Label text expression: z(geometry_n($geometry, @geometry_part_num ))
Label geometry expression: nodes_to_points($geometry)

Additional work

As part of this work QGIS’ handling of labelling multipart geometries will be refined. Development of the geometry generators will require major refactoring of multipart labels, so this opportunity will be used to fix some deficiencies in the current code. For instance, currently the option to “label every part of multipart geometries” does not respect any minimum feature size set for labels. This will be addressed as part of this proposal.

One important consideration for rendering multiple label for a single multipart geometry is correct handling of label obstacle geometries. Careful consideration will be made while developing this proposal to ensure that labels for each part of a multipart will correctly avoid being placed over the rendered symbol for a geometry where inappropriate.

Backwards Compatibility

N/A - label geometry generators will not be backwards compatible

Issue Tracking ID(s)

Relates to:
http://hub.qgis.org/issues/1780
http://hub.qgis.org/issues/3482

Votes

(required)

QGIS Enhancement 47: Label connectors

Date October 2015
Author Hugo Mercier
Contact hugo dot mercier at oslandia dot com
Last Edited
Status
Version QGIS 2.14

Summary

This proposal is aiming at adding to the labeling display engine the possibility for the user to define
"connectors" between labels and the features they label.

Current state

The current version of QGIS allows to manually place labels when the output of the automatic placement mechanism is not enough.

It relies on "data defined" properties. Indeed, the X and Y positions of a label as well as its rotation angle can be defined by a data column or any expression.

Some graphical controls allow then to manually manipulate these properties. The state of a label can be changed from being automatically placed to being manually placed (pinning/unpinning). When they are pinned, labels can then be moved or rotated by interaction of the user. Positions and rotations data are then updated in columns of the layer.

Proposed improvements

It may be desirable to have visual "connectors" between the label and the feature it refers to. They are a generalization of "arrows", as it can be found in drawing programs: single arrows, straight lines or curves. Connectors are "attached" to a label: if one of the two endings (either the label or the feature) is moved, the shape of the connector is recomputed and a display refresh is needed.

This QEP is not meant to give an exhaustive list of possible connectors. The idea is to have a generic implementation with the most common types of connectors and everything needed in the code to be able to add new types in the future.

Possible types of connectors:

• a straight line
• two connected straight lines, with an horizontal part and a free part
• three connected straight lines, with two horizontal parts and a vertical part
• a curve with one bending point
• a curve with two bending points

A connector will be defined in two steps :

1. How to place the two ends of the connector, and how to place additional control points
2. What style to apply

Placement of the two ends will then need additional properties:

• position of the ending on the side of the label, with different options
  • simple positioning: up, down, right, left, center, etc.
    • for each possible position, an optional x/y offset may be specified
  • "intelligent" quadrant positioning (see below)
• position of the ending on the side of the feature:
  • for points: it targets the point in itself and an x/y offset may be specified
  • for lines: beginning, middle, center
    • for each possible position, an optional x/y offset may be specified
  • for polygons: top-left, top-right, bottom-left, bottom-right, centroid of the polygon or on the border
    • for each possible position, an optional x/y offset may be specified

Illustration of the "intelligent" quadrant positioning. The positioning on the side of the label depends on the position on the side of the feature. It is chosen in order to minimize the length / complexity of the connector.

Some connectors may be defined by the help of additional control points : for straight lines with angles between them or for Bézier curves.

• No additional point : the connector is defined by a line from one end (the entity) to the other (the label)
• One additional point : this additional point may be :
  • placed freely by the user (a data defined property will be needed)
  • or automatically placed so that the connector is made of a straight line segment and a 45° line segment
  • or automatically placed so that the connector is made of two line segments with straight angles
• Two additional points that can be placed freely by the user with a data defined property

Each connector is then defined by a line made by a certain amount of points. They may be rendered by line symbol layers, such as « simple line » or « arrows ».

New line symbol layers will also be added to render the body of the connector (note that this could also be implemented as options to pre-existing symbol layers) :

• A quadratic Bézier line symbol layer where a Bézier curve is rendered for each 3 consecutive points, the second point forming the control point,
• An arc circle line symbol where the arc of a circle defined by each 3 consecutive points is rendered
• A cubic Bézier line symbol where a Bézier curve is rendered for each 4 consecutive points, the second and thirs points forming the control points

Arrow heads could be rendered by using marker symbol layers with correct options (new predefined « arrows » symbols will be added)

The user may also choose to render the connector with a « filled arrow » symbol layer, as it is done in the « arrows » symbol layer. The « arrows » symbol layer may be extended to support quadratic and cubic Bézier arrows.

Connector must be drawn with respect to the current map canvas rotation. If parts of the connector is supposed to be drawn "straight" (horizontally or vertically), they must be updated when the canvas is rotated

Implementation details

Connectors are drawn to the Qt canvas, using the QPainter API. The existing symbology code will be reused when possible.
In particular, lines of the connector will be drawn thanks to a QgsLineSymbolV2 with properly defined symbol layers (QgsMarkerLineSymbolLayerV2 for instance for head and tails)

The following classes / methods would be modified:

• QgsVectorLayerLabelProvider :
  • extract necessary property values and data-defined values for connectors
  • draw connectors
• QgsLabelFeature: new attributes for connectors

GUI Changes

New labeling properties will be available. They will be splitted into different parts : properties of the two connector ends, number and types of the control points and style.

(UI sketchup, not the final one)

Then the style could be defined by a set of symbol layers :

Each of the property could also be data-defined (not represented here).

From the user perspective, new tools will be available to easily manipulate label connectors. To be consistent with current label manual placement toolboxes, new controls will be added to manually place control points for connector types that need them.

Optionally, it would be nice for the user to be able to edit per-connector properties, i.e. properties that are data-defined via a GUI dialog. This could be achieved by reusing the connector properties dialog and connecting it to one particular feature.

Automatic creation of connectors

A new labeling option will allow to automatically create a label connector if the label is manually placed at some distance from its feature.
Automatic creation of fields for the handling of connector properties would be provided through the use of auxiliary fields (see corresponding QEP), meaning it would be transparent for the user to move a label from its automatic placement and see a connector appear (with default options)

QGIS Enhancement 40: Geometry redesign

Date 2014/10/30
Author Marco Hugentobler
Contact marco at sourcepole.ch
Last Edited 2014/10/30
Status Draft
Version QGIS 2.7
Sponsor Canton of Solothurn, Switzerland

Summary

The mmsql geometry standard describes some features not supported by the current geometry class:

• Curved geometries (circular arcs
• Compound types (compoundcurve, curvepolygon, geometrycollection)
• Z-coordinates
• M-coordinates (e.g. used for linear referencing)

This QEP proposes a redesign of the geometry system to support these features. As a positive side effect, the code should be more flexible and easier maintainable (e.g. possibility to add new geometry types likes splines).

Possible Applications

The following examples come to my mind (surely there are more):

1. In Switzerland, circular arcs and compound types (compoundcurve, curvepolygon) are important in public data like cadastral surveying data . These types are already supported in PostGIS, and work is underway to support it in OGR as well. In order to be able to see and edit this data, it is a logic step to support it in QGIS too.
2. It is common to have 3D data in PostGIS. Currently, depending on the edit method used, it can happen that QGIS converts a geometry to 2D during editing. QGIS should at least preserve the existing z-Values if possible (e.g. if a vertex is moved to a new position). To achieve that, it needs an internal geometry representation that supports z-values. Maybe we will see more 3D plugins if storing z-coordinates is supported by the core library.
3. Linear referencing is an important feature to avoid redundancy for geographic data. Currently, it is done either with views on DB side or, for shapefiles, interpreting z-values as m and then doing the computations in the analysis library (using event layer plugin as GUI). If the geometry model supports m-values, linear referencing can be solved properly and for all datasources which support m-values.

Proposed solution

It is proposed to build a class hierarchy according to the mmsql iso standard with QgsAbstractGeometryV2 as the base class. Working with inheritance, it will be straightforward to support nested types, like compoundcurve or geometry collecion. As the current geometry class is used in so many places in QGIS, it is proposed to keep the interface QgsGeometry and direct the calls to it to a new geometry object. This means that existing code should just continue working while offering the new possibilities.

For editing, it is planned to adapt the 'add feature' tool with the possibility to digitise circular arcs and to adapt the node-tool with the possibility to modify start/end/midpoints of circular arcs.

For geometry analysis and some edit methods, QGIS uses the geos library. As geos does not support circular arcs, curved geometries are segmentised if using one of those methods.

Class design

The UML diagram of the proposed solution contains the following elements:

• QgsGeometry has the same API as the current class. Internally, it holds a pointer to the new geometry class and redirects calls to it. To avoid performance overhead, QgsGeometry uses implicitely sharing.
• QgsAbstractGeometryV2 is the base class of the new geometry classes. Instantiable classes are QgsPointV2, QgsLineStringV2, QgsCircularStringV2,QgsCompoundCurveV2, QgsCurvePolygonV2, QgsPolygonV2, QgsGeometryCollectionV2, QgsMultiPointV2, QgsMultiSurfaceV2, QgsMultiCurveV2
• QgsGeometryEngine / QgsGeos handles analytical operations like buffer/union/intersection, etc. as well as import/export between geometry classes and geos. All this code is separated from the geometry classes now. The API has been further extened to prepare a geometry for better performance in case of repeaded topological operations.
• In QgsAbstractGeometryV2 and subclasses, methods like 'draw()', 'transform()', 'mapToPixel()' and (in future) 'vertices()' are provided. The idea is that code (e.g. for rendering) calls these methods instead of having a switch over the geometry types and pulling out the vertices. Old code (or code for very special rendering operations) can still call asPolygon(), asPolyline() etc. and receives a segmentized geometry in case of curves.

QgsAbstractGeometryV2 interface

• Geometry access: code should, wherever possible, call methods of QgsAbstractGeometryV2 (e.g. call QgsAbstractGeometryV2::transform() instead of retrieving the coordinates and transform them). In cases the content of the geometry really needs to be retrieved, there are the following access methods:
  • QgsPointV2::x(), QgsPointV2::y(), QgsPointV2::z(), QgsPointV2::m() to get the coordinates of a point.
  • QgsAbstractGeometryV2::is3D(), QgsAbstractGeometryV2::isMeasure() to test, if a geometry has z/m values.
  • QgsAbstractGeometryV2::coordinateSequence( QList< QList< QList< QgsPointV2 > > >& coord ) retrieves the vertices for all geometry types
  • QgsCurveV2::points( QList& pt ) retrieves vertices of curves.
  • QgsCurvePolygonV2::exteriorRing() and QgsCurvePolygonV2::exteriorRing() gets the rings for polygons
  • QgsGeometryCollectionV2::geometryN( int i ) accesses the parts of multigeometries
• Geometry construction
  • QgsPointV2( double x, double y, double m, double z )
  • QgsLineStringV2::setPoints( const QList& points ), QgsCircularStringV2::setPoints( const QList& points )
  • QgsCompoundCurveV2::addCurve( QgsCurveV2* c )
  • QgsCurvePolygonV2::setExteriorRing( QgsCurveV2* ring ), QgsCurvePolygonV2::setInteriorRings( QList<QgsCurveV2*> rings )
  • QgsGeometryCollectionV2::addGeometry( QgsAbstractGeometryV2* g )

Performance implications

First tests indicate that the rendering performance can be similar to the current state (2.6). It is planned to do some testing with qgis_bench to avoid significant performance regressions. It will however be impossible to test all possible rendering option.

Test implementation

A test implementation is in the branch https://github.com/mhugent/Quantum-GIS/tree/geometry_mmsql . At the moment, it is incomplete, not fully tested and editing of geometries is completely disabled. It should however already give a hint about the direction of development.

Questions and Answers

Q: Will the segmentising be done automatically so that plugins which also work on geometries but not with geos can also use this solution?

A: Yes, it will be done automatically with QgsGeometry::intersection(), QgsGeometry::asPolyline, etc. The only case where it is not automatically done is if the plugin calls asWkb() and parses the binary itself. In that case, the plugin still has the possibility to convert the geometry to a linear one (QgsGeometry::convertToStraightSegments)

Q: Looking at your branch I don't see any unit testing implemented for this work. Is this something which is planned? I strongly believe that something as fundamental as the geometry classes must be accompanied by a comprehensive test suite.

A: There are some tests in testqgsgeometryimport.cpp. It is indeed planned
to add much, much more tests once the architecture is stable.

Q: Why not reusing the existing classes to make the geometry hierarchy?

A: More flexibility and backward compatibility

Q: 'QgsGeometryEngine' is a more descriptive name than 'QgsVectorTopolgy'

A: Agreed, changed the class name

Q: The methods which are delegated to geos do not have to be in the geometry class interface

A: Agreed. The code is changed such that QgsAbtractGeometryV2 does not have a relation to QgsGeometryEngine

Q: there is a naming clash between QgsPoint and QgsPointV2 - they are meant for completely different purpose, yet QgsPointV2 looks like "improved" QgsPoint which is confusing. Especially, what will happen if we get rid of old QgsGeometry and the V2 suffix will be dropped?

A: The current QgsPoint will be removed and QgsPointV2 becomes QgsPoint

Q: Will the new geometry classes have implicitely sharing?

A: It is not planned. However it can be added to the new classes (needs to be done separately for each one)

Q: May I suggest to use iterator pattern for various methods that currently return (possily nested) containers? I mean methods like
QgsAbstractGeometryV2::coordinateSequence()

A: Added QgsAbstractGeometryV2::nextVertex to iterate over the vertices of a geometry

Q: Regarding the geometry access from QgsFeature - are the geometries still going to be accessed via f.geometry() or will there be a new method to get access directly to QgsAbstractGeometryV2 ?

A: Added QgsFeature.geometryV2()

Q: would it be possible to have a code examples in Python demonstrating the new geometry API? e.g. code that reads features and dumps geometry contents

A: Here are some examples (hope to add more in future):

QgsFeature::geometryV2(). So to dump the geometry content (in python), use code like this:

print feature.geometryV2().asWkt()

To create geometries in Python, do e.g. (for a linestring)

lineCoords = [QgsPointV2(100,100), QgsPointV2( 110, 100 ), QgsPointV2( 110, 110 )]
lineString = QgsLineStringV2()
lineString.setPoints( lineCoords )

A zm-Geometry can be created by having zm points in the coordinate list:

lineCoords = []
lineCoords.append( QgsPointV2( QgsWKBTypes.LineStringZM, 100, 100, 23, 1.0 ) )
lineCoords.append( QgsPointV2( QgsWKBTypes.LineStringZM, 110, 100, 33, 2.0 ) )
lineCoords.append( QgsPointV2( QgsWKBTypes.LineStringZM, 110, 110, 43, 3.0 ) )
lineString = QgsLineStringV2()
lineString.setPoints(lineCoords)

QGIS Enhancement 38: Property framework

Date 2014/11/22
Author Nyall Dawson
Contact nyall dot dawson at gmail dot com
Last Edited 2016/03/19
Status Draft
Version QGIS 3.0

1. Summary

This QEP concerns a system of managing and evaluating properties for an object. Properties are grouped in collections, and can be overridden in a specified order of precedence by storing them in a "stack" of collections. A specific property from a collection or stack is evaluated by passing a QgsExpressionContext to an evaluate function.

This proposal is being driven by a number of factors:

• to avoid the current multiple duplicate code paths involving storage, retrieval and evaluation of data defined properties and to make it easier to add data defined support to more things (eg diagrams) without incurring even more duplicate code. Currently labeling, symbology and composer all have their own methods for handling data defined properties, which makes maintenance of data defined code very difficult.
• to allow creation of other property types besides the current "data defined" (ie bound to field value or expression result) property, eg time based properties for a future in-built animation framework.
• to avoid the complexity of requiring users to write their own expressions to map values to colors, sizes, etc and apply scaling functions to these, and instead expose these to users in an interactive, flexible way. Think Mapbox studio's approach to zoom level styling:

but available for all property types. Eg data defined values can be set to preset ease in/ease out curves, or manually edited curves through an interactive GUI.

• enable the possibility of having live project wide colors. Ie a color palette could be created in the project properties, and color based properties "bound" to these colors. Altering the color would then automatically update every property which was bound to this preset color. This also brings the possibility of "color themes" for maps, eg binding properties to a predefined color types such as "highlights", "background features", etc, and then interactively changing all these color bound properties by applying a color theme to the project.
• to allow a system of inherited and overridden properties. Eg QGIS default label font overridden by a project default font and finally overriden by label font setting. The proposed composer rewrite (layouts work) would use this property inheritance to bind layout item properties to a dynamic template. Changes in the template would be reflected in all linked layouts, but individual items could overwrite the inherited properties as required. Layout item properties could then be set globally (eg, font size), per project (eg font family), via a "master template" and finally individually per layout item.
• the labelling engine has a need for predefined label styles. Label properties could be set globally, per project, via a predefined style, or overriden for a particular layer.

This proposal consists of both a framework for collecting, retrieving and evaluating properties, a GUI based method for transforming properties to colors/sizes/etc, and a GUI widget for controlling them.

2. Proposed Technical Solution

• Create an abstract base class "QgsAbstractProperty". Abstract properties have pure virtual methods for readXml, writeXml, creating and cloning, evaluating the property value against an expression context, and for returning any fields referenced by the property. The base class also implements setting a property as "active" or not.
• Classes derived from QgsAbstractProperty are QgsStaticProperty (a single unchanging value), QgsFieldBasedProperty (returns the value from the corresponding field), QgsExpressionBasedProperty (evaluates an expression and returns the result). In future a QgsTimeBasedProperty could also be added to handle calculation of a feature's "age" from the current map canvas time, in days/weeks/months/etc.
• A class "QgsPropertyCollection" would be created, which stores grouped map of multiple QgsAbstractProperty objects, each referenced by a integer key value. Properties within a collection are referenced by an integer key to avoid the cost of string creation and comparisons which would be required if a string key was used. The intended use case is that a context specific enum is cast to int and used for the key value (eg QgsDiagramLayerSettings::BackgroundColor). Collections have methods for storing/retrieving from XML, retrieving individual properties, returning a list of all fields referenced by active properties from the collection, and evaluating a contained property against a supplied expression context.
• A class "QgsPropertyCollectionStack" which stores an ordered list of QgsPropertyCollections. QgsPropertyCollectionStack has a method for retrieving the value from the first QgsPropertyCollection for which a particular property is active.

Transformers

QgsAbstractProperty objects may also have an optional QgsPropertyTransformer object set. QgsPropertyTransformer is an abstract base class for objects which transform the calculated value of a property. Initial implementations for transformers would include those which map a value into a scaled size (QgsSizeScaleTransformer) or color from a gradient (QgsColorRampTransformer).

The intention with transformers is to separate the calculation of a source value from a property (which may be taken from a static value, field, expression result, time based property, etc) from the calculations required to turn this value into a usable color, size, or numeric result.

GUI Changes

• The existing QgsDataDefinedButton class would be reworked to instead rely on QgsProperties instead of QgsDataDefined objects. In addition to the current options available from the data defined button, the new widget's drop down menu would also show all available QgsPropertyCollections for which the property can be inherited. For instance, for layouts, this would show "Disabled", "From master template", "From project" and "QGIS Default". For labelling, this would show "Disabled", "From style", "From project" and "QGIS Default".

The existing "size assistant" option would be rewritten to use properties and transformers. The dialog will be split into two sections, the top section would show the property's value source (ie, a dropdown with choice of fields and expression entry, and possibly in future a gui for calculating a feature's age from the canvas time property), and spin boxes for the expected min and max value calculated by the source (with a button to auto-calculate these).

The lower half of the dialog would show the transformer settings corresponding to what the property is being used for. Eg, for a property setting a size value, the existing options from the size assistant will be shown (min/max size, scale method, preview frame). For a property corresponding to a color, the transformer settings will include selection of a color ramp, color corresponding to null values, and an option to reverse the color ramp. Numeric properties (eg distances, offset values, etc) would show a simple numeric assistant with min/max output values.

Follow up work would involve adding a checkbox and collapsed group box between the value source section of the assistant and the transformer settings section for "apply curves". Checking and expanding this group would show a photoshop style interactive "curves" graph, which controls how input values are mapped to output values. The curves group would also include a combobox allowing for selection of presets like "ease in", "ease out", etc.

3. Examples

The following examples show how properties would be set for a vector layer's (vl) diagrams.

Setting a static override for a the diagram background color property:

vl.diagramLayerSettings().properties().setProperty(QgsDiagramLayerSettings.BackgroundColor, '#ff0000')

Setting an expression based outline width:

vl.diagramLayerSettings().properties().setProperty(QgsDiagramLayerSettings.OutlineWidth, QgsExpressionBasedProperty('rand(0,3)',True))

Field based outline width:

vl.diagramLayerSettings().properties().setProperty(QgsDiagramLayerSettings.OutlineWidth, QgsFieldBasedProperty('width_field',True))

Here's an example of a transformer used on a property. This will transform the value in the 'passengers' field (with values ranging from 0 to 10000) to an interpolated background color from the 'YlOrRd' ramp:

p=QgsFieldBasedProperty('passengers',True)
p.setTransformer( QgsColorRampTransformer(0, 10000, QgsStyleV2.defaultStyle().colorRamp('YlOrRd') ) )
vl.diagramLayerSettings().properties().setProperty(QgsDiagramLayerSettings.BackgroundColor,p)

4. Deprecations

The property system would deprecate the existing QgsDataDefined, QgsDataDefinedButton, QgsScaleExpression and QgsDataDefinedAssistant classes. Initially it would be implemented for diagrams, symbology, and labeling. Composer would adopt the properties framework when for the future "layouts" rework.

5. Voting History

(required)

QGIS Enhancement 29: Label line supports

Date October 2015
Author Hugo Mercier
Contact hugo dot mercier at oslandia dot com
Last Edited
Status
Version

Summary

A high level of refinement for manual label placement maybe required for some users, when words or even letters of a label have to be placed precisely for example.

We propose to rely on label paths (see QEP on label paths) to implement this feature.

Proposed solution

Here is a map with labels where such a free placement feature would be useful :

For such advanced labeling needs, where the user may want to place every word precisely, we propose to model it thanks to a label path.

This will only be available for labels that have a size expressed in fixed map units. It does not make sense to precisely place letters of a label if letters of this label have a size dependent on the current zoom factor.

This kind of label path is represented by a multi-linestring, called label support. Each linestring of the multi-linestring is a line where part of the text may be placed.

Each segment of a linestring represents a position and angle for the label letters. When one of the linestring "overflows" (no room left for the remaining letters), remaining letters will be placed on the next linestring.

This allows the user to introduce artificial horizontal or vertical spaces.

Since such representation is low-level and hard to handle for the end users, tools will help to create lines where the final label will be drawn for common scenarios (a multi-line paragraph). For more advanced uses, the underlying multi-linestring will be available for modification by the end user.

Core Changes

An new data-defined option to the labeling options, "Multilinestring for the free placement of letters" of type geometry will be added.

The labeling engine will be modified to draw these annotations directly, out of the global optimization. The glyphs will still be considered as obstacles for other automatically placed labels.

A helper function that outputs a multi-linestring of glyph placement for a multi-line paragraph (based on litterature on word wraping algorithms) will be added.

GUI Changes

A new "label support" toolbox will be added, only enabled when labels are pinned, with the following buttons:

• "Use label supports": toggle button. Toggle a label between simple x/y/angle positioning and positioning by line supports. If no data-defined property is set for the corresponding properties of the layer, they will be created in the auxiliary storage (see QEP on auxiliary storage)
  • If the label was positioned by x/y/angle, then a straight horizontal line is created as its initial support.
  • If the label was using a support, the x/y/angle of the first part will be used as x/y/angle for the simple positioning
• "Show label supports": toggle button to display support lines of labels, if available
• "Add part": add a line support at the end of the current chain. This tool will allow to draw a linestring with the mouse.
• "Remove part": remove a line support
• "Reorder part": When a part is clicked, it will allow to move it forward or backward in the list of line supports.
• "Paragraph box": Helper tool. The user will draw a box. A dialog will be shown to specify options of the paragraph (interline size, text direction, etc.). Line supports will be created out of this box.

Further considerations

Other helper functions could be added in the future, like a tool that outputs the medial axis of a polygon as a label path (for the labeling of large streets for instance)

Voting History

QGIS Enhancement 51: Generic Helping System for QGIS

Date YYYY/MM/DD
Author Richard Duivenvoorde
Contact [email protected]
Last Edited 2015/11/06
Status Draft
Version QGIS 2.13

Summary

This QEP describes a plan to create a generic QGIS Help component which is used by all 'help'-needed objects (context-help, expression-help, processing-help, plugin-help, dialog-help buttons etc etc).

GOAL

create a generic QGIS Help component which is used by all 'help'-needed objects (context-help, expression-help, processing-help, plugin-help, dialog-help buttons etc etc).

• help should be easily writable (rich content: images, links etc)
• help should be easily fixable by non tech users
• help should be translatable
• help could be online available
• help could be offline available as downloadable (one language per) zip
• help could open in default browser, or a special (Qwebview-based) help-dialog or in a widget in dialog

IDEA

• make the help CONTENT files part of the documentation project (either a separate project, or just pointing to current help sections)
• so help CONTENT is essentially written as rst files, compiled like the normal docs and pushed to transifex to be translatable
• one config file (JSON, YAML?) (some redirection) containing:
  • key (preferably recognizable, like: processing_algorithme_grass_whatever )
  • uri ( which an be expanded to either a url or a file into a zip (zip://......)
  • way of opening (defaulting to …?)
  • ???
• an simple static api like:
  openHelp(key, ...)
• this help component/module should be able to decide if it should go online to find the document, or find them in a zip which the user can download for a language which just has all docs in a zip (and kept as a zip?)
• the component has some configuration for things like: default way of opening, location of zip, maybe different css?
• the CONTENT rst can be pregenerated from the code, or some structured construct (JSON, YAML)
• to not end up with all small different html/help files it is preferred to group the help together in one big file, which contains anchors (deeplinks), so the url's will contain # like:
  • http://docs/qgis.org/en/site/vectortools.html#thistool or
  • http://docs/qgis.org/en/site/contexthelp.html#thisstuff or
  • http://docs/qgis.org/en/site/expressionhelp.html#addition

TODO

• create this component
• create a configuration
• create stubs or scripts to generate stubs
• put those in github repo for documentation
• write documentation (preferably images into it)
• move all current helping systems to this one

QGIS Enhancement 53 (was 35): WFS provider enhancements

Date 2016/01/25
Author Even Rouault (@rouault)
Contact even.rouault at spatialys.com
Status Draft
Funding Land Information New Zealand
Version QGIS 2.16/3.0+

Summary

The purpose of this enhancement proposal is to address a number of existing issues in the WFS provider (listed in the below "Issue Tracking IDs" section), as well as to extend it to support WFS 1.1 and 2.0 protocols. Changes to the QGIS WFS server side are out of scope of this proposal.

Proposed Solution

Due to the scope of changes described below, a substantial rework/rewrite of the WFS provider will be done.

WFS 1.1.0 support

WFS 1.1 brings support for exposing ( in "DefaultMaxFeatures" constraint of the GetCapabilities response ) the maximum number of features that a server can return in a single GetFeature response (when it has such limit). This is for example implemented by TinyOWS, MapServer or GeoServer. This can be used to detect when not all features intersecting the view are available, and so as to order for a new GetFeature request when zooming in. Currently the threshold to detect truncated responses is hardcoded to 500, which is rather arbitrary. Reading DefaultMaxFeatures will solve this.

The default format for a GetFeature response in WFS 1.1 is GML 3.1. QGIS only supports GML 2 currently. The impact is mainly on the parsing of geometries whose XML representation can be much more diverse than in GML 2. The QgsOgcUtils class has some support for GML 3 parsing but it is quite embryonic. It is proposed to rely on OGR_G_CreateFromGML() function that supports GML 3.1/3.2/3.3. Another important impact of WFS 1.1 is that coordinate axis order is assumed to be the one of the EPSG database, i.e. lat,lon order for geographic SRS (and Y/X for a few projected coordinate systems too). Due to issues in (old) server implementations, similarly to WCS and WMS connection configurations, options "Ignore axis orientation" and "Invert axis orientation" will be proposed to fix problems that might arise.

Not in scope (among other aspects of the WFS 1.1 spec):

• WFS-T 1.1.0

WFS 2.0.0 support / filter improvements

All main features of WFS 1.1.0 also apply to WFS 2.0.0. One interesting new feature of WFS 2.0 is the support for paged GetFeature requests, i.e. the client is able to browse through the features matching a filter with multiple GetFeature requests with explicit STARTINDEX parameter, even if the server implements a limit for the maximum number of features returned. The provider will make use of that capability when advertized by the server in its GetCapabilities response.
Note: some server implementations pre-dating WFS 2.0, like MapServer 6.4, have paging capabilities in WFS 1.0/1.1, but there is no standard way of advertizing it, so this is left out of scope.

WFS 2.0 servers can also advertize capabilities for doing server-side joins (attribute joins, spatial joins, ...). The provider will be upgraded to allow the user to specify such joins. The closest match to then Filter Encoding 2.0 specification used to send WFS 2.0 queries is a subset of SQL. So the "Filter" column in the list of layers will instead be replaced be a "SQL" column allowing both filters to be defined (WHERE clause), as well as joins. The GUI used to define the SQL will have at its top a text zone to enter the statement and in the lower part a "wizard" mode similar to the "SQL query builder" of DB Manager, modified so that join appear explictly.

The supported subset of SQL will allow statements like the following :

SELECT colX [AS alias], other.colY FROM mainLayer [AS]? aliasMainLayer]? [JOIN otherLayer [AS]? aliasOtherLayer]? ON mainLayer.foo = otherLayer.bar | ST_xxx(mainLayer.foo, ]* [WHERE filter_expr]? [ORDER BY [mainLayer.X [ASC|DESC]]+]?

Table(layer), columns and functions proposed will be determined from the capabilities exposed by the server.

There are SQL builders (for example from QGIS expressions), but no SQL statement analyzer in QGIS, so it is proposed to use the one of GDAL/OGR, which is already used by the OGR WFS client driver to support the above mentionned statements. The OGR SQL parser is however not in the public API of OGR (C++ interface only, not guaranteed to be stabled), so we will import the relevant code into the QGIS code base (within a dedicated namespace / function renaming, to avoid symbol clashes).

After adding a new WFS layer to the project, editing the SQL statement will be possible in the layer properties.

Not in scope (among other aspects of the WFS 2.0 spec):

• Support for using/creating/deleting WFS (server-)stored queries
• WFS-T 2.0

WFS-T support

The scope of this work doesn't include an upgrade of WFS-T to support WFS-T 1.1 or 2.0. So only WFS-T 1.0 will be supported. This will require the user to explictly select the WFS 1.0 version (see "Protocol version selection").

Protocol version selection

The version of the protocol chosen by default will be the version advertized by the server in its GetCapabilities response (typically the highest version it supports). The request sent to the server will include the parameter ACCEPTVERSIONS=1.0.0,1.1.0,2.0.0 so that the server advertizes a version that QGIS can support. If for some particular reason (need for WFS-T, defective implementation by the server, or any other compatibility problem), the user wants a particular version to be used, a combobox with choices Automatic/1.0/1.1/2.0 will be added in the Create/Modify WFS connection dialogs.

Profile of GML supported : Simple Feature Level 0

WFS 1.1 and 2.0 servers can return GML encoded features of potentially complex structure, and not a simple flat structure with a single geometry attribute and properties of simple types (integers, real numbers, strings...). This is the Simple Feature Level 0 profile of GML, the one currently supported by QGIS, and which will remain the target with this QEP.
The exception to this rule is when issuing a WFS 2.0 join request : the GetFeature response is a document with a more complex structure, containing fields from the main and joined layers.

Background GetFeature processing

Currently the result of a GetFeature request must be completely downloaded and parsed before the first feature is returned to the calling layer. This is not ideal as it is blocking for the user. It is proposed that the processing of the GetFeature response is done in a worker thread and the features are notified to the main thread iterating over the features. This way the UI will not be blocked when processing large requests (the progress dialog will be made a-modal). Cancelling a download (either through Cancel button, or zoom/pan operations requiring a new request) will cancel both the current request, and the following ones that would have occured with WFS 2.0 paging. (Note: non-GUI based mode will still be possible).

Caching

The WFS provider has 2 modes :

• the default mode, called "caching mode", where a single GetFeature request is sent without explicit bounding box filter for the whole layer.
• another mode, "non-caching", where a GetFeature request is sent with a BBOX parameter for current the bounding box.

The UI to select those modes is somewhat confusing as there is a "Cache filter" column (enabled by default) in the layer list and a general checkbox "Only request features overlapping the current view extent" (disabled by default). It appears that the later, when enabled, overrides the former, but this is confusing for users. It is proposed to remove the "Cache filter" column to only keep the general checkbox "Only request features overlapping the current view extent".

It is also proposed to offer the possibility to change the caching/non caching mode after adding the layer in the project by editing the layer properties.

One point to be aware of is that the terminology "caching" vs "non-caching" is somehow misleading since in both modes currently the features downloaded from the server in the last GetFeature response are kept in memory by the provider (allowing for example faster rendering if the view bounding box does not change in "non-caching" mode). This can become a problem when working with huge layers and servers without feature count limit (servers that can for example satisfy the request in streaming mode), when the amount of available RAM might not be sufficient to store all features. This issue will be also met with WFS 2.0 mode with paging. It is proposed that the downloaded features will be serialized instead into a temporary on-disk file (probably a Spatialite DB). This cache file will not be persisted among sessions. In non-caching mode, its content will be augmented with the new features downloaded from the servers each time a new request is needed (note to self: we will have to store BBOXes and remember those whose requests have completed until end from which ones have been partially processed so as to decide when a new download is needed)

Other changes

The QgsWFSProvider class seems to have had a support for a local file mode (opposed to HTTP/HTTPS WFS connections) working with a .gml file and its accompaying .xsd if present. I did not manage to use it through the QGIS UI, so it looks like dead code. This will be removed with this proposal as a clean-up. Similar functionnality can be obtained by opening the .gml file as a vector layer with the OGR GML driver.

Affected Files

At least:

• src/gui/qgsmanageconnectionsdialog.cpp
• src/core/qgsgml.*
• src/core/qgsogcutils.*
• src/providers/wfs/*

Performance Implications

Discussed in above "Caching" paragraph.

Tests

There are currently no tests in the QGIS tests for the WFS provider. It is intended to add ones, probably using a locally instanciated HTTP server serving predefined XML documents.

Backwards Compatibility

No major issue foreseen at this stage. Existing WFS connections in the configuration will be reused and existing .xml files with exported WFS connections will be imported even if lacking new options. Users that need WFS-T will have to manually select WFS 1.0 version if connecting to a server with WFS 1.1 and/or 2.0 capabilities. Existing projects with WFS layers should still be usable (probably at the expense of not displaying the existing filter due to the change of QGIS expression syntax to SQL)

Issue Tracking IDs

The following issues will be adressed per this proposal:

1. Panning a non-cached WFS layer causes selection to change - https://hub.qgis.org/issues/10106
2. WFS client not requesting new features when not-cached - https://hub.qgis.org/issues/9444
3. New WFS connection option - Max number of features returned - https://hub.qgis.org/issues/9450
4. WFS client implement simple logic for caching and non-caching - https://hub.qgis.org/issues/14121
5. WFS layers time out, even though the timeout is not reached - https://hub.qgis.org/issues/9395
6. $geometry parameter does not expand to layer geometry column name - https://hub.qgis.org/issues/7600
7. Ensure the OGC filter XML expression contains the GML namespace references - https://hub.qgis.org/issues/14119
8. Implement Better Filtering for WFS layers - https://hub.qgis.org/issues/14120 (Could make 6 redundant if implemented as proposed)
9. WFS 2.0 client provider: https://hub.qgis.org/issues/14122
10. WFS non cached: infinite flashing: http://hub.qgis.org/issues/14156

Documents and links of interest

• WFS 2.0.0 specification : http://portal.opengeospatial.org/files/?artifact_id=39967
• GML 3.2.1 specification : http://portal.opengeospatial.org/files/?artifact_id=20509
• Filter Encoding 2.0 spec. : http://portal.opengeospatial.org/files/?artifact_id=39968
• (UMN) MapServer WFS 2.0 implementation: http://mapserver.org/development/rfc/ms-rfc-105.html
• OGR WFS driver: http://gdal.org/drv_wfs.html

Votes

(required)

QGIS Enhancement 58: QGIS Styles, Symbols, and SVG Markers Sharing Repository

Date 2016/04/27

Author Akbar Gumbira (@akbargumbira)

Contact [email protected]

Status Draft

Version QGIS X.X

Summary

In this GSoC project, I will focus on making styles (.qml), symbols definition (.xml), and SVG markers sharing possible through web services that later can be accessed and used in QGIS. The project scope includes:

• Developing a web application to allow QGIS users (through OSGeo LDAP account) to upload their styles, symbols, and SVG’s.
• Integration in QGIS to allow users to fetch those resources and use them.

This project was partly done (styles and symbols sharing) in GSOC 2013, but unfortunately, the work was not merged in to master as some QGIS core developers thought that it was not production-ready, mainly on the server side. For the server side, I have moved the previous GSOC 2013 work to this repository: http://github.com/akbargumbira/QGIS-Sharing and will continue to work there.

GSoC 2013 Work

This is the work done on GSoC 2013. For symbol, user uploads xml file itself and the list of tags. For the styles, user submits a form with name, QML file, description, and tags.

• We can upload multiple symbols from one xml (we can export multiple symbols to one xml). In this case, it will create more than one row to the table.
  

Proposed Solution

Server Side

Authentication and user permissions

The behavior for authentication will follow what we have in Django plugin. User logs in using OSGeo acount.

User Management

• Admin/staff/superuser can:
  • trust a user. Resources that are uploaded by trusted users will not need approval from the admin,
  • untrust a user,
  • block a user. User would not be able to log in to the system (set is_active to False),
  • unblock a user.

Resource Management

• All users can:
  • browse, view, rate, and download public resources,
  • upload resources (after log in),
• Author can:
  • edit and remove their own resources
• Admin/staff/superuser can:
  • Approve and disapprove resources

Resource Classification

• Resource category will be hierarchy
• A resource (style, symbol, SVG marker, etc.) belongs to one or more categories
• User should be able to download a whole branch of the category tree
• User should be able to download a whole set of items sharing the same tag

Model

This section explains about the proposed model on the server side.

Author

Notes:

• Author row will be added after user uploads their first resource (symbol/style/svg). So, for every user, the first time they upload a resource, they will not be trusted and their resources will need to be approved by admin.

Category

Category will be hierarchical using django-mptt module (https://github.com/django-mptt/django-mptt)

Notes:

• Resource can have many categories. Category hierarchy could only be managed by admin.

Resource

This is an abstract model.

Symbol

This model inherits Resource abstract model.

Notes:

• To upload symbol(s), User submit a form with:
  • Name
  • An XML file (that could define multiple symbols)
  • Category (user choose defined category)
  • Tags (users type tags as they want)
  • Description
• One XML file could contain multiple symbols. All the extracted symbols from this XML (that defines multiple symbols) will have the same properties of name, category, tags, and description. Authors later could edit those properties if they want.

Styles

This model inherits Resource abstract model.

Notes:

• To upload style, User submits a form with:
  • Name
  • QML style file
  • Description
  • Category (users choose defined category)
  • Tags (users type tags as they want)

SVG Markers

This model inherits Resource abstract model.

Notes:

• To upload SVG, User submits a form with:
  • Name
  • SVG file
  • Description
  • Category (user chooses defined category)
  • Tags (user types tags as they want)
  • Checkbox is_authorized

Client Side - QGIS Desktop client (python plugin)

In QGIS Desktop, user should be able to:

• browse repository items (symbols, styles etc.) with preview (if possible)
• search/sort items by tag, category, name ...
• download and install selected items (from a search result)
• upload of styles and items

Notes from Mailing List

• ​Anita Graser (@anitagraser) : I think we need something similar to the plugin platform:
  • Anyone can upload but an admin has to check the uploads before they are made public.
  • Trusted users can upload and publish directly.
  • Users can ​vote on resources. (Resources being symbols, styles, or SVGs.)
  • There are tags to classify/search for resources.
• Alessandro Pasotti (@elpaso): The Python plugin repository has proven to be a valuable mean for sharing code and offer new functionalities, symbology is a fundamental part of a GIS system and system for sharing symbols would enhance user experience and improve cooperation between individuals.
  • create production-ready re-usable Django app for sharing symbols
  • create a Python plugin for browsing, uploading and downloading symbols from the repository
  • test and deploy on qgis.org
• Andreas Neumann (@andreasneumann):
  • Check openclipart.org for SVG markers
  • SVG scrubbers: https://github.com/codedread/scour for the uploaded SVG files to remove unused/unnecessary resources. It helps to keep the SVG icon files clean and small.
  • Some minimum quality standards:
    • Contains the necessary SVG parameters that allows replacing of fill, stroke, stroke-width and opacity
    • keywords and titles
    • author contact
    • source (if copied from another source)
    • checkbox where the uploading person confirms that (s)he is either the
    • original author of the symbol or has the rights to share the icon
    • should we propose a standard license on the icons?
• Richard Duivenvoorde (@rduivenvoorde):
  • Categorizing icons based on their themes (e.g oil-business icons)

Further Considerations

• Check Scour for SVG scrubber
• Web Service: REST or RPC?
• Integration in QGIS: with C++ in Style Manager or with Python like Python Plugin Manager?

Backwards Compatibility

User will need to upgrade to the QGIS target version to be able to use this new feature.

Votes

(required)

QGIS Enhancement 71: Save/restore colorramp name for singleband pseudocolor renderer

Date 2016/09/08

Author Alexander Bruy (@alexbruy)

Contact alexander dot bruy at gmail dot com

Maintainer @alexbruy

Version QGIS 3.0

Summary

Currently "Singleband Pseudocolor" renderer does not save name of the color ramp used for classification. So when user worked with multiple rasters and wants to modify classification settings for one of the rasters, he may forgot which color ramp was used for current raster.

Proposed Solution

This can be fixed easily by saving not only color map but also name of the original color ramp used for classification. Name of the color ramp also should be exported to the QML file and restore from it when style applied to raster layer.

Draft implementation can be found at https://github.com/alexbruy/QGIS/tree/save-colorramp

Affected Files

Affected files are:

• src/core/raster/qgsrastershader.cpp
• src/core/raster/qgscolorrampshader.h
• src/core/raster/qgscolorrampshader.cpp
• src/core/raster/qgssinglebandpseudocolorrenderer.cpp
• src/gui/raster/qgssinglebandpseudocolorrendererwidget.cpp

Also colorrampshader element of the QML files will be modified with addition of new attribute containing name of the color ramp.

Backwards Compatibility

Backward compatibility will be kept. New QML files will have additional attribute, if this attribute is not found — default color ramp will be used.

Votes

(required)

QGIS Enhancement 74: QGIS server code refactoring for QGIS 3.0

Date 2016/09/27

Author René-Luc D'HONT (@rldhont)

Contact rldhont at gmail dot com

maintainers @rldhont, @dmarteau

Version QGIS 3.0

Summary

The QGIS 3.0 version is the right time to refactor the QGIS Server code. A part of the QGIS Server code that needs a refactoring is the way Services has been added.

A better way to define Services in QGIS Server would be to add them as plugins like the layer providers. With this architecture, Services will be self declared and stored and could be written in C++ and Python.

Dependencies

Some part of the current QGIS Server code has to be cleaned up:

• the fcgi code to get inputs and to send outputs has to be clearly separated
• the project parsers and singletons qgis/qgis4.0_api#57 qgis/qgis4.0_api#42
  • WMS refactoring. Oslandia is working on it qgis/QGIS#4313
  • WFS refactoring. 3Liz is working on it qgis/QGIS#4344
  • WCS refactoring. Work done by 3Liz qgis/QGIS#4157
• the mapLayerRegistry is a singleton qgis/qgis4.0_api#23
  • fixed in qgis/QGIS@d56a97d
• the project is a singleton qgis/qgis4.0_api#8
  • fixed in qgis/QGIS@57ff36b
• server uses old map renderer qgis/QGIS#3809
  • qgis/QGIS#3886
• still too much dependencies to gui qgis/qgis4.0_api#66

Proposed Solution

QGIS Server Services as plugins like providers

Service provider module

Services provider are dynamically loaded modules. Services may register for one or more
services.

A Service module provide QGS_ServerProvider_Init() C entry point, this entry point return a QgsServiceModule instance:

class QgsServerProviderModule {

    /** Ask provider to register all provided services */
    void registerSelf( QgsServerProviderRegistry* );

    /** Finalize/cleanup module  */
    void unRegisterSelf( QgsServerServiceRegistry* )
};

Service registry

The server provider registry act as a manager for modules
and services:

class QgsServerProviderRegistry {

    /* Module management */
    /* ... */


    /** Retrieve a service by its name
    */
    QgsServerService* getService( const char* );


    /** Register a service by its name 

        This method is intended to  be called by modules for registering
        services. A module may register multiple services.
        The registry gain ownership of services. 
    */ 
    void registerService( const char* name, QgsServerService* );
};

Service

class QgsServerService {

    /** Anything informational */

    /* ... */

    /** Return allowed methods 

        Return flags indicating if method like GET or POST are allowed
    */
    bool allowMethod( QGServerMethodType ); 

    /** Execute a request
        @param request a QgsServerRequest instance
        @param response: a QgsResponse instance
        @return a result that could eventually be mapped to an HTTP response code 
     */ 
    int executeRequest( const QgsServerRequest& request, QGsServerResponse& response );
};

Requests

The request object encapsulate the input request

class QgsServerRequest {

    /** request url */
    const QString& url();

    /** url parameters */
    const QMap<QString, QString>& params();

    /** request header */
    const QMap<QString, QString>&  header();

    /** post data */
    const QByteArray* data();
};

Response

A wrapper around I/O far handling the response

class QgsServerResponse {

    /** Set header entry 
     */
    void setHeader( const QString& key, const QString value );

    /** Write a chunk of data 
    */
    qint64 write(const char * data, qint64 maxSize);

    qint64 write(const char * data);

    /** Return the underlying QIODevice 
     */
    QIODevice& ioDevice();

Example(s)

(optional)

Affected Files

(required if applicable)

Performance Implications

(required if known at design time)

Further Considerations/Improvements

(optional)

Backwards Compatibility

(required)

Issue Tracking ID(s)

(optional)

Votes

(required)

QGIS Enhancement 70: Multi map canvas for QGIS

Date 2016/09/08

Author Alvaro Huarte (@ahuarte47)

Contact ahuarte47 at yahoo dot es

maintainer @ahuarte47

Version QGIS 3.0

Summary

QGIS manages only a map (QgsMapCanvas class) and maptools and plugins work with it reading or editing its properties. The reference to this MapCanvas is provided by the QgisApp singleton instance and it doesn't change upto QGIS closes. Also, the user can't create new extra maps with their own layers (a project only manages a map).

This is partially fullfilled by DockableMirrorMap and other auxiliary plugins.

Proposed Solution

Support this feature in core, QgisApp will allow you to create and remove maps and will change the references to new active map canvas. The application have to provide to maptools and plugins some methods to query and manage the current collection of maps of the application.

The code will try to minimize the implication of this new capability using an automatic mechanism of sinchronization of the connection of slots and signals to new active map canvas.

In my proposed solution, MapTools automatically take care of this since QgsMapTool base class overrides the QObject::connect() and QObject::disconnect() methods to allow this automatic synchronization of references.

See experimental code:
qgis/QGIS#3465

Example(s)

See video demo with results of this experimental branch:
https://vimeo.com/179982647

Affected Files

Only the main tools of QGIS have been reviewed. Therefore there will be more files of other tools or plugins to change.

Performance Implications

This should not have significative implication on performance.

Further Considerations/Improvements

When the project saves non-floating maps, it saves the full window state and geometry of the application.

Currently Qt does not provide a method similar to restoreDockWidget() method to restore as better as possible the state of a specific widget (a widget with a map) without change the full window state of the application. Therefore, the loading of a project with non-floating maps may cause that windows previously closed appear, or visible windows disappear.

Backwards Compatibility

MapTools and plugins where there is currently an implicit assumption that only one canvas exists, must be revised. MapTools and plugins that need to update their own settings when the active canvas changes, must be edited as well.

Issue Tracking ID(s)

(optional)

Votes

(required)

QGIS Enhancement 45: Long term releases

Date 2014/10/03
Author Tim Sutton
Contact [email protected]
Last Edited 2014/10/03
Status Final Draft
Version QGIS 2.X

Summary

This proposal is aimed at providing more stable (as in seldom changing) releases
of QGIS for corporate and institutional users who prefer a slower release
cycle with a more stable version of QGIS that receives bug fix updates
over the life of the release.

Proposed Change

General procedure

It is proposed to adopt the following workflow for the release cycle:

• We will add a new branch 'stable' which will contain the current stable release.
• The existing cadence of 3 releases per annum will be continued
• Each 3rd release will be designated a long term release (LTR), which will be supported for the year that follows.
• The feature / freeze window for the LTR will be an inversion of the normal release, with a 1 month feature addition window and a 3 month feature freeze for documenting the new release and stabilising it.
• On a monthly basis a bug fix release will be made. This will be conditional on there being one or more commits to the release branch within that month period.
• Developers will be invited to backport bug fixes to the stable branch
• Documentation team will only release the documentation once per annum - for the LTR - and spend their time during the year documenting new features as they arrive in git, and then the three month freeze window polishing the documentation to be ready with the release.

Workflow implications

The diagram above illustrates the proposed workflow. The major impact will be on packaging (which is described below)
but there will also be implications on the general git workflow. In particular we will need to maintain more branches
and developers will need to be conscious of which branch fixes etc. need to be ported to.

At any given time there will be three active branches:

• the master branch - this is where all new code should arrive into the code base.
• the release branch - this will be branched from the master branch every four months. The non LTR
  releases will be for giving users early access to new features in a known version of QGIS. Bug fixes will be
  made against the release branch for the duration of their validity. The branch shall be named
  by the major and minor components of the release number e.g. 2.6.
• the long term release (LTR) branch - this will be branched from the community release branch after the first patch level
  release or 1 month, whichever is the soonest. The branch shall be named based on the branch from which
  it is derived with an additional 'LTR' suffix e.g. 2.6LTR. The branch shall exist for 1 year, wherapon the next
  Long Term Release shall be made.

Packaging implications

Unfortunately this rather simple proposal has rather large implications for packaging. 5 types of
packages will be made available as part of our official QGIS packaging effort:

1. Developer nightly packages. These packages will be built on a nightly basis against the master branch.
   They are intended to provide early adopters and testers with easy access to an installer that they can use
   to test upcoming features and new fixes to issues in the development branch.
2. Latest Release nightly packages. These packages will be built on a nightly basis against the latest release
   branch. They are intended to provide testers and folks doing QA with a platform to test the upcoming even numbered
   patch level release against the latest release branch.
3. Latest Release packages. These are release packages made at a fixed 4 month interval. They represent a snapshot
   of the development version with new features and an incremental feature release. The packages are released after a
   freeze period and in terms of stability reflect the second most stable packages we offer (after LTR packages).
4. LTR nightly packages. These are preview packages for the impending LTR patch level release. They are intended
   to provide a test environment for users and testers who wish to verify fixes that have made their way into the LTR
   branch work properly and have not introduced any side effects.
5. LTR packages. These are our 'best work' packages which provide a patch level package on the current Long Term
   Release.

For end-users we will simplify the choice by suggesting they download Long Term Release or the Latest packages.

Further Considerations

Life span of LTR's

The intent of this QEP is to provide an alternative for people who prefer
a slower release cycle. Currently I am only proposing to support each LTR
release for one year and that there should be no concurrent LTR releases. As
such we would only ever need to maintain one LTR release.

Policy for patches

The criteria for creating / backporting fixes to the stable branch should be as
follows:

• The patch should introduce no regressions to the test suite.
• The patch should not alter the API except in cases where the API is
  broken and the patch fixes it.
• The patch should not change the user interface except in cases where
  there is an error in the user interface and the patch fixes it.

.. note:: No new features will be allowed in the LRT after it is released.

Documentation

This QEP serves as the documentation for the LTR procedure, and will be migrated
to the project governance documentation.

Voting History

+1

• Gary
• Marco
• Tim
• Otto
• Paolo
• Anita
• Richard

No vote

• Jürgen

QGIS Enhancement 26: Support Of Virtual Layers

Date 2014/09/30
Author Hugo Mercier
Contact hugo dot mercier at oslandia dot com
Last Edited 2015/11/10
Status Draft
Version QGIS 2.14+
Sponsor MEDDE, France - "Ministry of sustainable development"
Sponsor URL http**//www.developpement-durable.gouv.fr

Summary

This enhancement proposal is about the addition of "virtual layers" to the QGIS core.

A virtual layer is defined to be very close to what is called a "view" in the context of relational database
management systems. It aims at offering a formatted view of pre-existing data originated from one or more layers, usually without copy of data involved.
A view can also bring new data computed on existing data, but without the need to store them explicitly.

Even if it comes from the database world, this feature does not focus on any particular data provider. It is not restricted to data providers that connect to databases.

The list of data representation and manipulation features are growing in QGIS. Such features usually already exists in database engines and are reproduced in QGIS to circumvent the lack of these functionnalities for popular data formats (shapefiles, text-based, etc.).
But supporting advanced database features by coding them into QGIS is probably not a good idea if one wants to keep a light and robust code base.

This is why we propose to embed a database engine. Once existing layers of a QGIS project can be exposed to this database engine, new interesting functionalities can be easily developed. It could also be used to refactor existing QGIS code that deals with database-oriented features.

For now the virtual layer is mainly seen for vector layers, but extension to raster layers could be considered in the future.

Possible use cases

The proposed design is based on these different use cases for vector layers:

• create a dynamic point layer based on X,Y(,Z) coordinates
• allow to filter features based on a SQL query, even for data sources not designed for that (CSV, ODS, etc.), and including spatial operators (Transform, Intersects, etc.)
• allow to join different layers with complex clauses (including spatial joins) and with different types of joins
• give access to attributes that are computed, based on the value of other attributes (this may be somehow redundant with the possibilities offered by virtual fields in expressions)

Virtual layers are designed to be easily usable by the end user as well as by some more advanced users.

Proposed Solution

A first working prototype has been implemented as a C++ plugin. Design and implementation details in this document results from it.

SQLite provides a very useful feature designed to embed its database engine in a third party application: virtual table.
It offers the ability to expose internal data as an SQLite table. Then any operations available on a regular table can also be applied to a virtual table. The implementation can then choose to apply or ignore some of the operations.
A C API (as well as a through a Python library) allows to create such a virtual table mechanism.

It makes a perfect candidate for the implementation of virtual layers in QGIS:

• the library is open source
• Spatialite, as an extension to SQlite is already used as a spatial format, with growing support in GIS applications
• it brings an embeddable powerful SQL engine

This proposal is inspired by the VirtualOGR_ driver for Spatialite that allows to open any OGR-supported format as a virtual table.

.. _virtual table: http://www.sqlite.org/vtab.html
.. _VirtualOGR: https://www.gaia-gis.it/fossil/libspatialite/wiki?name=VirtualOGR
.. _Python API: https://github.com/rogerbinns/apsw

Core changes

A new QgsVectorDataProvider (QgsVirtualLayerProvider) will be developed to handle virtual layers. This makes the changes very isolated and will then impact minimally the current code base.

Provider syntax

A new virtual layer can be created out of existing layers, possibly with a query. Every parameters are passed to the QgsVectorLayer constructor "source" argument as a string representing an URL with
a set of key-value pairs.

To reference a layer, use the key layer. The value is a string with three fields separated by a colon : provider:source[:name].

• The first field is the name of the QGIS provider to use
• The second field is the source string. If it contains special characters, and especially ":", "&", "?" or "=" it must be escaped with the percentage sign (see QUrl.toPercentEncoding)
• The third field is optional and is here to name the layer for reference in an SQL query. If no name is specified, each referenced layer will be named "vtabN" with an incrementing N integer.

A "live" layer already loaded by QGIS can also be referenced by using the key layer_ref. The corresponding value is layer_id[:name].

The geometry key is used to set the geometry column name, type and SRID. Its syntax is column_name[:type:srid]. The type field can be a string (point, linestring, etc.) or a QGis::WkbType.
SRID is an EPSG code.

A parameter allows to ignore any geometry column, resulting in an attribute-only layer : nogeometry

The uid key allows to specify which column must be used as an identifier for each feature. This is not mandatory. If no uid is specified, the underlying provider will autoincrement an integer for
each feature.

The query key allows to use an SQL query to setup the layer. It should also be escaped. Layer references are not strictly necessary. If the query uses names of existing QGIS layers (or their ID),
they will be automatically referenced. Name and type of the geometry column will also be detected.

Serialization

The definition of virtual layers can be written on disk and reopened later on (only if the definition does not include "live" layers).

When the source parameter contains a path and no other parameters, it is opened from disk.

When the source parameter contains a path and other parameters are set, the virtual layer is created as usual and saved into the given path.

SQL syntax

The supported SQL syntax is the SQL of SQLite. The version shipped with QGIS. No recursive CTE for now.

SQLite and Spatialite functions are supported.

The internal SQL parser tries to detect types of columns (and especially the geometry column). "CAST" and geometry casts expressions are supported.

Examples

Minimal syntax to set an SQL query out of an existing layer (Python syntax)

l = QgsVectorLayer( "?query=SELECT a,b FROM tab", "myvlayer", "virtual" )

A join between two shapefiles where the uid and the geometry column are set :

q = QUrl.toPercentEncoding("SELECT a.* FROM shp1 AS a, shp2 AS b WHERE Intersects(a.geometry,b.geometry)")
l = QgsVectorLayer( "?layer=ogr:/data/myshape.shp:shp1&layer=ogr:/data/myshape2.shp:shp2&query=%s&uid=id&geometry=geometry:2:4326" % q, "myjoin", "virtual" )

Minimal syntax to create a virtual layer on a single layer (equivalent to SELECT * FROM)

l = QgsVectorLayer( "?layer=ogr:/data/myfile.shp", "myvlayer", "virtual" )

Creation of a virtual layer and storage in a file :

l = QgsVectorLayer( "/myvlayer.sqlite?query=SELECT * FROM ta, tb", "myvlayer", "virtual" )

Opening of an existing virtual layer:

l = QgsVectorLayer( "/myvlayer.sqlite", "myvlayer", "virtual" )

Spatial index support

When the query uses a field from one of the referenced layer declared as a primary key, features are accessed via a QgsFeatureRequest with a "filterFid". It's then up to the underlying provider to support fast
access by ID.

For spatial indexes, a 'hidden' field named 'search_frame' is created for each virtual table (i.e. each referenced layer of the virtual layer). The bounding box of the given geometry will be used
to restrain the query to a particular region of space. This bounding box is passed to the underlying provider by a QgsFeatureRequest with a "filterRect".

The use of spatial indexes must then be explicit.

For example :

SELECT * FROM pt, poly WHERE pt._search_frame_ = poly.geometry AND Intersects(pt.geometry, poly.geometry)

GUI changes

• A new toolbox / entry in the layer menu will allow to create a new virtual layer
• A new toolbox / entry in the layer menu will allow to open an existing virtual layer
• A new entry in the context menu will allow to create a virtual layer out of the set of layers currently selected in the legend.
• In case the user wants to filter (add a WHERE clause) to a layer with QGIS joins, (s)he will be asked if (s)he wants to first convert its layer to a virtual layer and apply the filter on it.
• A new entry in DBManager will allow to run a query on any loaded layer.

The db manager SQL query window will be used if possible when creating a new virtual layer.

Python Bindings

Creation and use of virtual layers will be available through a new QGIS data provider.
There is no particular additional Python bindings needed.

Test Coverage

Uni tests will be available.
The current prototype implementation includes about 30 test cases.

Performance Implications

Virtual layers are first designed to add new functionalities, hard to obtain without. Speed is not the first priority.

Virtual layers add an intermediate processing. Values (and geometries) will be converted from source layers to the internal SQLite format and then again to QGIS format.

If performances appear to be a problem, a "bypassing" strategy (passing pointers to value rather than converting values) could for instance be investigating in the future.

Further Considerations/Improvements

• Virtual layers are read-only for now. For the simple case (see "Updatable views" here ), implementing feature update and addition should not be complicated. For the more general case where a virtual layer
  can be a join between layers, the concept of "updateable" views found in databases could be ported. It means it would require the user to specify triggers on UPDATE and INSERT.
  Creating a virtual layer with the layer_ref key allows to directly access already loaded QGIS layers (including memory layers). But internally, QgsDataProvider is used to access features, where QgsVectorLayer could be used, allowing to expose joins (and edit buffers)
• Spatial indexes must be set explicitly. The SQL parser code may be used in the future as a base for simple SQL transformations where use of geometry predicates (or bounding boxes operators like && in PostGIS) could be detected and spatial indexes added.
• QgsExpression functions are not supported in an SQL query. Addition of such a feature is planned, since SQLite virtual table mechanism allows to add/overload functions.

Voting History

(required)

QGIS Enhancement 49: Label mask

Date October 2015
Author Hugo Mercier
Contact hugo dot mercier at oslandia dot com
Last Edited
Status
Version

Summary

This QEP is about the integration of a special kind of object which serves as a mask for the labeling engine.

Thanks to symbology options like "inverted polygon fill" and/or "shapeburst fill", it is possible with the current QGIS version to display maps where some parts are "highlighted", because the uninteresting parts have been masked out.

However it may be desirable to also mask labels of the parts without much interest.

Here we can see that some labels are chosen to still be displayed (name of the departments), but other are filtered out by the mask.

Legend items should also be filtered out if they are currently masked.

One last need would be to also consider the mask geometry as an obstacle during label placements. In this case, not only features will be filtered out of the mask geometry, but no label could overlap it.

Proposed solution

The needs are:

• be able to hide labels based on a polygon geometry ("mask")
• be able to consider the mask geometry as an obstacle
• be able to visualize and manipulate this geometry (with a symbology)
• be able to set an option during atlas so that the mask geometry is the atlas polygon
• (option) be able to manually edit the mask geometry

We propose to add a concept of "mask geometry", that is global and unique for the current project.

The mask geometry can then be used by the labeling engine to hide some labels.

Core Changes

A new project variable "@mask_geometry" will be added and will store the geometry of the mask. And then the ability to store geometries in a variable will be added.

A new labeling property will be available for each vector layer to use or ignore the mask geometry as an obstacle for the automatic placement of labels.

GUI Changes

New functions will allow to easily convert a selection from a polygon layer to the mask layer. When multiple polygons are selected, the resulting mask geometry is the geometric union of all the polygon selections.

An option will also allow to make the geometry (the first one, or an union of all the geometries) of a vector layer the mask geometry.

A button will allow to show/hide the mask geometry as a canvas item and will allow the user to edit it with digitizing tools.

New labeling properties will be available for each vector layer allowing to use the mask geometry:

• a check box "Don't show labels inside/outside (with a dropdown) mask"
• a second nested check box enabled when the first is checked : "and consider the mask as an obstacle for labels"

A new toggle button in the legend will allow to filter out items if they are outside the mask.

A shortcut will be added in the application to create a (read only) vector layer from the mask geometry. This could be implemented by a virtual layer where the geometry is the @mask_geometry variable.

Atlas integration

An option will allow to use the mask geometry as the atlas feature. This will be done by overloading the global mask geometry in the atlas expression context and make it change when the atlas feature changes.

Then if there is a "mask" vector layer (through virtual layer) its geometry will then be updated to reflect the current atlas geometry (in the composer). As well as labels that will be masked based on the current atlas geometry (if the corresponding option is enabled)

Voting History

QGIS Enhancement 59 (was 40): Aggregate functions for QGIS expressions

Date 2016/04/29
Author Nyall Dawson (@nyalldawson)
Contact nyall dot dawson at gmail dot com
Version QGIS 2.16
Funding Kanton of Zug, Switzerland

Summary

This proposal covers implementing aggregate functions in the QGIS expression engine, allowing calculation of various statistical and summary values from the fields within a layer.

Aggregates will be implemented in two forms:

1. Aggregate functions which operate on the values from the current layer, including the ability to calculate grouped and relational aggregates.
2. A summary aggregate function which operates on all the values from a different layer.

Form 1: Current layer aggregates

The first set of aggregate functions will operate within the current layer only. They will support both calculation of the aggregate for ALL features within the layer, as well as for matching grouped features, and features from one of the layer's child relations.

The aggregates will be implemented through the addition of multiple aggregate functions, each representing a single aggregate calculation. For instance, “sum”, “count”, “mean”, etc.

1A Calculation of statistics

Numeric aggregates will include all statistics currently calculated by QgsStatisticalSummary (ie, those which are available in the “statistical summary dock”), including:

• Count (count(...))
• Sum (sum(...))
• Mean (mean(...))
• Median (median(...))
• Standard Deviation (stdev(...))
• Max (max(...))
• Min (min(...))
• Range (max - min) (range(...))
• Minority (least common value present) (minority(..))
• Majority (most common value present) (majority(...))
• Variety (count of distinct values) (variety(...))
• First (Q1) and Third (Q3) Quartile and Inter Quartile Range (Q3-Q1) (q1(...), q3(...), and iqr(...))

New classes for calculating string aggregates (QgsStringStatisticalSummary) and date/time aggregates (QgsDateTimeStatisticalSummary) will be created. These will be designed in a reusable manner so that the calculations can be reused by other parts of QGIS code, for instance allowing string and date/time statistics in the statistical summary dock.

QgsStringStatisticalSummary will calculate:

• count (number of non-null strings)
• count distinct strings
• count missing values (null strings)
• concatentation (ie, joining values with a delimiter, much like the Postgres expression array_to_string(array_agg(field), ',')

QgsDateTimeStatisticalSummary will calculate:

• min date/time
• max date/time
• range between min and max date/times

Both QgsStringStatisticalSummary and QgsDateTimeStatisticalSummary will have 100% unit test coverage.

1B Aggregate function syntax

Each of the aggregate functions will accept the same parameters:

• A sub-expression for the raw values used for calculating the aggregate. This sub-expression could be
  either a simple field reference or a calculation in itself
• An optional “group by” sub-expression. If specified, the aggregate will be calculated using only features with the same group by value as the current feature.
• An optional "filter" sub-expression. If specified, the aggregate will be calculated using only features which pass the filter criteria.

1C Aggregate function examples

• sum("passengers"): returns the sum of the “passengers” field from all features in the layer
• sum("passengers" / 7): returns the sum of the calculation “passengers” / 7 for all features in the
  layer
• sum("passengers", "town" ) ( or sum("passengers", group_by := "town" ) using named parameters) : returns the sum of the “passengers” field from all features in the layer with the same “town” value as the current feature.
• sum("passengers", left("town",1 )) ( or sum("passengers", group_by := left("town",1 )) : returns the sum of the “passengers” field from all features in the layer with the same first character in the “town” field as the current feature.
• sum("passengers",,"state"='vic' or "class" > 3) ( or sum("passengers", filter := "state"='vic' or "class" > 3) : returns the sum of the “passengers” field from all features in the layer where "state"='vic' or "class">3.

1D Caching of values

Aggregate calculation will be cached in short-term storage (within the passed QgsExpressionContext), so for instance the aggregate will only need to be calculated once for each unique group-by value for a single map render if used within data defined symbology, rather than once for every feature being rendered.

1E Relational aggregates

The aggregate functions will be accompanied by a specific aggregate_relation function, to simplify calculation of aggregates for related features. When using aggregate_relation the aggregate will be calculated using only the related child features from the matching layer relation.

Examples:

• aggregate_relation( 'my_relation', 'mean', "field_from_related_table" ) ( or aggregate_relation( relation:='my_relation', calculation := 'mean', expression := "field_from_related_table" ): returns the mean value of the “field_from_related_table” field for all matching child features using the 'my_relation' relation from the layer.
• aggregate_relation( 'my_relation', 'sum', "field_from_related_table" ) ( or aggregate_relation( relation:='my_relation', calculation := 'sum', expression := "field_from_related_table" ): returns the sum of the “field_from_related_table” field for all matching child features using the 'my_relation' relation from the layer.

Note: when using aggregate_relation the calculation sub-expression will ONLY be allowed to reference fields from the child layer.

Form 2: Summary aggregate function

The second aggregate function operates on all the values from a different layer, returning a single summary value. This will be implemented by the addition of a new aggregate function. The syntax of the aggregate function will be:

aggregate('layer name or id', 'aggregate_type', expression to aggregate, optional filter expression)

• The first parameter will be either the name or ID of a layer within the project from which to calculate the aggregate value.
• The 'aggregate_type' parameter will be a string representing the aggregate calculation to perform, eg 'sum', 'mean', 'count', etc.
• Both 'expression to aggregate' and the 'optional filter expression' are sub-expressions which reference the fields from the specified layer. The first represents the raw values to use for the aggregate calculation, the second an optional filter such that only features from the layer where the filter is true will be included in the aggregate calculation. Filters will take advantage of expression compilation to help speed up the calculation on supported providers. Note that both the expression to aggregate and the filter expression will ONLY be able to reference fields from the foreign table.

2B Examples

• aggregate('rail_stations','sum',"passengers") ( or aggregate( layer:='rail_stations',calculation:='sum',expression:="passengers")): would sum up all values from the “passengers” field in the 'rail_stations' layer
• aggregate('rail_stations','sum', "passengers"/7) ( or aggregate(layer:='rail_stations',calculation:='sum', expression:="passengers"/7)`) : would sum up a daily average of “passengers” by dividing the “passengers” field by 7 before summing the values
• aggregate('rail_stations','sum',"passengers","class">3) ( or aggregate(layer:='rail_stations',calculation:='sum',expression:="passengers",filter:="class">3)): would sum up all values from the “passengers” field from features where the “class” attribute is greater than 3 only

2C Caching

Aggregate calculation will be cached in short-term storage (within the QgsExpressionContext), so for instance the aggregate will only need to be calculated once for each map render if used within data defined symbology rather than once for every feature being rendered. The calculation will be designed to utilised expression compilation where available, so that if possible the calculation of the aggregate will be delegated to the provider.

Relation to virtual layers

While it is acknowledged that there is also a need for calculation of aggregates within virtual layers, expressions are prevalent throughout all areas of QGIS (data defined symbology, labeling, field calculator, reporting using atlas based compositions, etc) and there is a defined need for calculation of aggregates within expressions.

Accordingly this QEP covers only the inclusion of aggregates within the expression engine and modifications to the virtual layer feature will be out of scope.

Backwards Compatibility

N/A

Votes

(required)

QGIS Enhancement 50: Support for Enhanced Layer Metadata Management

Date 2015/03/26
Author Tom Kralidis
Contact tomkralidis at gmail dot com
Last Edited 2015/03/26
Status Draft
Version QGIS 2.10

Summary

This QEP aims to provide enhanced metadata management within the Layer Properties dialog, providing the following benefits:

• fulsome metadata representation of QGIS layer metadata at the core (no duplications or QGIS plugins)
• streamlining downstream plugins to operate against QGIS layer metadata at its core to provide value-added properties/metadata profiles, validation, etc.
• easier publishing (via MetaSearch)
• enhanced discovery (better metadata is more easily discovered)

Overview

QIGS by default provides a Layer Properties Metadata tab which allows a user to enter various values (and select from various select boxes) under various sections like Description, Attribution, MetadataUrl, LegendUrl. The Properties section is autogenerated from the layer and is ready only. These values are stored in the QGIS project file within the <maplayer> element.

Currently, there are no mechanisms to:

• save values in Metadata tab as a standalone standards-based (ISO, Dublin Core) metadata document (XML)
• interact with other QGIS plugins such as MetaSearch or Metatools

This leaves the QGIS layer metadata management story fragmented and spread across many internal mechanisms.

It is proposed to improve layer metadata management at the core of QGIS by enhancing the Layer Properties Metadata tab with a core QGIS metadata model, thus allowing for better metadata export, publishing, and use by downstream plugins seamlessly, efficiently, and in an authoritative manner.

This QEP will support an overall workflow per below:

manage Layer Metadata (QGIS core) -> publish/update/delete Layer Metadata (MetaSearch) to CSW

Proposed [Technical] Solution | Change

The Layer Properties Metadata tab will be reviewed to assess the following change improvements:

• design (tabs vs. tre)
• elements review

Implementation Details

The UI file will be reviewed and modified.

Changes to the metadata model will be applied to the QGIS project file.

Affected Files
...............

• https://github.com/qgis/QGIS/blob/master/src/ui/qgsvectorlayerpropertiesbase.ui
• https://github.com/qgis/QGIS/blob/master/src/ui/qgsrasterlayerpropertiesbase.ui
• https://github.com/qgis/QGIS/blob/master/src/app/qgsvectorlayerproperties.cpp
• https://github.com/qgis/QGIS/blob/master/src/app/qgsrasterlayerproperties.cpp
• https://github.com/qgis/QGIS/blob/master/src/mapserver/qgswfsprojectparser.cpp
• https://github.com/qgis/QGIS/blob/master/src/core/qgsmaplayer.cpp
• https://github.com/qgis/QGIS/blob/master/python/core/qgsmaplayer.sip
• https://github.com/qgis/QGIS/blob/master/src/core/qgsmaplayer.h

Test Coverage

(required for technical solutions/changes if applicable)

Performance Implications

No performance implications are expected at this time given the lightweight nature of this QEP to the codebase.

Further Considerations/Improvements

This QEP could be extended by adding deeper layer metadata support by providing direct ISO or Dublin Core support/validation.

Given the below workflow:

manage Layer Metadata (QGIS core) -> publish/update/delete Layer Metadata (MetaSearch) to CSW

QGIS MapServer could be enhanced to link directly to the metadata url generated by publishing to a CSW.

As well, a QGIS-pycsw module could provide a CSW service much like QGIS MapServer, in which case layer publishing to MapServer could also include metadata publishing inline.

Restrictions

Backwards Compatibility

Given that the layer metadata model will be enhanced and not changed, there will be little impact on backwards compatibility. There will be a UI design/change which will not break functionality.

Documentation

Documentation will be enhanced accordingly in http://docs.qgis.org/2.2/en/docs/user_manual/working_with_vector/vector_properties.html#metadata-menu

Issue Tracking ID(s)

(required)

References

(optional)

Miscellaneous

(optional)

Voting History

(required)

QGIS Enhancement 76: Project and Layer Registry Refactoring

Date 2016/10/24

Author Martin Dobias (@wonder-sk)

Contact wonder dot sk at gmail dot com

maintainer @wonder-sk

Version QGIS 3.0

Summary

This proposal aims to clean up the code responsible for handling of projects and layers associated with projects. Here is a brief list of the benefits that the refactoring should introduce:

• move QgsMapLayerRegistry functionality into QgsProject
• removal of QgsServerProjectParser (in future)
• allow opening multiple projects at the same time
• easier to use API (not using layer IDs)
• easier to make unit tests
• fix issues with embedded projects and conflicting layer IDs
• allow lazy loading of layers (?)

Merge QgsProject and QgsMapLayerRegistry

These two classes are dependent on each other: QgsProject writes all layers from layer registry when a project file is written, and it inserts all layers to layer registry when a project file is read (after clearing the registry first). It makes sense therefore to merge these two into one class to make things simpler. Also, one singleton less to deal with.

Support for Multiple Projects

It is currently (v2.x) not possible to have multiple project opened in QGIS environment because QgsProject is a singleton. This leads to various problems: QGIS Server implements its own QgsServerProjectParser for loading of projects, which naturally leads to incompatibilities between desktop app and the server. Another case is the use of embedded projects, where parsing of project files is again done independently from QgsProject. Having a chance to have multiple projects loaded at the same time would enable further functionality in the future, such as opening a project in browser dock and loading some layers from it to the current project.

The idea is to still have a static instance of QgsProject, but also to enable further instances to be created in order to load or save other projects.

References to Layers (and Other Objects)

One of the greatest issues with having multiple projects loaded in QGIS was the fact that layers from different projects may have used the same layer IDs internally, something that the map layer registry was not designed for.

We solve this issue by introducing a concept of references to arbitrary QObject-based objects. References are simple 128-bit universally unique identifiers (UUID) generated by a pseudo-random number generator. A mapping between UUIDs and actual objects is done using a global hashtable, to allow resolution of QObject instances from UUIDs.

References therefore can substitute layer IDs and they are more generic at the same time, as they can be used for lookup of any QObject-based object within QGIS.

References can be thought of as weak pointers to QObject-based classes. Why not to use std::weak_ptr or QWeakPointer instead? We would need to start using shared pointers for QObjects and that does not feel right from various reasons: QObject classes already have their parent-child memory management, all APIs use raw pointers to QObject and Python integration would get only more difficult (with Python's own memory management).

An important feature of references is that we can find out when a QObject is deleted (because a signal gets emitted), so it is possible to invalidate references to deleted objects. We may also store associated reference UUID in QObject's object name (or in a dynamic property).

The usage is simple: whenever a piece of code needs a reference, it calls QgsReference::ref(object) which will return QgsReference object that wraps assigned UUID. Later, when the object is needed, a call to QgsReference::resolve(ref) will return pointer to the object or null if it does not exist.

There is no need to register objects somehow before they can be used as references - they are automatically registered first time the reference is requested. We do not want explicit registration like with QgsMapLayerRegistry::addMapLayer() anymore because it sucks: it always caused confusion as to when layers need to be registered (e.g. required for rendering or for usage in layer trees, but not required to fetch features or to modify layer data).

It expected that code that previously used layer IDs and QgsMapLayerRegistry to resolve IDs to map layers will switch to storing QgsReference objects and use QgsReference::resolve(ref) whenever needed. It would be good idea to update public APIs to work with actual objects even if they only store references to those. For example, to keep track of map layers in QgsMapSettings class, we should have setLayers(QList<QgsMapLayer*>) and QList<QgsMapLayer*> layers() methods and internally store the list of layers as QList<QgsReference>.

To be decided: how to handle thread safety. We could make handling of references thread-safe, but there is a problem that using QObject that belongs to main thread in a worker thread will never be safe. So maybe better to explicitly forbid adding/resolving references in worker threads. Or have a separate hashtable of references for each thread.

Here is a sketch of the QgsReference class:

class QgsReference
{
public:

  //! returns ref for the object if it has been registered already - otherwise it will first
  //! generate UUID for the object, set it to object's name, add to internal hash and start
  //! listening to destroyed() signal to be able to detect when the object is deleted
  static QgsReference ref( QObject* obj );

  //! resolve object from a reference - resolves to a valid object or null if does not exist
  //! or was deleted already
  static QObject* resolve( const QgsReference& ref );

  //! constructs invalid reference
  QgsReference();

private:
  //! constructs a valid reference
  QgsReference( const QUuid& uuid );

  QUuid mId;

  static QHash<QUuid, QObject*> mRefs;  // + mutex for static methods
};

For QgsReference::resolve we could have a special variant to return objects casted to QgsMapLayer (or a general template variant).

Loading and Saving

OK so we have references to objects, but does that fit into loading and saving of projects? References cannot be stored in project files or layer style files because they are only valid during runtime of QGIS. We need a means to read files where some components reference objects from other components and also a means to write files with references. This will be done with a two-phase reading and writing. When writing data, our references with UUIDs will get a string identifier that will be valid within the scope of the file. Afterwards when reading data, the string identifiers will be kept and once everything is read, they will be resolved into true objects. This mechanism can be used not only for reading and writing of project files, but also any other documents where references may be used.

When writing a document:

• step 1: for each component, register any objects that may be referenced from other components into a context object. The context object basically keeps a mapping from QObject to QString identifiers.
• step 2: for each component, write the data (e.g. project XML file) and whenever a reference needs to be written, use the context to get the string identifier (it must have been registered in the step 1)

When reading a document:

• step 1: for each component, read the data (e.g. project XML file) and whenever a field that references other object is encountered, register its string identifier and get a reference. This reference object will always point to a temporary placeholder QObject, so any operations involving such object should be deferred. Components that "own" objects that can be referenced register those to the context object. The context here has a mapping of created objects (from QString to QObject) and a mapping of placeholders (from QString to QgsReference).
• step 2: now that all components have been read, we take all the placeholder objects and replace them by real ones. Finally all components are notified that the references have been resolved and they can finish initialization if it involves manipulation of referenced objects.

This is how an interface could look like to support loading/saving documents with references. Please note that it is not specific to usage with project files:

class QgsSerializable
{
  // read support

  class ReadContext
  {
    //! stores real objects to be used at a later time to resolve placeholders to real objects
    void addReferenceForId( QString id, QgsReference ref );
    //! references returned by this method are refs to placeholders that will be resolved once the reading has finished
    QgsReference referenceForId( QString id );
  };

  //! called to read
  virtual bool read(xml, context) = 0;
  //! called after all serializables have been read
  virtual void referencesResolved(context) {}

  // write support

  class WriteContext
  {
    // registers a reference that can be used later for writing
    void addReferenceForId( QString id, QgsReference ref );
    //! returns real objects previously registered in addReferenceForId()
    QgsReference referenceForId( QString id );
  };

  //! called before write() of any serializable
  virtual void registerReferences(context);
  //! called to write
  virtual bool write(xml, context) = 0;
}

Improve Project API

QgsProject class is one of the oldest in the QGIS API that has survived long without any significant changes. There are some bits of API that should be updated to make it more modern, for example to make API for reading/writing of properties more like QSettings and some other tweaks to make it easier to use.

One of the changes will be in improving reading and writing of projects. Currently, other components read or write data from/to project file be connecting to its readProject() and writeProject() signals. This would be replaced by a more object oriented approach:

• a new interface class QgsProjectComponent will be introduced with read/write methods (TODO: subclass of QgsSerializable ?)
• components would implement QgsProjectComponent and would be registered to a particular QgsProject instance
• read/write methods of project components will be called to allow reading/writing of references

Ownership of Layers

It is suggested that map layers are made children of projects they belong to (using QObject parent-child relationship). The QObject parent-child relationship makes sure that one layer belongs at maximum to one project and potentially allows access from map layer to its project. This can be extended to embedded projects, where QgsProject instances would have another QgsProject as their parent.

Backwards Compatibility

The proposal breaks API compatibility by changing QgsProject API and by changing how referencing of layers works. This is however not a problem as the proposal is aiming for QGIS 3.0.

Issue Tracking ID(s)

qgis/qgis4.0_api#8
qgis/qgis4.0_api#16
qgis/qgis4.0_api#23
qgis/qgis4.0_api#57

Votes

(required)

QGIS Enhancement 39: Layout rework (composer v2)

Date 2014/11/16
Author Nyall Dawson
Contact nyall dot dawson at gmail dot com
Last Edited 2014/11/16
Status Draft
Version QGIS 3.0
Sponsor TBC

.. note::

See :ref:`QEP 1 <qep1>` for description of QEP process.

1. Summary

QGIS' current composer functionality has a number of significant design limitations.
It is proposed that the composer functionality be rebuilt to overcome these limitations,
and to create a better foundation for this feature moving forward.

2. Summary of current issues/limitations

The composer functionality suffers from a substational number of design and API limitations.
Some of these include:

• Composer logic is tied up across the entire codebase, with logic being spread across core, gui and app.
  This makes features fragile, makes bug fixing difficult, and makes it very difficult for plugins to manipulate
  and export compositions without reimplementing large blocks of QGIS code within the plugin.
• There is large amounts of item-specific logic and handling scattered through QgsComposition, QgsComposerView and
  QgsComposer. This makes it impossible to have features like plugin generated item types, and makes maintenance difficult.
• Everything is coded to expect measurements and sizes in mm. It's not possible to implement other units (such as pixels
  or inches) without breaking api and resorting to a lot of hacks. (see #7367, #7959)
• Compositions which consist of mixed page sizes and orientation require an API and significant refactoring
  to implement.
• Undo/redo support is extremely fragile, and leads to hard crashes. This is impossible to fix without
  a large rework of how items are handled within compositions. (see #11371)
• QgsComposition should not require a QgsMapSettings/QgsMapRenderer. This should instead be set individually for map items.
  Doing so would pave the way for features such as reprojection support for individual map items.
• Items currently utilise the paint method for non-rendering purposes, such as setting a minimum or fixed size for the item. This
  prevents QGIS from utilising the existing caching methods for QGraphicsItems, leading to slower operation of the composer.
• The composition code is full of deprecated methods and legacy api. Over time the requirements of compositions
  have changed significantly, and the code is now quite messy and difficult to maintain. Moving forward, this will only get worse
  if the code is not refactored substationally.

3. Proposed Solution

The composer will be rebuilt to address these design limitations.

4. Implementation Details

New classes will be created:

• QgsLayoutUnit: base class for layout units, including page units such as millimetres and inches, and screen units (pixels)
• QgsLayoutMeasurement, QgsLayoutSize, QgsLayoutPoint: classes for single dimension measurements, sizes and points respectively.
  These classes will store the unit type alongside the measurement.
• QgsLayoutMeasurementConverter: handles conversion between different layout units. For conversion between screen and print
  units, QgsLayoutMeasurementConverter will utilise a dpi (dots per inch) property.
• QgsLayout: this will form the base class for all layouts, including both print compositions and report sections. Notable differences
  from QgsComposition include:
  • QgsLayout does not require a QgsMapSettings object.
  • Items removed from a layout are immediately deleted. Undo/redo functionality will work by storing/restoring an item's xml content,
    rather than a pointer to the item itself.
  • All export and item handling logic will be moved from QgsComposerView and QgsComposer to QgsLayout, with the exception of
    atlas export logic (which will be moved to QgsLayoutAtlas)
• QgsPrintLayout (or "QgsCompositionV2" - see note below): inherits from QgsLayout, with notable differences from QgsComposition including:
  • QgsPrintLayout will maintain a QgsPageCollection, which can consist of multiple pages with different size and orientation
• QgsLayoutContext: will store rendering flags (such as whether antialiasing should be enabled for the layout (#9281) ). QgsLayoutContext
  will also store pointers to the current feature and associated vector layer for a layout. For print layouts setting the current
  feature and layer will be handled by the atlas.
• QgsLayoutObject: base class for layout objects which can have data defined properties. This includes both layout items,
  and components of items (for instance, an individual map grid will be a QgsLayoutObject, to allow grid properties to be data defined).
• QgsLayoutItem: base class for layout items. Notable differences from QgsComposerItem include:
  • items are not expected to override the paint method, but instead to implement a pure virtual method "draw". All non-drawing
    code (such as item resizing) will be seperated from the painting code. New methods will be added for setting a minimum or
    fixed size for the item. New methods will be added "attemptResize" and "attemptMove", which will form a single code path for
    setting an item's size or position which respects overrides such as the item's reference point, minimum size, data defined overrides,
    etc.
  • item's will enable Qt's graphics scene caching via a "setCacheMode( QGraphicsItem::DeviceCoordinateCache )" call
  • methods and properties specific to rectangular items will be removed
• QgsLayoutRectItem: inherits from QgsLayoutItem, and adds methods specific to rectangular items, such as frame and background settings.
• QgsLayoutItemRegistry: singleton which handles registration of new items types (eg, plugin provided item types), also functions
  as a factory for creating new items
• QgsPageSizeRegistry: singleton which maintains a registry of known page sizes

5. Reporting framework

This proposed rebuild is also being driven by the planned work on adding a reporting framework into QGIS. This will be the subject
of an additional QEP.

6. Naming

Depending on the outcome of QEP #7 - Rename Composer, the reworked print layout class will either be QgsPrintLayout or QgsCompositionV2.

7. Python Bindings

Will be updated as required.

8. Affected Files

This work will be implemented in parallel to the current composer code. All work will consist of new classes, and the existing composer
code and API will remain until PSC agree to switch over to the new layout code.

9. Test Coverage

Since this will be implemented as a ground-up rebuild, we will aim for 100% unit test coverage for non GUI code.

10. Backwards Compatibility

Layouts will not be backwards compatible with older QGIS versions. Compositions will be automatically upgraded to layouts
on project load.

11. Voting History

(required)

QGIS Enhancement 34: Improve plot/graphs creation

:Date: 2015/03/19
:Author: Matteo Ghetta
:Contact: [email protected]
:Status: Draft
:Version: QGIS 3.0

. Summary

Right now the creation of high level plots and graphs is limited. Statist plugin and some Processing tools
(QGIS geoalgorithms, R scripts, shell scrips..) are able to produce some usable plot, but not high level ones.

. Proposed Change

Thanks to Processing one can add R scripts in order to produce high level plots, but it is worth to use some core
python library to create those plots (matplotlib, numpy, scipy...).

This enhancement proposal is for adding and additional menu (also embed it in Processing) so the user can create
high level plots with the attribute table fields.

Plots options:

• standard plots (histogram, scatterplots, pie charts, dnsity plots...)
• add trend lines
• add value lines based on statistical computation of the attribute fields (median, mean, standard deviation...)
• plot custiomization (color, size, title, axes name)
• axes options (logaritm, flip axes..)
• interaction plot - layer (e.g. click on the point of the scatterplot and see the point on the map)
• plot exportation (jpeg, png, svg...)

. Further Considerations/Improvements

It is also worth to include the possibility to visualize the plots in the print layout.

QGIS Enhancement: Processing UI refactoring

Date 2016/06/17

Author Arnaud Morvan (@arnaud-morvan)

Contact arnaud dot morvan at camptocamp dot com

maintainer @arnaud-morvan

Version QGIS 3.0

Summary

• Add a metadata attribute in parameters to allow using custom widgets in dialogs.
• Move widget initialization code from parameters panels to widget wrappers.

To discuss :

• Emit signals from parameters when value change.

Current situation

Actually in processing all parameter's widgets are hardcoded in
ParametersPanel, depending on parameter types. This is not flexible to
create custom parameters and related panels.

On top of this, we have a lot of code duplicated between:

• Standard algorithm dialog/panel
• Batch dialog/panel
• Modeler dialog/panel

Proposed Solution

A more modular vision could be to create a widget wrapper class for each parameter type.
ParametersPanel would only have to iterate over parameters and use appropriate wrapper to create the widget, get and set the value.

Using this, it could be easy to create new parameter types and related widget wrappers
without creating a custom dialog and panel class for each case.

Proposed design:

• Parameters will have a new metadata attribute.
• metadata attribute will contains a widget_wrapper property to choice the widget class to display in dialogs and optionally some extra options.
• widget_wrapper property will contain a string with the widget wrapper class path for dynamic import in dialogs (as we do not want to import ui classes in core ones).
• Each existing parameter class will have a default widget_wrapper value, to keep compatibility with existing algorithms.
• Parameter's constructors would allow to override the default metadata values, including the widget_wrapper one, in GeoAlgorithm.defineCharacteristics.

For a working example see : qgis/QGIS#3449

Benefits :

• We could use custom widgets for specific purpose without creating new parameter classes or altering the common ui classes (ParametersPanel, BatchDialog, ModelerDialog).
• Each processing plugin could provide it's own parameters types and related widgets.

Example(s)

For example, in postgis related algorithms, it could be possible to create custom widgets, capable of database introspection using existing GeoDB class to propose drop down lists of schemas and tables or primary keys and geometry column detection, without altering existing parameter types.

Here is an example of ParameterString class definition:

class ParameterBoolean(Parameter):

    default_metadata = {
        'widget_wrapper': 'processing.gui.widget_wrappers.BooleanWidgetWrapper'
    }

    def __init__(self, name='', description='', default=None, multiline=False,
                optional=False, evaluateExpressions=False, metadata={}):

Here is the widget wrapper class prototype:

class WidgetWrapper():

    def __init__(self, param, dialog=DIALOG_STANDARD, extra_values=[])

    def createWidget(self)

    def setValue(self, value)

    def value(self)

And a custom parameter instantiation, for a postgis table entry :

self.addParameter(ParameterString(self.TABLE,
                                  self.tr('table name'),
                                  'public',
                                  optional=True,
                                  metadata={
                                      'widget_wrapper': {
                                          class: 'processing.gui.PostgisTableWidgetWrapper',
                                          schema_parameter: self.SCHEMA
                                      }
                                  }))

Affected Files

QGIS/python/plugins/processing/core/parameters.py
QGIS/python/plugins/processing/gui/*

Performance Implications

This should not have significative implication on performance.

Further Considerations/Improvements

Adding a parameter's valueChanged signal is an option. Actually, processing dialogs update parameters value on dialog acceptation. To get a parameter signal when value change, we need that each widget update it's parameter's value in real time.

Here is a view of what we could get with this new strategy :

Dependant fields functionnality

Currently we have the possibility to give, in ParameterField, the parent layer parameter. Actually, the fields list update is hard coded in ParametersPanel.

In field widget, we could get the parent parameter related widget and bind the parent widget signal to the fields list update slot.

If we add the valueChanged signal on parameters, we could bind the parent parameter signal to the fields list update slot.

ParameterVector QgsVectorLayer cache

In ParameterVector class we could think to a getLayer method, responsible to create a QgsVectorLayer instance when needed (when value is a source uri, not a layer instance). In this case, even with a shapefile given by file path, we could get the fields list.

List of output fields

If we add the valueChanged signal on parameters, we can also think to list of output fields getter in GeoAlgorithm class.

Backwards Compatibility

Adding the metadata attribute with widget property should not break any compatibility.

Porting widget's initialization code from ParametersPanel to parameter's widgets could break custom dialogs depending on existing widgets.

Nothing here should break compatibility with existing algorithms, models or scripts.

Issue Tracking ID(s)

(optional)

Votes

(required)

QGIS Enhancement 77: Require use of new style Qt connections for newly added signals/slots

Date 2016/10/28

Author Nyall Dawson (@nyalldawson)

Contact nyall dot dawson at gmail dot com

Version QGIS 3.0

Summary

Many bugs in QGIS stem for incorrectly connected/disconnect signals. The signature of a signal or slot changes in a subtle way, and without any warning (assuming no unit test coverage of the affected code!) boom ... something breaks.

Qt5 brings a new method to connect signals and slots. The main benefit of the new method is a compile time error when the connection cannot be created. Having this breakage occur at compile time (and hence caught by the CI testing... and also... you know... people actually compiling their own changes) means that the issue is caught before the offending code is ever committed to the codebase. Win!

A good summary of the new connection method, including some of the advantages (and disadvantages), and possible traps, is available at https://wiki.qt.io/New_Signal_Slot_Syntax

Proposed Solution

We require use of the new style connections for ALL NEWLY INTRODUCED signals/slots ONLY.

The important point here is that this only applies to new signals and slots. Adding new code which connects existing signals to existing slots is not affected by this QEP. The rationale behind this distinction is that it is not possible to disconnect using the old way any connections which have been made using the new method. If careful consideration is not made when creating a connection using existing signals and slots then it is possible that existing code which disconnects these signals/slots will no longer function correctly. Rather then apply a blanket rule of "all new code must use the new method" it is instead proposed a safer approach of:

• any NEWLY introduced signals/slots must use the new style connects/disconnects
• you should continue to use old style connections for existing signals/slots UNLESS you carefully do due diligence to ensure that no old style disconnects are in place anywhere else in the existing code which may be affected by use of a new style connection

Example(s)

See the Qt docs at https://wiki.qt.io/New_Signal_Slot_Syntax

Old

connect(sender, SIGNAL (valueChanged(QString,QString)), receiver, SLOT (updateValue(QString)) );

New

connect(sender, &Sender::valueChanged, receiver, &Receiver::updateValue );

Backwards Compatibility

N/A

Votes

(required)

QGIS Enhancement 69: Improved Node Tool

Date 2016/08/26

Author Martin Dobias (@wonder-sk)

Contact wonder dot sk at gmail dot com

Maintainer @wonder-sk

Version QGIS 3.0

Summary

Node tool is an important map tool for editing of geometries. There are however various shortcomings
in the existing tool:

• it is not possible to use advanced digitizing dock for precise moving of nodes
• old snapping classes are used, accounting for slower performance and preventing removal of the old code
• features need to be explicitly selected before editing of nodes is possible
• editing of features from multiple layers requires switching between active layers
• extension of linestrings is not possible
• there is no feedback on mouse over

Proposed Solution

The plan is to replace the existing node tool implementation with a new one that addresses the above issues.

Implementation of a possible replacement has been developed as a QGIS python plugin which is available here:
https://github.com/wonder-sk/CadNodeTool

The existing python plugin would be ported to C++ and further extended where it still lacks full support
(e.g. for circular geometries)

Using Node Tool

A brief introduction on how the node tool works:

1. Moving the mouse around will highlight features/nodes that may be moved (from any layer that is editable)

   

2. Clicking a highlighted node will start moving it (any editable layer may be used). Nodes that are at the same position will be moved if topological editing is enabled.

   

3. Nodes can be added by hovering over a segment and clicking the marker that appears in the middle.

   

4. Another way to add vertex is to double-click on a segment (away from the mid-point)

5. Segments can be moved by single click on a segment.

6. Linestrings can be expanded by clicking marker that appears near the endpoints of linestring features.

   

7. Node coordinates may be edited using advanced digitizing dock.

   

8. Nodes can be removed by pressing Del key while a node is being moved. Also, nodes can get selected by dragging a selection rectangle and pressing Del afterwards. Whenever a node is selected, next node from the same feature is pre-selected, allowing for quick removal of several nodes in a row.

9. Topological editing is respected, so if it is enabled, nodes of different features at the same position
   are moved together. The features may be even coming from different layers, they are moved together as long as the layers are in editing mode.

   

10. Ctrl+click to highlight vertex without going to dragging mode (for deletion and for node editor integration)

Performance Implications

The node tool should become faster thanks to the use of new snapping classes.

Further Considerations/Improvements

Press/Release vs Click/Click

To move nodes with the existing node tool implementation:

1. user presses the left mouse button on a node,
2. drags the node (with mouse button still being pressed),
3. releases the button to place the node.

The new node tool would change this into slightly different pattern:

1. user clicks mouse button on a node,
2. drags the node (mouse button is released all the time),
3. clicks again to place the node (left click to accept, right click to cancel the change)

This change is necessary to allow the use of advanced digitizing dock in order to enter values. After little bit of testing, this approach feels very natural and has further advantages compared to the existing behavior:

1. it is easier to precisely place nodes (not having to apply force to the mouse button all the time)
2. one does not move nodes inadvertently
3. it is possible to cancel the operation
4. it allows to pan the map by pressing space bar while the node is being moved

Node Editor

As the new node tool does not have a concept of selected features, it is questionable whether it makes sense to keep the node editor dock widget tied to node tool. Numeric editing of X/Y coordinates is more easily done with advanced digitizing dock, but Z/M coordinates not covered. It is suggested that node editor would be available from identify tool, possibly also still having it integrated with new node tool, showing coordinates of the most recently used feature. The suggestion is that the node editor dock is hidden by default.

Backwards Compatibility

No problem at the code level - the existing node tool has lived within app source code, so there is not problem with backward compatibility.

Votes

TODO

QGIS Enhancement 62 (was 41): Curved labels improvements

Date 2016/06/01

Author Hugo Mercier (@mhugo)

Contact hugo dot mercier at oslandia dot com

maintainer @mhugo

Version QGIS 3.0

Summary

This QEP describes some improvements to the labeling engine for a better handling of curved labels.
A new option allowing to manually place these curved labels is proposed as well as a support for multiline curved labels.

Label anchor point

The labeling engine of QGIS allows to ask for labels to be placed along a line, either parallel or curved.
When the manual placement is enabled, an absolute X/Y coordinate is given by the user.

We propose to add an option so that the user can "orient" the placement of such curved or parallel labels.
The idea is to allow the user to set an "anchor point" that will be used by the labeling engine as a starting point of the label along the line.

The two fields used for the data-defined X/Y position will have a special meaning for curved / parallel labels. This coordinate will be projected onto the line to set the "anchor point". The PAL labeling engine will be modified to handle this.

It will be possible for the user to manipulate this anchor point for curved / parallel labels. The "move label" tool used for manual placement will display an anchor point constrained to be on the support curve/line.

Examples of curved labels with anchor points materialized

Multi-line support for curved labels

It is sometimes desirable to have multiple line labels for curved labels. But this feature is not supported yet by the labeling engine.

Following the discussion started here http://hub.qgis.org/issues/4442, we propose to handle multiline curved labels.

One option will be added to choose which algorithm is used to extend the linear support for labels that will be displayed below or above:

• either these additional curves are computed by shifting the main curve (using an averaged normal vector). In this case curvatures of additional curves
  stay identical to the main curve, but the distance between two lines is not constant and may result in labels too close from one line to another and leads to labels not shown.
• either they are computed by offsetting the main curve (using QgsGeometry::offsetCurve). In this case the distance between two curves are always the same.
  But it may results in curves with high curvatures that will also be ignored by the labeling engine. An option to specify the type of joins will also be available.

The labeling engine will then be modified to allow multiline candidates. In particular, the PAL code around the function createCurvedCandidatesAlongLine() will be modified to handle multiline labels.

The multiline labeling options will be reused to compute the shift or offset distance needed below and above the main line.
Placement candidates will be chosen in order to respect the alignment option.

Example(s)

Difference between lines computed by offseting and shifting the main support

Issue Tracking ID(s)

http://hub.qgis.org/issues/4442

QGIS Enhancement 31: QGIS Legal Entity

Date 2015/7/07
Author Tim Sutton
Contact [email protected]
Last Edited 2015/7/07
Status Final Draft
Version N/A

Summary

Proposal for the establishment of QGIS.ORG as a non-profit legal entity. This
document represents discussions held at the QGIS Hackfest in Nødebo, Denmark
in May 2015.

Proposed Change

Definitions

• QGIS User Group - A formal regional or national chapter, with its own web
  site. To be considered a user group, there should be a minimum of 10 members
  and be approved by the board as an official user group.
• QGIS User Community - Any active participant of the QGIS community
  including committers, patch providers, translators, document maintainers,
  advice givers, stack exchange contributors, etc.
• OSGEO - the umbrella organisation for FOSS GIS projects to which QGIS is a member.

Why establish a legal entity?

QGIS has matured from a small ‘hobby project’ started in 2002 by Gary Sherman
into a large, complex and professional project used by many thousands of people
in their personal and professional capacity all around the world. In this
transition, we have had to grow and mature not only in our code base and
technical implementation, but also in terms of our project governance and
management. Up until now, QGIS has been structured as a grassroots community
headed up by a project steering committee. There is no legal entity behind the
project and typically individual members have acted as a proxy for the project
for things like domain name registration, trademark acquisition, sign off of
agreements with donors and sponsors and so on. The lack of a legal entity has
also limited our ability to be more ambitious in our funding aspirations - for
example our ability to apply for grant funding.

What do we propose?

We propose the following:

• To establish a Swiss ‘Verein’ (association) as a non-profit entity known as ‘QGIS.ORG’.
• We will draw up a Charter using the Swiss QGIS User group as a model
• QGIS.ORG will consist of two tiers of membership:
  • ‘the board’
  • ‘voting members’

Composition, appointment and role of voting members

The voting members shall be appointed according to the following scheme:

• User Group Voting Members: One voting member per QGIS user group. Each
  user group will nominate who their voting member shall be. If a group does not
  nominate any voting member, that vote will be excluded until such time that
  they do.
• OSGeo Voting Member: One voting member representing the OSGeo foundation. We
  wish to reaffirm our commitment as a member of the umbrella OSGeo organisation
  and foster good collaboration between QGIS.ORG and OSGEO.ORG.
• Community Voting Members: For each user group voting member, there will be one
  voting community member elected. The intent here is to let the general community
  have equal representation to the user group voting membership within the total
  voting membership.

Voting memberships will be valid until either:

• The voting member resigns
• There is a motion and passed vote to remove the member

Election and eligibility of user group voting members

• In order to be recognised, a user group must be approved by the PSC, showing
  their foundation document and agreeing to abide by the guidelines defined for
  user groups.
• A user group may be a ‘combined activity’ group - for example a regional OSGEO
  user group can also register as an official QGIS user group too.
• A user group should be a minimum of 10 users

Election and eligibility of community voting members

Community voting members will be nominated and voted for by the established core
committers. The number of nominations for voted members should exceed the number
of available voter places by at least one. Any active QGIS community member will
be eligible for nomination as a QGIS.ORG voting member.

Composition of the Board

• Rotating chair: The board members will vote at every third annual general
  meeting to appoint or continue the existing appointment of the chair. Thus the
  chair shall be appointed for a minimum of three years at a time.
• Honorary member: Gary Sherman will retain a lifetime honorary seat on the
  board.
• PSC: The PSC will transform into the board of QGIS.ORG
• SWISS Member: There will be at least one Swiss board member in order to fulfil the
  requirements of the Verein legal entity.

Board remuneration: Board members would take no pay for their services.

Functions of the QGIS.ORG legal entity

The QGIS.ORG legal entity will serve the following functions:

• Hold an annual meeting where the annual budget will be presented for approval by voting members.
• Act as the holder of all trademarks etc. registered for QGIS
• Act as the holder for QGIS.org domain name
• Be able to sign contracts with external organisations on behalf of QGIS
• QGIS.ORG will not act as a consulting house. For example it will not pursue
  clients, provide deliverable services other than the distribution of funds to
  the QGIS project needed in order to carry out our activities.
• QGIS.ORG will apply for grants and other types of funding sources which can be
  disbursed to pay for developer meetings, sponsor QGIS developers, and other
  community members as needed to support the aims of the project.
• QGIS.ORG will sign any legal agreements as needed.

Documentation

This QEP serves as the documentation for the legal structure, along with the
Verein (legal entity) statutes, a draft of which is available for viewing
.. _here: https://docs.google.com/a/qgis.org/document/d/1FcChHYgoI4FlURrZRYBjD8kB6IBW_BaG0Ieg8TDT0pM/edit?usp=sharing
(this link allows public commenting but not editing).

This document and the linked document above will be migrated to the QGIS
Governance documentation in modified form. and will be migrated to the project
governance documentation.

Voting History

This QEP has been approved - see the loomio vote:

https://www.loomio.org/d/wyUpS942/motion-to-form-a-legal-entity

QGIS Enhancement 28: Processing UI/UX Improvements

Date 2015/10/23
Author Victor Olaya
Contact volayaf at gmail dot com

Last Edited 2015/11/07
Status Draft
Version QGIS 2.14

Summary

This QEP proposes improvements in the Processing UI to make it more intuitive and less confusing for users

Rationale

Processing is a complex tool and it features a large number of elements. This can be indimidating for some users and rather confusing.

To avoid this, a simplified interface was proposed, in which algorithm providers are not shown, and a single "Geoalgorithms" category is added to the toolbox. Not all algorithms are added under that group, but just a selected group of them, and certain algorithms are renamed.

This has proven to be a bad solution. Users are more confused, and when they find out that some algorithms are in the advanced mode but not in the simplified one, they report that as a bug. Mantaining the simplified interface is also a concern, since sometimes new algorithms are added but the corresponding entry for the simplified version is not added. That causes the algorithm to not be available in the simplified interface.

Additional problems have been reported, such as the lack of help. This QEP proposes some simple enhancements that might help aliviate that.

Overall, the idea is to make Processing easier to understand. This QEP does not include any new analytical feature or similar improvement.

Proposed Change

The following changes are proposed:

• Eliminate the simplified interface. Instead, the normal interface will be displayed all the time. To make it more friendly, only the "QGIS algorithms" will be enabled by default. A tip box on top of the toolbox will be added, telling first-time users that they can add more algorithms by enabling additional plugins.

When a search is performed and there are algorithms in disabled providers that match the search string, the user will be informed with a panel, and he will have the option of showing them

Disabled plugins can be enabled by just clicking on the Activate link.

This means that disabling a provider is now only a toolbox display option. Providers are always available, which will avoid errors when loading a model that contains an algorithm form a disabled provider (that failed before but will work fine now)

• Add a small help panel to the algorithm dialog. Currently, the algorithm panel has a help tab. This has two problems: the first one is that help is not seen inmediately after opening the dialog. The second, and most important one, is that the help is lacking for most algorithms, and a help template with parameter names is shown instead.

  Writing help files for all algorithms is a very long task, and we have not been succesful at it. It looks like full description of algorithms will not be available anytime soon. Instead of waiting for that and maintaining the help tab for a detailed algorithm content, a smaller panel is proposed. It would be placed in the side of the algorithm dialog, and it would display a short description of the algorithm. Having such a description for most algorithms (or at least for the ones in the "QGIS algorithms" group) is much more realistic and feasible. For those algorithms that have a full help file (for instance, GRASS algorithm, which have the original GRASS docs), the Help tab will be shown like it's done now, but this tab will be removed in case there is no help

See the picture below for an example

Short description are written in YAML format, like in this file:

https://github.com/volaya/QGIS/blob/ux_improvements/python/plugins/processing/algs/help/qgis.yaml

Adding more descriptions is just a matter of dropping a yaml file with a dictionary in that help folder, where keys are the algorithm console names and values are the short descriptions of the algorithms.

Backwards Compatibility

Since the change is only cosmetic, it wouldn't have any effect on analysis features themselves, and would not break models or scripts from previous versions

Prototype

An implementation of the above ideas can be found in this branch

https://github.com/volaya/QGIS/tree/ux_improvements

QGIS Enhancement 72: QgsMapCanvasCustomMap item for fast refreshing map layers

Date 2016/09/10

Author Nyall Dawson @nyalldawson

Contact [email protected]

maintainer Nyall Dawson

Version QGIS 3.0

Summary

The map map renderer within QGIS is designed to render multiple map layers in one pass, allowing for
interaction between the layers (such as labels from one layer avoiding overlap with the features from
another layer). While this approach works well for “basemap” style map renders, it does not work well for rendering dynamic layers with frequently changing features and geometry. The map refreshes require all layers to be re-rendered and re-composited, resulting in a “flashing” appearance with each map redraw.

This proposal covers implementing a new type of canvas item designed for rapidly refreshing map layers without visible redraw “flashing” effects.

While it's currently possible to achieve a similar result using a workaround of multiple map canvases (see eg https://gist.github.com/nyalldawson/1b389d87e2c70ef44bd7307365217dbc - credit to Nathan Woodrow for the original inspiration to the gist), this approach has shortcomings such as the additional canvases obscuring canvas items which should be drawn above all map items (eg identify highlights and map tool rubber bands).

Proposed Solution

The proposed solution consists of implementing a new type of QgsMapCanvasItem, specifically for rendering isolated map layers which can be refreshed and repainted without affecting other map layers or requiring a full map redraw.
The new item type, QgsMapCanvasCustomMap, is designed for use by plugin developers only and will not be accessible or configurable by end users. It will be exposed via the c++ and PyQGIS APIs, and will have no graphical GUI within the QGIS app for configuring or controlling which layers are displayed in this way. It will be left to QGIS plugin developers to implement use of QgsMapCanvasCustomMap within their plugins as required.

QgsMapCanvasCustomMap will render its contents on request, based on a specified subset of the project's map layers. The rendering will be isolated from the main map canvas render, so no interaction will be possible between the rendering of the main canvas and the contents of the QgsMapCanvasCustomMap. This includes label collision detection, so for instance labels from the layers within the QgsMapCanvasCustomMap may overlap labels and features from the underlying base map layers.

Additionally, QgsMapCanvasCustomMap items will always be rendered above the underlying base map, so it will not be possible for layers from the base map to be drawn above layers in the
QgsMapCanvasCustomMap.

To aid with use of animated, frequently refreshed QgsMapCanvasCustomMap items the rendering results of a QgsMapCanvasCustomMap will be cached and will only be redrawn following a completed render of the contents of the custom map item. Ie, the results of the previous render will be persistently shown on screen until the new render results are ready, avoiding any redrawing “flicker” effects.

Additionally, the z value for existing canvas items will be predefined to ensure correct placement of rubbers bands, etc when QgsMapCanvasCustomMap items are present in the scene.

Backwards Compatibility

N/A - this work will be targeted to QGIS 3.0 alone and will not be backported. Additionally, it will only be accessible though PyQGIS/c++ api and will not affect project files.

Votes

(required)

QGIS Enhancement 64: Move Geocoder Component into QGIS Core

Date 2016/06/11

Author Richard Duivenvoorde (@rduivenvoorde) et al.

Contact [email protected]

maintainer @rduivenvoorde

Version QGIS 3.0

Summary

With the proliferation of Geocoder componets, Paolo asked the develop list if it isn't time to incorporate geocoder functionality into core, see for more discussion this thread:

https://lists.osgeo.org/pipermail/qgis-developer/2016-June/043145.html

Current python geocoder plugins:

• http://plugins.qgis.org/plugins/GeoCoding/
• http://plugins.qgis.org/plugins/GeoCodingKorea/
• http://plugins.qgis.org/plugins/navidata/
• http://plugins.qgis.org/plugins/pdokbaggeocoder/
• http://plugins.qgis.org/plugins/ru_geocoder/
• http://plugins.qgis.org/plugins/geopunt4Qgis/
• https://github.com/sookoll/qgis_plugin_inads_geocoder/
• http://plugins.qgis.org/plugins/pdokservicesplugin/
• http://plugins.qgis.org/plugins/mmqgis/
• https://github.com/lutraconsulting/qgis-discovery-plugin (database-based, not webservice)

Proposed Solution

See mail thread.

Usable libs (mentioned in email thread):

• http://www.gdal.org/ogr__geocoding_8h.html (see: http://www.itopen.it/geocoding-with-gdal/)
• https://github.com/geopy/geopy (python)
• https://pypi.python.org/pypi/geocoder (python)

Ideas:

Richard:
What about something like that:

• we creating a gui-widget/toolbar for a geocoding search
• we create a python api for it (actually: implement a buttonless plugin
  which registres the geocoder, and registres some slot/signals and does
  (part of) implementation)
• by installing this plugin(s) you registre one or more geocoders, AND
  the plugin author actually implements the calling/retrieving/parsing of
  the results. We can maybe do some general standards in core (or via
  gdal): like OGC-OpenLS interface, or OSM-based ones or so
• the core-geocoder gui then can look like the search bar in Firefox
  a little dropdown in which you can select WHICH geocoder you run by default
• this core geocoder widget can then also implement a small 'multiple
  result' list...

A side note:

• some of the geocoders plugins Paolo lists, are 'one shot' geocoders:
  you sent one address and it returns one or two results. This is the
  geocoder I'm talking about
• some of the geocoder plugins accept a csv/list, you have to select
  some columnnames and you sent the list to a service and you get a list
  back (with info about misses etcetc). This is much harder I think to
  implement in a general way? Would be cool though...
  pdokgeocoder and MMQGISplugin have these...

Martin:
And I would not limit ourselves to just geocoding - there are other uses for that, e.g.:

• gazetteers - for example Discovery plugin [1]
• search in attributes of loaded layer(s)
  The API implementation could also take care of things like on the fly
  completion while user is still typing and running the queries in the
  background, so the GUI is not blocked. So a search plugin would only
  take care of processing search/geocoding/gazetteer query and response,
  all the rest would be handled by QGIS.

Nathan: I quite like how the Locator bar in Qt Creator work by also allowing
prefixs for searching:

• c - Class search
• . - method search

Alessandro: I guess that the most reasonable implementation would be having the most common geocoders (whole world coverage) available as core implementationn and create an API to register local/national geocoders by plugins.

Mederic: It could also be very interesting to have one or more Processing module(s) for geocoding (natives QGIS "algorithms"). We could then chain this with other algorithms (find all objects of a layer intersecting a 500m buffer from the positions of the adresses in a list for example). Of course this module would be easier to write if we would have a native QGIS API for this...

Example(s)

(optional)

Affected Files

(required if applicable)

Performance Implications

(required if known at design time)

Further Considerations/Improvements

(optional)

Backwards Compatibility

(required)

Issue Tracking ID(s)

(optional)

Votes

(required)

QGIS Enhancement 63: Symbol clipping

Date 2016/06/06

Author Hugo Mercier (@mhugo)

Contact hugo dot mercier at oslandia dot com

maintainer @mhugo

Version QGIS 3.0

Summary

This QEP is about the need for an advanced labeling and symbology use case where the user wants a special kind of "buffer" around a label or a marker symbol that is not opaque, but "cuts out" some other symbols.
This is meant to improve the readibility by avoiding the proximity mix between small elements of the
same color.

Example :
https://hub.qgis.org/attachments/download/7610

Here the "Wollishofen" label interacts with some underlying layers. In particular, there is no black symbols under the label that is drawn. It could be point or linear symbol layers as well as some lines that are part of a symbol layer (the outline of the red road line for instance).

The functionality must also be available for symbol markers, not only for labels.
https://hub.qgis.org/attachments/download/7611

GUI changes

For clipping buffers around labels, a new option will be available, with the same properties already existing for the "regular" buffer.
Then the user will choose which symbol layers this buffer around the label should "erase".

Regarding clipping buffers around markers, a new marker symbol layer will be defined with options to define a clipping buffer: its shape (ellipse or rectangle) and the list of symbol layers that have to be "erased" underneath. A later option could allow to define its shape as a buffer around a sub point marker.

Possible implementations

Deferred rendering

The main technical difficulty to face is that QGIS draws the map layer by layer, from the bottom to the top. With this new feature, some elements would have to be clipped by a region that could not be determined at the time of drawing.
The rendering process has then to be modified somehow so that two passes are made.

The main idea of this approach is to defer the drawing by stacking all drawing commands (thanks to a painter "proxy") and manipulating the drawing command stack before the actual rendering.
Some new drawing commands (in a QgsPainter class derived from QPainter ?) would allow to set a "clipping" region on some other symbol layers.
After the first pass where all the regular drawing commands plus the extra clipping commands will be resolved in a second pass so that the stack only contains regular drawing commands.
These commands will then be send to the original painting device.

Inspiration: https://majewsky.wordpress.com/2010/08/05/the-color-changing-qpaintdeviceproxy/
Proof of concept in Python : https://gist.github.com/mhugo/fbfa5d15b95787d20f5e8f7863441036
Using QPicture to store painting commands : https://gist.github.com/mhugo/f809970c952f318cb2abc57af1a2446d

Pros:

• High level of control about what layers could be "masked"

Cons:

• Complex implementation
• This technique would certainly cancel any parallel rendering for layers that are "clipped" by a symbol or a label of another layer (since it adds dependencies during the rendering)

Over painting approach

The idea here is to render the map in a first pass normally, ignoring any "clipping buffer" around labels or markers.
Then for each clipping buffer type, the whole map will be rendered again with some symbol levels deactivated. These renderings will be clipped by the clipping buffers and (opaque) painted over the background canvas.

Pros:

• May be easier to design and understand

Cons:

• Need one more rendering for each clipping buffer type (performance issue)

Symbol layer selection

When defining a clipping buffer (around a label or a marker), only symbol layers beneath the buffer can be cut. It simplifies the ergonomy and avoid cycles in dependencies between symbol layers.

QGIS Enhancement 54 (was 36): Dropping fTools and GDALTools core plugins in favor of the Processing

Date 2016/01/26
Author Alexander Bruy (@alexbruy)
Contact alexander dot bruy at gmail dot com
Version QGIS 2.16/3.0

Summary

Functionality of the fTools and GdalTools plugins now available in the Processing framework. Also Processing brings some nice additions e.g. batch execution, scripting which was not available with fTools/GdalTools. Both fTools and GdalTools plugins does not receive much attention since the introduction of the Processing.

Now we have duplicated codebase and when bugs found we need to fix them twice: in the Processing code and in the code of the fTools/GdalTools. This increases maintenance costs and there is a chance that bug will be fixed only in one place, causing problems for users.

There were several discussions about this including one at the hackfest in Essen:

• http://lists.osgeo.org/pipermail/qgis-developer/2014-June/033359.html
• http://lists.osgeo.org/pipermail/qgis-developer/2015-May/037657.html
• http://lists.osgeo.org/pipermail/qgis-developer/2015-September/039106.html

Proposed Solution

Consolidate developers efforts on Processing, as this is more feature rich and flexible plugin. fTools and GdalTools plugin will be removed (in fact GdalTools plugin will be kept, until we have multithreading support in Processing, as it does not block QGIS main window during long-running analysis). To keep most used tools in the usual place and avoid confusion Processing will create menu entries in the Vector and Raster menus, replacing removed algorithms with its own implementation. Algorithms available from menus will have same icons as old fTools/GdalTools tools, see screenshots below

This also allow our documentation writers easily update documentation, as there are no GUI changes.

Advanced users that need more control on menus (add/remove some entries, create their own menus/submenus, etc) can customize menus from Processing options.

It is possible to choose which algs to show and where to put them. Menus and submenus should be specified in format

MenuName/SubMenuName

NOTE: if existing menu/submenu entry has "&" symbol it should be also specified.

Affected Files

Directories python/plugins/fTools and python/plugins/GdalTools will be removed. Corresponding CMakeLists.txt file should be updated.

References to both plugins also should be removed from src/app/qgspluginregistry.cpp and ms-windows/python_plugins.nsh.

Processing code will be updated to create submenus in the Vector and Raster menus.

Further Considerations/Improvements

In future more improvements can be made:

• implement custom dialogs for algorithms that available from menus
• create additional submenus with frequently and widely used algorithms to make them more visible

Backwards Compatibility

As all tools will be available in the same menus with same icons and almost the same names there should not be any issues. The only difference is the GUI of the Processing dialogs, which are quite different from dialogs used in fTools/GdalTools. But as most of the users already familiar with Processing UI this also should not be a problem.

Prototype

An implementation of the above ideas can be found in this branch
https://github.com/alexbruy/QGIS/tree/drop-ftools-gdaltools

Votes

(required

.. _qep#[.#]:

QGIS Enhancement #: Fields And Forms Redesign

:Date: 2014/11/25
:Author: Matthias Kuhn
:Contact: matthias dot kuhn at gmx dot ch
:Last Edited: 2014/11/25
:Status: Draft
:Version: QGIS 2.8

1. Summary

The tab Fields on the vector layer properties dialog should be split into
two different tabs that separate data-related from form- and widget-related
user interface elements.
This is expected to improve user experience and usability.

2. Rationale

On the vector layer properties dialog there is a tab Fields where the user is
able to configure different things related to fields (data- and widget-related)
and forms (data- and form-related):

• The general layout of the form (autogenerated, drag and drop, ui file)
• The appearance of the individual widgets
  • Some weak constraints can be introduced that will limit acceptable input
    from widgets but will not be enforced by any other way to set a field's
    value (e.g. min/max values from the range widget)
• Adding fields and removing fields
• Drag and drop fields and relations to the form
• Python logic

This tab currently is in a very bad shape from a UX perspective. Problems
include:

• Having to scroll horizontally in the fields table when wanting to change
  the widget type or looking for information.
• The resizable containers on the right and at the bottom can be hidden
  (consciously and unconsciously) and confuse the user.

3. Proposal

The information and functionality that is now available on the Fields tab
will be split and put on two different tabs:

• Fields
• Form

3.1 Fields
.................

The Fields tab conatins anything that is related to data and fields.

In particular, this refers to:

• Changing fields
  • add
  • delete
  • calculate
• Information about fields
  • data type
  • comment
  • constraints (Future)
  • domain (Future)
  • exposure (WMS/WFS)
  • ...

Mockup

.. figure:: fields.png

   Mockup of the fields tab. This will be shown embedded in the vector layer properties dialog.

3.2 Forms
.................

The *Form* tab contains anything that is related to user interface and visual
representation and manipulation of a features fields.

In particular, this refers to:

- The widget type
- The widget configuration
- Form layout
- Python init script for the form
- ...

Mockups

.. figure:: form1.png

Mockup of the Form tab with autogenerated layout. This will be shown embedded in the vector layer properties dialog.

.. figure:: form2.png

Mockup of the Form tab with drag and drop designer layout. This will be shown embedded in the vector layer properties dialog.

.. figure:: form3.png

Mockup of the Form tab with ui file layout. This will be shown embedded in the vector layer properties dialog.

4. Scope

Weak constraints are currently defined in the widget properties. It is outside the
scope of this QEP to move constraints to the fields tab. The sole purpose of the
constraints part of the UI in the mockups is to highlight where I immagine this could
eventually lead to.
For the moment, the currently available weak constraints will still be defined on
the Form tab.

5. Documentation

Documentation would need to be updated to suit.

6. Voting History

(required)

QGIS Enhancement 66: Better editor widgets with support for arrays and maps

Date 2016/07/18

Author Patrick Valsecchi (@pvalsecc)

Contact [email protected]

maintainer [email protected]

Version QGIS 3.0

Summary

A simple INSPIRE WFS query returning GML Complex Features, mapped to a relational table, can lead to up to 50 tables due to the complexity of INSPIRE schemas and the presence of xlinks to additional info. In order to make such complex data usable in QGIS we need a few things in QGIS:

• Better support of array and map types for DB fields (at least for PostGIS and Spatialite). That would allow us to reduce the number of tables by mapping, for example, store translated texts into a hstore (lang=>text) in PostgreSQL instead of using a 1-N relation to an additional table (lang, text).
• Better default widgets used for editing feature attributes. For the moment, QGIS always use TextEdit. Using the native types of the DB, we could do better than that.
• Add support to lookup a table to specify the widgets to instantiate for editing a field. Something similar to the layer_styles table used to provide default stylings.

Proposed Solution

Support for arrays and maps

Spatialite has no support for those types. To work around that, the TEXT type can be used with a JSON representation of the list of map to be stored.

PostgreSQL has native types for those (arrays and hstore). Arrays are converted internally into a PostgreSQL specific representation before being read. This would be switched to a QList/QMap internal representation.

Other type of datasource is not considered for this project. But they must have a sane default behaviour in regard to those changes.

Functions will be added to the expression language to manage those types. Their naming will match their equivalent in PostgreSQL if they have one. For example:

• array_length
• array_contains
• array_find
• array_get

New tests for the PostGIS datasource will be added to check the support of those types (read/write). The new expression functions will be tested as well.

Better default edition widget

Change the isFieldSupported method to QgsEditorWidgetFactory allowing a factory to tell how good it is to edit/represent a given field:

• 0: not supported
• 5: barely supported (Datetime returns that for string fields)
• 10: barely supported (TextEdit returns always that)
• 20: perfect support
• …

Then, a QgsEditorWidgetRegistry::findBest(const QgsVectorLayer& vl, int fieldIdx) const could be created and used to find the best widget (internal or introduced by a plugin) for a given field. If more than one widget matches with the same rating, the first registered will be taken. This will be the subject of a few UTs.

A plugin system will be put in place so that findBest can be enhanced with datasource or customer specific logic.

Widgets will be added for editing array and maps. They will be represented as editable tables (one column for arrays and two for maps) with a toolbar to add/remove entries. Those widgets must be able to work with native QMap/QList and with a QString containing a JSON (for Spatialite). If doable, tests will cover those widgets.

Widget types from a table

The INSPIRE schemas could be used to automatically feed a metadata table that specifies specialized widgets to be used and their default configuration for every fields of every tables. A plugin for QgsEditorWidgetRegistry::findBest will looks if this table exists in the database of the given layer and search a row for the given table and field. If an entry is found, use the specified widget.

The place in the code where the data for that will be collected is not yet determined.

Affected Files

I’ve added a few TODO in the code where things need to be added/changed in this branch:
https://github.com/pvalsecc/QGIS/tree/editor_widgets_qep

Obviously, a few files will be added for the widgets for arrays and maps.

Performance Implications

Negligible. There is no complex computation done on very little data and not very often (at startup)

Further Considerations/Improvements

The feasibility to use sub-widgets (TextEdit, DateEdit, ...) for editing the array or map entries has not been studied for the moment. PostgreSQL supports only maps with strings for keys and values but is 100% flexible for arrays.

Backwards Compatibility

Current projects using PostGIS with arrays will be affected because of the modified internal representation from QString to QList. This could be a problem for custom plugins relying on that. We don't support hstore yet, so the usage of QMap won't be a problem.

Votes

(required)

QGIS Enhancement 37: Allow simultaneous editing of attributes for multiple features through the attribute table “form” view

Date 2016-03-16
Author Nyall Dawson (@nyalldawson)
Contact nyall dot dawson at gmail dot com
Status Accepted
Funding Kanton Basel Stadt in Switzerland
Version QGIS 2.16

Summary

Currently, QGIS attribute form only supports graphical editing of the attributes for one feature at a time.
There is no way to set the attributes for multiple features at once through the form view. Limited editing
for multiple features can be achieved using the “merge selected feature attributes” tool, but this tool is
geared toward statistical calculation for multiple feature attributes and it is cumbersome to edit large
amounts of attributes using this tool. Furthermore, the tool only provides for simple “text entry” of
attributes, and cannot be used with all the advanced editor widgets which are possible to use in QGIS
attribute forms (eg date/time widgets, relation widgets, unique value widgets, etc).

This QEP covers modifications to the attribute form and editor widgets to allow for simultaneously editing multiple features through the attribute form.

Relation to upcoming form search changes

Possible follow up work involves allowing the attribute form (including editor widgets) to be used for setting the parameters for searching and filtering layers. This QEP has been designed to allow this possible future work to hook in to the changes required for multi attribute editing.

Proposed Solution

Multiedit mode is initiated from the attribute form dialog through a new “Multiedit” button on the attribute table toolbar. It will work on selections made through the general QGIS feature selection tools or through the feature selection list shown on the left when in attribute form mode.

When multi-edit mode is activated, QGIS will scan all selected feature to determine whether each feature holds different values for a given attribute or whether all features have the same value.
If an attribute holds mixed values across the selection, the widgets will be update to reflect this. For widget types where it is applicable (Eg text edits), the initial value for the widget will be set to “mixed values”. Other widget types may indicate this state in different ways, eg for checkbox widgets the mixed values state will be indicated by the “partially checked” state. This will not be possible for all widget types, eg sliders and dials. For these widgets the value from the first selected feature will be shown instead.

For all widget types, when in multi-edit mode, a tool button with icon will be shown to the right of the widget. Initially, the button will show an icon indicating whether the selection has mixed values for the field (possibly the mathematical “not equal” symbol). If the field does not have mixed values, a tick icon will be shown instead. When the field has mixed values in the selection, clicking the button will show a popup-menu with a single action “Set [field name] for all selected features”. Activating this action will force the field value for all selected features to match the current widget value.

When the value for a widget is changed, the icon next to the widget will change to a “!” alert icon. The tooltip for this icon will show “Value for [field name] for all selected features will be overridden”. Clicking the button when in this state will show a popup menu with a single item “Reset to original values”.
Read-only widgets will stay disabled in the form and will never be changed. These widgets will either be shown in their “mixed value” state, or the value for the first selected feature if no mixed-state exists.

When saving multi-edits, only attributes that have a changed widget state (“!” icon) will be applied. Attributes which remain at their mixed value state will not be altered. Before applying the new attributes for all selected records the user has to confirm that they really want to change the attributes for all selected records and changed widgets.

All changes will be applied as a single edit command, so pressing undo will revert all changes made to all selected features at once.

The attribute table section (left panel) of the dual-view dialog will be active while multi-editing features. If the selection changes, the user will be prompted to save any current changes made to the previous selection.

Multi-edit mode will only be enabled for auto-generated and drag and drop forms. Custom forms will not be supported, due to the extra complexity added by custom layouts and python logic which could be present in these forms.

“Relation reference” widgets will be editable in multi-edit mode, however “relation editor widgets” will NOT be editable whilst in multi-edit mode.

Affected classes

• QgsDualView: will be modified to add a new multi edit toggle button and the corresponding signals/slots for enabling and triggering this button
• QgsEditorWidgetWrapper: a new virtual void setMixedState( bool mixed ) method will be added. The base implementation of this method will have no effect. QgsEditorWidgetWrapper subclasses can override this method to change their widget states to a "mixed value" state (eg QgsCheckboxWidgetWrapper will set its QCheckBox to the Qt::PartiallyChecked state). Other editor widgets will be updated to implement this method were appropriate (eg QgsTextEditWrapper, QgsClassificationWidgetWrapper, ... )
• QgsMultiEditToolButton: new QToolButton subclass for a widget which displays the state of an editor widget in multi edit mode. Eg shows the mixed values and changed values icons, and provides GUI to reset values to their original state.
• QgsAttributeForm: a new private method scanForEqualAttributes will be added, which will scan through a QgsFeatureIterator and return lists of both fields with mixed values and fields which all share the same values.
• QgsAttributeForm: a new enum will be added with corresponding getters and setters for setting the attribute form "mode". Options will include "SingleEditMode" (default, current behaviour) and "MultiEditMode". In future this will be extended to add "SearchMode".
• QgsAttributeFormEditorWidget: new QWidget subclass consisting of both an editor widget and additional widgets for controlling the behaviour of the editor widget depending on a number of possible modes. For instance, if the parent attribute form is in the multi edit mode, this widget will show both the editor widget and a tool button for controlling the multi edit results. (In future, when search mode is added, this widget will show the editor widget and a tool button for controlling search behaviour). Initially only DefaultMode (current behaviour - only the editor widget is shown) and MultiEditMode will be implemented. QgsAttributeFormEditorWidget will also have a method setIsMixed( bool mixed ), which if set will put both the editor widget and the multi edit button into the mixed values state.

Tests

Tests will be added for the non-GUI portions of this QEP (eg QgsAttributeForm::scanForEqualAttributes). Additionally, tests will be added for checking the logic of the QgsMultiEditToolButton modes.

Votes

(required)

QGIS Enhancement 67: API Documentation Guidelines

Date 2016/07/18

Author Martin Dobias (@wonder-sk)

Contact wonder dot sk at gmail dot com

maintainer @wonder-sk

Version QGIS 3.0

Summary

This QEP presents guidelines for QGIS API documentation that should be followed for classes that are available within QGIS API. The overall target is to improve the quality of the provided documentation so that API users are likely to understand purpose of the class and how to use it just from reading the API documentation.

Current Status

The API documentation is generated using Doxygen from the comments in class header files.

The main problems of the QGIS API documentation:

• missing overview pages to briefly introduce various aspects of API and how classes are related
• description of classes is often missing, is very brief or even wrong (outdated)
• many functions are not documented or the description does not give enough information
• code examples that would show real use of a class are very rare
• differences between C++ API and Python API are not covered (apart from note when a method is not available in Python)
• lack of images like diagrams, screenshots or other illustrations to help to understand the API
• often new methods lack QGIS version number when the method was added

Proposed Solution

The following sections set rules for doxygen comments in header files. In general the idea is to have API documented in a fashion similar to Qt API documentation.

Class Documentation

Class documentation should start with one sentence about the purpose of the class. This should be followed by brief description of the class, introducing the main methods of interest. If there are some pieces of functionality that are not obvious, they should be further explained in follow up sections. It is not necessary to explain common GIS concepts or design patterns (a link to other source may be provided if necessary).

It is useful to put the class into context of other related classes by mentioning those and how do they interact, with a link to API overview page (see below).

Classes that provide reusable widgets should normally include a screenshot of the widget. Classes involved in map rendering may include output generated by the class. It is encouraged to include diagrams or other illustrations in the class documentation if they may help in understanding the API.

Code examples are highly recommended, especially for classes that need non-trivial setup. As the main audience for the documentation are Python plugin developers, examples should be written in Python. Examples should be ideally self-contained so it is possible to just copy and paste them in Python console in QGIS without extra setup (for example without loading some layers). Example(s) should cover just the most common use cases to help people start using the API, it is not necessary to provide examples of advanced functionality. Example code may be written either inline or it may refer to snippets from external files. At some point the doxygen snippets should be automatically tested in the QGIS test suite to make sure they are still valid.

Classes should have a note in what QGIS version they have been introduced.

Method Documentation

Each method should have a description that explains its purpose in sufficient detail. Behavior in various special cases should be documented too, including reasons why the method may fail, what happens if an argument is a null pointer and so on.

It is preferred to write documentation about a method in third person, for example "Paints XYZ" or "Returns XYZ" (instead of "Paint XYZ", "Return XYZ").
Try to use full sentences in documentation as much as possible. I.e. "Sets the coordinates reference system for the layer" instead of "sets coordinate system" or "mutator for coordinate system". Use correct capitalization for sentences, and make sure there's a "." on the end!

Methods should have a not in what QGIS version they have been introduced if they were added later that the class:

@note added in QGIS 2.16

If there is any difference between Python API and C++ API, this must be clearly stated. Most commonly this is the case when a value is returned through an argument, in SIP file such argument is marked with /Out/ annotation. Other special cases may appear if SIP includes /Array/ annotation or %MethodCode directive. If Python method is not available for some reason, that should be noted as well:

@note not available in python bindings

In cases there is transfer of ownership of objects (e.g. a pointer passed to a method will get owned by its class or a method returns a new pointer), this fact should be documented. Similarly, if a returned piece of data is only temporary and may become invalid, this needs to be noted.

Try to link related methods, ie setWidth() should have a @see setWidthUnits() link. This makes the documentation much more useful as it gives a hint that there may be important notes in the related methods. Especially link getters and setters using "@see" tags ie setName should have a @see name(), and name() should have a @see setName().

Use correct capitalization for acronyms and terms (ie "DOM" not "dom", "XML" not "xml", "Python" not "python", "PostGIS" not "postgis" etc).

For documentation of arguments and return values, one should use @param and @returns
doxygen tags to document them.

API Overview Pages

Each are of API documentation should have a dedicated page in the API documentation that gives a brief overview of the API in question, for example: symbology API, map rendering API, raster API, map canvas API.

Doxygen Style

Doxygen supports Markdown syntax, this is the preferred way for markup compared to using Doxygen tags (Markdown is easier to read or write).

Some recommendations for Doxygen comments:

• /** XYZ */ is preferred comments style for multi-line comments
• //! is preferred comments style for one-line comments
• @param is preferred rather than \param
• @returns is preferred rather than @return
• Use @ref tag to link to other parts of documentation

We are not very picky about how exactly the multi-line comments should be written (e.g. First and last line only contain opening closing sequence, whether asterisks must be at the beginning). Sometimes people are get too opinionated about such details :-) (http://lkml.iu.edu/hypermail/linux/kernel/1607.1/00627.html)

Examples

Method documentation:

/** Returns text with occurences of "bar" replaced by "foo".
 * @param bar Input text to be processed
 * @returns Processed text after replacements
 * @note added in QGIS 2.16
 * @see foobar()
 */
QString foo(const QString& text);

Examples of well documented classes:

• QgsCoordinateReferenceSystem [not yet online]

When still unsure, have a look at documentation of Qt classes.

New and Existing Classes

Classes and methods newly added to the API should follow the new rules for the documentation. (Incomplete API documentation should be a showstopper for merge of a pull request.)

It will take a lot of effort to bring existing classes up to the new standards. Significant improvements or reworks of existing classes should therefore also improve the documentation.

Votes

(required)

QGIS Enhancement 35: Authentication configuration system with master password

:Date: 2015/01/01
:Author: Larry Shaffer
:Contact: lshaffer at boundlessgeo dot com
:Last Edited: 2015/09/20
:Status: Draft
:Version: QGIS 2.12
:Sponsor: Boundless - New York, NY USA
:Sponsor URL: http://boundlessgeo.com/

1. Summary

See Pull Request (PR) #2330 <https://github.com/qgis/QGIS/pull/2330>_

.. note::

The PR is already a completely functional core implementation of this
proposal, with Python bindings, unit tests, integration with PostGIS and OWS
service connections using username/password and support for PKI client
certificates (in PEM, DER or PKCS#12 format) for OWS HTTPS connections. In
addition, it has been built and run on most major platforms and has received
extensive functional testing over several months.

Currently, QGIS lacks any facility to store/retrieve authentication credentials
in a secure manner. Users either choose to save credentials insecurely in
connection profiles and/or as plain text in their project files, or choose to
not save credentials, thereby burdening themselves with correctly reentering the
credentials per work session.

Connection info can also be insecurely saved as part of a data resource's URI,
which is used to help define full connection profiles in a single URI, to aid
drag/drop operations, and to offer the user convenient/quick connections to data
sources. This can lead to accidental exposure of authentication credentials if a
project file, with credentials saved in it, is shared by a user who is unaware
of the consequences.

Similarly, there is no simple methodology in QGIS of representing complex
authentication configuration components (beyond just username and
password) within the scope of a single data source URI.

2. Proposed Technical Solution

Introduce a core singleton authentication (auth) manager that securely oversees
storage and retrieval of user auth configurations (configs) in a dedicated
SQLite database file, where such configs are encrypted on storage and decrypted
upon retrieval using a master password, known only to the user, that has its
salted and iterated SHA256 hash stored in the database for verification.

No access or usage of an auth config can take place without the user first
entering the correct master password, which unlocks access to the auth database
until the app is quit, or the cached master password is cleared mid-session.

Utilize an auth config's ID (a hash-like 7-character string, e.g. j2r5tvq),
generated during initial storage to the auth database, to abstractly represent
the config as a parameter in data source URIs and in application and plugin
settings; thereby allowing auth configs to be securely stored in plain text
application components, e.g. project and settings files, without disclosure of
credentials.

Add a variety of common authentication methods as plugins, e.g. HTTP Basic,
SSL/PKI, etc., and integrate resource access routines with calls to the auth
manager, which marshals an appropriate call to the method plugin associated to
any auth config ID assigned to the resource.

For example, if a WMS data source URI string contains authcfg=j2r5tvq and
such an auth config's method type was related to HTTP[S] connections, then
calls to the manager might update the connection's QNetworkRequest as
necessary. The details on how or why the method handles the update to network
objects are abstracted, with regards to the integrated routine creating the
network connection. Such a routine merely calls the manager when any updates to
network objects might need to happen.

User interaction with the auth system would be done via GUI widgets for:

• inputting the master password in a cross-thread manner (and via console)
• maintaining the auth configs in the database
• creating/editing individual auth configs, based upon ID
• selecting existing auth config IDs to associate with resources/servers

Here is example user documentation (from Boundless) for a PKI workflow, with an overview of the authentication system <https://github.com/dakcarto/QGIS-Enhancement-Proposals/blob/auth-system/extras/auth-system/pkiuser.rst>_.

.. _PR: qgis/QGIS#2330

3. Implementation Details

This describes the completed implementation in the current PR. Additional
possible refactorings are listed in 6. Further Improvements_.

The authentication database defaults to being created at::

~/.qgis2/qgis-auth.db

and contains two tables, one for the master password hash (single row only) and
another for auth configs. This is generated upon authentication manager's
init(), if it does not already exist.

Upon command line launch of qgis, the user can define::

--authdbdirectory | -a "path to directory for authentication database"

into which a qgis-auth.db will be created, if it does not already exist.

Alternatively, you can define the QGIS_AUTH_DB_DIR_PATH environment
variable, which has the same effect as the --authdbdirectory option.

3.1 Added Dependencies
......................

Adds required build/package dependency upon GPL2-licensed Qt Cryptographic Architecture_ (QCA) 2.0.3+, and a run-time dependency upon QCA's OpenSSL plugin
(qca-ossl). Latest QCA is 2.1.0, which builds on all major platforms using CMake
and includes all available plugins in the build process. A new FindQCA.cmake
module is added in the PR_ implementation, as well as build-time CMake
functions that check for qca-ossl if unit tests are to be built.

See PR_ for more details on building QCA 2.1.0. QCA's source repo browser_
at KDE.

QCA 2.1.0 is already compatible with Qt5_, while QCA 2.0.3 is not.

.. _Qt Cryptographic Architecture: http://delta.affinix.com/qca/
.. _source repo browser: http://quickgit.kde.org/?p=qca.git
.. _compatible with Qt5: https://projects.kde.org/projects/kdesupport/qca/repository/revisions/master/entry/README

3.1. Main Added Classes/Files
.............................

1. QgsAuthManager [src/core/auth/qgsauthmanager.h]

   • Singleton that oversees all master password and auth database functions and
     marshalling of auth methods
   • Instantiates in QgsApplication::initQgis() and cleans up in
     QgsApplication::exitQgis()

2. QgsAuthCrypto [src/core/auth/qgsauthcrypto.h]

   • Simple interface for hashing/verifying master password and encrypt/decrypt
     operations on auth configs with master password.
   • Currently uses QCA, though originally designed for CryptoPP, which was
     found to be way too finicky to build on Windows, especially for
     non-devs.

   .. _CryptoPP: http://www.cryptopp.com/
   .. _build on Windows: http://www.codeproject.com/Articles/16388/Compiling-and-Integrating-Crypto-into-the-Microsof

3. QgsAuthMethodConfig* [src/core/auth/qgsauthmethodconfig.h]

   • Class representing auth method configs
   • Has public properties that can generally be queried without requiring the
     user to input the master password
   • Has sensitive properties that become semi-public once the master password
     is set/verified and the config has been retrieved and decrypted from the
     auth database by the auth manager
   • Has sensitive properties that can be set and then encrypted and stored in
     the auth database by the auth manager

4. QgsAuthMethod [src/core/auth/qgsauthmethod.h] and QgsAuthMethodEdit

   • Class and edit widget that comprise an auth config method plugin
   • Each method accepts marshaled calls from the auth manager to update
     authentication-specific objects when needed, e.g. QNetworkRequest and
     DataSourceURI, during resource connections
   • Each method has an in-memory cache of authentication objects, generated
     during the processing of an auth config, that are stored upon first
     access/load of the config. Subsequent calls use the cached resource, e.g.
     generated SSL certificate, key and CA chain objects.

5. QgsAuthMethodRegistry [src/core/auth/qgsauthmethodregistry.h] and
   QgsAuthMethodMetadata

   • Singleton plugin registry modeled after QgsProviderRegistry
   • Loads plugins with lib*authmethod.(so|dll) name pattern

3.2 Main Added GUI Classes
..........................

1. Master password input dialog [src/gui/qgscredentialdialog.h]

   User is prompted whenever accessing the auth system, or whenever a layer is
   loaded/dragged/programmatically added that has an associated authcfg.

  When master password has not been set, nor its hash stored in auth
  database. **The master password can NOT be retrieved if the user looses
  it.**

  After master password has been configured and 3 incorrect attempts

1. QgsAuthConfigEditor [src/gui/auth/qgsauthconfigeditor.h]
   • An embeddable or standalone widget for directly managing auth configs in
     the auth database
   • Uses QSqlTableModel for its QTableView model
   • Offers utility functions for managing the auth database and master password

1. QgsAuthConfigEdit [src/gui/auth/qgsauthconfigedit.h]
   • An embeddable or standalone widget for creating/editing auth configs
     directly in the auth database
   • Depending upon method, does lightweight validation, e.g. cert issue dates

  Standalone config creation

  Standalone with existing config in edit mode

1. QgsAuthConfigSelect [src/gui/auth/qgsauthconfigselect.h]
   • An embeddable or standalone widget for selecting/adding/editing/removing
     auth configs in the auth database

  Standalone with no selection defined

  Integrated in WMS connection dialog, with config defined

1. Sundry GUI classes
   • QgsMasterPasswordResetDialog Embeddable or standalone widget for
     resetting master password and re-encrypting auth configs into a new auth
     database, with optional backup of old database (no Python binding)
   • QgsAuthGuiUtils Utility functions for managing the auth database and
     master password, and passing any messages to user via QgsMessageBar

3.3 PKI/SSL-related Added Classes
.................................

There are several editor widgets and included functionalities within QgsAuthManger
that support PKI and SSL authentication and configuration.

1. QgsAuthAuthoritiesEditor [src/gui/auth/qgsauthauthoritieseditor.h]
   • Manager for system Certificate Authorities and their trust policies
   • Offers ability to add new CAs from file or load into database
2. QgsAuthIdentitiesEditor [src/gui/auth/qgsauthidentitieseditor.h]
   • Manager for personal certificate/key PKI components stored in the database
3. QgsAuthSslErrorsDialog [src/gui/auth/qgsauthsslerrorsdialog.h]
   • New SSL errors dialog that allows saving of server certificate exceptions
   • User can review the SSL certificates associated with the error's connection
4. QgsAuthServersEditor [src/gui/auth/qgsauthserverseditor.h]
   • Manager for SSL server certificate configurations, e.g exceptions.

3.4 Authentication Methods Plugins
..................................

In the PR_ there are three auth method plugins. Additional plugins are now
easy to make and add. The system can also be updated to allow methods to be
registered via general application plugins (C++ or PyQGIS).

The current plugins in [src/auth] are:

Basic

• Basic username/password for HTTP[S] connections and database credentials

PKI-Paths

• Pure Qt SSL code, does not use QCA (CryptoPP was going to be lib supporting
  crypto functions in original implementation)
• Supports CA and client certificate/key in PEM or DER format (PEM is Qt native)
• Client key can be passphrase-protected

PKI-PKCS#12

• Uses QCA
• Supports client certificate/key bundles in .p12 or .pfx formats
• Bundle should not include any signing (CA) cert chain (this can be supported)
• Bundle can be passphrase-protected

Identity-Cert

• Uses PKI cert/key imported from PEM or PKCS#12 files into Identities table
• Uses QCA for importing PKCS#12

Adding more methods requires subclassing an existing base or method and
adding any new virtual functions to the base class that will handle new means of
applying authentication to integrated code elsewhere in the code base. Any new
virtual functions will need a single, similar marshaling function in the auth
manager to provide an abstracted call based solely upon the authcfg.

3.5 TLS/SSL Connection Authentication
.....................................

QGIS leverages QNetworkAccessManager_ via a custom subclass
QgsNetworkAccessManager for managing most network connections. This is a
higher level manager and does not offer a good means of responding to a TLS/SSL
server when it requests that a client provide a certificate for authentication,
e.g. like when Firefox prompts you to select a client cert from its embedded
cert manager in the middle of a connection.

A better TLS/SSL connection solution might be to use Qt's QSslSocket_ or
QCA's TLS class_. QCA::TLA has a nice certificateRequested signal.
However, implementing a new TLS/SSL client socket is beyond the scope of this
QEP/PR.

Instead, I chose to take a 'pre-configure' approach, where users need to define
auth configs for TLS/SSL connections prior to a server asking for the client
cert. This should generally not be an issue, though QGIS will not act
Web-browser-like in this regard.

Auth configs currently have an unused 'resource URI' property, which was
designed to be utilized later via a custom user-selected auth config type, e.g.
"Select configuration based upon URI", once support for that is
implemented. Such a feature would mitigate some of the annoyance of the
'pre-configure' approach.

.. _QNetworkAccessManager: http://qt-project.org/doc/qt-4.8/qnetworkaccessmanager.html
.. _QSslSocket: http://qt-project.org/doc/qt-4.8/qsslsocket.html
.. _QCA's TLS class: http://delta.affinix.com/docs/qca/classQCA_1_1TLS.html

3.6 Python Bindings
...................

All classes and public functions have sip bindings, except QgsAuthCrypto,
since management of the master password hashing and auth database encryption
should be handled by the main app, and not via Python.
See 5. Security Considerations_ concerning Python access below.

4. Test Coverage

Most coverage is provided by the current unit tests. However, a heavy amount of
functional testing was done for the OWS integration against current real-world
installs of PKI/SSL-enabled GeoServer installs (see PR_ for pre-configured
GeoServer test install). Those functional tests can be supplanted with automated
tests set up against a pre-configured lighttpd instance, which spawns
QGIS Server via FCGI.

5. Security Considerations

Once the master password is entered, the API is open to access auth configs in
the auth database, similar to how Firefox works. However, in the initial
implementation, no wall against PyQGIS access has been defined. This may lead
to issues where a user downloads/installs a malicious PyQGIS plugin or
standalone app that gains access to auth credentials.

The quick solution for initial release of feature is to just not include most
PyQGIS bindings for the auth system.

Another simple, though not robust, fix is to add a combobox in
Options -> Authentication (defaults to "never")::

"Allow Python access to authentication system"
Choices: [ confirm once per session | always confirm | always allow | never ]

Such an option's setting would need to be saved in a location non-accessible to
Python, e.g. the auth database, and encrypted with the master password.

Another option may be to track which plugins the user has specifically allowed
to access the auth system, though it may be tricky to deduce which plugin is
actually making the call.

Sandboxing plugins, possibly in their own virtual environments, would reduce
'cross-plugin' hacking of auth configs from another plugin that is authorized.
This might mean limiting cross-plugin communication as well, but maybe only
between third-party plugins.

Another good solution is to issue code-signing certificates to vetted plugin
authors. Then validate the plugin's certificate upon loading. If need be the
user can also directly set an untrusted policy for the certificate associated
with the plugin using existing certificate management dialogs.

Alternatively, access to sensitive auth system data from Python could never be
allowed, and only the use of QGIS core widgets, or duplicating auth system
integrations, would allow the plugin to work with resources that have an
authcfg, while keeping master password and auth config loading in the realm
of the main app.

The same security concerns apply to C++ plugins, though it will be harder to
restrict access, since there is no function binding to simply be removed as with
Python.

6. Further Improvements

General auth system improvements to be considered (no particular order):

• Have security guru/firm audit implementation (I'm no guru)
• Integrate auth system with more database connection configs
• Integrate auth system with Plugin Manager connections
• Integrate auth system with all HTTP connection classes
• Integrate auth system into QGIS Server, where user prompts are not supported
• Integrate master password entry with platform-specific password managers, so
  user does not need to enter it and automated pre-population scripts are easier
  to code
• Finish implementation of auth config 'resource' (auto-auth via matched
  resource URI)
• Try moving auth system integration code (when authcfg has been assigned) into
  QgsNetworkAccessManager instead of within individual data providers and
  pass authcfg to network manager
• Warn user when secure parts of auth system are accessed by Python (see
  above)
• PyQGIS plugins may need sandboxed to protect against, and selectively allow,
  access to auth configs
• Migrate all 'core' PyQGIS plugins that need to manage their own connections to
  using core auth system calls and/or core connections (which would auto-manage
  auth system calls), or isolate them from third-party plugins, if possible
• Switch master password memory allocation from QString (insecure) to
  QCA::SecureArray
• Add conversion button, to convert existing plain auth to auth config
• Add optional ability to edit the auth id for a configuration (user must
  confirm before operation)
• Add ability to change, edit or remove authcfg from existing layer in
  properties dialog
• Add simple read-only text field in layer properties dialog to quickly copy
  authcfg
• Add copy/paste/add/edit/remove of authcfg in layer contextual menu in Legend
  panel
• On layer load, notify user if any associated authcfg is missing in auth
  database
• Add layer authcfg (re)assignment in Handle Bad Layers dialog (can currently
  edit URI)
• Add authcfg as attribute of base QgsMapLayer class, so it can be queried
• Add checkValidity(bool verbose = false) to auth methods that emits
  messages
• Add Test Connection functions/buttons and connection debug dialog
• Add support for no-master-password encryption (or never add this?)
• Better auth system integration for all browser dock functions
• Find means of clearing cached connections in QgsNetworkManager

Specific to PKI and SSL certificate management:

• Add auth method for directly accessing user's OS-specific cert store
  (problematic if client key has passphrase on Windows)
• Better cert/key/trust chain validation in edit widget
• Check for expired/invalid cert/chain prior to connection

7. Restrictions

The confusing licensing and exporting issues_ associated with OpenSSL apply.
In order for Qt to work with SSL certificates, it needs access to the OpenSSL
libraries. Depending upon how Qt was compiled, the default is to dynamically
link to the OpenSSL libs at run-time (to avoid the export limitations).

QCA follows a similar tactic, whereby linking to QCA incurs no restrictions,
because the qca-ossl (OpenSSL) plugin is loaded at run-time. The qca-ossl
plugin is directly linked to the OpenSSL libs. Packagers would be the ones
needing to ensure any OpenSSL-linking restrictions are met, if they ship the
plugin. Maybe. I don't really know. I'm not a lawyer.

The auth system safely disables itself when qca-ossl is not found at
run-time.

.. _licensing and exporting issues: http://www.opensslfoundation.com/export/README.blurb

8. Backwards Compatibility

The proposed auth system causes no regressions nor backwards incompatibility.
With its initial PostGIS and OWS connection support it is side-by-side with the previous
username/password form widget in connection setup forms, allowing any existing,
older configurations to continue to work. Likewise, in the QNetworkRequest
and DataSourceURI updates, the new auth system configuration is only
prioritized once it has been utilized.

9. References

• Qt Cryptographic Architecture <http://delta.affinix.com/qca/>_
• QCA API Docs <http://delta.affinix.com/docs/qca/index.html>_
• QCA and Qt5 <https://github.com/JPNaude/dev_notes/wiki/Using-the-Qt-Cryptographic-Architecture-with-Qt5>_
• Salted Password Hashing - Doing it Right <http://www.codeproject.com/Articles/704865/Salted-Password-Hashing-Doing-it-Right>_
• QNetwork Module (SSL classes) <http://qt-project.org/doc/qt-4.8/qtnetwork.html>_

10. Voting History

(required)

QGIS Enhancement: NULL data symbol in graduated style

Date 2016/05/18

Author Adam Old (@adamold)

Contact [email protected]

maintainer @user

Version QGIS 2.14.2

Summary

When styling data as a chloropleth, it is important to be able to distinguish between measurements of 0 and NULL. There should be an easy option to add a null style.

My Section

Proposed Solution

add small button that would insert a style for either "other" falling outside the data range, or NULL, or ideally, both.

Example(s)

Affected Files

not sure

Performance Implications

none?

Further Considerations/Improvements

Backwards Compatibility

?

Issue Tracking ID(s)

(optional)

Votes

(required)

QGIS Enhancement 46: Labeling path

Date October 2015
Author Hugo Mercier
Contact hugo dot mercier at oslandia dot com
Last Edited
Status
Version

Summary

A new labeling option is proposed for addition here. It allows two things :

• an option to smooth or simplify geometries on the fly before doing curved labels.
• to use a user-defined linear path for the drawing of a label. A new data field would be necessary to store that path. That geometry field will require a new datadefined widget in labeling properties.

A user-defined path can be seen as an additional geometry on a layer, where the underlying model supports only one "geometry column" (see qgis/qgis4.0_api#21 )

Proposed solution

We propose the add a way for the user to use a different geometry object as support for the "curved" placement of a label.

The current version of QGIS allows to display labels along the linestring features in a "curved" manner.

However for linestrings defined with lots of control points (such as waterways) or with high curvatures, displaying correct labels can be either anaesthetic or very slow.

This is where using a path slightly different from the feature's geometry could be useful. For instance using a simplified version of the linestring could lead to a label faster to display or with smaller curvatures.

We then propose to allow the "curved" placement to use an arbitrary geometry, through a data-defined "label path" property. The data-defined property could point to a user-defined column or to an expression.

We also propose to add simplification / smoothing functions in core expression functions for that purpose.

This option for placement along a user-defined path would also be available for other types of features, like points and polygons.

Core Changes

A new data-defined labeling property of type QgsGeometry is added for the definition of the label path ("labeling/dataDefined/labelPath").

This data-defined property could be of type string or QgsGeometry. String values will be considered as the string representation of a WKB.

QgsPalLayerSettings::registerFeature will be modified in consequence: if this new property is set and curved placement is enabled, it will be used as a support.

New simplification functions will be available in expressions.

• simplify( geometry, tolerance ): Douglas Peucker
• (optional) visvalingam( geometry, tolerance): Visvalingam’s algorithm
• (optional) bendSimplify( geometry, tolerance): Wang Müller's algorithm

New smoothing functions will be available in expressions:

• smooth(geometry, iterations) : Chaikin subdivision algorithm (will smooth the curve, but increase the number of points)

GUI Changes

A new tool will be available in the labeling toolbox for point, linestring and polygon layers. This tool will allow to draw a user-defined label path. It will be enabled only if the label path property is defined from an editable field (not an expression).

A checkbox will allow to automatically enable on the fly smoothing for curved labels placement (shortcut to the definition of an expression-based label path)

Performance Implications

For the use case presented above, where the user wants to use an expression like "simplify($geometry)" as a label path, computing the simplification of the feature's geometry each time the label is drawn could be time consuming.

In this case, using an additional geometry field where the simplified geometry is stored (or multiple geometry fields for multiple scales) will be needed.
Providers supporting multiple geometry columns (PostGIS, Spatialite) will be modified to return QgsGeometry instead of WKT strings.

Moreover simplification / smoothing of a line needs to be done only on the visible part of the geometry on the current extent. Simplficiation / smoothing functions should be aware of the current visible extent for possible optimizations.

Test Coverage

Unit tests will be added to the core part to ensure the label path feature works correctly.

Voting History

QGIS Enhancement 30: Required unit testing for "critical" classes

Date 2015/10/20
Author Nyall Dawson
Contact nyall dot dawson at gmail dot com
Last Edited 2015/10/20
Status Accepted

Summary

This QEP describes a system of classifying certain classes within the QGIS codebase as critical
classes and proposes that no changes be allowed to these classes without accompanying unit
tests.

While QGIS contributor guidelines state that all changes to code within CORE be accompanied
by unit tests, this requirement is universally ignored. However, since the introduction of CI
testing of every commit and pull request to master the value of writing tests to prevent
regressions has tremendously increased. It is acknowledged that introduction of unit tests
does not automatically fix bugs, but it does help drastically reduce the potential of regressions.

This QEP describes a middle-ground approach to requiring compulsory unit tests for certain areas
of the codebase without placing too much additional burden on developers.

It is based on a well-defined minimal required test suite, and does not prevent
developers from writing additional tests or requiring tests for classes not
covered by this QEP.

Proposal

Any change to a class which is classified as "CRITICAL" MUST be accompanied by
unit testing to prevent regressions.

Pull requests which touch upon CRITICAL classes and which do not have unit
tests must NOT be merged to master. Commits pushed to master which
also violate this may be reverted if no suitable follow-up unit tests are added following discussion/consultation with the original committer.

Definition of CRITICAL classes

For a class to be classified as a CRITICAL class it must satisfy two requirements:

The class must relate to code which has:

• Potential for data loss for users
• Risk of incorrect calculations resulting in invalid analysis (eg, incorrect area
  or distance calculations)

This excludes code which relates to purely cosmetic components, eg symbology,
labeling, conditional styles, etc. The requirement is intended to identify areas
where regressions would result in significant harm to the reputation of QGIS
or potential risk of incorrect analysis products generated by QGIS and the wider
risks this entails.

The class must have 100% unit test coverage (or as close as possible)

This requirement is intended to reduce the workload on developers. While it is
relatively straightforward to adopt existing unit tests to cover new code, it
can be quite burdensome or complex to start creation of unit tests from scratch.
Requiring all changes to classes which satisfy 2.1.1 to have unit tests would likely
push away potential contributors who do not have the time or skill required
to write unit tests from scratch.

Accordingly, a class can only be categorised as CRITICAL when it already has
100% existing unit test coverage.

Proposed Technical Solution

When a class is categorised as CRITICAL, comments will be added to the source
code to warn developers that all changes must be accompanied by unit tests.
These would take the form of the line:

// CRITICAL code - changes must be accompanied with unit tests in testqgsfield.cpp

placed within the header, and between each function definition in the class cpp file.

The contributor guidelines would be updated to reflect this requirement.

For a class to be categorised as CRITICAL, a pull request will be opened which
includes the CRITICAL comment blocks for the class. The PR can then be used
for discussion regarding whether or not the class satisfies the requirements
for a critical class, specifically whether it satisfies 2.1.1.

Implementation Details

Initially, only the QgsField, QgsFields and QgsFeature classes would satisfy
the requirements for CRITICAL classes. It is the intention that the following
classes be prioritised to bring them up to CRITICAL status ASAP:

• all geometry classes
• QgsExpression
• QgsDistanceArea
• QgsStatisticalSummary

Voting History

+1

Andreas Neumann
Anita Graser
Otto Dassau

0
Paolo Cavallini
Richard Duivenvoorde
Marco Hugentobler

Other comments:

Andreas Neumann
If it helps to maintain/raise quality - I will support this QEP. Devs should be supportive, if a PR concerning core classes misses unit tests and the dev is not so familiar with unit tests. At least in the transition phase.

Jürgen E. Fischer
Generally I’m in favor of this. But the passage “and commits pushed to master which also violate this will be reverted.” is a nonono for me. Missing tests could probably be easily added with another commit - and reverting the commit in question IMHO isn’t necessary.
I think that this will probably (/hopefully) never happen in practice anyway and resolved by giving a dev in question a little nudge instead. But this is all about strict rules, so we would avoid rules that we don’t actually follow.

QGIS Enhancement 52 (was 33): Commit rule for new feature proposal

Date 11-06-2015
Author Hugo Mercier
Contact hugo dot mercier at oslandia dot com
Last Edited
Status
Version

Proposed change

This QEP is about a new rule for the commit of code change for a new feature.

The rule is the following: anyone proposing a patch that introduces a new feature, including developers with write access, must send it as a Pull Request to the QGIS github project.

In case the Pull Request is opened by a core developer, it is required (s)he:

• either requests a review (by tagging it)
• or includes a "merge date" information, delay after which (s)he intends to merge the changes

This "quarantine" delay allows other developers to be aware of the proposed changes and react and points problems if any. The delay must represent at least 2 (?) days, and should reflect the time needed to read it. The more important the proposed changes are, the longer the delay.

If accepted, the CODING document will be updated to reflect this new rule.

Voting History

QGIS Enhancement: Title

Date 2016/05/14

Author Adam Old (@adamold)

Contact [email protected]

maintainer @user

Version QGIS 2

Summary

Follow UI standards set by Adobe and other graphics software for navigating large documents while zoomed in and manipulating objects. In this case, improve navigating the map while using the "select features" tool or other tools.

My Section

(optional) Insert custom sections wherever needed

Proposed Solution

Press and holding spacebar should switch to the pan tool while spacebar is depressed. This allows immediate access to panning without resetting tool. This is a common solution, and is widely used by those with graphics background.

Example(s)

see photoshop/illustrator

Affected Files

none attached

Performance Implications

none

Further Considerations/Improvements

none

Backwards Compatibility

no effect on files.

Issue Tracking ID(s)

?

Votes

QGIS Enhancement 43: Rename Compositions to Print Layouts

Date 2014/10/17
Author Nyall Dawson
Contact [email protected]
Last Edited 2014/10/16
Status Draft
Version QGIS 3.0

Summary

"Compositions" in QGIS should be renamed to "layouts". This QEP relates
solely to an in-principle user visible rename of print composer, not to
any specific implementation or api changes.

Rationale

This change is being proposed for a number of reasons:

• The term "composition" is an awkward name for this feature and does not
  accurately reflect what this feature does. "Layout" is a much more widely
  used term in desktop publishing and consequently a rename would improve
  UX for new users who are unfamiliar with QGIS terminology. While existing
  QGIS users are familiar with the term and may not see an issue with it,
  new QGIS users often struggle to equate "composer" with creation of a
  printable map page.
• Other desktop GIS software packages (ArcGIS, MapInfo) use the term
  "layout" to refer to print designs. A rename to match this would aid
  the transition from these packages to QGIS for new users.
• The 3.0 release will see a major 'ground-up' rebuild of composer. This
  represents the perfect opportunity to rename composer as the new classes
  can all be named accordingly. A rename will also help users recognise that
  behaviour may differ from the composer in previous releases.
• During the 3.0 cycle work is planned on a reporting framework for QGIS.
  This framework would build off the existing composition features. While
  the term "print composition" works to a degree, the phrase "report
  composition" sounds horrible. "Print layout" and "Report layout" are much
  friendlier sounding names for these features.

Potential issues

• Documentation would need to be updated
• Existing users would need to be made aware of the change in name
• Existing blogs/stack exchange/mailing list answers would be outdated

Implementation Details

• User visible strings will be renamed for QGIS 3.0:
  • "Composition" -> "Print Layout"
  • "Print Composer" -> either "Print Layout" or "Layout Designer", depending
    on the context. Currently "Print Composer" is used inconsistently.
  • "Composer" -> "Layout Designer"
  • "Composer Manager" -> "Layout Manager"

Documentation

Documentation would need to be updated to suit.

Issue Tracking ID(s)

This proposal was originally filed as issue #5042.

Voting History

(required)

QGIS Enhancement 37: Fields And Forms Redesign

Date 2014/11/25
Author Matthias Kuhn
Contact matthias dot kuhn at gmx dot ch
Last Edited 2014/11/25
Status Draft
Version QGIS 2.8

1. Summary

The tab Fields on the vector layer properties dialog should be split into
two different tabs that separate data-related from form- and widget-related
user interface elements.
This is expected to improve user experience and usability.

2. Rationale

On the vector layer properties dialog there is a tab Fields where the user is
able to configure different things related to fields (data- and widget-related)
and forms (data- and form-related):

• The general layout of the form (autogenerated, drag and drop, ui file)
• The appearance of the individual widgets
  • Some weak constraints can be introduced that will limit acceptable input
    from widgets but will not be enforced by any other way to set a field's
    value (e.g. min/max values from the range widget)
• Adding fields and removing fields
• Drag and drop fields and relations to the form
• Python logic

This tab currently is in a very bad shape from a UX perspective. Problems
include:

• Having to scroll horizontally in the fields table when wanting to change
  the widget type or looking for information.
• The resizable containers on the right and at the bottom can be hidden
  (consciously and unconsciously) and confuse the user.

3. Proposal

The information and functionality that is now available on the Fields tab
will be split and put on two different tabs:

• Fields
• Form

3.1 Fields

The Fields tab conatins anything that is related to data and fields.

In particular, this refers to:

• Changing fields
  • add
  • delete
  • calculate
• Information about fields
  • data type
  • comment
  • constraints (Future)
  • domain (Future)
  • exposure (WMS/WFS)
  • ...

Mockup

.. figure:: fields.png

Mockup of the fields tab. This will be shown embedded in the vector layer properties dialog.

3.2 Forms

The Form tab contains anything that is related to user interface and visual
representation and manipulation of a features fields.

In particular, this refers to:

• The widget type
• The widget configuration
• Form layout
• Python init script for the form
• ...

Mockups

.. figure:: form1.png

Mockup of the Form tab with autogenerated layout. This will be shown embedded in the vector layer properties dialog.

.. figure:: form2.png

Mockup of the Form tab with drag and drop designer layout. This will be shown embedded in the vector layer properties dialog.

.. figure:: form3.png

Mockup of the Form tab with ui file layout. This will be shown embedded in the vector layer properties dialog.

4. Scope

Weak constraints are currently defined in the widget properties. It is outside the
scope of this QEP to move constraints to the fields tab. The sole purpose of the
constraints part of the UI in the mockups is to highlight where I immagine this could
eventually lead to.
For the moment, the currently available weak constraints will still be defined on
the Form tab.

5. Documentation

Documentation would need to be updated to suit.

6. Voting History

(required)