@davidoesch so far for the attribute "GROUP" you're setting the closest parent of a layer. Would it be possible to set this to empty is the closest parent is the root of the entire hierarchy?

Currently the service link contains some GetMap approach

do offer a solution that you can COPY the service, a GetCap URL would be nice. this is actually the source (with the version which is retrieved in the scraper.py)

Which information is available in most services? Based on this define attributes to scrape

To improve the discoverability: any chance to retrieve the all getFeature Variable names?

• recognize multilang services
• detect language
• translate using python libs like googletrans or deepl into missing lang DE FR IT EN

https://ckan.opendata.swiss/api/3/action/package_search?fq=res_format:WMS&rows=10000
https://ckan.opendata.swiss/api/3/action/package_search?fq=res_format:WFS&rows=10000
https://ckan.opendata.swiss/api/3/action/package_search?fq=res_format:WMTS&rows=10000

or
https://ckan.opendata.swiss/api/3/action/package_search?fq=organization:kanton-thurgau%20AND%20res_format:WMS&rows=10000

etc

Following https://www.jcchouinard.com/google-indexing-api-with-python/ add url to google. Or serve even geocat url ...

Add as well WFS ( can't be shown in map.geo.admin.ch)
is supported by OWSlib

falsche assignments (bzw. comparisons statt assignments)

.B. auf der zweitletzten zeile. im Beispiel ist version bei KeyError None. die zweitletzte zeile ergibt ein stills True. der return retourniert None. Sprich: macht hier glaub's nix aus.
btw, Vergleiche mit None sollte man wie folgt machen:

if x is None:

(aber auch anderes Zeug als None kann halt zu False evaluated werden, daher ist beim 2. viel vorsicht geboten)

thanks @rastrau for raising the issue

add new fields and new urls per service

write json to achieve an external topic geoadmin/mf-geoadmin3#4668 including search

    layer_data["CENTER_LAT"]=(service[i].boundingBoxWGS84[1]+service[i].boundingBoxWGS84[3])/2
    layer_data["CENTER_LON"]=(service[i].boundingBoxWGS84[0]+service[i].boundingBoxWGS84[2])/2
    layer_data["BBOX"]=' '.join([str(elem) for elem in (service.contents[i].boundingBox)])

-> check for WGS84 by default

Das csv ist zu gross
DB Lösung?
DuckDB?

based on abstract title

@davidoesch For some layers you're storing in the field "ABSTRACT" the text coming from the service abstract (I assume you do it when abstract at dataset level is missing).
Could you please fix this?
Abstract at dataset level is missing/empty --> Field "ABSTRACT" is empty

In source.csv:

"KT_AI","https://www.geoportal.ch/services/wms/ktai?SERVICE=WFS&REQUEST=GetCapabilities"
"KT_AR","https://www.geoportal.ch/services/wms/ktar?SERVICE=WFS&REQUEST=GetCapabilities"

Are not WFS services.

based ona a get map request check performance

Scrape STAC

Some layers are missing. To test set in source.csv only this url: https://geodienste.ch/db/lwb_rebbaukataster/deu?service=WMS&request=GetCapabilities&version=1.3.0
It should return 6 layers instead of 5.
The missing layer is: lwb_rebbaukataster

using NLP on abstract name title, try to create tags/categories

There are several NLP Python libraries that can be used to analyze text and add it to a predefined group. One popular library is the Natural Language Toolkit (NLTK).

NLTK provides a wide range of tools for natural language processing, including text classification. You can use NLTK's nltk.classify module to train a classifier on a dataset of labeled text, and then use the trained classifier to classify new text into predefined groups.

Another popular library is the scikit-learn, it's a machine learning library for Python, it provides various tools for natural language processing, including text classification. With the sklearn.feature_extraction.text.CountVectorizer and sklearn.feature_extraction.text.TfidfVectorizer classes, you can convert a collection of text documents to a matrix of token counts (or TF-IDF values) that can be used as input for a classifier. The sklearn.naive_bayes.MultinomialNB , sklearn.svm.SVC and sklearn.linear_model.LogisticRegression are some of the classifiers provided by scikit-learn which can be used for text classification.

You can also use other libraries such as spaCy, TextBlob, Gensim, etc, they all have their own features and capabilities which you can use to classify text into predefined groups.

It is important to note that, before using these libraries, you need to have labeled data to train the classifier and also preprocess the text data.

try to link to Metadata UUID of geocat for each layer, try to find the geocat entry for each service in the scraping process

Which fields in service configuration as an operator I do need to fill out?

Eg link to metadata, multilang, legend ( not only symbol but also description)
Etc

@davidoesch As far as I can see for the field "KEYWORDS" you are concatenating keywords at service level with keywords at dataset level.
This should be avoided. Could you please fix it?

see eg EMCWF datasets are all empty

Currently, def get_service_info(source) dos not assume the version paramter of OWS lib geopython/OWSLib#840 (comment)

so we need to check the availabe version
Scrapr.py is no hardcoed for the geoinfo services like KT AI and KT AR

Decide if TITLE or NAME makes sense to use in the MAP GEO description link.

Google and search all official and unofficial cantonal WMS WMTS WFS and try to add them