googlefonts / fontbakery-dashboard Goto Github PK

Originally submitted at fonttools/fontbakery#1264

I'd like to have a status page showing, for google/fonts, and for each repo upstream of it, and for all,

• what is the rate of new issues reported
• what is the rate of issues being closed
• what is the rate of new PR opened, passing all tests
• what is the rate of new PR opened, failing some test
• what is the rate of PRs merged

and have this per day/week/month and arbitrary times.

@m4rc1e developed a script to help divine some of this, so turning that into a web page is on the roadmap for sure :)

Originally submitted at fonttools/fontbakery#1297

MS-FontVal is not currently running on the worker container.

@graphicore There's an issue (and there will be a commit closing it pretty soon) here (fonttools/fontbakery#1336) that will provide some clues on how to set up the Microsoft Font Validator on the kubernetes cluster.

@m4rc1e said: "Hey @felipesanches ,

I think it's wise we stick to using Material Design for the UI."

To which I replied: "Yup! I think it's great."

And @davelab6 tipped at: "Material Design Components?"

And then said: "This is low Priority to me"

See fonttools/fontbakery#1311 and file issues based on those ideas

Originally submitted at fonttools/fontbakery#1237

We do collect all the data needed to build a burndown chart (and there's been some prototypes of that in the past) but right now all we show in the dashboard UI is info tagged with "HEAD": true in the database, meaning that we're just displaying the check results for the latest commit in the git repos.

Since we have the data, I think it would be wise to query that and render the chart. The best place to showcase the graph, in my oppinion, would be at the top of the family details page, where nowadays we only render the check-results summary pie-chart as seen in the screenshot below:

@davelab6: "Agreed"

It costs too much to run FB on the full commit history of projects, so nowadays we are only checking against the repo HEAD commit.

In order to make it acceptable to keep the full history, we would need to detect whether re-running the checks on a specific commit is really necessary. The criteria for deciding that would be: (a) the specific commit was never checked previously or (b) it was checked with an older version of fontbakery so it would be needed to re-run the checks and update the database entries.

https://letsencrypt.org/

We need a domain for this.

Not sure about inter-cloud communication, as long as we don't send something confidential it should be fine.

Originally submitted at fonttools/fontbakery#1235

Red is typically associated with "problem". It would be good to change the color-scheme of the generated pie-chart so that the "OK" check-results are represented by a portion of the pie coloured in green. While "ERROR" results should use the red color, and maybe blue for "INFO" and yellow for "Warning" and soft-blue for "SKIP".

@davelab6: "Agreed"

Originally submitted at: fonttools/fontbakery#1267

Even though family-names should be ordered alphabetically by default ( #1266 ), it would be useful to allow the user to sort the list by progress status as well.

A very old issue on the fontbakery repo suggested using Guiders-JS to document the user interface:

https://github.com/pickhardt/Guiders-JS

I consider this to be an extremely low priority issue in the "maybe-someday" category. I'm just filling this here because the corresponding issue on the original issue tracker (fonttools/fontbakery#23) does not apply anymore to that repo, which is now focused on FontBakery-core only.

Following fonttools/fontbakery#1537 the dashboard should have a way to show users what languages are/not supported by a family that is being tracked by fbd. This should have all the burn down chart stuff being discussed elsewhere in this issue tracker.

Per google/fonts#1215 we should include the Hawaiʻian language

There is a github-release tool that can be used to automate tagging and uploading releases on Github... Someday would be great for the dashboard to make it easy for upstream repo owners to make font releases that way :)

Originally submitted at fonttools/fontbakery#1241

After fixing fonttools/fontbakery#1240, I'd like to increase the list of font projects git repos being checked by the dashboard's worker container.

It seems to me that the canonical list of font projects to check against should be whatever we currently have at prod. I know it is 800-something families, but I don't know where to find an official list of them all other than automatically querying Google Fonts itself.

Is there a similar list for the git repos for each of those? I was able to compile a list of a bit more than 104 repos by scraping the google/fonts repo for URLs referencing other repos. (the list is here: fonttools/fontbakery#1158 (comment)) This would be roughtly only 13% of all projects. Does it mean 87% of the projects simply lack a git repo? Or the repos are there and I simply don't know exactly where?

NOTE: Please look at the original issue thread because there are plenty of additional comments in there!

Originally submitted at fonttools/fontbakery#1246

It is accessible only via direct public IP address. I looked up how to setup the domain name, but couldn't get it. I'll have to try again later. Dave said we can use fontbakery.com (which he owns)

I have a hunch that setting this up involves some setting in the kubernetes web-panel, or via some kubectl command. I think Kubernetes is probably responsible for updating the DNS when a new service is brought up in a node (especially when the externally accessible IP address changes). But I may be wrong. I'm new to this thing.

This seems to be a good way to do it:
https://github.com/kubernetes-incubator/external-dns

www.snap-ci.com and www.travis-ci.org and probably many other sites have great homepages and login pages.

Ours should

• look great
• explain the project
• invite users to login via github oauth

(was fonttools/fontbakery#245)

Originally submitted at fonttools/fontbakery#1236

While we display all check-results (for the sake of completeness), it is likely that the users may be much more interested in looking at what went wrong. So a button (or checkbox) for toggling the visibility of "OK" check results could be very useful in the dashboard UI.

@davelab6: "Agreed"

Originally submitted at fonttools/fontbakery#1286

@m4rc1e said: "Since we can now see the health of the whole collection, how about we allow the dashboard to look at different forks? This will give us the ability to see if a pr will improve the collection before we merge. I'm working on a collection wide hotfix, google/fonts#641. This would be incredibly useful."

Then I said: "Yes, I agree.

There are some challenges to achieve that, though.

The current implementation of the worker container involves checking out the last N commits of the full git repo of an upstream project and it expects to have a single family per repo. Multiple families in a single repo are handled by having multiple entries on the fontprojects.csv for the same repo, with differing font_prefix values. Loading up additional containers and re-cloning the repo is not so bad since the full size of the upstream font project repos tend to not be very large.

Dealing with the google/fonts git repo (or forks of it) is trickier since that repo is huge! A different approach would be needed for this. We might try fetching raw files from the Github repo via HTTP requests? I tend to dislike this approach."

@m4rc1e replied: "I understand this is no small task.

so the current implementation currently runs on upstream repos or downloadable families from fonts.google.com?

I'm just delving into the worker container to see the current implementation."

@davelab6 said: "We probably should have more persistent containers, so that the cloned git repo isn't lost on each run"

To which I replied: "A general mechanism for configuring the dashboard is the CSV file on the dispatcher container. If someone needs to setup the dashboard to monitor a different set of families, that can be achieved by providing a different CSV file. But for that, each family must be hosted in its own git repo."

https://github.com/urtd/fontinspector

The key feature of the 0.1.0 release of the dashboard for me is to be able to say, this is the FB score of the fonts in X family in production, and this is the FB score of the fonts in the upstream.

To do that, we need to develop a complete list of upstream URLs for every family in google/fonts.

See fonttools/fontbakery#291 and fonttools/fontbakery#1262

It has a builtin type for binary data (e.g. font files) BytesValue.

Consider also using gRPC which itself uses protocol buffers.

Also, evaluate NoFlo vs gRPC and see if they are any good together.

Originally submitted at fonttools/fontbakery#1273

Expected behaviour

These checks from 001 until 015 are "cross family" ones. For that reason, they are not stored on the individual font file report json outputs. Instead, they're saved in a file called CrossFamilyChecks.json

Observed behaviour

There should be a tab for those cross family checks. But it doesn't show up on the dashboard family details page.

@davelab6 said: "Agree"

• The global footer should link to an About page

In an About page:

• This about page should be generated by a script, eg by parsing requirements.txt, so it doesn't have to be maintained by hand and is updated as things change

• This should also parse the AUTHORS.txt file and place the list of Font Bakery authors at the top of the page.

-[ ] All fontbakery dependencies should be listed, in a table of contents/index (perhaps using a bootstrap sidebar navbar, or material side content index, which are both context-indicating)

• For each dependency, include its license text and copyright notice

See fonttools/fontbakery#1332

look at @mcdurdin 's http://github.com/mcdurdin/render-lizard seems nice, a possible candidate for integration with the regression testing thing.

• For each textarea, use CodeMirror and the https://github.com/emmetio/codemirror plugin
• For each textarea, append a small note, that mentions Emmet is enabled with a link to the docs on www.emmet.io

(was fonttools/fontbakery#277)

References:

• http://testmyfont.com
• https://github.com/typefacedesign/document-driven-typedesign
• fonttools/fontbakery#238
• #23

fonttools/fontbakery#1579 said:

@rsheeter has requested today a efficient visual regression tool, and there are a few related tasks for this

• a list of all roll backs in the longest possible period, and the reason why the roll back occurred
• fonts/tools repo split out to its own repo, #1466 mentioned this need
• gfregression turned into diffinator and made suitable for presubmit checking, with no dependencies on flask or FontBakery itself

Originally submitted at fonttools/fontbakery#1228

"gather from git" runs on the cmd-line specified TTF files once per commit.

Current shortcomings:

• A) It does not cover more than one family stored in a single repo.

• B) It does not follow git renames (I considered implementing that. It would be neat. But I did not think it was the most important thing to do in terms of priorities)

@davelab6 said: "Given a git repo URL, git commit hash id, a fontbakery version are recorded in a central database of burndown results, it seems that we can have the gather tool cache it's findings and return early if it finds an existing result for the given repo hash with a version more up to date or equal to current version"

To which I replied: "Sounds great!"

Originaly submitted at fonttools/fontbakery#1298

Fontforge checks are not currently running on the worker container.

Fontforge instalation commands were added to the worker container image (via Dockerfile apt-get install statements) but the fontforge python module is not found at runtime.

https://cloud.google.com/docs/authentication/

See fonttools/fontbakery#322

This may really be a separate project, to be integrated

Dashboard should show metrics on how many problems and what types of problems we actually catch early with FB

Originally submitted at fonttools/fontbakery#1123

@m4rc1e said:

"Hey Felipe and Dave,

I think we should merge my gfregression app into Font Bakery. Instead of having lots of seperate repos/apps/libraries, we should focus on consolidating them.

It also fits FB's brief nicely

Tools to prepare font families for inclusion in github.com/google/fonts

I use it for cross platform testing a collection of fonts vertical metrics, against the same fonts which are hosted on fonts.google.com.

Left: local fonts | Right: fonts.google.com

I also see we are pushing for a UI for FB. I've built this app with Flask and Jinja2, keeping everything as simple as possible. I would opt for AWS or Digiocean over App Engine, due to our dependencies.

Cheers,
Marc"

To which @davelab6 replied: "Agree, but I would like Felipe to focus on getting the checker working first and then build this comparison feature in.

Thus moving this to 0.4.0"

• switch total/percentages (default percentages)
• switch static latest finished, daily/live progress

Historical Data: the state of this table a year ago, three years ago, ...

The live updating view: will be implemented after the static one.

Sources, here the rows, will be implemented from right to left. API first ...

The Upstream sources will likely be augmented with METADATA.pb etc. from google/fonts:master

There will be a drill down. A column here can be expanded to a full collection report table. The imploded column shows the value of FAIL.

example collection report table:

Originally submitted at fonttools/fontbakery#1239

We need to decide how the check results for the production font files will be displayed on the dashboard. And then I need to implement whatever we decide.

I understand that we'll probably want to have the same level of detail for production files as we have with upstream git-repo files. So Ideally we would simply populate the same page template we already have. On the other hand, we'll probably want to compare side-by-side production against upstream.

@davelab6 do we need to compare in the single-font-file/individual-checks level ?
I can see the value of being able to observe in a glance that a certain check fails on upstream while passing on production (or vice versa).

We may also want to see a comparison in the global level like displaying the progress bars for the check-results summary of prod and upstream side-by-side somewhere in the UI.

This issue is focused on brainstorming this and later implementing it. For tracking the task of implementing the actual server-side fontbakery run on prod files we already have fonttools/fontbakery#1238

Please also take a look at the original issue thread at fonttools/fontbakery#1239 because there are some considerations there from me and from @davelab6 !

Originally submitted at fonttools/fontbakery#1287

@m4rc1e said: "I'd like to have the dashboard built on top of api views. I can then query the api directly in my hot fix scripts."

https://litmus.com/email-testing is a nice reference for email testing

Font Bakery now has a new output format, thus, we need to redo the report saving and rendering part. This will improve real time reporting and we also have the opportunity to do parallel execution of tests, which would be nice to have:

Things that need to be done (most "at the same time")

• update worker to invoke the new Font Bakery
• change test report format in the database
• create a new test reporter for the new report format
• update database report-record on a per test base (or per message?) (== real time pushes to browser)
• adopt test report rendering (in the browser/javascript) to new style tests
• We should have just one Font Bakery worker for drag and drop, dashboard etc. so make it reusable
• enable (worker) distributed/parallel test execution (per process distributed on a multi-core cpu would be awesome as well!)
• for parallel execution one worker would act as the master and dispatch subset-jobs via AMQP.

Parallel execution can be done after all the other things, when done carefully it'll just work 😄

Other Important

• all the crazy dependencies need to be in the workers: msfontvalidator, pyfontaine, fontforge, Apple ftxvalidator (use an external OSX-server cloud provider?, ask Apple to make a linux compatible binary 😹 )
• clean up repository if already possible, better docs
• begin to plan dashboard (issues has a lot of feature requests)
• It would be awesome to have self-updating workers 👿 why do a docker push etc.pp if the worker could just run pip install itself (... ok, this is playing the implications down). It would also make it more straight forward to re-run tests. As when the repository changes, we need to re-run when the workers are updated, it's basically both input for us.
• configuring development/minikube and deployment/gcloud is tortuous

Originally submitted at fonttools/fontbakery#1230

Currently the drag-n-drop webapp hosted at http://fontbakery.appspot.com/ does not run FontForge checks (or any other checks that require usage of the subprocess module) due to secutiry restrictions on Google App Engine.

If we migrate the webapp to Google Container Engine, we'll be able to run these additional checks, improving the coverage of the reports generated by this webapp.

Note: There's a whole bunch of messages in the original issue tracker entry. Please take a look at those as well!

Originally submitted at fonttools/fontbakery#1289

The bad formatting we get currently looks like this:

http://35.184.229.186/details/Miltonian/errorlog/

WARNING:root:fontforge python module is not available! To install it, see https://github.com/googlefonts/gf-docs/blob/master/ProjectChecklist.md#fontforge
INFO:root:OK: Miltonian-Regular.ttf is named canonically
INFO     OK: Miltonian-Regular.ttf is named canonically  
INFO:root:OK: All files are in the same directory.
INFO     OK: All files are in the same directory.  
ERROR:root:Did not find cached vendor list at: /tmp/fontbakery-microsoft-vendorlist.cache
ERROR    Did not find cached vendor list at: /tmp/fontbakery-microsoft-vendorlist.cache  
INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): www.microsoft.com
INFO     Starting new HTTPS connection (1): www.microsoft.com  
ERROR:root:'Miltonian-Regular.ttf' is missing a METADATA.pb file!
ERROR    'Miltonian-Regular.ttf' is missing a METADATA.pb file!  
INFO:root:OK: All font files have the same version.
INFO     OK: All font files have the same version.  
INFO:root:OK: Miltonian-Regular.ttf opened with fontTools
INFO     OK: Miltonian-Regular.ttf opened with fontTools  
INFO:root:OK: Font has post table version 2.
INFO     OK: Font has post table version 2.  
INFO:root:OK: OS/2 fsType is a legacy DRM-related field from the 80's and must be zero (disabled) in all fonts.
INFO     OK: OS/2 fsType is a legacy DRM-related field from the 80's and must be zero (disabled) in all fonts.  
ERROR:root:[FULL_FONT_NAME(4):WINDOWS(3)] entry: expected 'Miltonian Regular' but got 'Miltonian'
ERROR    [FULL_FONT_NAME(4):WINDOWS(3)] entry: expected 'Miltonian Regular' but got 'Miltonian'  
WARNING:root:VendorID string 'PYRS   Fontlab Ltd. / Made with FontLab' does not match nameID 8 (Manufacturer Name): 'Pablo Impallari'
WARNING  VendorID string 'PYRS   Fontlab Ltd. / Made with FontLab' does not match nameID 8 (Manufacturer Name): 'Pablo Impallari'  
INFO:root:OK: OS/2 VendorID is 'PYRS' and registered to 'PYRS   Fontlab Ltd. / Made with FontLab'. Is that legit?
INFO     OK: OS/2 VendorID is 'PYRS' and registered to 'PYRS   Fontlab Ltd. / Made with FontLab'. Is that legit?  
INFO:root:OK: No need to substitute copyright, registered and trademark symbols in name table entries of this font.
INFO     OK: No need to substitute copyright, registered and trademark symbols in name table entries of this font.  
INFO:root:OK: OS/2 usWeightClass value looks good!
INFO     OK: OS/2 usWeightClass value looks good!  
INFO:root:OK: OS/2 fsSelection REGULAR bit is properly set.
INFO     OK: OS/2 fsSelection REGULAR bit is properly set.  
INFO:root:OK: post table italicAngle is 0.0
INFO     OK: post table italicAngle is 0.0  
INFO:root:OK: OK: post table italicAngle is less than 20 degrees.
INFO     OK: OK: post table italicAngle is less than 20 degrees.  
INFO:root:OK: post table italicAngle matches style name
INFO     OK: post table italicAngle matches style name  
INFO:root:OK: OS/2 fsSelection ITALIC bit is properly set.
INFO     OK: OS/2 fsSelection ITALIC bit is properly set.  
INFO:root:OK: head macStyle ITALIC bit is properly set.
INFO     OK: head macStyle ITALIC bit is properly set.  
INFO:root:OK: OS/2 fsSelection BOLD bit is properly set.

Originally posted at:
fonttools/fontbakery#1406

See fonttools/fontbakery#276

Thought of this yesterday whilst helping a 3rd party use our tools.

Since we've already started modularising the different scripts, we should have them web based imo.

Aim should be to drag n drop a collection of fonts, run any sequence of scripts, then download the modified fonts. A log file should be included so the steps are repeatable.

Ported from fonttools/fontbakery#1454

Originally submitted at fonttools/fontbakery#1282

We're now dealing with almost a thousand font families. This is too much to show at once at the dashboard front page, so it would be good to add a pagination system to show just a few families per page. Maybe 50 or so...

@davelab6 said: "The webapp still available from http://fontbakery.com dealt with this already, with some nice lazy-loading code. I hope you can use it :)"

Having font updates use a serverside PageBot generating documents that use the fonts would be nice, so that designers could proof their work using standard templates on both web and print :)

This depends on the resolution of PageBot/PageBot#40

All from: http://104.197.213.179/report/9cabce25-26db-4d14-85c5-41b4ff3e42f3

Started/Finished should never change after it was written once.

Per google/fonts#1170 the dashboard should have a way to review the results of different CSS font-smoothing options

This is the continuation of fonttools/fontbakery#1306

Will be closed when setting up a local development environment is documented.

Originally submitted at fonttools/fontbakery#1233

@davelab6 reported: "There's a high priority set of work to fix fonts in families to be uniform glyph sets across the collection... so, a 2nd step after fonttools/fontbakery#1232 is to make something similar to the Missing Codepoints spreadsheet in the dashboard so we can burn down progress against that check. "

This must be done in the draganddrop branch at worker-fontbakery/Dockerfile.

To speed this up, I suggest to append a dockerfile CMD pytest path/to/Lib/fontbakery --verbose (including installing pytest) at the end of the file. Then you can test this outside of minikube and make it fast. You can also just add some files and run fontbakery check-googlefonts (probably you can run this directly from your host!).

These are the errors that need to be handled:

• com.google.fonts/test/035 Checking with ftxvalidator. message"ftxvalidator is not available!"
• com.google.fonts/test/036 Checking with ots-sanitize. "ots-sanitize is not available! You really MUST check the fonts with this tool. To install it, see https://github.com/googlefonts/gf-docs/blob/master/ProjectChecklist.md#ots Actual error message was: '[Errno 2] No such file or directory'"
• com.google.fonts/test/037 Checking with Microsoft Font Validator. "Mono runtime and/or Microsoft Font Validator are not available!"
• com.google.fonts/test/153 Check if each glyph has the recommended amount of contours. This test is useful to check if glyphs are incorrectly constructed. message"Failed with IOError: [Errno 2] No such file or directory: '/usr/local/lib/python2.7/dist-packages/fontbakery/desired_glyph_data.json'" status"ERROR" traceback"Traceback (most recent call last):\n File \"/usr/local/lib/python2.7/dist-packages/fontbakery/testrunner.py\", line 305, in _exec_test_generator\n for sub_result in gen:\n File \"/usr/local/lib/python2.7/dist-packages/fontbakery/specifications/googlefonts.py\", line 4164, in check_glyphs_have_recommended_contour_count\n from fontbakery.glyphdata import desired_glyph_data\n File \"/usr/local/lib/python2.7/dist-packages/fontbakery/glyphdata.py\", line 58, in <module>\n desired_glyph_data = _glyph_data(desired_glyph_data_path)\n File \"/usr/local/lib/python2.7/dist-packages/fontbakery/glyphdata.py\", line 34, in _glyph_data\n with open(json_file, 'r') as glyph_data:\nIOError: [Errno 2] No such file or directory: '/usr/local/lib/python2.7/dist-packages/fontbakery/desired_glyph_data.json'\n"
• com.google.fonts/test/054 Show hinting filesize impact. WARN ttfautohint is not available! You really MUST check the fonts with this tool. To install it, see https://github.com/googlefonts/gf-docs/blob/master/ProjectChecklist.md#ttfautohint

updated added com.google.fonts/test/054 wich is just a WARN

Originally submitted at fonttools/fontbakery#1297

MS-FontVal is not currently running on the worker container.