googlefonts / fontbakery-dashboard Goto Github PK
View Code? Open in Web Editor NEWA library-scale web dashboard for Font Bakery, no longer developed
License: Apache License 2.0
A library-scale web dashboard for Font Bakery, no longer developed
License: Apache License 2.0
It has a builtin type for binary data (e.g. font files) BytesValue.
Consider also using gRPC which itself uses protocol buffers.
Also, evaluate NoFlo vs gRPC and see if they are any good together.
https://litmus.com/email-testing is a nice reference for email testing
Originally submitted at fonttools/fontbakery#1237
We do collect all the data needed to build a burndown chart (and there's been some prototypes of that in the past) but right now all we show in the dashboard UI is info tagged with "HEAD": true in the database, meaning that we're just displaying the check results for the latest commit in the git repos.
Since we have the data, I think it would be wise to query that and render the chart. The best place to showcase the graph, in my oppinion, would be at the top of the family details page, where nowadays we only render the check-results summary pie-chart as seen in the screenshot below:
@davelab6: "Agreed"
It costs too much to run FB on the full commit history of projects, so nowadays we are only checking against the repo HEAD commit.
In order to make it acceptable to keep the full history, we would need to detect whether re-running the checks on a specific commit is really necessary. The criteria for deciding that would be: (a) the specific commit was never checked previously or (b) it was checked with an older version of fontbakery so it would be needed to re-run the checks and update the database entries.
Thought of this yesterday whilst helping a 3rd party use our tools.
Since we've already started modularising the different scripts, we should have them web based imo.
Aim should be to drag n drop a collection of fonts, run any sequence of scripts, then download the modified fonts. A log file should be included so the steps are repeatable.
Ported from fonttools/fontbakery#1454
Originally submitted at fonttools/fontbakery#1264
look at @mcdurdin 's http://github.com/mcdurdin/render-lizard seems nice, a possible candidate for integration with the regression testing thing.
Originally posted at:
fonttools/fontbakery#1406
This must be done in the draganddrop
branch at worker-fontbakery/Dockerfile.
To speed this up, I suggest to append a dockerfile CMD pytest path/to/Lib/fontbakery --verbose
(including installing pytest) at the end of the file. Then you can test this outside of minikube and make it fast. You can also just add some files and run fontbakery check-googlefonts
(probably you can run this directly from your host!).
These are the errors that need to be handled:
message"ftxvalidator is not available!"
"ots-sanitize is not available! You really MUST check the fonts with this tool. To install it, see https://github.com/googlefonts/gf-docs/blob/master/ProjectChecklist.md#ots Actual error message was: '[Errno 2] No such file or directory'"
"Mono runtime and/or Microsoft Font Validator are not available!"
message"Failed with IOError: [Errno 2] No such file or directory: '/usr/local/lib/python2.7/dist-packages/fontbakery/desired_glyph_data.json'" status"ERROR" traceback"Traceback (most recent call last):\n File \"/usr/local/lib/python2.7/dist-packages/fontbakery/testrunner.py\", line 305, in _exec_test_generator\n for sub_result in gen:\n File \"/usr/local/lib/python2.7/dist-packages/fontbakery/specifications/googlefonts.py\", line 4164, in check_glyphs_have_recommended_contour_count\n from fontbakery.glyphdata import desired_glyph_data\n File \"/usr/local/lib/python2.7/dist-packages/fontbakery/glyphdata.py\", line 58, in <module>\n desired_glyph_data = _glyph_data(desired_glyph_data_path)\n File \"/usr/local/lib/python2.7/dist-packages/fontbakery/glyphdata.py\", line 34, in _glyph_data\n with open(json_file, 'r') as glyph_data:\nIOError: [Errno 2] No such file or directory: '/usr/local/lib/python2.7/dist-packages/fontbakery/desired_glyph_data.json'\n"
WARN ttfautohint is not available! You really MUST check the fonts with this tool. To install it, see https://github.com/googlefonts/gf-docs/blob/master/ProjectChecklist.md#ttfautohint
updated added com.google.fonts/test/054 wich is just a WARN
(was fonttools/fontbakery#277)
Font Bakery now has a new output format, thus, we need to redo the report saving and rendering part. This will improve real time reporting and we also have the opportunity to do parallel execution of tests, which would be nice to have:
Parallel execution can be done after all the other things, when done carefully it'll just work 😄
Originally submitted at fonttools/fontbakery#1123
@m4rc1e said:
"Hey Felipe and Dave,
I think we should merge my gfregression app into Font Bakery. Instead of having lots of seperate repos/apps/libraries, we should focus on consolidating them.
It also fits FB's brief nicely
Tools to prepare font families for inclusion in github.com/google/fonts
I use it for cross platform testing a collection of fonts vertical metrics, against the same fonts which are hosted on fonts.google.com.
Left: local fonts | Right: fonts.google.com
I also see we are pushing for a UI for FB. I've built this app with Flask and Jinja2, keeping everything as simple as possible. I would opt for AWS or Digiocean over App Engine, due to our dependencies.
Cheers,
Marc"
To which @davelab6 replied: "Agree, but I would like Felipe to focus on getting the checker working first and then build this comparison feature in.
Thus moving this to 0.4.0"
There is a github-release tool that can be used to automate tagging and uploading releases on Github... Someday would be great for the dashboard to make it easy for upstream repo owners to make font releases that way :)
Per google/fonts#1170 the dashboard should have a way to review the results of different CSS font-smoothing options
Originally submitted at fonttools/fontbakery#1228
"gather from git" runs on the cmd-line specified TTF files once per commit.
Current shortcomings:
A) It does not cover more than one family stored in a single repo.
B) It does not follow git renames (I considered implementing that. It would be neat. But I did not think it was the most important thing to do in terms of priorities)
@davelab6 said: "Given a git repo URL, git commit hash id, a fontbakery version are recorded in a central database of burndown results, it seems that we can have the gather tool cache it's findings and return early if it finds an existing result for the given repo hash with a version more up to date or equal to current version"
To which I replied: "Sounds great!"
Originally submitted at fonttools/fontbakery#1297
MS-FontVal is not currently running on the worker container.
@graphicore There's an issue (and there will be a commit closing it pretty soon) here (fonttools/fontbakery#1336) that will provide some clues on how to set up the Microsoft Font Validator on the kubernetes cluster.
@m4rc1e said: "Hey @felipesanches ,
I think it's wise we stick to using Material Design for the UI."
To which I replied: "Yup! I think it's great."
And @davelab6 tipped at: "Material Design Components?"
And then said: "This is low Priority to me"
Originally submitted at fonttools/fontbakery#1246
It is accessible only via direct public IP address. I looked up how to setup the domain name, but couldn't get it. I'll have to try again later. Dave said we can use fontbakery.com (which he owns)
I have a hunch that setting this up involves some setting in the kubernetes web-panel, or via some kubectl command. I think Kubernetes is probably responsible for updating the DNS when a new service is brought up in a node (especially when the externally accessible IP address changes). But I may be wrong. I'm new to this thing.
This seems to be a good way to do it:
https://github.com/kubernetes-incubator/external-dns
We need a domain for this.
Not sure about inter-cloud communication, as long as we don't send something confidential it should be fine.
FAMILY | Upstream Pulls | Upstream Master | gh google/fonts pulls | gh google/fonts master | DEV API | Public API |
---|---|---|---|---|---|---|
Poppins | 7% | 6% | 5% | 8% | 9% | 17% |
Roboto | … | … | … | … | … | … |
TOTAL | 12% | 13% | 14% | 15% | 16% | 17% |
Historical Data: the state of this table a year ago, three years ago, ...
The live updating view: will be implemented after the static one.
Sources, here the rows, will be implemented from right to left. API first ...
The Upstream sources will likely be augmented with METADATA.pb etc. from google/fonts:master
There will be a drill down. A column here can be expanded to a full collection report table. The imploded column shows the value of FAIL.
example collection report table:
family_dir | total | ERROR | FAIL | WARN | SKIP | INFO | PASS |
---|---|---|---|---|---|---|---|
ofl/laosanspro | 124 | 4 | 8 | 5 | 47 | 3 | 57 |
ofl/miriamlibre | 222 | 10 | 8 | 12 | 4 | 6 | 182 |
ofl/orbitron | 418 | 16 | 43 | 21 | 24 | 12 | 302 |
ofl/pontanosans | 124 | 4 | 19 | 8 | 4 | 3 | 86 |
ofl/poppins | 1790 | 72 | 151 | 83 | 54 | 56 | 1374 |
Originally submitted at fonttools/fontbakery#1230
Currently the drag-n-drop webapp hosted at http://fontbakery.appspot.com/ does not run FontForge checks (or any other checks that require usage of the subprocess module) due to secutiry restrictions on Google App Engine.
If we migrate the webapp to Google Container Engine, we'll be able to run these additional checks, improving the coverage of the reports generated by this webapp.
Note: There's a whole bunch of messages in the original issue tracker entry. Please take a look at those as well!
Following fonttools/fontbakery#1537 the dashboard should have a way to show users what languages are/not supported by a family that is being tracked by fbd. This should have all the burn down chart stuff being discussed elsewhere in this issue tracker.
Per google/fonts#1215 we should include the Hawaiʻian language
Originally submitted at fonttools/fontbakery#1289
The bad formatting we get currently looks like this:
http://35.184.229.186/details/Miltonian/errorlog/
WARNING:root:fontforge python module is not available! To install it, see https://github.com/googlefonts/gf-docs/blob/master/ProjectChecklist.md#fontforge
INFO:root:OK: Miltonian-Regular.ttf is named canonically
INFO OK: Miltonian-Regular.ttf is named canonically
INFO:root:OK: All files are in the same directory.
INFO OK: All files are in the same directory.
ERROR:root:Did not find cached vendor list at: /tmp/fontbakery-microsoft-vendorlist.cache
ERROR Did not find cached vendor list at: /tmp/fontbakery-microsoft-vendorlist.cache
INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): www.microsoft.com
INFO Starting new HTTPS connection (1): www.microsoft.com
ERROR:root:'Miltonian-Regular.ttf' is missing a METADATA.pb file!
ERROR 'Miltonian-Regular.ttf' is missing a METADATA.pb file!
INFO:root:OK: All font files have the same version.
INFO OK: All font files have the same version.
INFO:root:OK: Miltonian-Regular.ttf opened with fontTools
INFO OK: Miltonian-Regular.ttf opened with fontTools
INFO:root:OK: Font has post table version 2.
INFO OK: Font has post table version 2.
INFO:root:OK: OS/2 fsType is a legacy DRM-related field from the 80's and must be zero (disabled) in all fonts.
INFO OK: OS/2 fsType is a legacy DRM-related field from the 80's and must be zero (disabled) in all fonts.
ERROR:root:[FULL_FONT_NAME(4):WINDOWS(3)] entry: expected 'Miltonian Regular' but got 'Miltonian'
ERROR [FULL_FONT_NAME(4):WINDOWS(3)] entry: expected 'Miltonian Regular' but got 'Miltonian'
WARNING:root:VendorID string 'PYRS Fontlab Ltd. / Made with FontLab' does not match nameID 8 (Manufacturer Name): 'Pablo Impallari'
WARNING VendorID string 'PYRS Fontlab Ltd. / Made with FontLab' does not match nameID 8 (Manufacturer Name): 'Pablo Impallari'
INFO:root:OK: OS/2 VendorID is 'PYRS' and registered to 'PYRS Fontlab Ltd. / Made with FontLab'. Is that legit?
INFO OK: OS/2 VendorID is 'PYRS' and registered to 'PYRS Fontlab Ltd. / Made with FontLab'. Is that legit?
INFO:root:OK: No need to substitute copyright, registered and trademark symbols in name table entries of this font.
INFO OK: No need to substitute copyright, registered and trademark symbols in name table entries of this font.
INFO:root:OK: OS/2 usWeightClass value looks good!
INFO OK: OS/2 usWeightClass value looks good!
INFO:root:OK: OS/2 fsSelection REGULAR bit is properly set.
INFO OK: OS/2 fsSelection REGULAR bit is properly set.
INFO:root:OK: post table italicAngle is 0.0
INFO OK: post table italicAngle is 0.0
INFO:root:OK: OK: post table italicAngle is less than 20 degrees.
INFO OK: OK: post table italicAngle is less than 20 degrees.
INFO:root:OK: post table italicAngle matches style name
INFO OK: post table italicAngle matches style name
INFO:root:OK: OS/2 fsSelection ITALIC bit is properly set.
INFO OK: OS/2 fsSelection ITALIC bit is properly set.
INFO:root:OK: head macStyle ITALIC bit is properly set.
INFO OK: head macStyle ITALIC bit is properly set.
INFO:root:OK: OS/2 fsSelection BOLD bit is properly set.
Originally submitted at fonttools/fontbakery#1287
@m4rc1e said: "I'd like to have the dashboard built on top of api views. I can then query the api directly in my hot fix scripts."
Having font updates use a serverside PageBot generating documents that use the fonts would be nice, so that designers could proof their work using standard templates on both web and print :)
This depends on the resolution of PageBot/PageBot#40
Originally submitted at: fonttools/fontbakery#1267
Even though family-names should be ordered alphabetically by default ( #1266 ), it would be useful to allow the user to sort the list by progress status as well.
A very old issue on the fontbakery repo suggested using Guiders-JS to document the user interface:
https://github.com/pickhardt/Guiders-JS
I consider this to be an extremely low priority issue in the "maybe-someday" category. I'm just filling this here because the corresponding issue on the original issue tracker (fonttools/fontbakery#23) does not apply anymore to that repo, which is now focused on FontBakery-core only.
Originally submitted at fonttools/fontbakery#1282
We're now dealing with almost a thousand font families. This is too much to show at once at the dashboard front page, so it would be good to add a pagination system to show just a few families per page. Maybe 50 or so...
@davelab6 said: "The webapp still available from http://fontbakery.com dealt with this already, with some nice lazy-loading code. I hope you can use it :)"
See fonttools/fontbakery#1311 and file issues based on those ideas
Dashboard should show metrics on how many problems and what types of problems we actually catch early with FB
Originally submitted at fonttools/fontbakery#1236
While we display all check-results (for the sake of completeness), it is likely that the users may be much more interested in looking at what went wrong. So a button (or checkbox) for toggling the visibility of "OK" check results could be very useful in the dashboard UI.
@davelab6: "Agreed"
The key feature of the 0.1.0 release of the dashboard for me is to be able to say, this is the FB score of the fonts in X family in production, and this is the FB score of the fonts in the upstream.
To do that, we need to develop a complete list of upstream URLs for every family in google/fonts.
Originally submitted at fonttools/fontbakery#1233
@davelab6 reported: "There's a high priority set of work to fix fonts in families to be uniform glyph sets across the collection... so, a 2nd step after fonttools/fontbakery#1232 is to make something similar to the Missing Codepoints
spreadsheet in the dashboard so we can burn down progress against that check. "
I'd like to have a status page showing, for google/fonts, and for each repo upstream of it, and for all,
and have this per day/week/month and arbitrary times.
@m4rc1e developed a script to help divine some of this, so turning that into a web page is on the roadmap for sure :)
www.snap-ci.com and www.travis-ci.org and probably many other sites have great homepages and login pages.
Ours should
(was fonttools/fontbakery#245)
Originally submitted at fonttools/fontbakery#1241
After fixing fonttools/fontbakery#1240, I'd like to increase the list of font projects git repos being checked by the dashboard's worker container.
It seems to me that the canonical list of font projects to check against should be whatever we currently have at prod. I know it is 800-something families, but I don't know where to find an official list of them all other than automatically querying Google Fonts itself.
Is there a similar list for the git repos for each of those? I was able to compile a list of a bit more than 104 repos by scraping the google/fonts repo for URLs referencing other repos. (the list is here: fonttools/fontbakery#1158 (comment)) This would be roughtly only 13% of all projects. Does it mean 87% of the projects simply lack a git repo? Or the repos are there and I simply don't know exactly where?
NOTE: Please look at the original issue thread because there are plenty of additional comments in there!
This may really be a separate project, to be integrated
All from: http://104.197.213.179/report/9cabce25-26db-4d14-85c5-41b4ff3e42f3
Started/Finished should never change after it was written once.
This is the continuation of fonttools/fontbakery#1306
Will be closed when setting up a local development environment is documented.
Originally submitted at fonttools/fontbakery#1286
@m4rc1e said: "Since we can now see the health of the whole collection, how about we allow the dashboard to look at different forks? This will give us the ability to see if a pr will improve the collection before we merge. I'm working on a collection wide hotfix, google/fonts#641. This would be incredibly useful."
Then I said: "Yes, I agree.
There are some challenges to achieve that, though.
The current implementation of the worker container involves checking out the last N commits of the full git repo of an upstream project and it expects to have a single family per repo. Multiple families in a single repo are handled by having multiple entries on the fontprojects.csv for the same repo, with differing font_prefix values. Loading up additional containers and re-cloning the repo is not so bad since the full size of the upstream font project repos tend to not be very large.
Dealing with the google/fonts git repo (or forks of it) is trickier since that repo is huge! A different approach would be needed for this. We might try fetching raw files from the Github repo via HTTP requests? I tend to dislike this approach."
@m4rc1e replied: "I understand this is no small task.
so the current implementation currently runs on upstream repos or downloadable families from fonts.google.com?
I'm just delving into the worker container to see the current implementation."
@davelab6 said: "We probably should have more persistent containers, so that the cloned git repo isn't lost on each run"
To which I replied: "A general mechanism for configuring the dashboard is the CSV file on the dispatcher container. If someone needs to setup the dashboard to monitor a different set of families, that can be achieved by providing a different CSV file. But for that, each family must be hosted in its own git repo."
In an About page:
This about page should be generated by a script, eg by parsing requirements.txt
, so it doesn't have to be maintained by hand and is updated as things change
This should also parse the AUTHORS.txt file and place the list of Font Bakery authors at the top of the page.
-[ ] All fontbakery dependencies should be listed, in a table of contents/index (perhaps using a bootstrap sidebar navbar, or material side content index, which are both context-indicating)
Originaly submitted at fonttools/fontbakery#1298
Fontforge checks are not currently running on the worker container.
Fontforge instalation commands were added to the worker container image (via Dockerfile apt-get install statements) but the fontforge python module is not found at runtime.
References:
fonttools/fontbakery#1579 said:
@rsheeter has requested today a efficient visual regression tool, and there are a few related tasks for this
Originally submitted at fonttools/fontbakery#1273
These checks from 001 until 015 are "cross family" ones. For that reason, they are not stored on the individual font file report json outputs. Instead, they're saved in a file called CrossFamilyChecks.json
There should be a tab for those cross family checks. But it doesn't show up on the dashboard family details page.
@davelab6 said: "Agree"
Originally submitted at fonttools/fontbakery#1297
MS-FontVal is not currently running on the worker container.
Originally submitted at fonttools/fontbakery#1239
We need to decide how the check results for the production font files will be displayed on the dashboard. And then I need to implement whatever we decide.
I understand that we'll probably want to have the same level of detail for production files as we have with upstream git-repo files. So Ideally we would simply populate the same page template we already have. On the other hand, we'll probably want to compare side-by-side production against upstream.
@davelab6 do we need to compare in the single-font-file/individual-checks level ?
I can see the value of being able to observe in a glance that a certain check fails on upstream while passing on production (or vice versa).
We may also want to see a comparison in the global level like displaying the progress bars for the check-results summary of prod and upstream side-by-side somewhere in the UI.
This issue is focused on brainstorming this and later implementing it. For tracking the task of implementing the actual server-side fontbakery run on prod files we already have fonttools/fontbakery#1238
Please also take a look at the original issue thread at fonttools/fontbakery#1239 because there are some considerations there from me and from @davelab6 !
Originally submitted at fonttools/fontbakery#1235
Red is typically associated with "problem". It would be good to change the color-scheme of the generated pie-chart so that the "OK" check-results are represented by a portion of the pie coloured in green. While "ERROR" results should use the red color, and maybe blue for "INFO" and yellow for "Warning" and soft-blue for "SKIP".
@davelab6: "Agreed"
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.