gap-system / gap-distribution Goto Github PK

If GAP has not been uninstalled, the exe installer will suggest the path to that existing GAP installation as the default installation path (see gap-system/gap#2482). Could this be avoided?

Perhaps using CALL_WITH_CATCH

If you (all from Windows explorer):

1. Download gap-4.9.1.zip
2. Unzip it
3. Try to delete the resulting directory

You'll find you can't delete the directory. The problem is files which are called 'Con', from carat.

NormalizInterface provided the https URL of the PackageInfo.f file instead of http. That resulted in the package not being picked up for the redistribution with the following text in the log:

Error: misformatted line
[ "NormalizInterface", 
 "https://gap-packages.github.io/NormalizInterface/PackageInfo.g" ]

This is not very helpful. If we do not permit https then ValidatePackageInfo should fail. If we do permit it, then this should work.

The script DistributionUpdate/fixpermissions currently is out of date -- and even if we update it, it likely will get out of sync with reality again and again in the future.

As it is, I don't see the point of this script these days: git tracks the executable bit for us.

In tools.tar.gz of GAP 4r8p3, the file doc/manualindex is not executable while it was so in GAP 4r7p9 (reported by Bill Allombert)

currently most are named like

  blah/

while the rest are named

  foo-1.2.3/

Do either way, but do not mix them.

1. before zipgapcore wraps the core archive, it should run autogen.sh. Two more files which it produces (configure and src/config.h.in) should be added to the archive.

2. makedoc builds GAP to build documentation - remove the call of autogen.sh from there, since it will be now redundant.

Add warnings for the situation when the distribution contains packages that require next release (for example, when GAP 4.8.8 distribution is including a package requiring GAP 4.9).

See gap-system/gap#2285

Having to download this script and change its permissions is a unnecessary chore. Why not just package them in the tarball?

In GAP 4.8.4, this script should be included in the distribution and made executable. Also remember to check installation instructions.

To collect TODOs - what can be better automated:

• copying main distribution archives to www.gap-system.org/pub/gap/

• setting of bootstrapping archives

• generation of the Mixer code to describe main GAP distro archives, or even a new mixer page

When making the release tarball, make sure the GAP version is displayed correctly in the banner. For GAP 4.8.8, this meant modifying cnf/mkversionheader.sh.

But this changed, in master, the place to modify is the script cnf/GAP-VERSION-GEN, and in there the line containing 4.dev.

To adjust the build date, edit src/gap_version.c.in and replace the today in there.

Some packages do not have http:// in the URLs specified in PackageInfo.g files, e.g. they have

WWWHome       := "place.somewhere.com/~username",

instead of

WWWHome       := "http://place.somewhere.com/~username",

Package overview pages are generated automatically using these details from PackageInfo.g files, and missing http results in broken links as they are interpreted as pointing to local files on the web-server.

Either http:// should be required in URLs and ValidatePackageInfo should check for that, or the website generating scripts should be updated to add http:// when it is missing.

As we see in frankluebeck/GAPDoc#17, under certain circumstances we may be interested to rebuild package manuals in GAPDoc format if they use MathJax. This may be straightforward for some packages (e.g. those using @fingolfin's ReleaseTools and having a makedoc.g script, and not so straightforward for others (but advised as a part of package updates for GAP 4.9 release), and perhaps may be automated with doc/BuildPackageManuals.sh script similarly to bin/BuildPackages.sh.

Besides ensuring that the MathJax URL is correct, this will also fix the problem when package manuals are build out of tree and have broken links to main GAP manuals.

It would be good to assign DOIs for GAP releases, using, for example, the workflow described at https://guides.github.com/activities/citable-code/.

With new packages for group libraries being needed to run GAP (at least for the transitional period) we need to be able to easily wrap a merged archive of needed packages. Before, it was simply done by renaming the GAPDoc archive; now it should be automated. I should add a new keyword only to mergePackages so that it could be called with a command like e.g.

./mergePackages only gapdoc=stable primgrp=latest smallgrp=latest transgrp=latest

to wrap together only specified packages. Appropriate commands should then be added to the Makefile.

If new package releases of the four (for now) required packages are picked up, then we should make sure that packages-required-BRANCH.tar.gz gets updated for all active branches. Ideally, this would be done fully automatically by a script running on Travis / GitHub Actions / Jenkins / whatever.

See also issue #102.

These are needed for the Jupyter interface, and are not yet ready for the release.

This comes from this discussion https://mail.gap-system.org/pipermail/gap/2016-March/000627.html (see also gap-packages/io#18). In my reply to @fingolfin's explanation, I've suggested to @ChrisJefferson and @james-d-mitchell to disable maintenance mode in their packages. If this can't work, instead one could set the modification time of all files in pkg (or in the whole GAP distribution) to the same modification time as some reference file (e.g. with time which will appear in the timestamp of the GAP distribution archive):

find PATH/TO/GAP/pkg -exec touch -r "PATH/TO/reference {} \;

They are now redundant (we do not use timestamps in GAP distributions), and having a fixed name for merged package archives would simplify handling them in various contexts.

To do: investigate which packages cause such messages in gap -r -A LinksOfAllHelpSections.g, for example

Invalid HTML path: [ "SONATA Tutorial: Designs", "8", fail, 
  "/Users/alexk/gap4r8p8/pkg/sonata/doc/tut/manual.pdf", 26, fail, 
  [ 8, 0, 0 ] ]

WritePackageWebPageInfos from https://github.com/gap-system/gap-distribution/blob/master/DistributionUpdate/PackageUpdate/PackageInfoTools.g should make use of gap-system/gap#7 and use new components SourceRepository, IssueTrackerURL and SupportEmail for package overview pages.

This is an issue to remind myself to do this for the GAP 4.8 public release, when there will be already packages using those new components.

Apparently, the package distribution is currently handled by a single mercurial repository which aggregates all packags. This repository seems to be private (e.g. I don't have access to it, in the sense that I do not even know where it is located).

This means that in particular virtually nobody can test the scripts in here.

Hence, I think we should make this repository public. And while at it, let's convert it from hg to git so that we us git everywhere (one less skill needed to work on the package distribution).

I am willing to help with both converting the package repository, and adapting the scripts to use git instead of hg; but alas, as I already said, I don't even know where that repository is, and cannot test the existing (mostly undocumented, it seems?) scripts without that.

Reported by Georg Konrad:

the windows installer of version 4.9.1 generates a wrong windows batch file, if GAP is installed in a non default directory. E.g. The windows batch file contains

set PATH=C:\gap-4.9.1\bin\i686-pc-cygwin-default32;%PATH%

instead of

set PATH=C:\Programme_(exe)\gap-4.9.1\bin\i686-pc-cygwin-default32;%PATH%.

TODO: fix this around the line.

and check other .bat files

Possible solutions for users experiencing this now: either install GAP in the default C:\gap-4.9.1 directory or edit the .bat file.

in GAP 4.10.0 tarball this file is read-only (seems to be the only one, at least in ctbllib.

This may cause some interesting script errors --- and we got this on Sage's ticket on GAP 4.10 and libgap

-r--r--r-- gap-jenkins/gap-jenkins   35292 2018-11-01 22:56 gap-4.10.0/pkg/ctbllib/tst/docxpl.tst

@alex-konovalov @ThomasBreuer - would you like to fix this (in 4.10.1 ?)

Version of GAP is now hardcoded and should be updated in too many places for the each minor release (before it was gap4r8 so only for major ones).

After recent changes by @markuspf and @fingolfin, the version number is not adjusted properly in the GAP source distribution and is reported as "unknown":

 ┌───────┐   GAP unknown of 2015-09-27 16:20:13 (BST)
 │  GAP  │   http://www.gap-system.org
 └───────┘   Architecture: x86_64-pc-linux-gnu-gcc-default64
 Libs used:  readline
 Loaded workspace: wsp.g
 Components: trans 1.0, prim 2.1, small* 1.0, id* 1.0
 Packages:   AClib 1.2, Alnuth 3.0.0, AtlasRep 1.5.0, AutPGrp 1.6, 
             Browse 1.8.6, CRISP 1.3.8, Cryst 4.1.12, CrystCat 1.1.6, 
             CTblLib 1.2.2, FactInt 1.5.3, FGA 1.3.0, GAPDoc 1.5.1, IO 4.4.4, 
             IRREDSOL 1.2.4, LAGUNA 3.7.0, Polenta 1.3.2, Polycyclic 2.11, 
             RadiRoot 2.7, ResClasses 3.4.0, Sophus 1.23, SpinSym 1.5, 
             TomLib 1.2.5
 Try '?help' for help. See also  '?copyright' and  '?authors'

IMHO the fix should be in https://github.com/gap-system/gap-distribution/blob/master/DistributionUpdate/updateversioninfo

There are "pattern" files in https://github.com/gap-system/gap-distribution/tree/master/DistributionUpdate which contain references to files/directories that already had been removed.

It could be based on 76.25 Package release checklists from the reference manual https://www.gap-system.org/Manuals/doc/ref/chap76.html#X82CE0A518440CCBB and placed here (on Wiki) to help package authors and reviewers of packages submitted for the redistribution with GAP.

configure and src/config.h.in are produced by running autogen.sh, and the latter is only meant for use from git. For software releases, the packagers should run autogen.sh, and ship the generated files to the user.

As Leonard Soicher pointed out, http://www.gap-system.org/Packages/sla.html has now release date in the future - 29/12/2016. This page is generated automatically from PackageInfo.g file, so I am not going to fix this on the website since it will be overwritten after the next release again.

There are several kinds of misuses that may happen with the date:

1. The string is not of the form NN/NN/NNNN where N is a digit (for example, 7/1/16 instead of 07/01/16). This will be rejected by ValidatePackageInfo:

TestMandat( record, "Date",
    x -> IsString(x) and Length(x) = 10 and x{ [3,6] } = "//"
             and ForAll( x{ [1,2,4,5,7,8,9,10] }, IsDigitChar ),
    "a string of the form `dd/mm/yyyy'" );

2. The date swaps day and month, or contains numbers for non-existing day/month etc. This may be improved by checking that all numbers are within proper bounds, with the same effect that PackageInfo.g validation fails in case the date is wrong.

The next two issues are too specific to the package update mechanism - they should better be implemented separately, and not inside the ValidatePackageInfo:

3. The date is in the future. In this case, package update is not picked up, and an error message is produced to be passed to the authors.

4. The date is in the past. Often this happens when the author incremented the version but forgot to increment the date. But this is a bit more subtle. Maybe it is several days old, but check for package updates was not run since last week? Or maybe it is the same as the date of the last release, but it was yesterday, and the author prepared next release during the same day? Or maybe the package was published a month ago on the author's website, which then was not available for a month and we were unable to pick it up? So it seems like in this case there heuristic should be applied and a warning, not an error, should be produced. For example, I could easily add a warning for packages with release date not in the same month with the day of the check for updates.

a typical release of packages has a lot of needless stuff that just takes up space,
such as docs as pdf/dvi (and TeX work files), generated xml...

e.g. there are 108 log files

gap4r8/pkg]$ find . -name *.log | wc -l
108

similarly:
gap4r8/pkg]$ find . -name *.dvi | wc -l
32
gap4r8/pkg]$ find . -name *.pdf | wc -l
244

there are 11 copies of convert.pl script (why are they there at all? isn't it a part of GAP proper?)

there is even an .exe file in
grape/bin/i686-pc-cygwin-gcc-default32/dreadnautB.exe
(and we talk about SOURCE, for Pete's sake)

Before a package can make it into Sagemath packages, all this has to cleaned.
Is it possible to produce a source release tarball with "make distclean" ran on
each pkg ?

The timestamps in GAP release archives are superfluous these days (they historically served a purpose, but not anymore). Let's drop them for GAP 4.9.0. That'd lead to filenames like gap4r9p0.tar.gz. I'd go one step further, and change this to gap-4.9.0.tar.gz, to match what "everybody else" does, and has been doing for a few decades now.

That would then suggest that the directory in there also be named gap-4.9.0 (and not gap4r9 or gap4r9p0, but that might require further changes to tooling, and some people might not like it (I think some people want to just extract new GAP tarballs "over" their existing tarballs, as a cheap upgrade mechanism). So as a compromise, we could for go gap-4.9 (and perhaps on a future date, if/when we have a "proper" update system, change this again).

The links we use on package pages (for example https://www.gap-system.org/Packages/atlasrep.html ) refer to the gap4 directory ( https://www.gap-system.org/pub/gap/gap4/tar.bz2/packages/atlasrep1r5p1.tar.bz2 ). This is a symlink, which means it breaks when we release a new version of GAP.

We should instead either:

1. (my preference) make /pub/gap/gap4 a directory containing all packages from gap4, rather than a symlink to the most recent version.

2. Change these links to refer to /gap/gap49 (or whatever the current version is) so they don't break when we release a new version of GAP.

To remind me of Homebrew/homebrew-core#21928 to be revisited when GAP 4.9.1 will be out.

merged packages archive tar.gz increased from 317.98 to 330.59 MB after picking up [ "curlInterface", "2.1.0" ], [ "HAP", "1.12.7" ], [ "IO", "4.5.3" ], [ "matgrp", "0.51" ], [ "recog", "1.3.1" ], [ "Semigroups", "3.0.19" ].

In particular, the following appeared in the log:

pkg/Hap1.12/test/examples/image1.3.2.eps: up to 144 MB of RAM may be required to manage this file
(use 'hg revert pkg/Hap1.12/test/examples/image1.3.2.eps' to cancel the pending addition)

and

pkg/recog-1.3.1/examples/c3c5benchmarks_mats.g: up to 66 MB of RAM may be required to manage this file
(use 'hg revert pkg/recog-1.3.1/examples/c3c5benchmarks_mats.g' to cancel the pending addition)

Need to look at large files in these packages.

XGAP has a new homepage: https://gap-packages.github.io/xgap

Since I have no access to the old homepage, this will have to be moved manually.

We check currently for non-standard permissions in package archive at the release wrapping stage, where it may be too late to ask package authors to fix them.

It will be better to check for permissions when the new version of the package is picked up.

This should complement gap-system/gap#1305

Also clean up references to these files in pattern files (see #22).

Graham Ellis wrote: "Some years back I extracted, corrected and stored in ~/pkg/Hap1.11/lib/HapPrime that portion of HAPprime that is used in HAP. So HAP never calls the HAPprime package and I would guess that no user ever calls HAPprime. [...] Is it possible to just drop HAPprime from the distribution?"

Instead of making a shallow clone and then deleting its .git dir, I recommend using git archive instead.
That is also what I use in https://github.com/fingolfin/ReleaseTools

This script prints a summary of package tests:

With more packages with tests, and even more packages with passing tests, it would be good to separately display a list of packages with failing tests, for them to be seen easier.

packages-required-stable-4.11.tar.gz and packages-required-master.tar.gz need to be updated. They contain outdated versions of GAPDoc, smallgrp, primgrp -- this recently bit me, when code started to fail when I switch from a full to a minimal bootstrap because of the outdated GAPDoc.

An idea - it wouldn't be difficult to extract from the GAPInfo record details about packages:

• authors/maintainers overview
• time since the last release
• availability of the test file
• using GAPDoc for the documentation
• missing css files in documentation
  etc.

After doc/manualindex will be restored in gap-system/gap#642, in the release should go to the same archive as gapmacro.tex and convert.pl

When package is just added to redistribution, there is no "stable" revision marked yet (that will happen only after it will be included in a public release), but this is not reflected properly in the autogenerated report.

Check if we can provide more packages with precompiled binaries - both with kernel modules and with stand-alone executables.

What are these README.blah doing there? By right,

 mv README.blah blah/README 

will clean a lot up...