GitHub has a rate limit of 5000 API queries per hour. We currently make 3 requests per dist and the updater script runs twice per hour. This means we'll hit the limit when our ecosystem reaches 833 modules, which is about 80% more modules than we currently have.

Taking care of this Issue will be easy, once the new-db-builder branch is merged. We'll just perpetually run the build script, with sufficient delays between fetching info for each dist so we don't hit the API limit. The cron job will be there just to pull updates for the repo and ensure the build script is alive.

The modules page should at the very least link to http://testers.perl6.org/

But if it could somehow incorporate those PASS Ratio graphs that would be fantastic.

in Dist::Source:

author_id     => $dist->{author}
                            // (@{ $dist->{authors}||[] })[0] // 'N/A',

dies on https://raw.githubusercontent.com/cjfields/bioperl6/master/META.info

Somewhere in the pipeline Unicode chars get mangled. See description of, say http://modules.perl6.org/#q=Math::Tau

The sort direction toggle behavior is a little inconvenient.

• The default for date-based columns should be descending sort (recent first)

• The last-used order should be used when clicking on a non-sorted column. For example,

  1. Load modules.perl6.org, notice that it is sorted by Name ascending
  2. Click Description header, it is sorted by Description ascending
  3. Click Name header, it is sorted by Name descending

  It should only toggle the sort order when clicking on the current sort header.

It would be neat if the site could give useful response when core types are looked up.

For example, I search for "Socket" and got a couple fo dists, but I wasn't aware the core had a few Socket modules too.

So search results could prepend results from core stuff search as well (distinguished from regular modules with some highlight maybe).

Similarly, the module links in the app version of the site could be made to work for core stuff as well, so that, e.g., http://modules.perl6.org/repo/IO::Socket::INET redirects to http://docs.perl6.org/type/IO::Socket::INET

Search option probably also looks into url strings. It is not possible to find modules related to HTTP because all urls start with http: so typing it will not affect anything

#34 is recurring, looks like a case of encode('utf-8', decode('latin-1', $utf8_bytes)) today.

Currently, looks like the latest build is tracked by travis, but we need to track master, which that's what I believe panda installs.

The current issue is the modules page shows, say FTP module as failing, but in reality master passes and it's only the develop branch that's failing.

The code is a bit too hard to follow and probably has too much influence on by the dist's META file contents.

Also, Kwalitee metrics are currently not cached.

It points to https://github.com///

The current ecosystem seems to have no transition path for merging with CPAN. We seem stuck with a system which clearly can be improved, yet we as a community seem unable to envision any way ahead to integrate the current system with CPAN.

I'd like to be able to type something like http://modules.perl6.org/l/runge and get to the github repository for Math::RungeKutta.

e.g.

That would allow linking from a README.md to the module in the ecosystem if that module is a dependency or for general propaganda.

If we look at ecosystem list, there are currently 389 lines.

The newly added module count shows 385 modules.

And the page apparently lists only 382 modules based on running

perl -MMojo::UserAgent -wlE 'say Mojo::UserAgent->new->get("http://modules.perl6.org/")->res->dom->find(".name")->size'

The data shows there are 7 modules missing from the final list; being lost at various stages of the process.

This should be an easy fix (just don't have the time at the moment). We have this CSS that takes care of too-long distro names on small screens. That code currently pushes the logotype pic down onto a new line.

The fix likely involves just showing the distro logo inside the <a> with the distro name.

To repeat:

1. click "Updated" to sort by that column
2. scroll to the middle of the page
3. click a link to visit a module page
4. click back in the browser
5. see page sort by name

Perhaps use a #sort-col anchor, to avoid re-sorting?

metacpan does that, and it's quite handy, since it's much easier to type - than ::: https://metacpan.org/search?q=file-find-rule&search_type=modules

Bootstrap3 plugin includes all the standard Bootstrap JS files; we need to toss all we're not actually using.

Now that we have more info on the page, we should look into optimizing the shown info to get more screen real estate:

1. The badges: some of them might not be as useful. The 90-day commits badge might be misleading, since some modules now could be "finished" and not require commits. The panda badge might not need to be there.
2. The travis badges could be shrunk down by removing the redundant "build" word.

This was the discussion on IRC: http://irclog.perlgeek.de/perl6/2015-11-03#i_11475494

Currently, there exists a possible race condition with the new database builder:

1. User pushes new commits and Travis starts building
2. While that has NOT completed, our builder fetches the dist, grabs new commit and asks the Travis postprocessor to fetch new build status
3. We get a not yet completed status into the database
4. When dist is built again, we don't see any new commits, so we don't run the travis postprocessor and the wrong status is stuck with us forever!

This will be especially noticable in dists that take a long time to finish travis build.

Possible solution: run travis postprocessor regardless of whether there were new commits to the dist. We'll be hitting Travis API much more often, so we'd need to check for what limits there are.

The Problem

There are many modules in the ecosystem that lack any useful documentation. I've actually heard someone refer to this as "Perl 6's culture of no documentation."

The Solution

We're about to have many new comers and having a simple way to contribute would be good for the growth of the community.

1. Same as with the Most Wanted Modules Wiki, we should have one for Documentation Wanted.
2. Those who are new to the language could study those modules, figure out how they work, document the behaviour, and submit a doc PR.

In summation, I think this can be a great entry-level task for new comers to contribute to, while at the same time eradicating the "no documentation culture".

On slow connections the site loads rather bare, even if you visited it recently.

It's likely our main CSS isn't getting cached properly. It might be because currently we're copying repo files to another dir, so first thing to check would be whether this issue still exists after we toss the now-deprecated output-dir option to the build script.

...instead of the URL to repo's travis build status... which of course won't exist.

To solve #21, I went with custom edits to the table sorter plugin. Modifying it will also allow us to resolve #18 and #17

As a result, the file is no longer minified.

We should add automatic minification of JS files (and possibly CSS as well), so we won't have to do that manually on each edit.

use github username.

See if we can add OpenSearch support:

http://irclog.perlgeek.de/perl6/2015-11-24#i_11588796
http://www.opensearch.org/Home

Since panda currently doesn't precompile, it's worth mentioning zef and the whole precompilation issue somewhere.

rakudobrew build-zef

Example list of modules (with one module):

{
    "Benchmark::Plot": {
        "last_updated": "2011-04-20T13:26:41-07:00",
        "badge_has_tests": null,
        "badge_is_fresh": "",
        "name": "Benchmark::Plot",
        "badge_panda": "1",
        "badge_is_popular": "",
        "description": "Simple benchmark graphing script",
        "badge_has_readme": "http://github.com/colomon/Benchmark-Plot/blob/master/README",
        "auth": "colomon",
        "url": "https://github.com/colomon/Benchmark-Plot/",
        "home": "github",
        "repo_name": "Benchmark-Plot"
    }
}

What is the problem? Instead of booleans true and false (JSON::true and JSON::false), 1 and "" is used instead (in one case it's even null). The problem is that it's implementation detail, not every language has those values as truthy or falsy. While it isn't important in Perl or JavaScript, it is important, for example in Ruby where 0 and "" is true.

It would be handy if you could look up docs for a module (assuming a repo has the README) by visiting to http://modules.perl6.org/repo/Foo::Bar::Baz where Foo::Bar::Baz is the module you want to look up.

This saves going to modules site, searching for module, then clicking on repo link.

The build script can easily generate .htaccess with Apache's mod_rewrite rules for the redirects.

@moritz is mod_rewrite module enabled on the Apache that hosts the site? I could write up some code to make this feature.

We currently have a SHELTER in the ecosystem repo that seems to be a collection of abandoned modules, available for adoption.

We should add a different Dist Source that handles these and display them on the site slightly different, to attract the attention of potential adopters.

We likely should add a source field to the Dists model to know where dists is from and display it accordingly on the site.

P.S.: we currently do have two sheltered dists

@szabgab there's a problem with the current implementation of Travis badges. Travis doesn't seem to like 385+ simultaneous requests for badges and responds with 429s (HTTP Too Many Requests).

We need to amend the implementation and only show badges on request for specific modules (on-click or on-hover). Then users would be able to look up build status on a module they're interested in

perl build-project-list.pl --no-app-start says
database generated; app restarted

Some of the site's features aren't entirely obvious right away, but can be valuable.

We need to add "tips" that would advertize these features. For example, how GitHub uses their "ProTips" to give hints on search queries:

Here are currently-implemented features to include:

• Link to a search query: http://modules.perl6.org/#q=<SEARCH TERM> (e.g. http://modules.perl6.org/#q=Test)
• Link to dist's repo (to look at docs/issues): http://modules.perl6.org/repo/<DIST NAME> (e.g. http://modules.perl6.org/repo/Parse::Selenese)

As the site is fleshed out, more tips will be added.

For placement, these two locations look inviting:

Here are some problems / potential improvements for the current sorting plugin:

• #21 needs custom edits to use a sprite instead of individual images. We don't actually need images there at all; we can use Unicode chars. If we do that, we likely can get rid of custom edits (just use CSS content property to display proper Unicode char)
• We have Issues #18 (inconvenient sorting) and #17 (sort not preserved on Back button). #23 is sorta related too

@azawawi also mentioned http://datatables.net/ that provides pagination. So, instead of doing anything about the above, we should see if the datatable stuff is better.

Just a note, in case anyone notices the current breakage and starts trying to fix it:

The breakage was done by the commit implementing #21 . The new code requires Mojolicious::Plugin::AssetPack (and supporting modules), which currently are not installed. Server admin @moritz will install those when they have a chance, and then the site should work fine.

To resolve the issue:

apt-get install libpng12-dev;
cpanm Imager::File::PNG Mojolicious::Plugin::AssetPack 

Cheers,
ZZ

There's a bit of bit rot in the app that powers the site. New major version of Mojolicious::Plugin::AssetPack uses a different interface and the old interface our app uses, while still works, is deprecated and will be removed in the future.

On further inspection, other stuff has bitrotted too: Bootstrap plugin and AssetPack no longer seems to support sprites

(this was wishlisted on Raku/ecosystem#76 (comment))

The recent page exists, but it's currently pretty unknown and I don't see it linked to from anywhere on http://modules.perl6.org/recent.html

Alternatively, why not get rid of recent.html and just make the module list sortable?

Currently, each page load generates 51 HTTP requests to fetch images. That number is growing any time someone adds a logotype to their repo.

We can reduce all of these to a single request if we sprite the images, thereby improving load speed of the page and hammering the server less.

I can easily add this feature if the server admin ( @moritz ?) installs additional Perl modules and Debian packages needed for the module that can auto-generate sprites.

This is what's needed to be installed:

apt-get install libpng12-dev;
cpanm Mojolicious::Plugin::AssetPack Imager Imager::File::PNG Imager::File::GIF Imager::File::JPEG;

If you want to test that the spriter works, you can download and run this app:

cd $(mktemp -d);
git clone https://github.com/zoffixznet/debug.git .;
git checkout c91f35a87dd64297a43de1766330eb514251e699;
morbo app.pl;

The app will now start up on port 3000 and when visiting http://localhost:3000 you should see a page with two pics on it:

Let me know whether the extra software above can be installed and I'll write this feature.

Cheers,
ZZ

Instead of asking people to make PRs to the ecosystem to add their dists, we can make a special page on the modules.perl6.org where users can submit their dists for inclusion into the ecosystem. This will also allow much easier inclusion of dists from other Dist Sources (e.g. GitLab).

How it Works

1. User visits, say /upload, and gives the site a link to the repo of their dist.
2. The site normalizes the link (we could get a link to repo using git:// or https://, to META, to raw META, etc)
3. The site looks at the repo contents to analyze for common problems (JSON errors in META or missing required fields)
4. Potentially, the site can also make analysis and offer the dist author suggestions for improving it (adding a README and tests).
5. The site then makes a commit to the ecosystem, adding the dist.
6. ???
7. PROFIT!

Benefits

1. Automated check for problems in dists that immediately gives feedback to dist author instead of just showing up in logs
2. Encourage people who do not like GitHub to contribute their dists using different Dist Sources
3. Make it easier for people who are unfamiliar with git or making PRs to add dists
4. Less work for members of Perl 6 GitHub Organization, since we won't need to check each PR for problems and manually merge it.

Currently, if you do a search -> visit a module -> hit browser Back button, you're shown all modules instead of search results.

The status column has been changed to text recently, but some kind of colourful indicator would be nice to have (eg using these images).

One could even make it into a badge, though this would mean we would lose the ability to sort by the status.

It says:

You can link to a search query, like so: http://modules.perl6.org/#q=Test

Why not provide the current link instead of rather useless link to q=Test? :)

It is not good that webpage suggests that list is full while it isn't. User may think that module that provides expected functionality does not exist.

Tested in Mozilla Firefox and Internet Explorer.

List::UtilsBy
We also need to document the several options of getting sass for for AssetPack's Sass handler

Just noticing this in the build log: we have four numbers 458, 459, and 460. Either those are off-by-one errors or we're missing a dist or two from final build for whatever reason.
Total: 460

458/460 JSON::Pretty
Updated since last check
A minimal JSON (de)serializer that produces easily readable JSON

ok - 459
nok - 0

Both my Avro and Compress::Brotli modules are displayed as not set up on modules.perl6 but if you click on them it does take you to the correct travis page ?

I'm often interested in seeing what github user is behind a given module (for example if many modules have similar names, but i'm sure i want the one by user "foo")

Sadly, the /repo/ url hides that fact from me; i'd have to open a new tab, switch to it, look at the url, close it again :(

Is there a way to make the authority visible? Is that useful for anyone besides me?