pgxn / pgxn-api Goto Github PK

Maintain and serve a REST API to search PGXN mirrors

Perl 94.71% HTML 5.29%

pgxn-api's Introduction

PGXN

PGXN, the PostgreSQL Extention network, is a central distribution system for open-source PostgreSQL extension libraries. It consists of four parts:

A network of mirrors for the geographic distribution of extension packages. The master mirror is here.
PGXN Manager is a web application via which users can upload packages for distribution on the network (source).
PGXN API is a web application that syncs itself to a PGXN mirror and provides a RESTFUL interface for use by PGXN clients (source).
PGXN is the main web site for the network, and client to the API. It includes pages describing all the contents of the network, including all released distributions and their documentation, and is fully searchable. It is currently under heavy development (source).
A command-line client for installing packages distributed via the network. Development of this piecs has not yet begun.

Project Status

PGXN is now in production. As such, this repository is longer worked on, but this page remains as a pointer to all of the sub-projects that encompass the whole of PGXN.

Who’s Doing This?

I am. I’m David Wheeler, inveterate Perl and PostgreSQL hacker. I love the extensibility of PostgreSQL and have long been a fan of CPAN, the Perl community’s distributed collection of Perl software and documentation. But PostgreSQL’s extensibility is not well-known, and it’s difficult to find the extensions that do exist. I’d like to solve that problem. Care to help?

pgxn-api's People

Contributors

Stargazers

Watchers

Forkers

dsteinbrunner pombreda

pgxn-api's Issues

Indexer Confused by Symbolic Links

The recent release of pg_acl v0.1.0 triggered an error in the indexer. It chokes on pg_acl-0.1.0/test/pgxntool:

binmode() on closed filehandle GEN63 at /var/virtuals/pgxn/perlbrew/perls/perl-5.14.2/lib/5.14.2/x86_64-linux/IO/File.pm line 200.
at /var/virtuals/pgxn/perlbrew/perls/perl-5.14.2/lib/5.14.2/x86_64-linux/IO/File.pm line 200.
    IO::File::binmode(IO::File=GLOB(0x2f14de0)) called at /var/virtuals/pgxn/perlbrew/perls/perl-5.14.2/lib/site_perl/5.14.2/Archive/Zip.pm line 373
    Archive::Zip::_binmode(IO::File=GLOB(0x2f14de0)) called at /var/virtuals/pgxn/perlbrew/perls/perl-5.14.2/lib/site_perl/5.14.2/Archive/Zip/Member.pm line 1021
    Archive::Zip::Member::extractToFileHandle(Archive::Zip::ZipFileMember=HASH(0x2f0d9e0), IO::File=GLOB(0x2f14de0)) called at /var/virtuals/pgxn/perlbrew/perls/perl-5.14.2/lib/site_perl/5.14.2/Archive/Zip/Member.pm line 489
    Archive::Zip::Member::extractToFileNamed(Archive::Zip::ZipFileMember=HASH(0x2f0d9e0), "www/src/pg_acl/pg_acl-0.1.0/test/pgxntool") called at lib/PGXN/API/Sync.pm line 200
    PGXN::API::Sync::unzip(PGXN::API::Sync=HASH(0x1466118), "dist/pg_acl/0.1.0/pg_acl-0.1.0.zip", HASH(0x2ef5bb0)) called at lib/PGXN/API/Sync.pm line 156
    PGXN::API::Sync::validate_distribution(PGXN::API::Sync=HASH(0x1466118), "dist/pg_acl/0.1.0/META.json") called at lib/PGXN/API/Sync.pm line 79
    PGXN::API::Sync::update_index(PGXN::API::Sync=HASH(0x1466118)) called at lib/PGXN/API/Sync.pm line 40
    PGXN::API::Sync::run(PGXN::API::Sync=HASH(0x1466118)) called at blib/script/pgxn_api_sync line 41

I downloaded that zip and unzipped it manually, and saw:

 unzip pg_acl-0.1.0.zip 
Archive:  pg_acl-0.1.0.zip
541ff97f51766f3d471471adab4baa8c44f13c46
   creating: pg_acl-0.1.0/
  inflating: pg_acl-0.1.0/.gitignore  
  inflating: pg_acl-0.1.0/LICENSE.md  
  inflating: pg_acl-0.1.0/META.in.json  
  inflating: pg_acl-0.1.0/META.json  
 extracting: pg_acl-0.1.0/Makefile   
  inflating: pg_acl-0.1.0/README.asc  
  inflating: pg_acl-0.1.0/pg_acl.control  
   creating: pg_acl-0.1.0/pgxntool/
 extracting: pg_acl-0.1.0/pgxntool/.gitignore  
  inflating: pg_acl-0.1.0/pgxntool/JSON.sh  
  inflating: pg_acl-0.1.0/pgxntool/JSON.sh.LICENCE  
  inflating: pg_acl-0.1.0/pgxntool/LICENSE  
  inflating: pg_acl-0.1.0/pgxntool/META.in.json  
  inflating: pg_acl-0.1.0/pgxntool/README.asc  
  inflating: pg_acl-0.1.0/pgxntool/_.gitignore  
  inflating: pg_acl-0.1.0/pgxntool/base.mk  
  inflating: pg_acl-0.1.0/pgxntool/build_meta.sh  
  inflating: pg_acl-0.1.0/pgxntool/meta.mk.sh  
  inflating: pg_acl-0.1.0/pgxntool/setup.sh  
   creating: pg_acl-0.1.0/pgxntool/test/
   creating: pg_acl-0.1.0/pgxntool/test/pgxntool/
  inflating: pg_acl-0.1.0/pgxntool/test/pgxntool/psql.sql  
  inflating: pg_acl-0.1.0/pgxntool/test/pgxntool/setup.sql  
  inflating: pg_acl-0.1.0/pgxntool/test/pgxntool/tap_setup.sql  
   creating: pg_acl-0.1.0/sql/
  inflating: pg_acl-0.1.0/sql/acl.sql  
   creating: pg_acl-0.1.0/test/
 extracting: pg_acl-0.1.0/test/deps.sql  
   creating: pg_acl-0.1.0/test/expected/
  inflating: pg_acl-0.1.0/test/expected/acl_type.out  
  inflating: pg_acl-0.1.0/test/expected/build.out  
  inflating: pg_acl-0.1.0/test/expected/compat.out  
  inflating: pg_acl-0.1.0/test/expected/rights.out  
    linking: pg_acl-0.1.0/test/pgxntool  -> ../pgxntool/test/pgxntool 
   creating: pg_acl-0.1.0/test/sql/
  inflating: pg_acl-0.1.0/test/sql/acl_type.sql  
  inflating: pg_acl-0.1.0/test/sql/build.sql  
  inflating: pg_acl-0.1.0/test/sql/compat.sql  
  inflating: pg_acl-0.1.0/test/sql/rights.sql  
finishing deferred symbolic links:
  pg_acl-0.1.0/test/pgxntool -> ../pgxntool/test/pgxntool

Note the last two lines: turns out that pg_acl-0.1.0/test/pgxntool is a symbolic link:

> ll test 
total 16
-rw-r--r--@ 1 david  staff    65B Jan  7 15:05 deps.sql
drwxr-xr-x@ 6 david  staff   204B Jan  7 15:05 expected/
lrwxr-xr-x  1 david  staff    25B Jan  8 09:10 pgxntool@ -> ../pgxntool/test/pgxntool
drwxr-xr-x@ 6 david  staff   204B Jan  7 15:05 sql/

Weirdly it seems to point to the wrong place; ../pgxntool/test/pgxntool does not exist, though ../pgxntool does.

Anyway, tweak the indexer to either resolve links to other files or directories in the zip file, or to ignore them altogether if they resolve to files outside of the zip file.

Mongo FDW Source in the Wrong Place

The mongo_fdw 1.0.0 src was unwrapped into /src/mongo_fdw but should have been in /src/mongo_fdw-1.0.0. I fixed it manually, but should figure out WTF it did it that way.

Add JSONP Support

Allow any request to use JSONP. Should be simple with Plack::Middleware::JSONP.

Distribution names in URLs should resolve case-insensitlvely

This works:

http://api.pgxn.org/dist/pgtap/0.94.0/META.json

But this does not:

http://api.pgxn.org/dist/pgTAP/0.94.0/META.json

Seems like, since release and extension names are case-insensitive, maybe the should be allowed to be any case in the URI. Perhaps even this?

http://api.pgxn.org/dist/PGTAP/0.94.0/META.json

Same goes for these?

http://api.pgxn.org/src/pgTAP/pgtap-0.94.0/
http://master.pgxn.org/dist/pgTAP/1.0.0/META.json

Version Bug in amqp.json

This looks wrong. Note that version 0.2.0 is listed in pg_amqp 0.3.0. This is wrong. There is no amqp 0.2.0 extension, only pg_amqp 0.2.0 and amqp 0.3.0. Might have been broken somehow when I reindexed with a new zip file that had a broken META.json with amqp 0.2.0. So probably the reindexing needs to properly delete an extension that was listed in the previous indexing and not in the new one.

Source Permissions can be Wrong

Noticed that http://api.pgxn.org/src/email/email-0.1.0/Makefile was returning NOT FOUND. Reason: email-0.1.0 did not have its execution bit set for all. It's owned by the pgxn user, and the pgxn_api user could not cd into it.

So probably need to audit the distribution unpacking code to make sure that permissions are sufficient for newly-unpacked distributions for anyone to be able to change directories and read files.

Search Parser Chokes on Null Byte

This request returns an error:

https://pgxn.org/search?q=..%00&in=extensions

The error output is:

Failed to parse ' '
	S_parse_flat_string at /build/liblucy-perl-XPo8ef/liblucy-perl-0.3.3/core/Lucy/Search/QueryParser.c line 301
	at PGXN/API/Searcher.pm line 52.

Not Using docfile for Extension

Omnipitr 0.1.0 specifies doc/howto.pod as the docfile for the "omnipitr" extension. Yet the site isn't linking the extension to that doc. Why not?

Ensure Tag Case is Correct

Just got this error from pgxn_api_sync:

/etc/cron.hourly/sync_pgxn:
Cannot open www/mirror/tag/Italian.json: No such file or directory
run-parts: /etc/cron.hourly/sync_pgxn exited with return code 2

Curiously, while the master mirror has an italian.json, the API mirror does not. And neither has Italian.json. So I suspect it's not doing the right thing. It should be lowercasing the tag name for the JSON file at all times. Right? Confirm with the PGXN Manager code.

t/indexer.t may fail (non-English locale?)

Some of my smokers report the following failure, probably only when a German locale is in effect, and probably also only for perl <= 5.22:

#   Failed test 'Should get exception with file names for bad copy'
#   at t/indexer.t line 166.
# expecting: Regexp ((?^:Cannot copy t\/test_doc_root\/mirror\/dist\/nonexistent\/0\.1\.0\/nonexistent\-0\.1\.0\.zip to t\/test_doc_root\/dist\/nonexistent\/0\.1\.0\/nonexistent\-0\.1\.0\.zip: No such file or directory))
# found: Cannot copy t/test_doc_root/mirror/dist/nonexistent/0.1.0/nonexistent-0.1.0.zip to t/test_doc_root/dist/nonexistent/0.1.0/nonexistent-0.1.0.zip: Datei oder Verzeichnis nicht gefunden
# Looks like you failed 1 test of 249.
t/indexer.t .. 
Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/249 subtests

Detect Other Special Files

Other special files to detect:

INSTALL
LICENSE
COPYING
AUTHORS

Binmode Warning During Sync

Seen a few of these. The latest, with stack trace:

binmode() on closed filehandle GEN16 at /var/virtuals/pgxn/perlbrew/perls/perl-5.14.2/lib/5.14.2/x86_64-linux/IO/File.pm line 200.
at /var/virtuals/pgxn/perlbrew/perls/perl-5.14.2/lib/5.14.2/x86_64-linux/IO/File.pm line 200.
    IO::File::binmode('IO::File=GLOB(0x3060190)') called at /var/virtuals/pgxn/perlbrew/perls/perl-5.14.2/lib/site_perl/5.14.2/Archive/Zip.pm line 370
    Archive::Zip::_binmode('IO::File=GLOB(0x3060190)') called at /var/virtuals/pgxn/perlbrew/perls/perl-5.14.2/lib/site_perl/5.14.2/Archive/Zip/Member.pm line 1039
    Archive::Zip::Member::extractToFileHandle('Archive::Zip::ZipFileMember=HASH(0x3051360)', 'IO::File=GLOB(0x3060190)') called at /var/virtuals/pgxn/perlbrew/perls/perl-5.14.2/lib/site_perl/5.14.2/Archive/Zip/Member.pm line 498
    Archive::Zip::Member::extractToFileNamed('Archive::Zip::ZipFileMember=HASH(0x3051360)', 'www/src/json_enhancements/json_enhancements-1.0.1/doc/json_en...') called at lib/PGXN/API/Sync.pm line 200
    PGXN::API::Sync::unzip('PGXN::API::Sync=HASH(0x19f8118)', 'dist/json_enhancements/1.0.1/json_enhancements-1.0.1.zip', 'HASH(0x3040380)') called at lib/PGXN/API/Sync.pm line 156
    PGXN::API::Sync::validate_distribution('PGXN::API::Sync=HASH(0x19f8118)', 'dist/json_enhancements/1.0.1/META.json') called at lib/PGXN/API/Sync.pm line 79
    PGXN::API::Sync::update_index('PGXN::API::Sync=HASH(0x19f8118)') called at lib/PGXN/API/Sync.pm line 40
    PGXN::API::Sync::run('PGXN::API::Sync=HASH(0x19f8118)') called at blib/script/pgxn_api_sync line 41

Handle Tags with Slashes

Slony-I was just released with the tag "pl/pgsql". The sync choked on that:

Cannot open www/tag/pl/pgsql.json: No such file or directory

So it needs to be modified to escape platform directory separators.

Ignore Docs with No Text

The kmeans distribution includes a Perl script, plot.pl. The indexer found it and ran it through Text::Markup, with this result. Not very useful. So either Text::Markup needs to return nothing when no text was found, or the indexer should notice it and not index it.

Show unstable distributions on PGXN when no stable

People are often confused when they upload a non-stable distribution and it fails to show up in search on the site. This is because the API does not index non-stable distributions for search. However, I think we should make an exception and index a non-stable distribution if there are no stable releases for the distribution at all.

Add Suggest Support to Search

Maybe use LucyX::Suggester.

Search Query Crashes Lucy

This search makes Lucy barf:

http://pgxn.org/search?q=json%00'%7C%7CSLeeP(3)%26%26'1&in=docs

Consolidate Search

The search functionality is a bit non-intuitive, unfortunately. I think this is mainly because so many extension ship with no documentation, so the default search doesn’t find them.

So I think we need to consolidate the into a single search instead of five. Results should probably just show extensions and users. The info for extensions will still link to the dist page, of course. Maybe tags, too?

Consider Indexing Non-Stable Releases When None are Stable

Maybe allow non-stable releases to be indexed for FTS if there are no stable releases?

Sync Order Matters

I tried to do a fresh sync on a new box, but it failed, saying something like "Cannot open www/users/foo.json: File does not exist". Running the sync with -VVV, I saw that it was syncing a distribution, and as part of that, wanted to update the user JSON file. But the user JSON file has not been synced, and so did not yet exist!

So the sync code needs to either copy the user JSON file over if it does not exist, or sync user JSON files before syncing distributions. And maybe extensions and other stuff too?

Not Found on Search

Go on pgxn.org
search for "json" -> http://pgxn.org/search?q=json&in=docs
the first hit is "Test types" (www_fdw 0.1.6)
click on the first link -> http://pgxn.org/dist/www_fdw/test/readme.html
not found

Add README to documentation Index if no other docs

Quite a few distributions have only one extension and no documentation file other than the README. Authors can get around this by adding the README under the docfile key in the provides section, but it seems silly.

So, add a rule to the indexer such that, if the distribution:

Has only one extension
Has no documentation file for that extension
Has a README

Then automatically index the README as documentation.