dspace / xoai Goto Github PK

View Code? Open in Web Editor NEW

28.0 28.0 50.0 1.55 MB

OAI-PMH Java Toolkit

Java 99.96% XSLT 0.04%

dspace java oai-pmh repository

xoai's Introduction

DSpace

DSpace Documentation | DSpace Releases | DSpace Wiki | Support

Overview

DSpace open source software is a turnkey repository application used by more than 2,000 organizations and institutions worldwide to provide durable access to digital resources. For more information, visit http://www.dspace.org/

DSpace consists of both a Java-based backend and an Angular-based frontend.

Backend (this codebase) provides a REST API, along with other machine-based interfaces (e.g. OAI-PMH, SWORD, etc)
- The REST Contract is at https://github.com/DSpace/RestContract
Frontend (https://github.com/DSpace/dspace-angular/) is the User Interface built on the REST API

Prior versions of DSpace (v6.x and below) used two different UIs (XMLUI and JSPUI). Those UIs are no longer supported in v7 (and above).

A maintenance branch for older versions is still available, see dspace-6_x for 6.x maintenance.

Downloads

Backend (REST API): https://github.com/DSpace/DSpace/releases
Frontend (User Interface): https://github.com/DSpace/dspace-angular/releases

Documentation / Installation

Documentation for each release may be viewed online or downloaded via our Documentation Wiki.

The latest DSpace Installation instructions are available at: https://wiki.lyrasis.org/display/DSDOC7x/Installing+DSpace

Please be aware that, as a Java web application, DSpace requires a database (PostgreSQL) and a servlet container (usually Tomcat) in order to function. More information about these and all other prerequisites can be found in the Installation instructions above.

Running DSpace 7 in Docker

NOTE: At this time, we do not have production-ready Docker images for DSpace. That said, we do have quick-start Docker Compose scripts for development or testing purposes.

See Running DSpace 7 with Docker Compose

Contributing

See Contributing documentation

Getting Help

DSpace provides public mailing lists where you can post questions or raise topics for discussion. We welcome everyone to participate in these lists:

[email protected] : General discussion about DSpace platform, announcements, sharing of best practices
[email protected] : Technical support mailing list. See also our guide for How to troubleshoot an error.
[email protected] : Developers / Development mailing list

Great Q&A is also available under the DSpace tag on Stackoverflow

Additional support options are at https://wiki.lyrasis.org/display/DSPACE/Support

DSpace also has an active service provider network. If you'd rather hire a service provider to install, upgrade, customize or host DSpace, then we recommend getting in touch with one of our Registered Service Providers.

Issue Tracker

DSpace uses GitHub to track issues:

Backend (REST API) issues: https://github.com/DSpace/DSpace/issues
Frontend (User Interface) issues: https://github.com/DSpace/dspace-angular/issues

Testing

Running Tests

By default, in DSpace, Unit Tests and Integration Tests are disabled. However, they are run automatically by GitHub Actions for all Pull Requests and code commits.

How to run both Unit Tests (via maven-surefire-plugin) and Integration Tests (via maven-failsafe-plugin):
```
mvn install -DskipUnitTests=false -DskipIntegrationTests=false
```
How to run only Unit Tests:
```
mvn test -DskipUnitTests=false
```

How to run a single Unit Test

# Run all tests in a specific test class
# NOTE: failIfNoTests=false is required to skip tests in other modules
mvn test -DskipUnitTests=false -Dtest=[full.package.testClassName] -DfailIfNoTests=false

# Run one test method in a specific test class
mvn test -DskipUnitTests=false -Dtest=[full.package.testClassName]#[testMethodName] -DfailIfNoTests=false

How to run only Integration Tests

mvn install -DskipIntegrationTests=false

How to run a single Integration Test

# Run all integration tests in a specific test class
# NOTE: failIfNoTests=false is required to skip tests in other modules
mvn install -DskipIntegrationTests=false -Dit.test=[full.package.testClassName] -DfailIfNoTests=false

# Run one test method in a specific test class
mvn install -DskipIntegrationTests=false -Dit.test=[full.package.testClassName]#[testMethodName] -DfailIfNoTests=false

How to run only tests of a specific DSpace module

# Before you can run only one module's tests, other modules may need installing into your ~/.m2
cd [dspace-src]
mvn clean install

# Then, move into a module subdirectory, and run the test command
cd [dspace-src]/dspace-server-webapp
# Choose your test command from the lists above

License

DSpace source code is freely available under a standard BSD 3-Clause license. The full license is available in the LICENSE file or online at http://www.dspace.org/license/

DSpace uses third-party libraries which may be distributed under different licenses. Those licenses are listed in the LICENSES_THIRD_PARTY file.

xoai's People

Contributors

Stargazers

Watchers

xoai's Issues

HTTPS oai_dc namespace is not supported

I'm trying to parse the records list of a OAI Repository, but XOAI returns all records as empty records. This happens because the response namespace is declared as HTTPS.

To give an example, the XML response I get is:

<OAI-PMH xmlns="https://www.openarchives.org/OAI/2.0/" (...) />

This makes XOAI return a list of empty records. If I change the header to:

<OAI-PMH xmlns="http://www.openarchives.org/OAI/2.0/" (...) />

Then all records are correctly parsed.

XOAI should support the ability to parse records in HTTPS namespace.

The repository where I'm getting this error is: https://journalsporl.com/index.php/sporl/oai

Metadata xml snippet misses defined namespace(s) defined in parent xml elements

The defined namespace xsi is not copied when extracting the mods:mods element within the metadata section. When this extraction is used in a namespace aware parser it will fail because the attribute xsi is not bound (xsi:schemaLocation).
Any defined namespaces in upper xml element should be copied to the metadata extract.

<OAI-PMH
    xmlns="http://www.openarchives.org/OAI/2.0/"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.openarchives.org/OAI/2.0/ http://www.openarchives.org/OAI/2.0/OAI-PMH.xsd">
  <responseDate>xxxx</responseDate>
  <request from="xxxx" set="xxx" metadataPrefix="mods" verb="ListRecords">xxxx</request>
  <ListRecords>
    <record>
      <header>
        <identifier>xxxx</identifier>
        <datestamp>xxxx/datestamp>
        <setSpec>xxxx</setSpec>
      </header>
      <metadata>
        <mods:mods xmlns:mods="http://www.loc.gov/mods/v3" version="3.3" xsi:schemaLocation="http://www.loc.gov/mods/v3 http://www.loc.gov/standards/mods/v3/mods-3-3.xsd">
          <mods:identifier type="nbn">xxxx</mods:identifier>
          ......
        </mods:mods>
      </metadata>
    </record>
  </ListRecords>
</OAI-PMH>

Request for minor release

I fixed an issue in 4.1.0 a long time ago:
#30

I just noticed however that the release 4.2.0 wasn't performed yet.

Since this is already fixed and there are quite a few pull requests, would it be possible to integrate these and perform a minor release?

Delete ConfigurationManager

There is no need for the ConfigurationManager class. A better alternative is to have a method in the Configuration which would write/read to/from a file.

Error on Identify with optional elements

When a response for the identify request does not have the optional elements 'compression' and 'description' an exception 'NoSuchElementException' is thrown

Maven is missing jar starting version 4.0.0

The latest version (4.1.0) of XOAI starting 4.0.0 is not available on Maven central. It shows an entry but does show actually have the jar file.

If I create a maven/gradle project with xoai dependency 4+, no library gets downloaded

http://mvnrepository.com/artifact/com.lyncode/xoai/4.1.0

Can you please check?

MetadataSearch doesn't expose the entire metadata index

As a OAI repository importer
I want to see the entire OAI record metadata tree
So that I can easily get all metadata entries for that record

Create XmlOutputContext.emptyContext with granularity parameter

Just a tech task.

When using other than 'oai_dc' metadataPrefix only the first response is correct

The Parameters.withResumptionToken removes the metadataPrefix therefore the second response retrieves the data in the oai_dc and not in the original metadataPrefix.

ServiceProvider.listRecords(..) throws NoSuchElementException

Provoking test case:

    @Test
    public void listRecords() throws NoSetHierarchyException, BadArgumentException, HttpException {
        OAIClient oaiClient = new HttpOAIClient("http://www.ssoar.info/OAIHandler/request");

         Context context = new Context()
                .withOAIClient(oaiClient)
                .withMetadataTransformer( "xoai", identity() );

         ServiceProvider ssoarOaiPmhEndpoint = new ServiceProvider(context);

         ListRecordsParameters parameters = new ListRecordsParameters();
         parameters.withMetadataPrefix("xoai");
         ssoarOaiPmhEndpoint.listRecords(parameters);
    }

Log (truncated to last lines. Notice that HTTPClient's Wire class does not output a full tag on the last line):

...
DEBUG [main o.a.http.wire o.a.h.i.c.Wire 2015-10-26 17:53:14,933]  << "[\r][\n]"
DEBUG [main o.a.http.wire o.a.h.i.c.Wire 2015-10-26 17:53:14,933]  << "1c4[\r][\n]"
DEBUG [main o.a.http.wire o.a.h.i.c.Wire 2015-10-26 17:53:14,933]  << "me="identifier">oai:gesis.izsoz.de:679</field>[\n]"
DEBUG [main o.a.http.wire o.a.h.i.c.Wire 2015-10-26 17:53:14,934]  << "        <field name="lastModifyDate">2012-08-29 23:40:31.76</field>[\n]"
DEBUG [main o.a.http.wire o.a.h.i.c.Wire 2015-10-26 17:53:14,934]  << "    </element>[\n]"
DEBUG [main o.a.http.wire o.a.h.i.c.Wire 2015-10-26 17:53:14,934]  << "    <element name="repository">[\n]"
DEBUG [main o.a.http.wire o.a.h.i.c.Wire 2015-10-26 17:53:14,934]  << "        <field name="name">SSOAR - Social Science Open Access Repository</field>[\n]"
DEBUG [main o.a.http.wire o.a.h.i.c.Wire 2015-10-26 17:53:14,934]  << "        <field name="mail">[email protected]</field>[\n]"
DEBUG [main o.a.http.wire o.a.h.i.c.Wire 2015-10-26 17:53:14,934]  << "    </element>[\n]"
DEBUG [main o.a.http.wire o.a.h.i.c.Wire 2015-10-26 17:53:14,934]  << "</metadata>[\n]"
DEBUG [main o.a.http.wire o.a.h.i.c.Wire 2015-10-26 17:53:14,935]  << "</metadata>[\n]"
DEBUG [main o.a.http.wire o.a.h.i.c.Wire 2015-10-26 17:53:14,935]  << "        </record>[\n]"
DEBUG [main o.a.http.wire o.a.h.i.c.Wire 2015-10-26 17:53:14,935]  << "        <record>[\n]"
DEBUG [main o.a.http.wire o.a.h.i.c.Wire 2015-10-26 17:53:14,935]  << "            <header>[\n]"
DEBUG [main o.a.http.wire o.a.h.i.c.Wire 2015-10-26 17:53:14,935]  << "                <identifier>oai:gesis.izsoz.de:1542</identifie"

Failure stacktrace:

    at org.codehaus.stax2.ri.Stax2EventReaderImpl.throwEndOfInput(Stax2EventReaderImpl.java:453)
    at org.codehaus.stax2.ri.Stax2EventReaderImpl.nextEvent(Stax2EventReaderImpl.java:242)
    at com.lyncode.xml.XmlReader.next(XmlReader.java:134)
    at org.dspace.xoai.serviceprovider.parsers.MetadataParser.parseElement(MetadataParser.java:42)
    at org.dspace.xoai.serviceprovider.parsers.MetadataParser.parseElement(MetadataParser.java:43)
    at org.dspace.xoai.serviceprovider.parsers.MetadataParser.parseElement(MetadataParser.java:43)
    at org.dspace.xoai.serviceprovider.parsers.MetadataParser.parseElement(MetadataParser.java:43)
    at org.dspace.xoai.serviceprovider.parsers.MetadataParser.parseElement(MetadataParser.java:43)
    at org.dspace.xoai.serviceprovider.parsers.MetadataParser.parseElement(MetadataParser.java:43)
    at org.dspace.xoai.serviceprovider.parsers.MetadataParser.parseElement(MetadataParser.java:43)
    at org.dspace.xoai.serviceprovider.parsers.MetadataParser.parseElement(MetadataParser.java:43)
    at org.dspace.xoai.serviceprovider.parsers.MetadataParser.parseElement(MetadataParser.java:43)
    at org.dspace.xoai.serviceprovider.parsers.MetadataParser.parseElement(MetadataParser.java:43)
    at org.dspace.xoai.serviceprovider.parsers.MetadataParser.parseElement(MetadataParser.java:43)
    at org.dspace.xoai.serviceprovider.parsers.MetadataParser.parseElement(MetadataParser.java:43)
    at org.dspace.xoai.serviceprovider.parsers.MetadataParser.parseElement(MetadataParser.java:43)
    at org.dspace.xoai.serviceprovider.parsers.MetadataParser.parseElement(MetadataParser.java:43)
    at org.dspace.xoai.serviceprovider.parsers.MetadataParser.parseElement(MetadataParser.java:43)
    at org.dspace.xoai.serviceprovider.parsers.MetadataParser.parseElement(MetadataParser.java:43)
    at org.dspace.xoai.serviceprovider.parsers.MetadataParser.parseElement(MetadataParser.java:43)
    at org.dspace.xoai.serviceprovider.parsers.MetadataParser.parseElement(MetadataParser.java:43)
    at org.dspace.xoai.serviceprovider.parsers.MetadataParser.parseElement(MetadataParser.java:43)
    at org.dspace.xoai.serviceprovider.parsers.MetadataParser.parseElement(MetadataParser.java:43)
    at org.dspace.xoai.serviceprovider.parsers.MetadataParser.parseElement(MetadataParser.java:43)
    at org.dspace.xoai.serviceprovider.parsers.MetadataParser.parseElement(MetadataParser.java:43)
    at org.dspace.xoai.serviceprovider.parsers.MetadataParser.parseElement(MetadataParser.java:43)
    at org.dspace.xoai.serviceprovider.parsers.MetadataParser.parseElement(MetadataParser.java:43)
    at org.dspace.xoai.serviceprovider.parsers.MetadataParser.parseElement(MetadataParser.java:43)
    at org.dspace.xoai.serviceprovider.parsers.MetadataParser.parseElement(MetadataParser.java:43)
    at org.dspace.xoai.serviceprovider.parsers.MetadataParser.parseElement(MetadataParser.java:43)
    at org.dspace.xoai.serviceprovider.parsers.MetadataParser.parseElement(MetadataParser.java:43)
    at org.dspace.xoai.serviceprovider.parsers.MetadataParser.parseElement(MetadataParser.java:43)
    at org.dspace.xoai.serviceprovider.parsers.MetadataParser.parseElement(MetadataParser.java:43)
    at org.dspace.xoai.serviceprovider.parsers.MetadataParser.parseElement(MetadataParser.java:43)
    at org.dspace.xoai.serviceprovider.parsers.MetadataParser.parseElement(MetadataParser.java:43)
    at org.dspace.xoai.serviceprovider.parsers.MetadataParser.parseElement(MetadataParser.java:43)
    at org.dspace.xoai.serviceprovider.parsers.MetadataParser.parseElement(MetadataParser.java:43)
    at org.dspace.xoai.serviceprovider.parsers.MetadataParser.parseElement(MetadataParser.java:43)
    at org.dspace.xoai.serviceprovider.parsers.MetadataParser.parseElement(MetadataParser.java:43)
    at org.dspace.xoai.serviceprovider.parsers.MetadataParser.parseElement(MetadataParser.java:43)
    at org.dspace.xoai.serviceprovider.parsers.MetadataParser.parseElement(MetadataParser.java:43)
    at org.dspace.xoai.serviceprovider.parsers.MetadataParser.parseElement(MetadataParser.java:43)
    at org.dspace.xoai.serviceprovider.parsers.MetadataParser.parseElement(MetadataParser.java:43)
    at org.dspace.xoai.serviceprovider.parsers.MetadataParser.parseElement(MetadataParser.java:43)
    at org.dspace.xoai.serviceprovider.parsers.MetadataParser.parseElement(MetadataParser.java:43)
    at org.dspace.xoai.serviceprovider.parsers.MetadataParser.parseElement(MetadataParser.java:43)
    at org.dspace.xoai.serviceprovider.parsers.MetadataParser.parseElement(MetadataParser.java:43)
    at org.dspace.xoai.serviceprovider.parsers.MetadataParser.parseElement(MetadataParser.java:43)
    at org.dspace.xoai.serviceprovider.parsers.MetadataParser.parseElement(MetadataParser.java:43)
    at org.dspace.xoai.serviceprovider.parsers.MetadataParser.parseElement(MetadataParser.java:43)
    at org.dspace.xoai.serviceprovider.parsers.MetadataParser.parseElement(MetadataParser.java:43)
    at org.dspace.xoai.serviceprovider.parsers.MetadataParser.parseElement(MetadataParser.java:43)
    at org.dspace.xoai.serviceprovider.parsers.MetadataParser.parseElement(MetadataParser.java:43)
    at org.dspace.xoai.serviceprovider.parsers.MetadataParser.parseElement(MetadataParser.java:43)
    at org.dspace.xoai.serviceprovider.parsers.MetadataParser.parseElement(MetadataParser.java:43)
    at org.dspace.xoai.serviceprovider.parsers.MetadataParser.parseElement(MetadataParser.java:43)
    at org.dspace.xoai.serviceprovider.parsers.MetadataParser.parseElement(MetadataParser.java:43)
    at org.dspace.xoai.serviceprovider.parsers.MetadataParser.parseElement(MetadataParser.java:43)
    at org.dspace.xoai.serviceprovider.parsers.MetadataParser.parseElement(MetadataParser.java:43)
    at org.dspace.xoai.serviceprovider.parsers.MetadataParser.parse(MetadataParser.java:34)
    at org.dspace.xoai.serviceprovider.parsers.RecordParser.parse(RecordParser.java:56)
    at org.dspace.xoai.serviceprovider.parsers.ListRecordsParser.next(ListRecordsParser.java:60)
    at org.dspace.xoai.serviceprovider.handler.ListRecordHandler.nextIteration(ListRecordHandler.java:70)
    at org.dspace.xoai.serviceprovider.lazy.ItemIterator.hasNext(ItemIterator.java:32)
    at org.dspace.xoai.serviceprovider.lazy.ItemIterator.<init>(ItemIterator.java:22)
    at org.dspace.xoai.serviceprovider.ServiceProvider.listRecords(ServiceProvider.java:57)
    at org.gesis.dda.harvester.harvesting.xoai.XoaiTest.listRecords(XoaiTest.java:94)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
    at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
    at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
    at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
    at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
    at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:86)
    at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:459)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:675)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:382)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:192)

Sets are added to an item independently from the filter

If an oai context contains for example 2 virtual sets.
S1. Items with dc.identifier.doi
S2. Items without dc.identifier.doi
and wie have 2 Items:
A with doi
B without doi
then as expected:
S1 only contains item A
S2 only contains item B
But if you look at Item A with GetRecord than the record shows that the item is part of S1 and S2 which is false.
This is due to a bug in version 3.2.9 of the xoai library:
https://github.com/lyncode/xoai/blob/xoai-3.2.9/src/main/java/com/lyncode/xoai/dataprovider/data/internal/ItemIdentifyHelper.java#L23
It add's a set to an item independent of the filter.

Examples in Service-Provider wiki page are broken

The examples don't seem to exist in the source code. The corresponding wiki page consequently throws an exception.

XLS to map xoai to oai_dc?

Hello,

I am trying to use xoai-data provider as the implementation of oai protocol. In version 3.x in order to map internal metadata format xoai to oai_dc oai_dc.xsl was used. It looks like in version 4.x the internal metadata format was changed and oai_dc.xsl from version 3.2.x does not work any more. By any chance, can you forward where I can find that implementation ?

Thank you in advance.

setSpec not properly parsed

com.lyncode.xoai.serviceprovider.util.XMLUtils.getHeader()

Line 152

else if (item.item(i).getNodeName().toLowerCase().equals("setSpec"))

should be:

else if (item.item(i).getNodeName().toLowerCase().equals("setspec"))

Notice all lower characters!

How to filter with metadataPrefix

I don't see how can we retrieve the metadataPrefix from the request to the methods getItems, getIem and getItemIdentifiers.

I just want to make a query to the data source with the metadataPrefix.

Record header does not need a setSpec

If a record does not have a setSpec in its header then
Expected:
The record is harvest
Actual:
It is thrown an exception (no such element).

(See http://www.openarchives.org/OAI/openarchivesprotocol.html#Record)

How do you retrieve a record from a repository?

I am playing around with the framework and trying to retrieve a record (see test case below), but it is failing with something to do with the Transformers:

    at com.lyncode.xoai.xml.XSLPipeline.process(XSLPipeline.java:36)
    at com.lyncode.xoai.serviceprovider.parsers.RecordParser.parse(RecordParser.java:67)
    at com.lyncode.xoai.serviceprovider.parsers.GetRecordParser.parse(GetRecordParser.java:65)
    at com.lyncode.xoai.serviceprovider.handler.GetRecordHandler.handle(GetRecordHandler.java:45)
    at com.lyncode.xoai.serviceprovider.ServiceProvider.getRecord(ServiceProvider.java:61)

What is missing in the example? This is definitely that would be tackled in #13 .
Note that I've also tried without the context.withTransformer(t). withTransformer the code will reach the exception cycle with two transformers in the list, the one supplied and a null.

public class RemoteRepoTest {

    ServiceProvider underTest;

    @Before
    public void before() throws TransformerConfigurationException, TransformerFactoryConfigurationError{
        Context context = new Context().withBaseUrl("http://www.rcaap.pt/oai");
        Transformer t  = TransformerFactory.newInstance().newTransformer();
        context.withTransformer(t);
        HttpOAIClient client = new HttpOAIClient(context);
        context.withOAIClient(client);

        underTest = new ServiceProvider(context);

    }

    @Test
    public void remoteRepoTest() throws Exception {

        Identify identify = underTest.identify();
        assertEquals(identify.getRepositoryName(), "RCAAP - Repositório Científico de Acesso Aberto de Portugal");
    }

    @Test
    public void getRecord() throws Exception {
        GetRecordParameters parameters = new GetRecordParameters().withIdentifier("oai:digituma.uma.pt:10400.13/9").withMetadataFormatPrefix("oai_dc");
        Record record = underTest.getRecord(parameters);
        record.getAbouts();
    }
}

don't encode parameters other than set name in resumptionToken

This enables requesting another page of results more easily.

Context Filters not working in 3.2.7

Probably a typo in configuration parser. "red" should be "ref"
https://github.com/lyncode/xoai/blob/xoai-3.2.7/src/main/java/com/lyncode/xoai/dataprovider/xml/xoaiconfig/parse/ContextConfigurationParser.java#L25

The 4.0 has different structure so I'm not sure whether it's still an issue. But future version of @DSpace seems to be prepared for 3.2.7

Allow setting the User Agent in HTTP requests

Some repositories will return a HTTP 403 Forbidden when HttpOAIClient tries to execute requests. This occurs because the User Agent property is not properly configured.

Therefore, to avoid this issue, HttpOAIClient should allow changing the User Agent.

Example of OAI repository where this error occurs: http://repositorio.insa.pt/oaiextended/request

MalformedByteSequenceException

Hello
I get a MalformedByteSequenceException executing this:

`String url = "http://www.investigacionesgeograficas.com/oai";

    OAIClient oaiClient = new HttpOAIClient(url);

    Context context;
    try {
        context = new Context()
         .withOAIClient(oaiClient)
         .withMetadataTransformer(FORMAT, KnownTransformer.OAI_DC);

        ServiceProvider underTest = new ServiceProvider(context);

        ListRecordsParameters parameters = ListRecordsParameters.request();
        Calendar cal = Calendar.getInstance();
        cal.add(Calendar.DAY_OF_MONTH, -50);
        Date from = cal.getTime();
        parameters.withFrom(from);
        parameters.withMetadataPrefix("oai_dc");

        Iterator<Record> it = underTest.listRecords(parameters);
        while(it.hasNext()){
            Record record = it.next();
            System.out.println(record.getMetadata().getValue().searcher().findAll("dc.title"));
        };


    } catch (BadArgumentException e) {
        e.printStackTrace();
    }catch (TransformerFactoryConfigurationError e1) {
        e1.printStackTrace();
    }`

I am using xoai 4.1.0

The stacktrace:

com.sun.org.apache.xml.internal.utils.WrappedRuntimeException: Invalid byte 3 of 3-byte UTF-8 sequence.
at com.sun.org.apache.xalan.internal.xsltc.dom.XSLTCDTMManager.getDTM(XSLTCDTMManager.java:460)
at com.sun.org.apache.xalan.internal.xsltc.dom.XSLTCDTMManager.getDTM(XSLTCDTMManager.java:248)
at com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.getDOM(TransformerImpl.java:542)
at com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.transform(TransformerImpl.java:725)
at com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.transform(TransformerImpl.java:336)
at com.lyncode.xoai.xml.XSLPipeline.process(XSLPipeline.java:37)
at com.lyncode.xoai.serviceprovider.parsers.RecordParser.parse(RecordParser.java:73)
at com.lyncode.xoai.serviceprovider.parsers.ListRecordsParser.next(ListRecordsParser.java:70)
at com.lyncode.xoai.serviceprovider.handler.ListRecordHandler.nextIteration(ListRecordHandler.java:83)
at com.lyncode.xoai.serviceprovider.lazy.ItemIterator.hasNext(ItemIterator.java:40)
at com.lyncode.xoai.serviceprovider.lazy.ItemIterator.(ItemIterator.java:30)
at com.lyncode.xoai.serviceprovider.ServiceProvider.listRecords(ServiceProvider.java:65)

The problem is in the method "parse" from RecordParse. It is getting bytes without encoding:
inputStream = new ByteArrayInputStream(content.getBytes());

GDCC fork with version 5 - polished codebase, new features, Java 11+ only

Just a note: for Dataverse also relying on this library, we have been hacking on it to create a new version 5.0 and are near to a first release on Maven Central.

You may find it here https://github.com/gdcc/xoai

Would be nice to get in touch!

MetadataSearchImpl creates repeated values in index

Example for an oai xml containing:
dc:title Some title /dc:title

MetadataSearchImpl.consume will create an index with two items:
dc.title - Some title
dc.title:value - Some title

Shouldn't the expected index contain one entry only?

javadoc warnings

[WARNING] Javadoc Warnings
[WARNING] /home/dspace/svn/xoai-common/src/main/java/com/lyncode/xoai/common/dataprovider/core/Set.java:40: warning - @param argument "description" is not a parameter name.
[WARNING] /home/dspace/svn/xoai-common/src/main/java/com/lyncode/xoai/common/dataprovider/data/AbstractItem.java:33: warning - @return tag has no arguments.
[WARNING] /home/dspace/svn/xoai-common/src/main/java/com/lyncode/xoai/common/dataprovider/data/AbstractItemIdentifier.java:39: warning - @return tag has no arguments.

Error while parsing metadata

Giving the following response XML from an OAI repository:

<?xml version="1.0" encoding="UTF-8"?>
<OAI-PMH xmlns="http://www.openarchives.org/OAI/2.0/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.openarchives.org/OAI/2.0/ http://www.openarchives.org/OAI/2.0/OAI-PMH.xsd">
   ...
   <ListRecords>
      <record>
         <header>
            ...
         </header>
         <metadata>
            <crossref xmlns="http://www.crossref.org/xschema/1.1" xsi:schemaLocation="http://www.crossref.org/xschema/1.1 http://www.crossref.org/schema/unixref1.1.xsd">
               ...
            </crossref>
         </metadata>
      </record>
   </ListRecords>
</OAI-PMH>

We get an error if we try to parse it using XOAI.

ERROR: 'The prefix "xsi" for attribute "xsi:schemaLocation" associated with an element type "crossref" is not bound.'

From what I could deduce from reading the source code and debugging, each metadata node is parsed individually by the XOAI library. And, in that context, the xsi prefix is indeed not bound, as it is only defined in parent node OAI-PMH.

There should be a way to configure the XML reader to ignore namespace errors.

xoai documentation for 4.1.x

I would like to implement a standalone oai-pmh version 2 data provider server using xoai 4.1.x without DSpace.

Can anyone direct me to any documentation for doing this or to a reference implementation?
Thanks David

Error retrieving Sets

When I request the ListSets and in the response the setName element comes first than the setSpec the Sets are not retrieved because the ListSetsParser expects that setSpec is always first than the setName

Parameters always converts dates to same format

Parameters uses UTCDateProvider to convert dates. But the method used always converts to "yyyy-MM-dd'T'HH:mm:ss'Z'".

If the repository has a different granularity this will yield in an unsuccessful harvest.

Several potential NullPointerException bugs.

Hi all,

Our tool has found several potential NPE bugs.

In the statement filters.add(new ScopedFilter(context.getSet(setSpec).getCondition(), Scope.Set)); at line 168, context.getSet(setSpec) may return null according to its definition at line 110.
Similar to 1, statements String until = request.getString(Until); and String from = request.getString(From); may return null according to the definition of method getString at line 112. The usage of variables until and ```from`` is as follows (from lines 92 to 97):

        String until = request.getString(Until);
        String from = request.getString(From);
        if (isTrueThat(until, is(not(nullValue())))
                && isTrueThat(from, is(not(nullValue())))
                && from.length() != until.length())
            throw new BadArgumentException("Distinct granularities provided for until and from parameters");

The method formatForPrefix may return null. The following code in method createHeader may use the null because method canDisseminate :

        MetadataFormat format = getContext().formatForPrefix(parameters
                .getMetadataPrefix());
        if (!itemIdentifier.isDeleted() && !canDisseminate(itemIdentifier, format))
            throw new InternalOAIException("The item repository is currently providing items which cannot be disseminated with format "+format.getPrefix());

Note that method canDisseminate uses the argument format by return !format.hasCondition() || format.getCondition()...; at line 129

Thanks.

IdentifierIterator ignores a 'status' attribute of a header

I use HarvesterManager#listIdentifier to list identifiers (headers). It returns a ListIdentifiers which gives access to IdentifierIterator via iterator method. IdentifierIterator uses XMLUtils#getIdentifier to unmarshall xml to Identifier object. Unfortunately XMLUtils#getIdentifier ignores a 'status' attribute of a header. However XMLUtils#getRecord handles the 'status' attribute properly. According to the OAI-PMH v2.0 specification (http://www.openarchives.org/OAI/openarchivesprotocol.html#ListIdentifiers) "... a returned header may have a status attribute of "deleted" if a record matching the arguments specified in the request has been deleted." I think that a 'status' attribute should not be ignored if you list identifiers.

Request for minor release 4.1.1

Hi,

I'm currently using latest updates in 4.1.1-SNAPSHOT. Whenever I want to build my system on a new machine I have to check out the project and do a maven install which is a bit inconvenient. I am wondering if you have any plans to release 4.1.1 ?

Thanks,
Masoud

XOAI Data Provider Configuration could return an InputStream.

Utility

Compilation issue

First, it seems to me that running "mvn install" is not idempotent. At
least three times it happened to me, that I got a bunch of errors like
this one:

/home/dspace/svn/xoai-common/src/main/java/com/lyncode/xoai/common/dataprovider/data/AbstractItem.java:[33,21]
generics are not supported in -source 1.3
(use -source 5 or higher to enable generics)
public abstract List getAbout();

I was able to fix it by adding:

maven-compiler-plugin
2.3.2

1.6
1.6

and running mvn install again, which resulted in a successful compile.
Then I checked out pom.xml again (so it was without my addition listed
above), I ran mvn install again, but this time it ran successfully!
What's going on here?

listSets() resumptionToken causes IllegalStateException

I'm trying to receive the list of sets from a OAI handler. As the particular repository I'm querying has more than 100 sets, its handler provides sets in chunks of 100 with an additional resumptionToken element.
I use XOAI like in the following snippet:

        OAIClient oaiClient = new HttpOAIClient("http://www.example.com/oai/request");

         Context context = new Context()
                .withOAIClient(oaiClient);

         ServiceProvider ssoarOaiPmhEndpoint = new ServiceProvider(context);         
         Iterator<Set> sets = ssoarOaiPmhEndpoint.listSets();

         sets.forEachRemaining(  x -> logger.info( x.getName() )  );

This will return the first 100 sets, but then I get an IllegalStateException:

java.lang.IllegalStateException: Invalid use of BasicClientConnManager: connection still allocated.
Make sure to release the connection before allocating another one.
        at org.apache.http.util.Asserts.check(Asserts.java:34)
        at org.apache.http.impl.conn.BasicClientConnectionManager.getConnection(BasicClientConnectionManager.java:160)
        at org.apache.http.impl.conn.BasicClientConnectionManager$1.getConnection(BasicClientConnectionManager.java:142)
        at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:423)
        at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:106)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)
        at com.lyncode.xoai.serviceprovider.client.HttpOAIClient.execute(HttpOAIClient.java:42)
        at com.lyncode.xoai.serviceprovider.handler.ListSetsHandler.nextIteration(ListSetsHandler.java:67)
        at com.lyncode.xoai.serviceprovider.lazy.ItemIterator.hasNext(ItemIterator.java:40)
        at com.example.XoaiTest.makeSimpleOaiPmhConnection(XoaiTest.java:58)

How do I resolve this issue?

Hostname in certificate didn't match exception

When connecting to some https repositories, the following exception occurs:
javax.net.ssl.SSLException: hostname in certificate didn't match

Examples where this occurs:
https://revistaclinica.hff.min-saude.pt/index.php/rhff/oai
https://spmfrjournal.org/index.php/spmfr/oai

This issue can be solved by updating the httpclient dependency to the latest version:

<dependency>
    <groupId>org.apache.httpcomponents</groupId>
    <artifactId>httpclient</artifactId>
    <version>4.5.3</version>
</dependency>

No timeout is defined when a OAI repository is not available

XOAI will try forever to connect to a OAI repository that is not available

NoSuchElementException when parsing xoai

Stacktrace as follows:

Exception in thread "main" java.util.NoSuchElementException
	at org.codehaus.stax2.ri.Stax2EventReaderImpl.throwEndOfInput(Stax2EventReaderImpl.java:453)
	at org.codehaus.stax2.ri.Stax2EventReaderImpl.nextEvent(Stax2EventReaderImpl.java:242)
	at com.lyncode.xml.XmlReader.next(XmlReader.java:134)
	at org.dspace.xoai.serviceprovider.parsers.MetadataParser.parseElement(MetadataParser.java:43)
	at org.dspace.xoai.serviceprovider.parsers.MetadataParser.parseElement(MetadataParser.java:44)
	at org.dspace.xoai.serviceprovider.parsers.MetadataParser.parseElement(MetadataParser.java:44)
	at org.dspace.xoai.serviceprovider.parsers.MetadataParser.parseElement(MetadataParser.java:44)
	at org.dspace.xoai.serviceprovider.parsers.MetadataParser.parseElement(MetadataParser.java:44)
	at org.dspace.xoai.serviceprovider.parsers.MetadataParser.parseElement(MetadataParser.java:44)
	at org.dspace.xoai.serviceprovider.parsers.MetadataParser.parseElement(MetadataParser.java:44)
	at org.dspace.xoai.serviceprovider.parsers.MetadataParser.parseElement(MetadataParser.java:44)
	at org.dspace.xoai.serviceprovider.parsers.MetadataParser.parseElement(MetadataParser.java:44)
	at org.dspace.xoai.serviceprovider.parsers.MetadataParser.parseElement(MetadataParser.java:44)
	at org.dspace.xoai.serviceprovider.parsers.MetadataParser.parseElement(MetadataParser.java:44)
	at org.dspace.xoai.serviceprovider.parsers.MetadataParser.parseElement(MetadataParser.java:44)
	at org.dspace.xoai.serviceprovider.parsers.MetadataParser.parseElement(MetadataParser.java:44)
	at org.dspace.xoai.serviceprovider.parsers.MetadataParser.parseElement(MetadataParser.java:44)
	at org.dspace.xoai.serviceprovider.parsers.MetadataParser.parseElement(MetadataParser.java:44)
	at org.dspace.xoai.serviceprovider.parsers.MetadataParser.parseElement(MetadataParser.java:44)
	at org.dspace.xoai.serviceprovider.parsers.MetadataParser.parseElement(MetadataParser.java:44)
	at org.dspace.xoai.serviceprovider.parsers.MetadataParser.parseElement(MetadataParser.java:44)
	at org.dspace.xoai.serviceprovider.parsers.MetadataParser.parseElement(MetadataParser.java:44)
	at org.dspace.xoai.serviceprovider.parsers.MetadataParser.parseElement(MetadataParser.java:44)
	at org.dspace.xoai.serviceprovider.parsers.MetadataParser.parseElement(MetadataParser.java:44)
	at org.dspace.xoai.serviceprovider.parsers.MetadataParser.parseElement(MetadataParser.java:44)
	at org.dspace.xoai.serviceprovider.parsers.MetadataParser.parseElement(MetadataParser.java:44)
	at org.dspace.xoai.serviceprovider.parsers.MetadataParser.parseElement(MetadataParser.java:44)
	at org.dspace.xoai.serviceprovider.parsers.MetadataParser.parseElement(MetadataParser.java:44)
	at org.dspace.xoai.serviceprovider.parsers.MetadataParser.parseElement(MetadataParser.java:44)
	at org.dspace.xoai.serviceprovider.parsers.MetadataParser.parse(MetadataParser.java:34)
	at org.dspace.xoai.serviceprovider.parsers.RecordParser.parse(RecordParser.java:56)
	at org.dspace.xoai.serviceprovider.parsers.ListRecordsParser.next(ListRecordsParser.java:60)
	at org.dspace.xoai.serviceprovider.handler.ListRecordHandler.nextIteration(ListRecordHandler.java:71)
	at org.dspace.xoai.serviceprovider.lazy.ItemIterator.hasNext(ItemIterator.java:32)
	at org.dspace.xoai.serviceprovider.lazy.ItemIterator.<init>(ItemIterator.java:22)
	at org.dspace.xoai.serviceprovider.ServiceProvider.listRecords(ServiceProvider.java:57)

Minimum reproducible:

OAIClient oaiClient = new HttpOAIClient("http://repository.abertay.ac.uk/oai/request");
context.withOAIClient(oaiClient);
ServiceProvider ssoarOaiPmhEndpoint = new ServiceProvider(context);
ListRecordsParameters parameters = new ListRecordsParameters();
parameters.withMetadataPrefix("xoai");
ssoarOaiPmhEndpoint.listRecords(parameters);

Example record at: view-source:http://repository.abertay.ac.uk/oai/request?verb=GetRecord&metadataPrefix=xoai&identifier=oai:repository.abertay.ac.uk:10373/1861

parseElement is failing at: parsing of license. Example is

<element name="license"><field name="bin">Tk9URTogVGhpcyBpcyB0aGUgZGVmYXVsdCBsaWNlbmNlIHRoYXQgdGhlIFVuaXZlcnNpdHkgb2YgQWJlcnRheSAKRHVuZGVlIHJlcXVpcmVzIGFsbCBzdWJtaXR0ZXJzIHRvIGdyYW50LgoKTk9OLUVYQ0xVU0lWRSBESVNUUklCVVRJT04gTElDRU5DRQoKQnkgYWdyZWVpbmcgYW5kIHN1Ym1pdHRpbmcgdGhpcyBsaWNlbmNlLCB5b3UgKHRoZSBhdXRob3IocyksIApjb3B5cmlnaHQgb3duZXIgb3Igbm9taW5hdGVkIGFnZW50KSBncmFudHMgdG8gVW5pdmVyc2l0eSBvZiBBYmVydGF5IApEdW5kZWUgKFVBRCkgdGhlIG5vbi1leGNsdXNpdmUgcmlnaHQgdG8gcmVwcm9kdWNlLCB0cmFuc2xhdGUgCihhcyBkZWZpbmVkIGJlbG93KSwgYW5kL29yIGRpc3RyaWJ1dGUgeW91ciBzdWJtaXNzaW9uIChpbmNsdWRpbmcgdGhlIAphYnN0cmFjdCkgd29ybGR3aWRlIGluIHByaW50IGFuZCBlbGVjdHJvbmljIGZvcm1hdCBhbmQgaW4gYW55IG1lZGl1bSwgCmluY2x1ZGluZyBidXQgbm90IGxpbWl0ZWQgdG8gYXVkaW8gb3IgdmlkZW8uCgpZb3UgYWdyZWUgdGhhdCBVQUQgbWF5LCB3aXRob3V0IGNoYW5naW5nIHRoZSBjb250ZW50LCB0cmFuc2xhdGUgdGhlCnN1Ym1pc3Npb24gdG8gYW55IG1lZGl1bSBvciBmb3JtYXQgZm9yIHRoZSBwdXJwb3NlIG9mIHByZXNlcnZhdGlvbi4gCllvdSBhbHNvIGFncmVlIHRoYXQgVUFEIG1heSBrZWVwIG1vcmUgdGhhbiBvbmUgY29weSBvZiB0aGlzIApzdWJtaXNzaW9uIGZvciBwdXJwb3NlcyBvZiBzZWN1cml0eSwgYmFjay11cCBhbmQgcHJlc2VydmF0aW9uLgoKWW91IHJlcHJlc2VudCB0aGF0IHRoZSBzdWJtaXNzaW9uIGlzIG9yaWdpbmFsIHdvcmssIGFuZCB0aGF0IHlvdQpoYXZlIHRoZSByaWdodCB0byBncmFudCB0aGUgcmlnaHRzIGNvbnRhaW5lZCBpbiB0aGlzIGxpY2VuY2UuIFlvdSAKYWxzbyByZXByZXNlbnQgdGhhdCB5b3VyIHN1Ym1pc3Npb24gZG9lcyBub3QsIHRvIHRoZSBiZXN0IG9mIHlvdXIgCmtub3dsZWRnZSwgaW5mcmluZ2UgdXBvbiBhbnlvbmUncyBjb3B5cmlnaHQuCgpJZiB0aGUgc3VibWlzc2lvbiBjb250YWlucyBtYXRlcmlhbCBmb3Igd2hpY2ggeW91IG9yIHlvdXIgcHVibGlzaGVyCmRvIG5vdCBob2xkIGNvcHlyaWdodCwgeW91IHJlcHJlc2VudCB0aGF0IHlvdSBoYXZlIG9idGFpbmVkIHRoZQp1bnJlc3RyaWN0ZWQgcGVybWlzc2lvbiBvZiB0aGUgY29weXJpZ2h0IG93bmVyIHRvIGdyYW50IFVBRCB0aGUKcmlnaHRzIHJlcXVpcmVkIGJ5IHRoaXMgbGljZW5jZSwgYW5kIHRoYXQgc3VjaCB0aGlyZC1wYXJ0eSBvd25lZAptYXRlcmlhbCBpcyBjbGVhcmx5IGlkZW50aWZpZWQgYW5kIGFja25vd2xlZGdlZCB3aXRoaW4gdGhlIHRleHQgb3IKY29udGVudCBvZiB0aGUgc3VibWlzc2lvbi4KCklGIFRIRSBTVUJNSVNTSU9OIElTIEJBU0VEIFVQT04gV09SSyBUSEFUIEhBUyBCRUVOIFNQT05TT1JFRCBPUiAKU1VQUE9SVEVEIEJZIEFOIEFHRU5DWSBPUiBPUkdBTklaQVRJT04gT1RIRVIgVEhBTiBVQUQsIFlPVSBSRVBSRVNFTlQgClRIQVQgWU9VIEhBVkUgRlVMRklMTEVEIEFOWSBSSUdIVCBPRiBSRVZJRVcgT1IgT1RIRVIgT0JMSUdBVElPTlMgClJFUVVJUkVEIEJZIFNVQ0ggQ09OVFJBQ1QgT1IgQUdSRUVNRU5ULgoKVUFEIHdpbGwgY2xlYXJseSBpZGVudGlmeSB5b3VyIG5hbWUocykgYXMgdGhlIGF1dGhvcihzKSBvciBvd25lcihzKSAKb2YgdGhlIHN1Ym1pc3Npb24sIGFuZCB3aWxsIG5vdCBtYWtlIGFueSBhbHRlcmF0aW9uLCBvdGhlciB0aGFuIGFzIAphbGxvd2VkIGJ5IHRoaXMgbGljZW5jZSwgdG8geW91ciBzdWJtaXNzaW9uLgo=</field>
</element>

which contains the mime encoded contents of the license.

v4.2.1-SNAPSHOT cloned from git repo today.

Any ideas?

Handle CDATA results

Example: http://www.scielo.oces.mctes.pt/oai/scielo-oai.php?verb=ListSets&metadataPrefix=oai_dc

The result text is inside CDATA elements. How should this result be handled?

0003-2573