Git Product home page Git Product logo

ihsn / nada Goto Github PK

View Code? Open in Web Editor NEW
36.0 17.0 9.0 73.18 MB

National Data Archive (NADA) is an open source data cataloging system that serves as a portal for researchers to browse, search, compare, apply for access, and download relevant census or survey information. It was originally developed to support the establishment of national survey data archives.

Home Page: http://nada.ihsn.org

License: MIT License

HTML 0.99% PHP 84.71% Hack 0.56% CSS 2.48% JavaScript 7.21% XSLT 3.46% TSQL 0.59%
data-catalog data-catalog-management data-catalog-backend data-portal data catalog catalogue nada data-documentation-initiative ddi-codebook

nada's Introduction

NADA - Data Catalog

NADA is an open source microdata cataloging system, compliant with the Data Documentation Initiative (DDI) and Dublin Core’s RDF metadata standards. It serves as a portal for researchers to browse, search, compare, apply for access, and download relevant census or survey datasets, questionnaires, reports and other information.

Getting Started

Server Requirements

  • PHP version 7.3 or later
  • MySQL or Microsoft SQL Server database
  • Apache, IIS or NGINX

Documentation

User and admin guide - https://ihsn.github.io/nada-documentation/

Installation

System requirements and steps for installation - NADA installation guide.

Upgrading from older versions of NADA

Documentation for upgrading from various versions - NADA upgrade guide.

Schema guide

NADA supports multiple data types that include Microdata (DDI CodeBook 2.5), Document, Table, Geospatial, Timeseries, Visualization and Image. For all data types, documentation is available in the draft guide (https://mah0001.github.io/schema-guide/). See our demo catalog show casing all support data types - https://nada-demo.ihsn.org/index.php/catalog/

API documentation

The API documentation is available in OpenAPI/Swagger format here - https://ihsn.github.io/nada-api-redoc/catalog-admin/

For Getting started with the API, read the Administrator Guide section which covers both using the web interface and the API - https://ihsn.github.io/nada-documentation/getting-started/#publishing-a-document

API Client tools

Versioning

For the versions available, see the tags on this repository.

Authors

License

This project is licensed under the MIT License - see the license.txt file for details

Acknowledgments

  • CodeIgniter
  • Bootstrap
  • todo - other libraries used

nada's People

Contributors

ch-ricardor avatar mah0001 avatar masghar78 avatar mehmood avatar unisante-gp avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

nada's Issues

Studies list navigation link in collection broken

When I select a collection and then click on "dataset", I get a list of the studies affiliated to the collection. But if I need to navigate to the next page (either by clicking on "page 2" or "next page"), I'm sent back to the page 2 of the "Central Data Catalog", loosing the collection variable. If I put back the collection name in the url manually (/index.php/catalog/COLLECTIONNAME?&page=2) it's working without problem.

Add abstracts to catalog cards

Display the abstract on the catalog cards (screenshot below). This information will consist of 2 lines in small fonts, with the text “… more” to show the full abstract. Add a configuration option to enable/disable abstract display.

image

Upload CESSDA CV to NADA

It would be incredibly useful (and reduce alot of errors!) if we could upload the CESSDA CV or AN Other directly into NADA

Error adding a study

Installed 5.0.5, I get an error when I add new study after selecting DDI file & RDF file and then click submit.
I get an error which says:
This page isn't working.
Server is currently unable to handle this request.
HTTP ERROR 500

Potential security issue

Hello 👋

I run a security community that finds and fixes vulnerabilities in OSS. A researcher (@rafaelcintralopes) has found a potential issue, which I would be eager to share with you.

Could you add a SECURITY.md file with an e-mail address for me to send further details to? GitHub recommends a security policy to ensure issues are responsibly disclosed, and it would help direct researchers in the future.

Looking forward to hearing from you 👍

(cc @huntr-helper)

issue when editing metadata

Dear NADA team, One issue:
I'm using v5.2.

When editing metadata in the Site admin: the description of scope is populating the abstract field and description of scope is empty. The data looks correct on the front end, and in the XML file:

In XML (located: C:\inetpub\wwwroot\nada\datafiles\central[guid]\ ):
stdyDscr>
stdyInfo>
abstract> (appears correctly on front end, Identification.Abstract)
(Does not appear in metadata on admin page)

stdyDscr>
stdyInfo>
notes> (appears correctly on front end, Scope.Notes)
(Appears in metadata “Overview.Abstract” on admin page)

It could be an error on my part, but I'm having trouble figuring out what is causing it.
Thanks!
Fred

pdf generation error

Hi Mehmood,
when run pdf generator "......./index.php/admin/pdf_generator/setup/401"
I get error "HTTP ERROR 500"
my environment is IIS 8.5 , mbstring enabled..
How can I track down I couldn't find any error msg in php or iis log's
where can I get nada error logs...

Variable view selecting "data availible from external repo" generate error

Hello,

After working some time on the last nada upgrade, I gave to our survey team the task to look for any other bug in the application. They found some, for instance this one :

When you go on the search page, under "variable view" and you select "Data available from external repository", you get the following error :

ERROR - 2022-09-06 16:28:08 --> Query error: Unknown column 'forms.formid' in 'where clause' - Invalid query: SELECT SQL_CALC_FOUND_ROWS v.uid, v.name, v.labl, v.qstn, v.vid, surveys.title as title, surveys.idno, surveys.nation, v.sid
FROM `variables` as `v`
INNER JOIN `surveys` ON `v`.`sid` = `surveys`.`id`
LEFT JOIN `repositories` ON `surveys`.`repositoryid`=`repositories`.`repositoryid`
WHERE `forms`.`formid` in (5)
 AND `published` = 1
ORDER BY `title` ASC
 LIMIT 15

Upgrade from 4.2 to 5.0.5

Hi Mehmood - We conducted an upgrade from 4.2 to 5.0.5 and picked up some issues.

  1. Search by keyword feature not working as expected. It takes forever just to search a particular dataset with keyword. Not sure if the dataset tags are added during the upgrade.
  2. It takes forever to move from one page to the next. You can only do this if one refreshes the button while it is searching. E.g. you are on the Central Data Catalog and you want to go the next page.
  3. Searching for Variables is also not working - looking at the upgrade script the variables table is dropped and then recreated. How do we re-insert our existing project variables?
  4. When creating a report for all the study downloads, you can only create a report for the downloads that done after the upgrade. I know the data is there but is there a possibility of creating a report for all the data downloads including those prior to upgrade?
  5. In the 5.0.5 release notes point 4 indicates that dd_projects table needs to altered but I am not finding it.

Cheers
B

Error when calling collection history

When I try to access a collection history from the dashboard, I get the following error :

An uncaught Exception was encountered

Type: Error

Message: Call to a member function result_array() on boolean

Filename: /var/www/html/nada5/application/models/Repository_model.php

Line Number: 706

Backtrace:

File: /var/www/html/nada5/application/controllers/admin/Repositories.php
Line: 746
Function: repo_survey_list

File: /var/www/html/nada5/index.php
Line: 343
Function: require_once

Studies resources abstract ignore when importing RDF

When importing a RDF into NADA 5.0.4 to add resources to a study, the abstract field is ignored.

By checking the dc2sql.xslt file used by the RDF parser, I found out a <dc:abstract /> tag is expected. Unfortunately, RDF files we get from Nestar use a <dcitems:abstract /> tag instead.

So abstracts generated by Nestar are ignored.

Variables groups ?

In nada4 under "data description", you had the option to select "variable groups". I can't find this option in nada5. Do you plan to implement this feature ? Or am I missing something ?

Showing incorrect survey numbers on upgrde to 5.2

Hi Mehmood,

The upgrade shows incorrect count i.e. 1-15 of 1 instead of 1-15 of (all the survey count). Let's assume that I have 1000 surveys in the catalog and hence catalog page should show 1-15 of 1000 but currently it is showing as 1-15 as 1.

Please find attached image for your reference.
NADA issue

notifications and alerts

Implement notifications/alerts for registered users:

  • Inform users when entries are added to the catalog
  • Inform users if any changes are made to specific study metadata or data
  • Users should be able to manage the notifications
  • Options to subscribe/unsubscribe

http://localhost:8080/nada5/index.php/install giving me a blank screen

Hi,

I am using WAMP 3.2.0 on a windows 7, 64 bit machine to install NADA 5.0.4.

WAMP Icon is green, presumably showing that all the configurations are ok?

localhost:8080 gives me the contents in the picture:

wamp

I am not sure how to sort out the error displayed about 8080 not being an Apache Listen port.

localhost:8080/nada5/index.php/install_ gives me a blank screen.

How do i get past this to at least see errors that need fixing?

Values displayed in the catalogue are strange or incorrect.

image

Issue observed in UNHCR microdata library here:
https://microdata.unhcr.org/index.php/catalog/797/data-dictionary/Household_Data?file_name=Household%20Module

  1. The reported percentages are 'strange'. I have included what I expected to see instead.
  2. The bars appear 'strange'. The blue bar corresponding to the value {1=no} appears to be larger than the blue bar corresponding to the value {2=yes} despite the frequency for {2=yes} is 4 times larger than that of {1=no}.

Wrong mysql driver when upgrade

Hello,
I followed your upgrade guide from 4.4 to 5.0 and the code still produced errors caused by mysql_connect() call. It took me some time to find out that I had to change the $db['default']['dbdriver'] var to "mysqli" in order to work with php 7.
Maybe you should precise this requirement in the doc ?
Thanks

jQuery version

Hi again :)

Our antivirus pointed this file :

/var/www/html/nada5/javascript/superfish/js/jquery-1.2.6.min.js: SecuriteInfo.com.Exploit.22396.7953.UNOFFICIAL FOUND
/var/www/html/nada4/javascript/superfish/js/jquery-1.2.6.min.js: SecuriteInfo.com.Exploit.22396.7953.UNOFFICIAL FOUND

----------- SCAN SUMMARY -----------
Known viruses: 10560234
Engine version: 0.101.1
Scanned directories: 23846
Scanned files: 156859
Infected files: 2
Data scanned: 8341.41 MB
Data read: 8169.42 MB (ratio 1.02:1)
Time: 1905.744 sec (31 m 45 s)

The reason is that jQuery version 1.2.6 was released more than 10 years ago and several XSS vulnerabilities has been found in it since then. Maybe you should upgrade your jQuery version ...

probleme with sqlsrv

hi mahmood,

I have installed nada 5.0.4 with sql server, but i have a problem in nada/index.php/home, bellow the error

**An uncaught Exception was encountered
Type: Error

Message: Call to a member function row_array() on boolean

Filename: C:\wamp64\www\nada-master\application\models\Stats_model.php

Line Number: 64

Backtrace:

File: C:\wamp64\www\nada-master\application\views\static\home.php
Line: 3
Function: get_survey_count

File: C:\wamp64\www\nada-master\application\controllers\Page.php
Line: 185
Function: view

File: C:\wamp64\www\nada-master\application\controllers\Page.php
Line: 71
Function: static_page

File: C:\wamp64\www\nada-master\index.php
Line: 343
Function: require_once**

this url works fine when using database mysql

thank you

Missing data display in 5.2

Hello! We are upgrading from 5.05 to 5.2 and we noticed that when browsing the data in the new version, the result of the surveys are not displayed anymore.
5.0.5 :
image
5.2 :
image
Is this behaviour intended ? Are these data availible elsewhere ? I didn't found anything about that in the changelog.
Thanks!

Support for complex IDNo

Using 5.01 it looks like IDNo format such as
doi:10.7910/DVN/27883
is breaking the import.

It looks like the validation is being applied as one would for the ID attribute i.e. xs:ID rather than content which can accept pretty much anything, which I dont think is correct.

Boolean OR operator in catalog search (MySQL)

Hello!

I'd like to ask about the catalog search functionality, specifically regarding Boolean full-text searches using MySQL. In Catalog_search_mysql.php, it appears that keywords are being fixed so that the default behavior is a conjunction of search terms. Since a disjunction of search terms (i.e. operator OR) is the default in MySQL fulltext search, the keyword modifications make it syntactically impossible to build an expression with Boolean OR functionality.

My question is whether I'm parsing the behavior correctly and whether there is a capability to search with OR that I may be missing. Thank you for any insights.

Generating PDF with over 4000 variables timeout

Hi Mehmood - We have nested study publication with four data files and unable to generate the Metadata in PDF - It seems to timeout. Are there any limitations with publishing study with over 4k variables?

Incidentally, we see this behaviour with studies with 2k variables as well but not for studies with 1k variables or less. Seems 1k is the sweet spot. Any ideas where I should be looking at or is this limitation in NADA?

Any plans to support DDI 3+ (Lifecycle)?

Are there any plans to develop NADA to support a fully DDI-based solution for both study-level and variable-level metadata, (e.g., Lifecycle, or perhaps the forthcoming CDI standard)?

I've been discussing with an organization that is looking to adopt a more comprehensive metadata solution. Given the content they oversee, a more pure social science metadata paradigm would probably be ideal for them. NADA still seems promising as one of the best options for them right now, but as they're growing they're trying to think carefully about future directions as well.

Export full metadata to JSON for microdata

On study info pages, the link to download JSON only provides partial metadata for Microdata projects. The JSON export should include full study, data file and variable level metadata similar to the DDI.

image

Error 200 on DDI batch refresh

Hi again :)
During the DDI batch refresh, I get a huge amount of "Error: 200 - Ignore errors and continue?" messages. Only 41/903 entries receive the {"success":"UPDATED: XXX"} messages, other ajax responses are empty. I have no errors in apache or php server side.
When I go to the studies, I have no "data description" tab in these failed update entries. Do you have a clue on what's happening ?
Thanks.

Unable to upload studies

Hello everyone!
Please I need some help with NADA 5.0.6 configuration. It was installed in Linux Centos 7 with mysql 5.7 and php 7.4, everything was fine until the upload files in add studies was tested. I can't find the right folder where the study files are uploaded, datafiles, imports and files folders have 777 permissions but nothing works. Other attachment upload actions like citation files and bulk import ddi works but not this one. Please help me to find the path or missing configuration. This is the image of the error.

Screenshot_20210222_120904

thanks.

Current status of version 5

Hello,
I noticed that the ihsn.org still propose version 4.4 for download. Does it mean that the version 5 is not yet considered as production-ready ?
If so, when do you think that version 5 will be released for production ? In one month, six months, ... ?
Thanks in advance for your answer,
Best

mysql and InnoDB version

Hello,
Mysql 5.6 use InnoDB Antelope by default, therefore I got some warnings during mysql nada5 update :

InnoDB: ROW_FORMAT=DYNAMIC requires innodb_file_format > Antelope.
InnoDB: assuming ROW_FORMAT=COMPACT.

Do you thing it is safe to ignore these warnings or do you thing dynamic row format is mandatory and we should deploy Barracuda on our servers ? (which would represent quite some additional work).

mysql UPDATE TABLE `survey`

Hi,
During the mysql database upgrade from 4.4 to 5, the following line :

install/nada5-upgrade-mysql.sql:56

UPDATE TABLE `surveys` set type='survey';

produce an error. I corrected it this way :

UPDATE `surveys` set `type`='survey'; 

Is this correct ?

License Request Reviewer -- notifications

Dear Nada team,
I have created accounts for our license request reviewers. They seem to be working properly. However, when a user makes a new request, the Reviewer does not receive an email notification that there is a pending request. I was trying to look up this feature in the documentation. Is this a feature?

When the reviewer logs in and processes a request, the notificaiton back to the user is working properly. Also, the email associated with accounts and passwords is working properly, so I don't think it is an email issue.

Thanks! Fred

PDF Generation - error message when including Variable description

Hi Mehmood

We picking up that studies with over 2000 variables are timing out when we include variable description. I have increased the memory_limit and max_execution_time in php.ini. This error doesn't occur when we only choose variable list and external resources.

`An uncaught Exception was encountered
Type: Mpdf\MpdfException
Message: The HTML code size is larger than pcre.backtrack_limit 1000000. You should use WriteHTML() with smaller string lengths.

Filename: /var/www/html/nada5/vendor/mpdf/mpdf/src/Mpdf.php
Line Number: 26661

Backtrace:
File: /var/www/html/nada5/vendor/mpdf/mpdf/src/Mpdf.php
Line: 13159
Function: AdjustHTML

File: /var/www/html/nada5/application/libraries/Pdf_report.php
Line: 145
Function: WriteHTML

File: /var/www/html/nada5/application/controllers/admin/Pdf_generator.php
Line: 134
Function: generate

File: /var/www/html/nada5/application/controllers/admin/Pdf_generator.php
Line: 63
Function: _export_pdf

File: /var/www/html/nada5/index.php
Line: 343
Function: require_once`

Is sorting by year working ?

Hi again,

Just a small question, it seems that in the search page, sorting by year is not working. In the code in application/libraries/Catalog_search_mysql.php line 1033 I can see that the 'year' parameter is replaced by 'title'. Maybe it's better not to have the option if it's really not used, not to confuse users ?

Best

Search filters not carried on when no result

Hi,

Another bug spotted :

When you do a search with parameters, as long as you are getting result the said parameters are kept with the hidden input forms :

<input type="hidden" name="sort_by" id="sort_by" value=""/>
<input type="hidden" name="sort_order" id="sort_order" value=""/>
<input type="hidden" name="ps" id="ps" value="15"/>
<input type="hidden" name="repo" id="repo" value="XXX"/>
<input type="hidden" name="sid" id="sid" value=""/>

But as soon as you have no result, the hidden fields are not displayed and the parameters are lost, which can be quite confusing for the user.

Small PHP warning when sending mails during license request

Got this PHP warning showing when trying to send an email during the license requeset process :

A PHP Error was encountered
Severity: Warning
Message: sprintf(): Too few arguments
Filename: admin/Licensed_requests.php
Line Number: 411

It's just a sprintf() call that needs to be removed.

We probably shouldn't show PHP warnings in production, but it's a small fix.

TinyMCE editor changes links automatically

When editing a page using the TinyMCE editor, links are automatically modified.

For instance, a correct link we use is : https://microdata.pacificdata.org/documentation/what_is_microdata_and_the_pacific_data_hub_microdata_library.pdf

But even if I set the A tag attribute to be
href="/documentation/what_is_microdata_and_the_pacific_data_hub_microdata_library.pdf"
it saves as
href="/index.php/documentation/what_is_microdata_and_the_pacific_data_hub_microdata_library.pdf"

The extra index.php is always added by the editor.

The only way I found to fix it was by disabling the WYSIWYG editor.
But this is certainly not as convenient for our editors.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.