danicat / read.dbc Goto Github PK

An R package for reading data in the DBC (compressed DBF) format used by DATASUS.

C 73.04% R 20.05% Makefile 6.91%

read.dbc's Introduction

read.dbc

Author: Daniela Petruzalek
e-mail: [email protected]
License: AGPLv3

Introduction

read.dbc is a R package to enable importing data from DBC (compressed DBF) files into data frames. Please note that this is the file format used by DATASUS and it is not related to Microsoft FoxPro or CANdb DBC file formats.

DATASUS is the name of the Department of Informatics of Brazil's Healthcare System (Sistema Unico de Saúde - SUS). They are responsible for publishing Brazilian public healthcare data. Besides DATASUS, the Brazilian National Agency for Supplementary Health (ANS) also uses this file format for its public data.

This code was tested using files from both DATASUS and ANS to ensure compliance with the format, and hence ensure its usability by researchers.

This project is based on the work of Mark Adler (blast) and Pablo Fonseca (blast-dbf).

Neither this project, nor its author, is related in any way to the Brazilian government.

Changelog

For a complete description of the changes, please check CHANGELOG.md.

Repository Contents

README.md: this file.
CHANGELOG.md: change history.
src/blast.c: decompression tools for PKWare Data Compression Library (DCL).
src/blast.h: blast.c header and usage notes.
src/dbc2dbf.c: the main program to decompress the dbc files to dbf.
R/read.dbc.R: the code for reading .dbc files within R.
R/dbc2dbf.R: a helper function to decompress the .dbc files, it works as a wrapper to the "blast" code.
man/*: package manuals
inst/*: test and misc files

Installation

As of June, 7 of 2016, this package officialy became part of CRAN (The Comprehensive R Archive Network). Therefore, it's current stable version can be installed by running install.packages:

    install.packages("read.dbc")

In case you want to install the development version of this package, you still can do it using the devtools library:

    devtools::install_github("danicat/read.dbc")

Usage

Reading a DBC file to a data frame:

    # The 'sids.dbc' file is the compressed version of 'sids.dbf' from the "foreign" package.
    sids <- read.dbc(system.file("files/sids.dbc", package="read.dbc"))
    str(sids)
    summary(sids)

    # The following code will download data from the "Declarations of Death" database for
    # the Brazilian state of Parana, year 2013. Source: DATASUS / Brazilian Ministry of Health
    url <- "ftp://ftp.datasus.gov.br/dissemin/publicos/SIM/CID10/DORES/DOPR2013.dbc"
    download.file(url, destfile = "DOPR2013.dbc", mode = "wb")
    dopr <- read.dbc("DOPR2013.dbc")
    head(dopr)
    str(dopr)

Decompressing a DBC file to a DBF:

    # Input file name
    in.f  <- system.file("files/sids.dbc", package = "read.dbc")
    
    # Output file name
    out.f <- tempfile(fileext = ".dbc")

    # The call return logi = TRUE on success
    if( dbc2dbf(input.file = in.f, output.file = out.f) ) {
         print("File decompressed!")
         file.remove(out.f)
    }

Contact Info

If you have any questions, please contact me at [email protected].

Developer Information

Mac OS X

Setup:

Install Xcode
Install R: https://cran.r-project.org/bin/macosx/
Install Rstudio: https://posit.co/download/rstudio-desktop/
Run make setup to install R dependencies
Run make check to verify the package

You can also run make help to see a list of available commands.

Submitting to CRAN

First make sure all the checks are passing by running make cran.

Once ready, use devtools::submit_cran(). This needs to run from RStudio or the R interpreter itself as the tool doesn't allow non-interactive runs.

read.dbc's People

Contributors

Stargazers

Watchers

read.dbc's Issues

Error to find file with read.dbc

Hi,
I used setwd("C:/Users/leand/Desktop/dbc") for set the path with my files
After, I put library("read.dbc") but when I try to pass for the row with x <- read.dbc(system.file("PASC2007.dbc")), I'm receiving this message: Error in dbc2dbf(file, out) : Input file does not exist.

In the folder, the file is there.

Code:
setwd("C:/Users/leand/Desktop/dbc")

library("read.dbc")
x <- read.dbc(system.file("PASC2007.dbc"))
str(x)

Console:

setwd("C:/Users/leand/Desktop/dbc")
library("read.dbc")
x <- read.dbc(system.file("PASC2007.dbc"))
Error in dbc2dbf(file, out) : Input file does not exist.

download.file keeps corrupting data

I'm trying to design a loop to download all data available from a certain category in DATASUS, the download occurs, but when I try to read the dbc file, the function returns "Error in dbc2dbf(file, out) : Corrupted file? Blast error code: -1". Is there any solution to this?

Thanks in advance,
Luis Paese

read.dbc removed from CRAN

Hello Daniela. How are you doing? I hope well. Daniela, the read.dbc package was removed from CRAN on 11/20. Do you need some help with this package? I don't know if there were some issues to help/fix. If so, I can help you if necessary. See you! Thanks!

Blast error code: -3

Dani,

I tried to replicate the usage procedure as described in readme.me. As soon as I ran the read.dbc function, I got the following error: blast error code: -3

I'm using R version 3.3.1 under windows 64 bit

Minor code cleanups

While the code seems to be running fine, I would like to provide a few suggestions of minor code improvements in read.dbc:

Raise an error if input file does not exist, so that read.dbc does not even call dbc2dbf.
Use on.exit() to avoid nesting code.
Raise an error if dbc2dbf returns 0.
Avoid unnecessary return()

Also, in both read.dbc and dbc2dbf I used a stopifnot(), which might or might not suit your preferences.

Closing connection in corrupted files

Hi danicat!

I have been using read.dbc function from read.dbc package and it is working fine for me (btw, thank you very much for that contribution!).

Unfortunatelly, sometimes the file to be read is corrupted. To handle this, I use the try statement. The problem is that the corrupted file keeps "open" in R until I finish my R session, in a way I can't delete the file until the session is over.

My code is:

ReadFile <- try(read.dbc("C:/LocalPath/A_File.txt"))

Is there a way to "close" the connection prematurely (without the need to end the session)?

More details: RStudio v0.99.484, R v3.3.1 (x64), OS Windows 7 Enterprise SP1

PS.: To reproduce the error simply try to read any file that is not a dbc file, a TXT file for example.

UPDATE:
AEF from Stackoverflow made a contribution that might help.

Erro ao tentar installar o pacote no Anaconda 3

tentando a URL 'https://cran.r-project.org/src/contrib/read.dbc_1.0.5.tar.gz'
Content type 'application/x-gzip' length 20988 bytes (20 KB)
==================================================
downloaded 20 KB

* installing *source* package ‘read.dbc’ ...
** package ‘read.dbc’ successfully unpacked and MD5 sums checked
** libs
/home/yves/anaconda3/envs/importacao/bin/x86_64-conda_cos6-linux-gnu-cc -I/home/yves/anaconda3/envs/importacao/lib/R/include -DNDEBUG   -D_FORTIFY_SOURCE=2 -O2 -I/home/yves/anaconda3/envs/importacao/include   -fpic  -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -pipe -I/home/yves/anaconda3/envs/importacao/include  -c blast.c -o blast.o
make: /home/yves/anaconda3/envs/importacao/bin/x86_64-conda_cos6-linux-gnu-cc: Comando não encontrado
/home/yves/anaconda3/envs/importacao/lib/R/etc/Makeconf:160: recipe for target 'blast.o' failed
make: *** [blast.o] Error 127
ERROR: compilation failed for package ‘read.dbc’
* removing ‘/home/yves/R/x86_64-conda_cos6-linux-gnu-library/3.4/read.dbc’

The downloaded source packages are in
	‘/tmp/Rtmp1yBGfN/downloaded_packages’

Acho que deva estar faltando alguma dependência para compilar o blast, mas não sei qual é. Estou usando o Ubuntu Gnome 16.04.

Corrupted file? Blast error code: -1

Hi Daniela,
thank you for this package.

I was trying to read a dbf file and got this error:
"Corrupted file? Blast error code: -1"

I am in a Mac. I saw someone reported this error before and you suggested a download mode to binary in Windows. This file is the preliminary SIM for 2019, so it is not hosted on the ftp yet. But it is here

http://svs.aids.gov.br/dantps/centrais-de-conteudos/dados-abertos/sim/

Any suggestion?

Thank you again,
Otavio

Package removal from CRAN - read.dbc

Hello,

I noticed that the read.dbc package has been removed from the CRAN repository due to a policy violation (https://cran.r-project.org/web/packages/read.dbc/index.html). I was hoping to use this package for my project, but it seems that it is no longer available through the standard R channels. I would like to know the reason for the policy violation and if there are any plans to address it and bring the package back to CRAN.

In the meantime, I also noticed that the package is still available on Github. Is it safe to use this package from Github, or are there any known issues that I should be aware of?

Thank you for your time and help in this matter.

Error: could not find function "read.dbc"

in R for Mac I used
install.packages("read.dbc")
and then
x <- read.dbc(system.file("/Users/username/Downloads/DNRAC94.DBC", package="read.dbc"))

a result was
Error: could not find function "read.dbc"

Could you clarify, what I did wrong? Thank you a lot.

blast.c code broken in Solaris x86 and Solaris SPARC

The blast.c algorithm is broken in the Solaris x86 and Solaris SPARC platforms. The command line utility (blast-dbf) is also broken.

The blast.c code must be cleaned or rewritten to not rely on any undefined behaviour.

error decompressing file: 2

Dani,

I was able to download the CNES data from the ftp, but when I try to read it in R it appears to have something wrong. When I try to use the funciont read.dbc the following error shows up: "Error in dbc2dbf(file, out) : error decompressing file: 2".
Do you have any idea of why this error shows up?

Thanks,
Guilherme Carvalho

add stringsAsFactors = F functionality

The package is great.
For the SIM data, all the variables are imported as factor variables.
It would be nice to add the "stringsAsFactors", so that people can turn that behavior off (stringsAsFactors = F)