unb-knedle / dodfminer Goto Github PK
View Code? Open in Web Editor NEWData extractor of PDF documents from the Official Gazette of the Federal District, Brazil.
License: GNU General Public License v3.0
Data extractor of PDF documents from the Official Gazette of the Federal District, Brazil.
License: GNU General Public License v3.0
The title/subtitle hierarchy is constantly broken when the TitleExtractor is run over those PDFs.
Specifically, seems to happen when there are several titles within the same page.
Subtitles are also affected similarly.
Steps to reproduce the behavior:
Expected behavior
Rebuild the title/subtitle hierarchy correctly.
Additional context
Attached are the
Describe the bug
A clear and concise description of what the bug is.
To Reproduce
Steps to reproduce the behavior:
1.
2.
3.
Expected behavior
A clear and concise description of what you expected to happen.
Screenshots
If applicable, add screenshots to help explain your problem.
Desktop (please complete the following information):
Additional context
Add any other context about the problem here.
Missing dodf_data causes crashing on SemEfeitoAposentadoria
Expected behavior
If FLEX_DATE regex does not matches, the subsequent code will crash.
Aviso 3ª SESSÃO
Publicação aviso julgamento final das propostas
Publicação Aviso Recurso Contra Propostas de Preço
Publicação Aviso Julgamento Recurso e Convocação 4ª Sessão
Publicação Aviso Julgamento Habilitação e Resultado Final
Publicação Aviso Recurso Contra Habilitação
Publicação Aviso Julgamento Recurso Habilitação
Feature description
Given a list of title and subtitles, create a JSON with information from blocks of each DODF.
Additional context
AVISO PROSSEGUIMENTO LICITAÇÃO
AVISO REABERTURA LICITAÇÃO
Publicação Aviso 2ª Sessão –
Publicação Aviso Resultado Julgamento Propostas Técnicas
Publicação Aviso Recuros contra Resultado Julgamento Propostas Técnicas
Publicação Aviso Julgamento Recusos
Aviso Adiamento com Nova Abertura – CC n.º 01/2020 –
Aviso de Licitação –
Aviso de Alteração – Convocação 5.11.2019 –
Resultado de Recurso Fundac e Convocação
Aviso DODF
Aviso de Resultado – Habilitação e Convocação
Erro ao executar o código ext = ExtratorTituloSubTitulo(file)
.
Erro acusado: list index out of range
Erro no terminal:
Erro na extração de titulos do pdf: ../data/dodfs/2019/01_Janeiro/DODF 021 30-01-2019 SUPLEMENTO B.pdf list index out of range Erro na extração de titulos do pdf: ../data/dodfs/2019/01_Janeiro/DODF 002 08-01-2019 EDICAO EXTRA.pdf Não começa com títulos Erro na extração de titulos do pdf: ../data/dodfs/2019/01_Janeiro/DODF 005 15-01-2019 EDICAO EXTRA.pdf list index out of range Erro na extração de titulos do pdf: ../data/dodfs/2019/01_Janeiro/DODF 001 01-01-2019 EDICAO ESPECIAL.pdf list index out of range
Passos para reproduzir o erro:
Comportamento esperado:
Retornar um objeto com um dicionário contendo títulos e subtítulos do pdf passado no argumento.
Notebook Vivobook:
Implement PDF2Image configs to be passed via cli
Update README.md file:
It would be useful to have a CLI flag to extract only pdfs that are below a certain size or page number. Big pdfs uses as lot of memory and some computers might not be able to handle.
Aviso Resultado do Julgamento da Habilitação
Aviso Resultado de Julgamento das Propostas e Convocação
Aviso Interposição Recursos – Propostas Técnicas
Aviso Resultado Julgamento Propostas Técnicas
Aviso Convocação Segunda Sessão
Aviso Alteração Subcomissão Técnica
Describe the bug
When you opt to run dodfminer with the option -t -with-titles
, it does not work.
To Reproduce
Steps to reproduce the behavior:
dodfminer -t -with-titles
Expected behavior
Create json with titles
Desktop (please complete the following information):
It'd be interesting if the repo's requirements.txt could be generated locally instead of containing all project dependencies once there are modules which are independent from each other and therefore should not rely on one single requirements.txt file.
deve ser feita classe que extraia os títulos/subtítulos de exemplares do DODF
Describe the feature you would like the improvement and the reasons to it.
Output the JSON file is a little bit unfriendly. User willing to read it have to manually change the encoding, also documents might get messy.
Solution you'd like
Implements the output to be saved to a MongoDB database in a BSON file, where each pdf is an instance. This would enable a development of a module to create user interaction with the database.
Majority props are no longer present on the .csv produced by DODFMiner. Also even the props which appear in the dataframes have only nan.
To Reproduce
dodfminer extract -s path_to_pdf sem_efeito_aposentadoria
Expected behavior
An dataframe containing also the columns:
and not having all the inermediate collumns filed with NaN.
File: https://ufile.io/ycq50e9p
Page 9.
PROBLEM: Mixed titles:
These titles must be separated.
Expected solution: take page column into account when mounting titles/subtitles.
Aviso Errata 02 –
Aviso de Licitação
Aviso de Convocação para devolução de Invólucros
Aviso de Revogação
2.º Aviso Alteração Subcomissão Técnica
1.º Aviso Alteração Subcomissão Técnica
criar arquivo contendo todos os títulos dos PDF do DODF no período [2001, oo]
Implement download only one single pdf for tests in bad internet connection
Add the possibility to change tesseract configs on execution
Extrair a informação de hierarquia entre os títulos e subtítulos.
Criar o CLI de execução do programa
Aviso de reabertura
Aviso de suspensão
Aviso de nova abertura –
Aviso de suspensão –
AVISO SORTEIO SUBCOMISSÃO TÉCNICA
AVISO SUSPENSÃO LICITAÇÃO
Os arquivos de extração de títulos foram mal posicionados e precisam ser refatorados para sem implementado como código fonte.
o pylint ficou de ser usado; porém, qual o arquivo configuração a ser usado?
Executei a ferramenta no extrator localmente mas como a identação padrão difere da minha, deu uma pontuação bem negativa.
Passar --indent-string=' ' não resolveu, mesmo estando meu TAB configurado para 2 espaços no VSCode.
the problem was caused by a mistype extract_title_subtitle intead of extract_titles_subtitles.
These issue should be fixed ASAP because the currently extractor version contains that bug and does not work since this is a critical bug.
Progress Bar is Crashing when download finishes
Encapsulate core functions in class to prevent user access
It has been determined that the Dash interface will be removed and a React page will be created to replace it. To use React as a front-end, an API is required.
The solution found is create an api with Flask
Código não segue alguns padrões de documentação e PEP8.
Realizar as seguintes melhorias na branch de refatoração, nos arquivos title_extractor e title_filter
Melhorias:
Currently there are two tests failing:
It is important to make the tests pass for coverage improvement.
Some documentations are missing and some does not follow the google format.
http://google.github.io/styleguide/pyguide.html#38-comments-and-docstrings
It's necessary to replace Dash. The actual method is inefficient, slow and difficult to work in multiple fronts.
To resolve this problem, has been decided to replace Dash by a React Page in Frontend
Prerequisites:
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.