Planning our activities with issues that don't fit in a specific repository yet.
License: GNU General Public License v3.0
You should probably just look here: https://github.com/Police-Data-Accessibility-Project
This will largely be up to the legal team, but it would be useful to be able to store personally identifying information such as the arresting officer, judge, and attorneys in an anonymized but consistent form.
By retaining that data we would be able to track trends in, for example, an officer's ratio of arrests by race, or a judge's sentencing by race.
One useful trend to track would be an officer's frequency of charging for "Resisting an Officer/Arrest", because this charge can be an indicator of a more violent or abusive officer. Especially if this correlates with the charge being applied more often to POC.
With this sort of data we could draw useful conclusions about the behavior of states, counties, courts and departments without naming individuals.
As a developer, it would be ideal if I could easily spin up different scrapers using a common interface so we can easily test and deploy each scraper in a common way.
One possible path to accomplish this is to use docker and build deployable containers. This will allow a CI system to quickly build and test scrapers without needing to know specific details about each one.
Each container will need to install the specific dependencies for each scrapper.
The biggest challenge will be setting up the browsers in a headless manner. As of today the node scraper uses puppeteer with chrome, and the python scrapper uses selenium with firefox.
After scraping continually for a day, my scraper for Bay County stopped working properly.
This was because some case numbers on the portal result are associated with more than one case...
To reproduce:
Go to the portal's search page and search for case 19002535 and a single result will appear. Now search for case 19002536 and a search window with two cases will appear: 19002536CFMB and 19002536CFMA, a behaviour that was not seen before.
This should not be hard to resolve, each case number on this 'Case Search Results' window should be opened and scraped.
Currently, the search_portal() function treats arriving at the Case Search Results window as a failed search, as the same window opens when there are 0 search results. This assumption must also be changed.
If you intend to reproduce the behaviour, take a cell in the bay-county-scraped.csv output and change the bottom row's CaseNum to 19002535 and it will start scraping from the broken record.
I have created an issue as I do not intend to immediately resolve this. Others are free to have a go, I know @mcsaucy has been contributing to this scraper a lot. :) If no one takes it up, I will get around to it.
Forgive me if we already have such things, but I think it'd be good to come up with a set of design goals/high-level tenets for the various components of the system.
It can help a lot with making a choice when presented with many differing opinions on direction. Since this is a super grass-roots organization that's trying to ramp up quickly, we're bound to end up with many differing opinions on technical direction.
Things like the below. I'm only using them as examples:
It can help keep team members on the same page, and help ramp new contributors more quickly by helping them understand the high level reasons for why we might've made certain decisions.
Hi @ktynski , I have a question about the felonies data, I downloaded it to play around and see if I could get any interesting insight out of it. What location is this from?
I made a scraper for Bay County Florida in #15, this was merged in #50. One limitation is that Bay County Florida do not show many of the minimum scraped fields requirements on their portal. Instead, these are embedded in PDFs attachments in the case dockets.
My scraper downloads these and appends the case number to the beginning. Eg: 20000113-CASE FILED 01102020 CASE NUMBER 20000113CFMA.pdf
Here's an example case filing PDF. The blue fields contain the arrest date, arresting officer, officer ID, and DOB, all of which are missing from the scraped data at present. The forms are typed, not hand-written, but text is not embedded as a textbox, but a image/scan of the printed form.
By extracting these fields the scraped data will be more complete.
This would require the following:
I have no experience scraping data from PDFs, some people with experience doing this have discussed this briefly in Slack. Feel free to take on this task or private message me (Oscar) if you are willing to lend advice.
This needs to be made up to date, or removed. Does Security need its own Component in Github?
Unable to join. Please generate a new invite link and update the links in README.md and CONTRIBUTE.md.
Seems this link has expired.
Since we opted to switch from a mono-repo project to a multi-repo project, code related to scrapers has moved to Police-Data-Accessibility-Project/Scrapers.
In light of this, some text has become inaccurate, such as:
This repository now seems to primarily consist of organisation and meeting minutes.
Structured, semi-structured, RDBMS or flat files...
Consider layers of the data landscape: ingestion, processing, and serving.
I propose a flat file data lake for raw data. The data will likely have different schemas and therefore raw data should be curated in the processing layer by an ELT/ETL tool. Finally, data should be stored in a performant warehouse for serving.
All counties in SC appear to use the same platform. You can get to each counties records from this page... https://www.sccourts.org/caseSearch/
I'd like to see a scraper built for this, as the single scraper could cover the entire state.
Discussion for building risk assessment for holding PII. I'd like to focus on distilling this towards a format similar to:
Please be creative, threat analyze to your heart's content.
Project Decisions about PII, technical or non-technical - i.e. PDAP will do this approach to PII
Ex:
Thanks!
'Counties/Florida' is wonky, but we'll definitely hit county name collisions otherwise. Maybe we should have Municipalities/Florida/Bay or something? We should also probably ween ourselves off of spaces (and dashes) as path components, as many languages don't like that.
This could end up looking like:
Municipalities/Florida/Bay/...
Municipalities/Florida/St_Johns/...
Municipalities/New_York/New_York_City/...
Provide a package for web scrapers to validate processed data.
Please add/comment on features scoped to CSV FORMATTING ONLY in this issue.
I'll update this as we think through what is required for this. Refer to the README for the current list of minimum fields (and information about expected PII)
To project owners: Don't be afraid to step on my toes here and edit/update this issue however you want.
The #database_general slack channel has a link to the first database meeting and the meeting minutes can be found here: https://github.com/Police-Data-Accessibility-Project/Police-Data-Accessibility-Project/blob/master/Meeting%20Minutes/db_team_minutes/5_31_2020.md
This can be compiled from the #_introduceyourself slack channel, just needs someone to do the tedious work of going through it all.
I'm working on a scraper for Bay County Florida. This portal is based on Benchmark by Pioneer Technology, so the scraper shouldn't require many changes to work on other counties with this same portal.
If anyone would like to assist please let me know. I'm mostly making this issue so that others don't begin work on the same portal system.
As language/libraries have not yet been defined, I am using Python with Selenium as the scraper library.
Currently: https://airtable.com/shrUAtA8qYasEaepI/tblx8XaKnFTphWNQM
This is an interface which helps people find what they need in our Data Sources db. Most of it is a table with filters for "faceted search".
As a Scraper coder, I should be able to find a data source and its ID using search filters.
As a data consumer, I should be able to find list of data sources pertaining to a geographic or topic area.
As a PDAP volunteer, I should be able to find out which data sources have been archived, and which have scrapers.
Police-Data-Accessibility-Project/data-sources-app#15
Police-Data-Accessibility-Project/data-sources-app#17
We're moving more things from Slack to github issues (like suggestions and design discussions). It may be helpful to have labels which clearly denote the types of issues that are present.
This also lets us clearly denote issues which are good for folks just getting involved (it's kinda rough presently).
To build a generalised scraper that works on Tyler Technologies outsourced court record portals (or other widely used portal systems), it would be a good first step to know which counties use which portal system.
What are the obvious indicators of a Tyler Technologies portal?
So far I've observed Calaveras Superior's domain (cacalaverasportal.tylerhost.net) has tylerhost in its name, though this is not the case for all Tyler Technologies portals.
The bottom right of the page has an "Empowered by Tyler Technologies" logo.
The bottom left of the page says '© 2020 Tyler Technologies, Inc.'.
Where should this list be compiled?
I propose adding an extra column to Privacy_Public Access to Court Records State Links.csv detailing each portal's vendor.
We could create an email template supporting the National Police Misconduct Database
https://www.vice.com/en_us/article/889gva/defund12-tool-emails-city-councilmembers-with-one-click
The outreach, recognition, and copywriting teams can help.
To: [AUTOMATICALLY POPULATE RELEVANT CONGRESS PEOPLE]
Subject: [*** INSERT UNIQUE SUBJECT LINE ***]
Message:
Dear Representatives,
My name is [YOUR NAME] and I am a resident of [CITY/TOWN/DISTRICT]. I am writing to you in support of the National Police Misconduct Database introduced in [TBD].
[TODO: WRITE REQUEST FOR THEM TO SUPPORT BILL AS WELL]
[TODO: EXPLAIN WHY THE BILL IS IMPORTANT]
[TODO: ]
Thank you,
[YOUR NAME]
[YOUR ADDRESS]
[YOUR EMAIL]
[YOUR PHONE NUMBER]
Having a Python linter would help us maintain good code health. Before we (or immediately after) do that, we should hit everything with black once and manually tidy up what's necessary so we start out healthy.
If we want to use black more going forward, I'd be down for that, but I don't have a strong opinion.
For anyone looking at scraping the court records for Alameda County, CA: it seems the link on the spreadsheet is only for civil trials which I am assuming we are ignoring. Criminal trials are available at: https://publicportal.alameda.courts.ca.gov/publicportal/ and they seem impossible to scrap by day.
Currently I think the best option is to go:
Search Hearings > Search type = Judicial Officer; Go through all cases under each individual officer in the drop-down from 2005(oldest available record) to present
This only provides case numbers though which we will have to plug back into their "smart search" to actually get case data from. These case records DO NOT contain any information about the arresting officer so I don't know if it is even worth scrapping.
I tried joining the slack, but it doesn't seem to be emailing me when it claims it is. Have attempted several times. Is link expired?
Possible things to define:
Where data should land when a scraper is run?
$OUTPUT_DIR system variable should be respected by the scraper and data should be put there. Then some common sidecar process when the scraper is run in the cloud can handle watching that directory and moving the data towards its final destination.Should there be a common way to specify where to start? It isn't clear to me whether this has to be done as whole-sale scraping A-Z, or if certain data sets can be targeted in batches. Somewhat similar to message queue offsets.
The site has an extensive form we can use to query records. Some desired data, such as
may only exist within court documents (summons, complaint, etc.) and more human-based approaches might be necessary.
Recommended by a cyber security expert on our board.
Maybe Travis?
What is your view of the PDAP's Mission Statement
If you have an idea of the Mission Statement, please contribute in this form below - succinct and measured is best. If you'd like to contribute dialogue towards what other Mission Statements have been suggested, keep it civil and facts-based.
Example Mission Statement idea:
(Have these two)
Mission Statement: PDAP's Mission Statement is......
Why I think this:
(Suggested)
What will this Mission Statement stop us from doing:
How does this Mission Statement help us grow:
Establish logging on all server assets in edition to the already existing base DO logging. My plan for this is to setup a Splunk server (free tier) and then put the Splunk Forwarding Agent on all servers
Please use this issue to discuss obtaining legal counsel and the risks associated with the groups activities.
I have picked up bits and pieces but let's start this thread to define the requirements of what we are building.
Chicago: citizens police data project (data via FOIA)
National: Police Data Initiative (from the departments themselves)
I've added an adaptation of this post from the #_announcements channel in the team slack to the Wiki as the "Organization" page: https://policeaccessibility.slack.com/archives/C014C0S31A9/p1590889096146300
These changes can be viewed on my fork of the wiki.
I have made some changes that I'd like to point out for the sake of transparency:
I replaced all instances of "Product team" with "Engineering team" per the discussion here: https://policeaccessibility.slack.com/archives/C014X1CQ63B/p1591373296396400
When referring to the "Product" itself, I changed the text to instead refer to "the PDAP system". Instead of saying the project has a "product focus", I said it has a "software-deliverable focus".
I shortened the name of the "Security & Operations" team to "Operations", since the original writeup itself did shorten that name in the description.
Instead of saying that "the conversation around the mission statement is being discussed offline", I said "the conversation around the mission statement is being discussed among the leadership". This was done with the purpose of expressing more transparency.
The links to join the Slack channel in the documentation appear to have expired. They both link to a Slack page with the following message:
This link is no longer active
To join this workspace, you’ll need to ask the person who originally invited you for a new link.
What: build 1-sheet recommendation of tool, actions, and points of contact on Security Engineering for volunteers working on PDAP
Why: start building communication bridges and trust b/t SecEng and the rest of the project, enable volunteers to safely participate in the project with least amount of possible friction and acceptable risk
How:
--> aggregate ideas here
--> split into separate docs base on role in the project
--> Split b/t 'must do,' 'important to do,' 'consider doing,' or similar idea (ranked suggestions)
--> distribute docs
Follow-on work: ID what can be enforced from these recommendations, start using this for threat vector planning, etc.
We're considering using the Notion API or Django here. Notion is configurable without code, and one less app to deploy. Django is more customizable.
Dolt SQL API: https://docs.dolthub.com/dolthub/api
saw reddit thread in /r/javascript about this repo and a user in there commented about a token being exposed.
adding this just to highlight it in the actual repo
there's a token exposed in the r.js file under the scraper
I'm working on a scraper for Missouri with an emphasis on St. Louis (for now). It's a really old Java + JS site, and it's rather arcane. Once I'm done with the 22nd circuit (STL), I'll be able to generalize the scraper and capture the rest of MO.
current link[1] in CONTRIBUTING.md is expired - can someone provide a new one? Thanks!
https://join.slack.com/t/policeaccessibility/shared_invite/zt-eji7fh9w-slynNpPJtcGLUUhbhBmbTg
Transferring issues is a thing. Can someone transfer the scraping-related issues over to the Scraping repo? I lack the permissions. 😢
As a data scraper or interested stakeholder, I should be able to get a birds-eye view of how many scrapers are written, active, etc. for the quantity of datasets.
I should be able to drill down and find specific areas of contribution.
This may just be a summary of a few key metrics above, below, or overlaid on the map.
Great initiative!
Just wanted to suggest some sort of way people could sponsor any of the compute/api/etc cost to help support this.
Hey everyone, I've started to see people talk about opening PRs and actually merging code (e.g. scrapers) already, but I'm concerned about the fact that there's no code license in this repository yet. IANAL, but my loose understanding is that if the repository does not have an explicit license, then the authors of each contribution have full copyright over their work, and others (i.e. the project owner) would not be able to assign a license to the project without the explicit approval of all the existing contributors. So I suppose I'd like to open up this issue as a place to discuss licenses for the project.
I've heard talk that it might be desirable to license the software in one way and the collected data in another. @KMK on slack suggested GNU GPLv3 for the software license, which I think would be great for this project. For anyone who's not familiar, here's a good summary of it from https://choosealicense.com/licenses/gpl-3.0/
Permissions of this strong copyleft license are conditioned on making available complete source code of licensed works and modifications, which include larger works using a licensed work, under the same license. Copyright and license notices must be preserved. Contributors provide an express grant of patent rights.
I came across your project on Reddit and figured I'd submitted to Hacker News. As I'm writing this, it's the top submission. This is an awesome project and I love the idea.
Just wanted to let you folks know and I apologize if you didn't want the attention yet.
We should change the dataset_status table to be these instead, in this order:
Cannot be scraped now
1 invalid URL
2 legal blocker
3 not started
4 scraper out of date
Can be scraped now
5 scraper and sample data exist
6 scraper can be run manually
7 scraper is run automatically
I've transcribed the following document that was posted in Slack and added it to the wiki home page
You can preview the wiki page on my fork. Once you're ready to merge, you can use this URL for the forked wiki repo
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.