stl-public-meetings / city-scrapers-stl Goto Github PK
View Code? Open in Web Editor NEWScrape, standardize and share public meetings from local government websites in St. Louis
License: MIT License
Scrape, standardize and share public meetings from local government websites in St. Louis
License: MIT License
URL: https://www.stlouis-mo.gov/government/departments/planning/planning/planning-commission/index.cfm
Spider Name: stl_planning
Agency Name: St. Louis Planning Commission
See the contribution guide for information on how to get started
URL: https://www.claytonmo.gov/government/boards-and-commissions/economic-development-advisory-committee
Calendar URL: https://www.claytonmo.gov/calendar/meetings/-seldept-3/-toggle-all
Spider Name: clay_economic_development
Agency Name: Clayton Economic Development Advisory Committee
See the contribution guide for information on how to get started
URL: https://www.claytonmo.gov/government/boards-and-commissions/board-of-adjustment
Calendar URL: https://www.claytonmo.gov/calendar/meetings/-toggle-all/-seldept-8
Spider Name: clay_adjustment_board
Agency Name: Clayton Board Of Adjustment
See the contribution guide for information on how to get started
URL: https://www.claytonmo.gov/government/mayor-board-of-aldermen
Calendar URL: https://www.claytonmo.gov/calendar/meetings/-toggle-all/-seldept-2
Spider Name: clay_aldermen
Agency Name: Clayton Board Of Aldermen
See the contribution guide for information on how to get started
URL: https://www.stlouis-mo.gov/government/departments/public-service/index.cfm
Spider Name: stl_public_service
Agency Name: St. Louis Board of Public Service
See the contribution guide for information on how to get started
URL: https://crevecoeurcitymo.iqm2.com/Citizens/Calendar.aspx?From=1/1/1900&To=12/31/9999
Spider Name: cc_audit
Agency Name: Creve Coeur Audit Committee
See the contribution guide for information on how to get started
URL: https://crevecoeurcitymo.iqm2.com/Citizens/Calendar.aspx?From=1%2f1%2f1900&To=12%2f31%2f9999
Spider Name: cc_venable_park
Agency Name: Creve Coeur Venable Park Task Force
See the contribution guide for information on how to get started
URL: https://www.stlouis-mo.gov/government/departments/planning/cultural-resources/preservation-board/index.cfm
Spider Name: stl_preservation
Agency Name: Planning and Urban Design Agency
See the contribution guide for information on how to get started
URL: https://crevecoeurcitymo.iqm2.com/Citizens/Calendar.aspx?From=1%2f1%2f1900&To=12%2f31%2f9999
Spider Name: cc_horticulture
Agency Name: Creve Coeur Horticulture, Ecology and Beautification Committee
See the contribution guide for information on how to get started
Spider needs to scrape phone conferences
URL: https://www.stlouis-mo.gov/government/departments/sldc/boards/Local-Development-Company.cfm
Spider Name: stl_development
Agency Name: St. Louis Local Development Company
See the contribution guide for information on how to get started
URL: https://www.stlouis-mo.gov/government/departments/sldc/boards/port-authority-commission.cfm
Spider Name: stl_port_authority
Agency Name: SLDC Development Boards
See the contribution guide for information on how to get started
URL: https://www.stlouis-mo.gov/government/departments/airport/index.cfm
Spider Name: stl_airport
Agency Name: St. Louis Airport Commission
See the contribution guide for information on how to get started
URL: https://crevecoeurcitymo.iqm2.com/Citizens/Calendar.aspx?From=1/1/1900&To=12/31/9999
Spider Name: cc_building_code
Agency Name: Creve Coeur Building Code Board of Appeals
Fix classification for existing spider
URL: https://crevecoeurcitymo.iqm2.com/Citizens/Calendar.aspx?From=1%2f1%2f1900&To=12%2f31%2f9999
Spider Name: cc_planning
Agency Name: Creve Coeur Planning and Zoning Commission
See the contribution guide for information on how to get started
URL: https://www.stlouis-mo.gov/government/departments/health/board/index.cfm
Spider Name: stl_health
Agency Name: St. Louis Joint Boards of Health and Hospitals
See the contribution guide for information on how to get started
URL: https://www.stlouis-mo.gov/events/past-meetings.cfm?department=359
Spider Name: stl_mechanical
Agency Name: St. Louis Mechanical Section
See the contribution guide for information on how to get started
URL: https://crevecoeurcitymo.iqm2.com/Citizens/Calendar.aspx
Spider Name: cc_economic_development
Agency Name: Creve Coeur Economic Development Committee
See the contribution guide for information on how to get started
URL: https://crevecoeurcitymo.iqm2.com/Citizens/Calendar.aspx?View=List&From=1/1/1900&To=12/31/9999
Spider Name: cc_arts
Agency Name: Creve Coeur Arts Committee
Needs to scrape cancelled meetings.
URL: https://www.stlouis-mo.gov/government/departments/sldc/boards/Tax-Increment-Financing-Commission.cfm
Spider Name: stl_tax_financing
Agency Name: St. Louis Tax Increment Financing Commission
See the contribution guide for information on how to get started
URL: https://www.claytonmo.gov/government/boards-and-commissions/sustainability-advisory-committee
Calendar URL: https://www.claytonmo.gov/calendar/meetings/-seldept-10/-toggle-all
Spider Name: clay_sustainability
Agency Name: Clayton Sustainability Advisory Committee
See the contribution guide for information on how to get started
URL: https://crevecoeurcitymo.iqm2.com/Citizens/Calendar.aspx?From=1/1/1900&To=12/31/9999
Spider Name: cc_finance
Agency Name: Creve Coeur Finance Committee
See the contribution guide for information on how to get started
URL: https://crevecoeurcitymo.iqm2.com/Citizens/Calendar.aspx?View=List&From=1/1/1900&To=12/31/9999
Spider Name: cc_arts
Agency Name: Creve Coeur Arts Committee
URL: https://www.stlouis-mo.gov/government/departments/parks/parks/forest-park-advisory-board.cfm
Spider Name: stl_forest_park_advisory
Agency Name: Parks, Recreation and Forestry
See the contribution guide for information on how to get started
URL: https://www.claytonmo.gov/government/boards-and-commissions/clayton-recreation-sports-and-wellness-commission
Calendar URL: https://www.claytonmo.gov/calendar/meetings/-toggle-all/-seldept-7
Spider Name: clay_sports_wellness
Agency Name: Clayton Recreation, Sports And Wellness Commission
See the contribution guide for information on how to get started
Request: Spelling errors/invalid links
URL: https://crevecoeurcitymo.iqm2.com/Citizens/Calendar.aspx?From=1%2f1%2f1900&To=12%2f31%2f9999
Spider Name: cc_parks_preservation
Agency Name: Creve Coeur Parks and Historic Preservation Committee
See the contribution guide for information on how to get started
URL: https://www.claytonmo.gov/government/boards-and-commissions/uniformed-employees-retirement-board
Calendar URL: https://www.claytonmo.gov/calendar/meetings/-seldept-5/-toggle-all
Spider Name: clay_uniform_retirement
Agency Name: Clayton Uniformed Employees Retirement Board
See the contribution guide for information on how to get started
URL: https://www.stlouis-mo.gov/government/departments/sldc/boards/enhanced-enterprise-zone-commission-july-26-2016.cfm
Spider Name: stl_enterprise_zone
Agency Name: St. Louis Enhanced Enterprise Zone Board
See the contribution guide for information on how to get started
URL: https://www.claytonmo.gov/government/boards-and-commissions/community-equity-commission
Calendar URL: https://www.claytonmo.gov/calendar-6/-seldept-2
Spider Name: clay_equity_commission
Agency Name: Clayton Community Equity Commission
See the contribution guide for information on how to get started
URL: https://crevecoeurcitymo.iqm2.com/Citizens/Calendar.aspx?From=1%2f1%2f1900&To=12%2f31%2f9999
Spider Name: cc_stormwater
Agency Name: Creve Coeur Stormwater Committee
See the contribution guide for information on how to get started
URL: https://www.stlouis-mo.gov/government/departments/sldc/boards/Clean-Energy-Development-Board.cfm
Spider Name: stl_energy_development
Agency Name: SLDC Development Boards
See the contribution guide for information on how to get started
URL: https://www.stlouis-mo.gov/government/departments/affordable-housing/index.cfm
Spider Name: stl_affordable_housing
Agency Name: St. Louis Affordable Housing Commission
See the contribution guide for information on how to get started
URL: https://crevecoeurcitymo.iqm2.com/Citizens/Calendar.aspx?From=1%2f1%2f1900&To=12%2f31%2f9999
Spider Name: cc_adjustment
Agency Name: Creve Coeur Board of Adjustment
See the contribution guide for information on how to get started
URL: https://www.stlouis-mo.gov/government/departments/aldermen/
Spider Name: stl_aldermen
Agency Name: St. Louis Board of Aldermen
See the contribution guide for information on how to get started
Great work getting this up and running so quickly! One of the aspects of the City Scrapers project we haven't documented very well is our use of a Python package we created, scrapy-wayback-middleware
.
The overall goal of the City Scrapers project is to improve transparency and create an archive not just of upcoming meetings, but past meetings and related documents as well as how they change over time. An important part of that for us has been archiving (almost) every page and document we scrape on the Internet Archive's Wayback Machine as well as in our static output.
Having a second, more public and accessible location makes the meeting information more available regardless of how long the project goes. We've even used it to track potential violations of open meetings laws, since it provides an external source for seeing what content was or was not on a website at a given time. Here's an example of snapshots of the Chicago Plan Commission's website over time.
The con of this approach is that it can make cron builds take significantly longer, but currently we're well under the 6 hour GitHub Actions time limit with over 100 scrapers on the main City Scrapers repo.
If you're interested, you can add scrapy-wayback-middleware
as a dependency, and then you'll likely want to subclass the middleware to also scrape any documents you find like we've done in our main middleware.py
. Then you can add it in your settings/prod.py
like we did in our settings.
We're only activating it when the WAYBACK_ENABLED
environment variable is set, and the template cron.yml
file already sets this so once it's added in your settings file you should be good to go!
Let me know if you have any questions, and I'm happy to put in a PR for this if it's helpful
URL: https://crevecoeurcitymo.iqm2.com/Citizens/Calendar.aspx?From=1%2f1%2f1900&To=12%2f31%2f9999
Spider Name: cc_personnel_appeals
Agency Name: Creve Coeur Personnel Appeals Board
See the contribution guide for information on how to get started
URL: https://www.claytonmo.gov/government/boards-and-commissions/clayton-century-foundation
Calendar URL: https://www.claytonmo.gov/calendar/meetings/-toggle-all/-seldept-12
Spider Name: clay_community_foundation
Agency Name: Clayton Community Foundation
See the contribution guide for information on how to get started
URL: https://www.stlouis-mo.gov/government/departments/sldc/boards/Land-Clearance-for-Redevelopment-Authority.cfm
Spider Name: stl_redevelopment
Agency Name: SLDC Development Boards
See the contribution guide for information on how to get started
URL: https://www.claytonmo.gov/government/boards-and-commissions/parks-and-recreation-commission
Calendar URL: https://www.claytonmo.gov/calendar/meetings/-seldept-7/-toggle-all
Spider Name: clay_parks
Agency Name: Clayton Parks And Recreation Commission
See the contribution guide for information on how to get started
URL: https://crevecoeurcitymo.iqm2.com/Citizens/Calendar.aspx?From=1%2f1%2f1900&To=12%2f31%2f9999
Spider Name: cc_horticulture
Agency Name: Creve Coeur Horticulture, Ecology and Beautification Committee
See the contribution guide for information on how to get started
URL: https://www.stlouis-mo.gov/government/departments/sldc/boards/Planned-Industrial-Expansion-Authority.cfm
Spider Name: stl_industrial_expansion
Agency Name: SLDC Development Boards
See the contribution guide for information on how to get started
URL: https://www.stlouis-mo.gov/government/departments/sldc/boards/Land-Reutilization-Authority.cfm
Spider Name: stl_land_reutilization
Agency Name: St. Louis Land Reutilization Authority
See the contribution guide for information on how to get started
URL: https://www.claytonmo.gov/government/boards-and-commissions/non-uniformed-employees-retirement-board
Calendar URL: https://www.claytonmo.gov/government/boards-and-commissions/non-uniformed-employees-retirement-board
Spider Name: clay_nonuniform_retirement
Agency Name: Clayton Non-Uniformed Employees Retirement Board
See the contribution guide for information on how to get started
URL: https://www.stlouis-mo.gov/government/departments/sldc/boards/Industrial-Development-Authority.cfm
Spider Name: stl_industrial_development
Agency Name: St. Louis Industrial Development Authority
See the contribution guide for information on how to get started
URL: https://crevecoeurcitymo.iqm2.com/Citizens/Calendar.aspx?From=1%2f1%2f1900&To=12%2f31%2f9999
Spider Name: cc_police_safety
Agency Name: Creve Coeur Police and Safety Committee
See the contribution guide for information on how to get started
URL: https://www.claytonmo.gov/government/boards-and-commissions/plan-commission-and-architectural-review-board
Calendar URL: https://www.claytonmo.gov/calendar/meetings/-seldept-8/-toggle-all
Spider Name: clay_plan_arb
Agency Name: Clayton Plan Commission And Architectural Review Board
See the contribution guide for information on how to get started
URL: https://crevecoeurcitymo.iqm2.com/Citizens/Calendar.aspx?From=1/1/1900&To=12/31/9999
Spider Name: cc_building_code
Agency Name: Creve Coeur Building Code Board of Appeals
See the contribution guide for information on how to get started
URL: https://www.stlouis-mo.gov/government/departments/public-safety/building/zoning/index.cfm
Spider Name: stl_zoning
Agency Name: St. Louis Zoning Section
See the contribution guide for information on how to get started
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.