Committee assignments? about congress-legislators HOT 52 CLOSED

I've got them in NYT API, but house is working on them. Senate XML is up to
date on cmte-by-cmte basis, at least.

On Saturday, February 9, 2013, Joshua Tauberer wrote:

Anyone know if the House is working on posting them and if the Senate XML
up to date?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/36..

from congress-legislators.

Okay I guess from laziness I'll wait for the House.

from congress-legislators.

The Senate's XML seems to match the HTML, and include top-level committee
assignments for new members. New members don't seem to have been assigned
to subcommittees yet, though. Derek, do you know if the Senate assigned
subcommittees in their resolutions? Anyway, I'm fine with doing repeated
sweeps over time as stuff gets added.

But as written, the committee_memberships script erases House memberships
entirely, because it loads in the whole memberships file, looks for all
current things, and saves it. It'd need refactoring to support doing one
chamber at a time. So I've also been letting it sit. People keep writing in
about them missing, though...

On Sat, Feb 9, 2013 at 7:37 PM, Joshua Tauberer [email protected]:

Okay I guess from laziness I'll wait for the House.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/36#issuecomment-13342010..

Developer | sunlightfoundation.com

from congress-legislators.

The subcommittee assignments are not in the resolutions, no, and while they are in the committee XML, they do not appear to be complete. The House resolutions are already out of date as we've had a couple of late additions and committee resignations.

from congress-legislators.

I updated Senate committee assignments, and this actually did include subcommittee assignments. So I think that's done? House is still not updated...

from congress-legislators.

Really appreciate that.

from congress-legislators.

I've added some late additions/changes to House cmte assignments to NYT API, but yeah, still not updated on Clerk's site.

from congress-legislators.

@dwillis, any objections or warnings about me using the NYT API to temporarily fill in our committee assignments for the House?

from congress-legislators.

I think we're up to date, so no. We do have ppl who served but left a cmte in the response, so anything where the end date is less than today is an active member.

from congress-legislators.

I think I crashed it. :( http://api.nytimes.com/svc/politics/v3/us/legislative/congress/113/house/committees.json?api-key=#### with a proper API key goes boom, either directly, or using the API Request Tool.

from congress-legislators.

Nope, wasn't you. Fixed now!

from congress-legislators.

Thank you!

On Mon, Mar 11, 2013 at 5:06 PM, Derek Willis [email protected]:

Nope, wasn't you. Fixed now!

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/36#issuecomment-14742721
.

Developer | sunlightfoundation.com

from congress-legislators.

And this is just top-level committee members, not subcommittees, right?

from congress-legislators.

@konklone: Any progress? Can I help?

from congress-legislators.

I stalled when I saw it was just top-level committees. I can just accept that for now, and finish this up tomorrow. Thanks for the poke.

from congress-legislators.

@dwillis - I think you have a bug in your membership listings. The Permanent Select Committee on Intelligence, with an official url of intelligence.house.gov, has Dianne Feinstein as the chair:

http://prototype.nytimes.com/gst/apitool/index.html?api_id=4&request_id=4&congress-number=113&chamber=house&committee-id=&resp_format=json&perform_request=Make+Request&use_pp=on

from congress-legislators.

Gaah! Will fix.

Fixed now. Sorry about that.

from congress-legislators.

Another, smaller one for you - Jo Ann Emerson has resigned, but her membership is still present in HSAP:
http://prototype.nytimes.com/gst/apitool/index.html?api_id=4&request_id=4&congress-number=113&chamber=house&committee-id=HSAP&resp_format=json&perform_request=Make+Request&use_pp=on

from congress-legislators.

I believe I have this done in d67656c, and it looks okay. I'm ignoring any members whose ID doesn't appear in our legislators-current file, and preserving all Senate data (committees whose IDs begin with J or S). Gonna try loading it into Sunlight's stuff now - do feel free to try it out and spot check.

from congress-legislators.

Cool. No data for HLIG (House Permanent Select Committee on Intelligence) though.

from congress-legislators.

I think that's cause of the bug Derek's working on. When he's fixed it, the
script can be re-run and it should (I think) include it.

On Wed, Mar 20, 2013 at 3:25 PM, Joshua Tauberer
[email protected]:

Cool. No data for HLIG (House Permanent Select Committee on Intelligence)
though.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/36#issuecomment-15197771
.

Developer | sunlightfoundation.com

from congress-legislators.

Oh I see it's HSIG in the NYT API but HLIG on THOMAS. I can do the fix but I don't want to step on your toes, Eric.

from congress-legislators.

Ok, I've fixed this issue - the former_members array now shows Emerson (and two Democrats as well), and for other cmtes.

from congress-legislators.

On the HSIG/HLIG issue, I think HSIG is how the House clerk refers to it, but I could be persuaded to use HLIG if that's what ppl want.

from congress-legislators.

Spot check- Andy Harris is listed in four committees, but according to his web page he left three to join Approps.
http://harris.house.gov/about-me/committees-and-caucuses

Other spot checks look good!

from congress-legislators.

Let's just special-case HSIG/HLIG here.

from congress-legislators.

Fixed with a workaround for now, where it uses HLIG to query the NYT API, but saves it as HSIG.

That's a good spot check, though I think we'll have to accept the risk of outdated information until the Clerk updates its information. I don't want to make manual adjustments to the NYT-downloaded data, if we're going to replace it.

from congress-legislators.

Sure. On GovTrack I used to only update membership a few times a year, so I'm not worried about it. Definitely don't want to manually modify it here. Thanks guys! (It's coming up on GovTrack momentarily.)

from congress-legislators.

Harris is correctly listed as a former member for Science, Transportation and Natural Resources in the API.

from congress-legislators.

So he is - but I'm only using current_members, not former_members. Looking into it...

from congress-legislators.

Fixed it, I had cached some relatively recent version of the NYT responses which didn't have them listed as former members yet. I wasn't passing the force-download flag to the download call, so it was always using the cache and never re-downloading the updated data. The data now excludes what the NYT lists as former members.

from congress-legislators.

Harris still appears in HSII and HSPW (i.e. one correctly removed, two remain).

from congress-legislators.

@dwillis is clearly on this, they've now been removed from the NYT API, along with a bunch of other fixes. Committed an update.

from congress-legislators.

Thanks guys! (I think we can close the ticket.)

from congress-legislators.

I don't know - this doesn't feel done until the House updates and we get subcommittees. Anyone have any idea when that'll happen?

from congress-legislators.

I don't, but we can probably ask them on Friday.

from congress-legislators.

Joint committees are currently only listing Senate members.

Do we have a listing for House members of joint committees anywhere?

from congress-legislators.

I don't know if I've ever seen House membership of joint committees in my
life. :/ Anyone else have any insight on this?

On Thu, Mar 21, 2013 at 11:50 AM, schmod [email protected] wrote:

Joint committees are currently only listing Senate members.

Do we have a listing for House members of joint committees anywhere?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/36#issuecomment-15246737
.

Developer | sunlightfoundation.com

from congress-legislators.

Hasn't been posted on the Clerk site since the 111th Congress.

Eric Mill [email protected] wrote:

I don't know if I've ever seen House membership of joint committees in
my
life. :/ Anyone else have any insight on this?

On Thu, Mar 21, 2013 at 11:50 AM, schmod [email protected]
wrote:

Joint committees are currently only listing Senate members.

Do we have a listing for House members of joint committees anywhere?

—
Reply to this email directly or view it on
GitHubhttps://github.com//issues/36#issuecomment-15246737
.

Developer | sunlightfoundation.com

---

Reply to this email directly or view it on GitHub:
#36 (comment)

Sent from my Android phone with K-9 Mail. Please excuse my brevity.

from congress-legislators.

Derek says he's got them in the NYT API, from the resolutions. I'll update the script to add them.

from congress-legislators.

Right now, the NYT has full House and Senate membership for the Joint Economic Committee (JSEC), but only Senate members for the Joint Committee on Taxation (JSTX) and no members for the Library (JSPR) or Printing (JSLC) joint committees.

from congress-legislators.

FYI: http://www.govtrack.us/congress/bills/113/sres88/text

from congress-legislators.

I'll check on the House assignments.

from congress-legislators.

The House Clerk has posted two things, as of April 16 - an HTML list of (top-level) committee assignments, and a PDF of all committee assignments, including subcommittees.

The PDF is text-based and appears pretty scrapable - but geez, I'd rather not. Still, unless anybody has any alternative suggestions, it's probably best to assume these are the best we'll get in the near future.

from congress-legislators.

In previous years, this page would have links to the committees that would contain assignments, including subcommittees. So that might be coming, and would be easier to handle than the PDF, I'd think.

from congress-legislators.

I noticed recently there is committee info buried:

http://clerk.house.gov/member_info/mem_contact_info.aspx?statdis=CA20
http://clerk.house.gov/committee_info/index.aspx?comcode=AP00

But I haven't checked if it's up to date.

from congress-legislators.

AHA! That is indeed up to date.

from congress-legislators.

Yeah, I just did some spot checks and it does look up to date. Thanks for
saving Christmas, Josh!

On Mon, Apr 29, 2013 at 3:31 PM, Derek Willis [email protected]:

AHA! That is indeed up to date.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/36#issuecomment-17188718
.

Developer | sunlightfoundation.com

from congress-legislators.

For all we know this has been available all Congress.....

from congress-legislators.

I just overwrote the committee membership file with info from the House Clerk pages: e883c30

It was hard to verify that nothing is worse off now because the diff was not very friendly. I think we gained a few missing subcommittees' memberships, but we must have lost JSEC House members.

I'm thinking of closing this issue since I don't know that we can or want to push this toward completeness for joint committees. Maybe we'll open another issue for House members of joint committees if someone cares enough to work on it?

from congress-legislators.

Btw, the change to the scraper to make it work for House committees is in 6526705: rather than getting the committee ID list from the Clerk page, just assume we already have the list in committees-current.yaml and go straight to the committee membership pages.

from congress-legislators.

Awesome, thank you. Yeah, let's close it and move specific improvements into new threads.

from congress-legislators.