security-force-monitor / sfm-cms Goto Github PK

Rather than double the amount of fields that each step in the wizard (see #8) has in order to capture a confidence score for each field, one way of doing this would be all at once at the end. So, once a user completes extracting all the info, we could render a table with each field added, the value and then a slider (or select box) for the confidence score.

On /create-source there are two fields for URL and Archive URL. Does the input expect protocol (e.g. https://[...]) or will it populate it automatically on submit?

Add a way to enter parent organizations to the page for entering Organizations. This suggests a bit of UI complexity given that it seems pretty likely that parent organizations will be entered at the same time as their children (and one could imagine that several layers of hierarchical information could be contained within a single source). @tonysecurityforcemonitor Does it make sense to just instruct users to start with the "largest" or "highest" (not sure what the best term is there) and then, as they add organizations, allow them to create relationships with Organizations that are already entered into the form? So, for instance, you would enter NYPD, then when you enter Patrol Borough Manhattan South, you could add NYPD as the parent and then when you add the 9th Precinct, you could add the Patrol Borough Manhattan South as the parent.

• Add division_id
• Add notes as TextField
• Replace id as integer with UUID
• superior_unit (should be another Organization)
• subordinate_unit (inverse of superior_unit)
• OrganizationMembership
• headquarters: (through which it is related to Site)
• jurisdiction or area_of_operation, relates to area association Area
• other_affiliations (Organizations can be members of other organizations like joint task forces, operations or UN peacekeeping assignments, for example. These can in turn be members of other organizations)
• memberships through which it is related to people

@evz

Right now it's a question in my mind if we are better off combining this into the 2-Organizations page. It makes sense in my mind to have all the data related to an organization in one screen. Either way there is a “break” in data entry to go from Organization to Person to Person’s Relationships, back to Organization's site/area of operations...

For the Geography Type drop down can rename Area to Area of Operations? It would be clearer for the researcher.

Is Name Site:Name (The site's name)? I should try and think of something that is more immediately descriptive for the researcher (perhaps Base Name or Facility Name). Just a note for the future - but don't let me forget!

The start date and end date placeholder text could be darker/more prominent - I had a hard time seeing it.

1. Go to /organization/edit/3/ and click Cancel.
2. Fields labelled Source for your changes and classification move to an error state (red boxed).

Expected behaviour on cancel is that if there are no changes, I'm returned to / and the expanded list item I was attempting to edit.

When users are entering dates, they should be allowed to enter just a year or just a month and a year. The fields that should validate this way are:

• Association_AssociationStartDate
• Association_AssociationEndDate
• Composition_CompositionStartDate
• Composition_CompositionEndDate
• Emplacement_EmplacementStartDate
• Emplacement_EmplacementEndDate
• MembershipPerson_MembershipPersonFirstCitedDate
• MembershipPerson_MembershipPersonLastCitedDate
• Organization_OrganizationFoundingDate
• Person_PersonDeathDate
• Violation_ViolationStartDate
• Violation_ViolationEndDate

From Tony:

First is this article - https://web.archive.org/web/20141112091927/http://leadership.ng/news/389581/fire-guts-sokoto-military-barracks

This is a type of article that we would come across frequently. It's short, but has several important keywords (battalion, brigade, commander) that would pop up in a scrap of the website. The article names several people (including their ranks and affiliations) as well as gives information related to several organizations.

This article has the following elements:

Person:Name - Tasi’u Ibrahim, Adebowale Ajagbe
Organization:Name - 26 Battalion, 1 Brigade
Organization:Classification - Army, Military
PersonMembership:organization_id - 1 Brigade, 1 Brigade Garrison (this organization is not directly named, but implied by "garrison commander, 1 Brigade, Lt.Col. Adebowale Ajagbe")
PersonMembership:Role - Commander
PersonMembership:rank - Brigadier General (it is abbreviated to Brig-Gen), Lieutenant Colonel (it is abbreviated to Lt.Col.)
Organization:Site - Sokoto (for both the 26 Battalion and 1 Brigade)
OrganizationOrganization:child_id - 26 Battalion, 1 Brigade Garrison
OrganizationOrganization:parent_id - 1 Brigade

Depending on what else is in the database these other fields could be involved as well

Person:other_names - Tasi’u Ibrahim is actually a Person:other_names - the Person:Name for this person is Mohammed Tasiu Ibrahim (which only came about from cross referencing multiple articles)
PersonMembership:date_first_cited - if we didn't have any other data in the database about these people this citation would be used as the PersonMembership:date_first_cited
PersonMembership:date_last_cited - this article is actually the PersonMembership:date_last_cited for Mohammed Tasiu Ibrahim connection with the 1 Brigade (again only known because of other research)
OrganizationOrganization:date_first_cited
OrganizationOrganization:date_last_cited
OrganizationSite:date_first_cited
OrganizationSite:date_last_cited

Here is another article that is very common - short crime reporting that is helpful for mapping police forces: https://web.archive.org/web/20140906071324/http://www.vanguardngr.com/2014/03/gunmen-kill-tricycle-rider-aba/

Person:Name - Ibrahim Adamu
Organization:Name - Ndiegoro Police Division
Organization:other_names - Abia Police Command
Organization:Classification - Police
PersonMembership:organization_id - 
PersonMembership:Role - Commander
PersonMembership:title - Commissioner of Police
PersonMembership:rank - Commissioner of Police
Organization:Site - Aba (for Ndiegoro Police Division
OrganizationArea:organization_id
OrganizationArea:area_id
Area:geonames_name - Aba South Local Government Area (we'd have to double check Geonames, but this would be the Area for Ndiegoro Police Division - anytime a crime/event happens in an area and a organization responds we assume that that area is part of the organization's area of operations)

Similar to the article above, depending on what else is in the database these other fields could be involved as well:

Person:other_names
PersonMembership:date_first_cited
PersonMembership:date_last_cited
OrganizationOrganization:date_first_cited
OrganizationOrganization:date_last_cited
OrganizationSite:date_first_cited
OrganizationSite:date_last_cited
OrganizationArea:date_first_cited
OrganizationArea:date_last_cited

And then sometimes we'll get articles that cover large personnel shifts:
https://web.archive.org/web/20130508014613/http://www.vanguardngr.com/2013/05/ig-orders-redeployment-of-11-cps-with-immediate-effect/

Or that look back at the career of an individual:
https://web.archive.org/web/20140726071006/http://www.vanguardngr.com/2014/07/police-job-best-nigeria/

All of these articles could also result in a researcher suddenly realizing that several different people are actually the same person - like in the last article I shared. Another common thing that happens is we have an article where we stumble across information that contains quite a bit of information that we need to update several different People or Organizations.

This email has already gotten very long - but the one of the last things I'll mention is there are articles which mention both police and military organizations/people and otherwise will mention different branches of security forces.

As another note - while we are extracting a lot of data from an article that doesn't necessarily mean there is a lot of data out there - as of right now I have 7 different sources for the Organization:Site for the 1 Brigade - and that is after a scan of articles from Nigerian newspapers from 2002 to 2016. To put that in more perspective, a brigade is a major military unit, ranging from roughly 3000 to 5000 soldiers.

Each models seems to have a search method which does the right thing. Lets see if this is sufficient for a global search and if not make something that is.

@evz - From a workflow perspective I think it might be better to have this screen merged with 3-People. That way a researcher is entering everything about a person on one screen.

I think re-arranging the first and last citations and renaming the real context buttons would improve workflow.

Perhaps having First Citation and then a Start Date? button - then the start context field - then the Last Citation and then an End Date? button

The thinking on how to determine when an organization has dissolved has advanced and this is no longer required.

This will enable auto complete widgets to only return things that are relevant to that area. We can do this with a geonames ID or an OCD division ID

So far we've been committing things as they come in from the form. This probably isn't the best practice. Let's define a formset_valid method where needed and commit changes there.

@evz
@tlongers

It appears that the dropdown is the same if an organization already exists in the database or if its being entered for the first time. There is a question in my mind of how the workflow will look when adding to an existing organization.

• name
• geometry (point)
• GeoNames name
• GeoNames ID
• Add division_id validated to be 2 lower char.
• Add uuid Field : autogenerate and unique between all uuid Fields.
• Add site_admin_level_1_name and site_admin_level_1_id
• Add relation to Organization via OrganizationSite

For the name autoselect in the create-orgs page (so, presumably, also the other single-value autoselects; possibly some of the multi-value ones like alias too), it breaks in strange ways when you fail to validate and the value you put in is repopulated into the field. Then selecting anything no longer updates the hidden fields, and selecting "newly created" ones doesn't even change the visible field - also in some strange circumstances the newly created options don't have the correct ids (they remain the same as the value text, rather than being changed to -1 or whatever appropriate integer). Probably it has something to do with the event handlers not binding correctly.

I think this behavior/problem can also be replicated outside of the context of repopulation after failure to validate by selecting the same new organization twice in a row.

After examining the documents that Tony shared with us regarding the existing data entry workflow, we decided to take slightly different approach on how to get information into the CMS which involves a guided data entry wizard. Since everything in the system ultimately starts with a source (and most everything is attached to a source), we decided to have things cascade from there and take the following steps (I've made this a checklist to show what we have roughed out so far):

• Step 1: Describe the source itself.
• Step 2: Describe organizations.
• Step 3: Describe people and tell us what organizations they belong to.
• Step 4: Describe the relationship between the people and the organizations (so, for instance, what rank do they have, what is their title, etc)
• Step 5: Describe Sites and Areas from the source and associate them with organizations.
• Step 6: Describe any events mentioned in the source.

This assumes a pretty radical departure from what existed in the CMS previously and is obviously open for revision.

• Organization
• Person
• MembershipPerson
• Event

• Organization model includes realfounding and just founding dates. What is the difference?
• Same thing for dissolution
• What's the difference between OrganizationComposition and OrganizationOrganization?

If a user really messes something up, we ought to leverage the versioning stuff that we have built in to allow an admin to roll it back.

In the various documentation that we've been given, there is a concept of Events which can be associated with Organizations. In the backend code, it seems like the same thing is called Violations. Which should be use?

The way things are currently setup, there's really no way to define a new boundary for an area. Do we expect that users will draw the boundaries on a map or do we want to have some kind of gazetteer that users can pick from? Or both?

Should violations be allowed to have arbitrary combinations of perpetrators/perpetrator organizations? i.e. could we list people A, B, C, and organizations X, Y, Z as perpetrators, with none of A, B, C belonging to any of X, Y, Z? @tonysecurityforcemonitor

Verification acts erratically (or not at all) if organization name is not entered. Possibly applies to other forms?

1. On / choose an existing entry.
2. Expand it using the + widget and choose any of the MembershipPerson values listed to navigate to a page which enables you to edit the data (e.g. /membershipperson/edit/1/)
3. Edit the data. Neither the Cancel nor Save buttons do anything.

The Wayback machine has an API which will return archived snapshots of arbitrary URLs. So, for instance:

http://archive.org/wayback/available?url=example.com

Returns the closest archived snapshot of example.com. In the case where there is no snapshot, there is a URL that you can hit that will force the wayback machine to create a snapshot:

https://web.archive.org/save/http://example.com

The one snafu is that the actual archive URL contains a timestamp from when it was actually snapshotted which means that you'd have to wait a couple seconds and then hit the API again to get the actual archive URL.

Anyways, something to stew on...

I have a version of this stubbed out for Organizations but we ought to allow an admin user to go through and edit things as needed.

I see skip and continue options to move the data entry workflow forward. I'd expect a Go back a step or Previous step button. Prefer this to using the browser's back button. Guess the question is "why would I go back a page?" Likely because I've messed something up, or want to double-check.

I know you've not got to that yet but had a few questions.

• If you delete a specific source, do you also delete all datapoints derived from it?
• Where a datapoint has multiple sources, which we expect will be common, how will the deletion of source work?

There are two kinds of classifications to record about an organization: the classification of a relationship between organizations and classification of an organization itself.

Related to #41 and #45

Date published - will a researcher be able to just enter month and year and/or just year and/or no date?

No date sources are often webpages where the date we use is the date crawled or date accessed -- currently the citations look like this:

"BERRIOZÁBAL.". Enciclopedia de Los Municipios y Delegaciones de México. Instituto para el Federalismo y el Desarrollo Municipal, Secretaría de Gobernación. http://www.inafed.gob.mx/work/enciclopedia/EMM07chiapas/municipios/07012a.html (accessed 28 August 2015) Internet Archive link: https://web.archive.org/web/20150829011338/http://www.inafed.gob.mx/work/enciclopedia/EMM07chiapas/municipios/07012a.html

"Zonas Militares". Secretaría de la Defensa Nacional (SEDENA). https://web.archive.org/web/20060821201706/http://www.sedena.gob.mx/ejercito/comandancias/zon_mil.htm (captured 21 August 2006)

This isn't the date published per se... I'm not sure if the url and archive url entries would allow this none url text to be entered at the end (and if that would mess up the url or not)

@evz
Also --- @tlongers - some method examples here

We'll need to add the ability to just extract events from sources. It sounds like we can maybe just skip ahead to the Sites/Areas forms and then advance to the Events form.

When a page for data entry loads, focus the cursor on the first available data entry field. Just speeds things up.

In / on the the staging site, see the source called "Source Title Test 2" - expanding it shows there are no other entities related to this source. It's possible to just add a source, skipping the steps where the user extracts content from the source. There should never be a source that has no info derived from it, but I wonder whether there are situations when a workflow to resume data extraction from a source would be useful.

On my wishlist (Related to #21): could the tool also grab the full html and render a PDF of the webpage of a source?

This is kinda built into the way we are building out the wizard but, once we build out the "skip" and "back" functionality, we need to enforce this.

Right now the event models are not related to the source table in the way that all of the other relations are. @tonysecurityforcemonitor Is that on purpose?

@evz

Data model doesn't have death date in it anymore...

Wizard forms need function "skip" and "go back" buttons

The datepicker on create-sources doesn't style correctly, despite all the apparent reqs being there.

When loading one of the later pages in the wizard with no source in context, an error about dictionary values appears instead of redirecting to the create-source page with the message box.

Some fields suggest that there is a finite list of choices that should be available for a user to be able to enter. For these fields, we could include initial sources of data that we could load in when setting up the app and it would cut down a bit on the complexity of things under the hood. These fields include:

• Rank & Role for the MembershipPerson data type
• Classifications for the Organization data type
• Violation type for the Violation data type

There may be others. @tonysecurityforcemonitor Are these something that we can pre-poulate? Or will we need to be dynamically creating these based upon user input?

• Add division_id
• Replace interger id with UUID as unique identifier / primary key
• Add notes as TextField
• Make alias a One-to-Many relationship

Found on /create-source

1. Tab to a field that is combobox and start typing
2. Nothing happens. WHAT....
3. Mouse click combobox
4. Typing ability restored! Options listed/filtered! Success!

I'd expect to just be able to start typing when the focus is on the combobox.

1. On /create-orgs there is a blue button called Add organization+, which magics a new field group onto the page.
2. After doing so, there is an orange button called Remove organization-. Nothing happens when I click on this.

Expected behaviour would be that Remove organization removes the field group if all fields in it are empty. If any field is populated I would expect to be asked to confirm the removal of the field group.

• Add an organization that is a member of another org. (member_id)
• The other org. of which the organizations is a member (organization_id)
• The date of first cited for the relation (date_first_cited)
• The date of the last citation for the relation (date_last_cited)

• Death date from Person model
• Founding date from Organization model

Others?

It appears as if the Event model currently has a text field for both the perpetrator and perpetrator organization. It seems like this ought to be changed to a many-to-many type relationship for both of these relations.

Got this error - there was more, including the trace, but this is what I saved:

MultiValueDictKeyError at /en/create-events/
"'form-0-geoname'"
Request Method: POST
Request URL:    http://sfm.datamade.us/en/create-events/
Django Version: 1.9.7
Exception Type: MultiValueDictKeyError
Exception Value:
"'form-0-geoname'"
Exception Location: /home/datamade/.virtualenvs/sfm/lib/python3.5/site-packages/django/utils/datastructures.py in __getitem__, line 85
Python Executable:  /home/datamade/.virtualenvs/sfm/bin/python3.5
Python Version: 3.5.1
Python Path:
['/home/datamade/sfm-cms',
'/home/datamade/.virtualenvs/sfm/bin',
'/home/datamade/.virtualenvs/sfm/src/complex-fields',
'/home/datamade/.virtualenvs/sfm/lib/python35.zip',
'/home/datamade/.virtualenvs/sfm/lib/python3.5',
'/home/datamade/.virtualenvs/sfm/lib/python3.5/plat-linux',
'/home/datamade/.virtualenvs/sfm/lib/python3.5/lib-dynload',
'/usr/local/lib/python3.5',
'/usr/local/lib/python3.5/plat-linux',
'/home/datamade/.virtualenvs/sfm/lib/python3.5/site-packages']

Server time:    Sat, 25 Jun 2016 09:23:08 +0000

Hrmm. Where's this UX going? Keeping as a list, or a map-based selector?