socialplanning / opencore Goto Github PK

Somehow we sometimes have multiple user accounts with the same email address.

Need to investigate.

I don't know offhand if it's possible to preserve original creation dates and creators when reimporting content from an export. If we can't, it's not a blocker, but it would be a shame to lose that data, so it's worth investigating.

This pertains to all major content types -- members, projects, mailing lists, wiki pages -- I assume the answer (and technique, if there is one) will be the same for all content types.

e.g. http://openfsm.net/projects/varal-da-violncia/summary

Need to create as admin instead, and then:

• manually set project creator string
• manually delete admin from project team

Wiki attachments are currently losing some information in export:

• Creator
• Creation Date
• Title (as distinct from filename/ID)

These need to be somehow added to the export, and then also factored into the reimport code.

A simple way to do this would be adding a microformatted index.html page to each page subdirectory that has attachments, e.g.:

<ul>
  <li><a href="attachment-1.pdf">Title For First Attachment</a> uploaded on <time pubdate datetime="2005-11-13T09:00Z"></time> by <span class="author">ejucovy</span></li>
  ...
</ul>

We could then just parse that on reimport.

Include metadata upon reimport.

• Creator
• Creation date
• Filename

In some test runs of Listen export->reimport, I noticed what look like some content encoding issues -- blobby-question-mark entities appearing in the reimported archives where they were not present in the original.

Need to reproduce this bug and fix it.

• Birthday
• Site name
• FQDN
• Email from address
• Email blacklist

• Team
• Site members requesting to be members
• Site members invited
• Email invites
• Membership requests pending confirmation of new site account

Make sure to preserve dates for all, and listed + role status for full fledged team members.

Many people on the /manage_membership page of a mailing list are displayed by email, even though the email-to-user-account function should be able to find their accounts.

Write a validation/migration script to run through the whole site database to find & correct any other manifestations of bug #43

Remaining to do:

• site role (admin/member)
• confirmation code for unconfirmed members? maybe?
• last login date

i.e. apply modifyportalcontent permission to site admin accounts

After I implement a solution for socialplanning/opencore-listen#1 upstream in the Listen codebase, I need to merge that code to the opencore build profile and test it in my zopectl script.

Ideally, we'd scan each project and determine if it has been modified at all since its last export. This looks like it will be very difficult though. Many objects don't cause the project' lastModifiedDate to change when the objects change (e.g. new mailing list messages, changes to team membership) and some objects evidently don't even update their OWN lastModifiedDate to reflect all changes (e.g. promoting a team member to admin)

I think a good enough solution would be to just check whether the project's wiki needs to be re-exported. This should be pretty easy, and also will give us the most speedup -- wiki export is the slowest part of project export by far. So the entire project would be exported into a new zipfile, but we would check if an existing wiki export is up-to-date and, if so, just move that wiki export into our new zipfile instead of re-exporting the wiki.

• Export wiki history
• Export metadata for each page (maybe handled in history)
• Export file attachments
• Import history
• Preserve metadata on import
• Import file attachments

In a project export, there will be a lists/ directory. Within the lists/ directory, there will be a subdirectory for each mailing list in the project.

Each exported mailing list contains a settings.ini file generated from this template: https://github.com/socialplanning/opencore/blob/master/opencore/export/export_list_conf.ini.tmpl -- which should contain all data necessary to reconstruct a mailing list, including its mailto, list of managers, and workflow policies.

So, the import code will need to iterate through the subdirectories of lists/ and create a new mailing list object for each subdirectory, based on that subdirectory's settings.ini data.

We need to support reimport of wikis from a project export.

The structure of the project export means that data will be in two places.

The primary data will all live in the wiki_history/ subdirectory of the export. This is a BZR repository with full wiki history, preserving original commit dates, messages and authors. File contents are preserved in their "unbaked" wiki formats.

Meanwhile, file attachments will live in subdirectories of the pages/ subdirectory of the export. Historical revisions of file attachments are not preserved (by design). Current versions of attachments are organized by the pages that they are attached to. So attachments on the "project-home" page live in the pages/project-home/ subdirectory of the export.

Import can therefore be done in two phases:

1. Complete content import from BZR history from first revision to last
2. File attachment import from pages subdirectory, in any order

The pages/ subdirectory also contains current wiki page contents, transformed into "baked" HTML with headers and necessary CSS links, so that the export's pages/ subdirectory is suitable for standalone drop-in publication to a web server. These should be discarded during import: they are lossy, and redundant with the lossless BZR history.

This would be included in the join request email that's sent to project admins.

See e.g. http://openfsm.net/projects/ui18n/lists/ui18n-discussion/archive/2009/04/1240516617333/forum_view vs -- on the latest test-reimport site -- /projects/ui18n/lists/ui18n-discussion/archive/2009/04/1240516617000/forum_view

Allow admins to specify a wiki page that should receive a serialization of the project's team, automatically updated every time the team changes.

Need to think more about security here.

All of them are missing a lot of project-membership listings.

It seems like performing a reindex on membrane_tool's project_ids index fixes the @@team page, but not @@Profile. (I'm not sure about @@manage-team.)

I'm trying to reindex all indexes on portal_catalog now to see if that impacts @@Profile.

Allow admins to specify a mailing list that would receive project membership request emails, instead of sending it (out-of-band) to project admins.

Include this in project export and reimport.

Need to investigate whether the existing Listen export functions handle message attachments. If not, we need to build that in to both export and import; in some cases it's essential content.

We'll need a script that exports all projects.

Here's some code that looks like it works, given constants PROJ_ID and OUTDIR:

>>> from opencore.utils import setup
>>> app=setup(app)
>>> from opencore.export.export_utils import *
>>> project = app.openplans.projects[PROJ_ID]
>>> status = get_status(project.id)
>>> status.features = ["wikipages", "mailinglists", "wikihistory"]
>>> outdir = OUTDIR
>>> tmpfd, tmpname = tempfile.mkstemp(suffix='.zip', prefix=TEMP_PREFIX, dir=outdir)
>>> tmp = os.fdopen(tmpfd, 'w')
>>> z = ZipFile(tmp, 'w')
>>> proj_dirname = badchars.sub('_', project.id)
>>> exporter = ContentExporter(project, None, status, proj_dirname, z)
>>> exporter.save()
>>> z.close()
>>> tmp.close()

When reimporting mailing lists, we need to make sure to include and fix up:

• Creator
• Creation date
• Last modified date
• Description

We currently support export of projects' contents, but do not export the metadata about the project itself.

This needs to include:

• Project ID/slug
• Project title
• Project description
• Project creation date
• Project security policy
• Installed featurelets
• Project homepage
• Project team -- including member roles and privacy ("listed") settings, as well as member join dates
• Project creator
• Project logo
• Project modification date
• Project location

Find them by code like:

for mem in app.openplans.portal_memberdata.keys():
  if not IHandleMemberWorkflow(app.openplans.portal_memberdata[mem]).is_unconfirmed():
    if IAnnotations(app.openplans.portal_memberdata[mem]).get("opencore.member.pending_requests", {}).items():
      print mem, IAnnotations(app.openplans.portal_memberdata[mem]).get("opencore.member.pending_requests", {}).items() 

• Export news history
• Export metadata for each page (maybe handled in history)
• Export file attachments
• Import history
• Preserve metadata on import
• Import file attachments

Not sure what content there is other than "News" -- I'm pretty sure About/Contact/etc are all views instead of content, but we need to make sure we don't miss any.

The listen moderation queues (both senders/subscribers, and messages) are, I believe, not currently exported or imported in any code.

Does anybody care?

Listen already has import features built in, but they aren't tested at all in an opencore setting.

The built-in export and import features pertain to:

• Mailing list archives (in MBOX format)
• Mailing list subscribers and allowed senders (in CSV format)

The built in export/import features do not include metadata and settings on the list itself (including list managers) -- these need to be handled separately.

Right now I'm just hardcoding all projects to have listen, and nothing else, on reimport.

And I'm currently setting the homepage for all projects to /summary.

Need to pull these preferences from the settings.ini instead.

They're already there on export, but not in a very machine-friendly format -- they just exist in an <h1> on the baked export.

Should add it to the JSON map file as well. (I think I have a JSON somewhere?)

Add site manager views to list all unconfirmed members.

For each unconfirmed member, show:

• User ID
• Email address
• Creation date
• Pending membership requests, with messages

For each unconfirmed member, allow manager to:

• Delete member (with all pending membership requests)
• Delete one or more pending membership requests
• Resend confirmation code email
• Generate confirmation link for direct sending
• Auto-confirm member (making sure to properly convert all pending membership requests, create people folder content)

• Filename
• Creation date
• Modification date
• Creator

We have an export of user photos, but we don't currently reimport them.

On the summary page, wiki pages end up being randomly ordered after reimport. The last modified dates are all correct on the pages themselves, but the summary page (and presumably the underlying RSS feed) isn't sorting by that somehow. Maybe we just need to reindex some catalog after import.

Allow admins to specify a mailing list to receive wiki change notices, or send (out of band) to project admins, or none.

Include this in project export and reimport.

We need to preserve:

• titles
• creators
• creation dates

for all attachments upon reimport. Right now dates are not being set properly upon reimport.