google / support-tools Goto Github PK

What steps will reproduce the problem?
1. Export google code project with questions like this "What steps will 
reproduce the problem?" in the description
2. Import into Bitbucket

What is the expected output? What do you see instead?
In Bitbucket, I see HTML like this: <b>What steps will reproduce the 
problem?</b>
Instead it should make the stuff bold, probably by replacing HTML with Markdown.

What version of the product are you using? On what operating system?
revision with sha #937ced33b1eb4e7dba8265eeb76f1da7fc0448d1

Due to the poor quality of issues imported in GitHub (See #28), could you add 
an option to the GitHub importer script to tag all the issues that were 
imported with a custom tag?
That would at least allow users browsing issues on GitHub to understand why 
they were all raised by/commented on by the repository owner and nobody else, 
if all issues were tagged with "Imported-From-GoogleCode" or something.

Thanks.

Hi,

Is there any possibility to use "proper" user names in the issues (and 
comments) once imported to GitHub? Currently all the issues and comments belong 
to the repository owner, which makes then mostly useless...

Thanks,

Nico

Link:

http://storage.googleapis.com/google-code-attachments/tint2/issue-1/comment-2/ti
nt_read_all_xdg_config_dirs.patch

Shows:

<Error>
<Code>NoSuchKey</Code>
<Message>The specified key does not exist.</Message>
</Error>

For issue:

https://code.google.com/p/tint2/issues/detail?id=1

I understand that the GitHubIssueImporter cannot create issues and comments as 
the original authors, but it would be nice if it would add a note in the 
content that stated who *actually* wrote it?

What steps will reproduce the problem?
1. Click Export to Github on any repository which is in Russian

What is the expected output? What do you see instead?
Russian characters should be visible as usual
But all of them become "?"

What version of the product are you using? On what operating system?
Latest. Windows 8

I am trying to import issues in bitbucket. All users get replaced with the 
provided default username. I saw that there is an "user_file_path" option to 
give a mapping. But I found no documentation what syntax this file should have. 
Please provide a wiki page explaining this.

The procedure to migrate wiki is not described in the FAQ.
The wiki is migrated with syntax changes to separate branch, OK.
However, you need to make manual adjustments, some pages look strange.
As the pages are migrated to a separate branch only and not added to the GitHub 
wiki, some more steps are required.

I am sure someone has thought about this already, but not finding good links. 

Please add to the FAQ.

What steps will reproduce the problem?
1. Start export and receive error GitHub code import took too long

What is the expected output? What do you see instead?
Expect if import fail converter should revert all changes.

Hi there,

I'm migrating our old support projects for Summer of Code to GitHub and have 
noticed that the downloads aren't being moved across. Arbitrary blobs are being 
retained if they're attached to issues, so our GHOP repo is intact ( 
https://github.com/MatthewWilkes/google-highly-open-participation-plone )

Downloads, however, aren't being added as releases, as I'd expect. For example:

https://code.google.com/p/google-summer-of-code-2008-plone/downloads/list
https://github.com/MatthewWilkes/google-summer-of-code-2008-plone

Can we get an update to the migrator to handle these old OSPO repositories 
correctly?

Cheers,

Matt Wilkes
(Plone)

What steps will reproduce the problem?
Attempt to export https://code.google.com/p/ftpii/ to 
https://github.com/joedj/ftpii/

What is the expected output? What do you see instead?
Expected output is a successful project import.
Observed output is "There was an error importing commits" at 
https://import.github.com/joedj/ftpii/import

So everyone can see it, this is a copy of 
https://code.google.com/p/support/issues/detail?id=33898.

What steps will reproduce the problem?
Step 1. Move a project using the "Project moved" button at 
https://code.google.com/p/mmisw/adminAdvanced
Step 2. Click on the "Publish project" button, which then appears, presumably 
to 'unmove' the forward link ("All project contents will become visible to all 
users.")
Step 3. Try to access the project

What is the expected output? What do you see instead?
Rather than restoring access to the project as it was before, all access 
attempts (whether to the root path, or to the redirect path) result in a 
redirect failure. The URL is correct 
(https://code.google.com/hosting/moved?project=mmisw in my case) but the page 
shows 404 error.  
  If you then go back and hit the Project moved button, and enter an external URL, everything looks like after step 1 above -- people get a page linking forward.  If you enter an internal project name (mmisw), we're back to the 404 error.
  While we wee in 403/404 error mode, the contents within the project were visible to direct links (at least, seemed to be for a while; but no longer).


What browser (or git/hg/svn client) are you using? On what operating
system?
Chrome on Mac 10.9.

Please provide any additional information below.

This is a copy of https://code.google.com/p/support/issues/detail?id=33898.

We do still need access to the information in this project, and would not have 
done the move if we'd realized it was irreversible and effectively removed the 
project content.

The 404 error contains the following:

404. That’s an error.

The requested URL /hosting/moved?project=mmisw was not found on this server. 
That’s all we know.

The issue exporter script takes both a github user token and a username, when 
the username can be derived from the token by looking up the current user's 
information (https://developer.github.com/v3/users/#get-the-authenticated-user)

This would leave one less option for the user to specify and would simplify 
logic to guarantee that the current user is the user they claim to be.

The importer tries to wrap the lines in the wiki.
However, the multibyte unicode characters gets broken when lines are wrapped.

Old issue:
    https://code.google.com/p/ctex-doc/issues/detail?id=1&can=1
New issue:
    https://github.com/Leo-Liu/ctex-doc/issues/1

Some characters at the end of lines are converted to Unicode replacement 
characters “���”.

As seen in Revision ec307471f338

My old hg project log are in Chinese:

    https://code.google.com/p/zhmakeindex/source/list

But in the new project

    https://github.com/Leo-Liu/zhmakeindex/commits/master

all the Chinese characters are ??? in the log.

Branch name: development-oud

Purpose of code changes on this branch:

Updated comments and added naming discrepancy notice to module docstring.

After the review, I'll merge this branch into:
/trunk

What steps will reproduce the problem?
1. run the github_issue_converter.py migration tool
2. in the middle of transferring comments, abort the script. In my case it was 
aborted due to the abuse filter kicking in.
3. run the github_issue_converter.py migration tool once more

What is the expected output? What do you see instead?
Expected that the tool picked up where it left of by continuing the migration 
of not yet transferred comments. But instead it skips this issue completely and 
continues with the next.

What version of the product are you using? On what operating system?
Linux *** 3.13.0-43-generic #72~precise1-Ubuntu SMP Tue Dec 9 12:14:18 UTC 2014 
x86_64 x86_64 x86_64 GNU/Linux

The --project_name flag appears to be required, but is not mentioned in 
GitHubIssueImporter?

[deleted issue]

Sorry for being picky about this.

I have 249 attachments in my GoogleCodeProjectHosting.json.

Using the info from issue 39 (comment indices are 0-based) I managed to obtain 
links to 122 attachments (listed in good-links.txt).

Apparently another 112 of the attachments actually use 1-based indices in the 
links  (listed in bad-fixable.txt).

Yet there are still 15 attachment links that *do not work at all* (listed in 
bad-unfixable.txt).

See the attached files for the links. I used the projects tint2 and pmn.

--

As a workaround I tried to use directly the links visible on the Google Code 
webpage, which look like this:

https://pmn.googlecode.com/issues/attachment?aid=10004000&name=magic&token=remov
edthesecrettoken

and download all of them to my computer. However I *have to* include the secret 
token parameter in the URL otherwise the script gets redirected to a Sign In 
page which it cannot handle (as it is not a browser). It is not clear how to 
obtain that token.

Finally, I tried copy-pasting the token from the browser to my script but 
apparently a separate token is required for each attachment, so my conclusion 
is that it is impossible to automate this process except maybe with a browser 
extension that crawls the whole issue tracker.

Branch name: development-oud

Purpose of code changes on this branch:
Fixed naming discrepancies and style issues.

After the review, I'll merge this branch into:
/trunk

Provide a way for project admins to remove the "Export to GitHub" button.

What steps will reproduce the problem?

# mkdir /tmp/unicodetest
# cd /tmp/unicodetest
# git clone https://code.google.com/p/support-tools/
# hg clone https://wiki.ipv6-hosts.googlecode.com/hg wiki --insecure
# python ./support-tools/wiki_to_md/wiki2gfm.py 
--input_file=./wiki/DoIHaveToUseIt.wiki --output_file=o.md
Traceback (most recent call last):
  File "./support-tools/wiki_to_md/wiki2gfm.py", line 121, in <module>
    main(sys.argv)
  File "./support-tools/wiki_to_md/wiki2gfm.py", line 117, in main
    converter.Convert(input_stream, output_stream)
  File "/tmp/unicodetest/support-tools/wiki_to_md/impl/converter.py", line 117, in Convert
    input_line = self._ExtractPragmas(input_line, input_lines, output_stream)
  File "/tmp/unicodetest/support-tools/wiki_to_md/impl/converter.py", line 156, in _ExtractPragmas
    pragma_value.strip())
  File "/tmp/unicodetest/support-tools/wiki_to_md/impl/pragma_handler.py", line 50, in HandlePragma
    .format(pragma_value))
UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-4: 
ordinal not in range(128)




What is the expected output? What do you see instead?

# python ./support-tools/wiki_to_md/wiki2gfm.py 
--input_file=./wiki/DoIHaveToUseIt.wiki --output_file=o.md
Warning (line 1 of input file):
A summary pragma was used for this wiki:
        **************
Consider moving it to an introductory paragraph.

Warning (line 2 of input file):
The following pragma has been ignored:
        #labels *************
Consider expressing the same information in a different manner.



What version of the product are you using? On what operating system?

Revision 937ced33b1eb4e7dba8265eeb76f1da7fc0448d1

GNU bash, version 4.1.2(1)-release-by_tst_tlinux_v1004 
(x86_64-unknown-linux-gnu)

Distributor ID: Ubuntu
Description:    Ubuntu 14.04.2 LTS
Release:        14.04
Codename:       trusty


Please provide any additional information below.

Both attachments (exported by different git commands) represent the same commit.

Tools does not work with python v3.
Please add to project description compatible python versions.

It would be nice to know what could go wrong before doing the import and ending 
up with a half-way result. Here's some of the things I would have liked, in my 
experience trying out the GitHubIssueImporter tool.

I wanted to provide a username mapping, to keep as much of the information as 
possible, but it wasn't possible to know which mappings were even needed before 
starting? I modified the script to raise an exception in _GetIssueAssignee when 
a mapping is not found, instead of returning the --github_owner_username, 
adding entries to the mapping files as errors were found.

Issues can only be assigned to users that have commit access to the repository. 
If there was a dry-run mode, it would be nice if it could check whether that's 
the case, or at least, list the users that will be needed, so it can be 
verified that they have commit access ahead of time.

What steps will reproduce the problem?

Export https://code.google.com/p/streamflyer/ to github.



What is the expected output? What do you see instead?

The following error text appears:

    Aw, Snap!
    There was an error migrating the project to GitHub.
    The error is as follows:
        Error creating repo:
        PUT https://api.github.com/repos/rwitzel/streamflyer.downloads/collaborators/GoogleCodeExporter: 404 Not Found []



What version of the product are you using? On what operating system?

Did this export today (2015-03-16) with the new "Export to GitHub" button. 
Repeated it twice in a row and got the same the result.

What steps will reproduce the problem?
1. Try to export the project proyecto-val to GitHub

What is the expected output? 
The project migrated to Github

What do you see instead?
I got an e-mail with the folowing error:

Aw, Snap!
There was an error migrating proyecto-val to GitHub.
The error is as follows:
Project: proyecto-val
GitHub code import failure: Error! There was an error importing commits.


The project is using 13.9 MB of the quota and I checked and it doesn't have any 
100MB+ file, So I don't know what is causing this.

I plan to migrate a project from GoogleCode to Bitbucket. The main reason not 
choosing GitHub was it's issue handler doesn't support file attachments(just 
pictures). But Bitbucket does! The GoogleCodeProjectHosting.json file created 
by Takeout contains all necessary info to concatenate an attachment url ,so it 
would be nice to have a command line option to migrate file attachments of 
issue comments too.

Currently you cannot export projects under /a/eclipselabs.org. This feature is 
in-progress. This bug is for tracking purposes.

What steps will reproduce the problem?
1. Use the 'Export to GitHub' tool

Old wiki at:
https://code.google.com/p/skype4pidgin/w/list

Not showing at:
https://github.com/EionRobb/skype4pidgin/wiki

What steps will reproduce the problem?
Exporting my projects to github with the github exporter have all failed and 
created redirect loops trying to access my repo when I'm logged in. If I'm 
logged out, I can access the repo.

I see this output in Chrome when surfing to the repo:

The webpage at https://import.github.com/auth/github has resulted in too many 
redirects. Clearing your cookies for this site or allowing third-party cookies 
may fix the problem. If not, it is possibly a server configuration issue and 
not a problem with your computer.

This is on Linux, using Chrome 41

What steps will reproduce the problem?
1.trying to export jbrout project to my GitHub account
2.
3.

What is the expected output? What do you see instead?
expecting: something nice (not sure exactly), export should finish without 
problems
observing:

Aw, Snap!

There was an error migrating the project to GitHub.

The error is as follows:

Error getting GitHub user.

You can try to export the project again, read our FAQ for any known issues, or 
contact Google for assistance.


What version of the product are you using? On what operating system?
NA

Please provide any additional information below.

What steps will reproduce the problem?
1. Run the https://code.google.com/export-to-github/ export

What is the expected output? What do you see instead?
Expect to see issues copied, nothing has happened

Old issues list:
https://code.google.com/p/skype4pidgin/issues/list

Nothing at:
https://github.com/EionRobb/skype4pidgin/issues

Is it possible to export the SVN repository (not a checkout of the repo, but 
the actual repo) so that it can be hosted elsewhere? 

What steps will reproduce the problem?
1. Try to export to GitHub
2. Wait for failure email

What is the expected output? What do you see instead?
Not sure, I haven't managed to make the exporter run correctly.

What version of the product are you using? On what operating system?
Yours? OSX? 

Please provide any additional information below.
Tried to use the standard export option, failed to migrate issues. The repo is 
mlode. 

Branch name: development-oud

Purpose of code changes on this branch (generate_user_map.py):
1. Added a tool to auto-generate a file users.json, which maps Google Code user 
to BitBucket user.

Example:

{
    "users": {
        "[email protected]": "oud-google",
        "oudavid86": "oudavid86"
    }
}

Please note that the user must change the right-side user, otherwise it will 
just default to the Google Code username.

2. Added a tool to convert issues data from Takeout into a format accepted by 
BitBucket (bitbucket_issue_converter.py).

I extracted the issues into a class GoogleCodeIssue and comments into 
GoogleCodeComment. These two classes can/should be shared with the GitHub issue 
importer. There are also two corresponding classes BitBucketIssue and 
BitBucketComment. Below is an example of the JSON that is generated from 
bitbucket_issue_converter.py.

Example:

{
    "comments": [
        {
            "content": "As the input size increases exponentially, we need to make sure that the algorithm isn't unnecessarily computing sums it doesn't need to.",
            "created_on": "2014-07-01T21:37:12.000Z",
            "id": 0,
            "issue": 1,
            "updated_on": "2014-07-01T21:37:12.000Z",
            "user": "oudavid86"
        },
        {
            "content": "Assigning issue to oudavid86.",
            "created_on": "2014-07-01T21:40:05.000Z",
            "id": 1,
            "issue": 1,
            "updated_on": "2014-07-01T21:40:05.000Z",
            "user": "oudavid86"
        }
    ],
    "issues": [
        {
            "assignee": "oudavid86",
            "content": "Need unit test to check that run-time complexity does not exceed O(N^2)",
            "content_updated_on": "2014-07-01T21:40:05.000Z",
            "created_on": "2014-07-01T21:37:12.000Z",
            "id": 1,
            "kind": "bug",
            "priority": "critical",
            "reporter": "oud-google",
            "status": "open",
            "title": "Need unit test to check that run-time complexity does not exceed O(N^2)",
            "updated_on": "2014-07-01T21:37:12.000Z"
        }
    ],
    "meta": {
        "default_kind": "task"
    }
}

A user can take this json, zip it up with attachments (if any), and import them 
into BitBucket here: 
https://bitbucket.org/<USER>/<PROJECT>/admin/issues/import-export

After the review, I'll merge this branch into:
/trunk

GitHub seems to reject empty comments, which causes some errors to appear 
during the importation.

See: https://code.google.com/r/kgschwebke-plv8js/adminAdvanced

Expected: You can set the project moved field.
Observed: It is not present.

There seems to be no possibility to map the SVN user names in commits? I added 
my gmail-Adress to the Github settings, but on every commit, it still shows: 
"Invalid author email".

Docs on 
https://help.github.com/articles/why-are-my-commits-linked-to-the-wrong-user/ 
say: This means the email address in your local Git configuration settings is 
either blank or not formatted as an email address. You will need to change the 
email address in your local Git configuration settings, then add the new 
address to your GitHub email settings to link your future commits. Old commits 
will not be linked.

This is not good, because by that I loose my committer credits on OpenHUB / and 
others.

What steps will reproduce the problem?
1. go to https://code.google.com/a/eclipselabs.org/p/gps-tracker-tau/
2. click Export to GitHub
3. Click Start Export button.

What is the expected output? What do you see instead?
I'm expecting successful migration to GitHub repository. 
Instead I see “Error getting project "gps-tracker-tau".
I've also tried to import "eclipselabs.org/p/gps-tracker-tau/" without success.

What version of the product are you using? On what operating system?
https://code.google.com/export-to-github/request-export/gps-tracker-tau?
Windows 8.1, Chrome 43.0

Please provide any additional information below.

python3 googlecode-issues-exporter/github_issue_converter.py <a lot of options>
  File "googlecode-issues-exporter/github_issue_converter.py", line 423
    print "\nDone!\n"
                    ^
SyntaxError: Missing parentheses in call to 'print'

---

Reported on google-code-shutdown@

What steps will reproduce the problem?
1. Export https://code.google.com/p/opencaching-api/ to github, wait for 
"export complete" email
2. Compare the original and exported contents of some issues, e.g. issue 70:
https://code.google.com/p/opencaching-api/issues/detail?id=70
https://github.com/wrygiel/opencaching-api/issues/70

What is the expected output? What do you see instead?

1. The original issue currently has 183 comments. The exported version is 
truncated after 102 comments.
2. The original issue was closed (fixed) in comment 153. The exported version 
is marked as closed in comment 5.

What version of the product are you using? On what operating system?
Did this export today (2015-03-14) with the new "Export to GitHub" button. 
Repeated it twice in a row and got the same the result.

What steps will reproduce the problem?
1.Go to https://code.google.com/p/utooth/ project home
2.Click Export to github


What is the expected output? What do you see instead?
Project should be exported to github. I see internal server error

The tool did silent about the license of my projects was exported. It is wrong. 
If it do not must do the commit of a license file,  at least the exporter must 
comment about that at GitHub information about licensing should gain a 
different visibility.

Looks like eclipselabs projects 
(https://code.google.com/a/eclipselabs.org/hosting/) can't be exported - I only 
get an error "Error getting project <name>" after trying to use "Export to 
Github->Start Export" button.

Example project: https://code.google.com/a/eclipselabs.org/p/plugindependencies/

What steps will reproduce the problem?
1. run wiki2gfm.py with wiki input from 
https://code.google.com/p/fusioncatcher/wiki/Manual 

What is the expected output? What do you see instead?

It is expected to end without error. The following error is shown:


Warning (line 1 of input file):
A summary pragma was used for this wiki:
    Manual
Consider moving it to an introductory paragraph.

Warning (line 3 of input file):
A table of contents plugin was used for this wiki:
    <wiki:toc max_depth="2" />
The Gollum wiki system supports table of content generation.
See https://github.com/gollum/gollum/wiki for more information.
It has been removed.

Traceback (most recent call last):
  File "/apps/bin/support-tools/wiki_to_md/wiki2gfm.py", line 122, in <module>
    main(sys.argv)
  File "/apps/bin/support-tools/wiki_to_md/wiki2gfm.py", line 118, in main
    converter.Convert(input_stream, output_stream)
  File "/apps/bin/support-tools/wiki_to_md/impl/converter.py", line 123, in Convert
    input_line = self._ProcessBody(input_line, input_lines, output_stream)
  File "/apps/bin/support-tools/wiki_to_md/impl/converter.py", line 215, in _ProcessBody
    output_stream)
  File "/apps/bin/support-tools/wiki_to_md/impl/converter.py", line 346, in _ProcessLine
    output_stream)
  File "/apps/bin/support-tools/wiki_to_md/impl/converter.py", line 513, in _ProcessMatch
    handler(input_line, match, output_stream)
  File "/apps/bin/support-tools/wiki_to_md/impl/converter.py", line 604, in _HandleBold
    self._HandleTag(input_line, "Bold", output_stream)
  File "/apps/bin/support-tools/wiki_to_md/impl/converter.py", line 1281, in _HandleTag
    self._CloseTag(input_line, tag, output_stream)
  File "/apps/bin/support-tools/wiki_to_md/impl/converter.py", line 433, in _CloseTag
    handler(input_line, output_stream)
  File "/apps/bin/support-tools/wiki_to_md/impl/formatting_handler.py", line 297, in HandleBoldClose
    self._HandleFormatClose(input_line, output_stream, "Bold")
  File "/apps/bin/support-tools/wiki_to_md/impl/formatting_handler.py", line 960, in _HandleFormatClose
    self._Write("{0}{1}{0}".format(tag, format_buffer), output_stream)
UnicodeEncodeError: 'ascii' codec can't encode character u'\u2013' in position 
54: ordinal not in range(128)

Branch name: development-oud

Purpose of code changes on this branch:
Applied gpylint style changes.

Please see in-line comments.

After the review, I'll merge this branch into:
/trunk

This is probably more of a GitHub issue, but I don't know where to report it to 
them and I see that you forward stuff here to them where suitable, so I'll 
report it here.

I noticed that once you start the export to GitHub, you can, although not 
advertised anywhere, go to the GitHub import tool and specify the username 
mapping to GitHub users.

Steps:
1. Say you export my project "altdrag" using the export tool.
2. Then once the export has begun, navigate to 
https://import.github.com/stefansundin/altdrag/import (replace with your 
username)
3. You may have to refresh if the import hasn't begun yet. Keep refreshing 
until you can see it doing the import.
4. Usernames show up below and you can map them to GitHub users.

Unfortunately, once the import has fetched all the commits, you can briefly see 
the status "Updating authors", and after which, any changes to the mapping 
doesn't have any effect. It would be nice to have a few extra minutes to update 
this information correctly. Doing it afterwards is painful and involves git 
filter-branch commands.

In addition, the mapping form should probably be disabled when changes stop 
having effect.

It would be awesome if you guys could forward this to the appropriate person. 
Thank you!

What steps will reproduce the problem?
1. Migrated project d7ux from google code to github.
2. https://code.google.com/p/d7ux/source/list?num=25&start=230 shows it has 230 
commits
3. https://github.com/goba/d7ux/commits/master shows the migrated one has 8 
commits

What is the expected output? What do you see instead?

230 commits migrated.

What version of the product are you using? On what operating system?

Used the github migration tool provided.

Please provide any additional information below.

Attempted to fix the authors but a local checkout of the repo does not reveal 
more commits either.