Get https://coveralls.io/ working on this repo.

Publish version 0.1 to Maven Central.

This project works in the domain of rewriting Subversion history. It fills in the tooling gap that exists in this domain, especially around the problems of upgrading Subversion 1.6 repositories to be used with newer Subversion versions, removing large binaries from Subversion history, and getting a Subversion repository in a proper form to be converted to Git.

Part of the documentation should some sort of graphical representation of how we modelled the domain, and the process of how an SVN dump file is read. We can use these documents to bootstrap a common language that people can use when speaking about this project. The chart should be easy to update over time, as needed.

Reading large files that are part of an SvnNode takes a disproportionately long time. We currently read the file content of an SvnNode with the readByteArray() method. This is slow for some reason, and I don't know why.

Need to fix this, so that we don't take a very long time reading streams.

svn:mergeinfo is a SVN property that can be present on any SvnNode, but usually on root directories.

A well known problem happens when using Subversion 1.7+ with repositories that were created with Subversion 1.6. An svn merge will get the following error:

svn: E200020: Invalid revision number '0' found in range list

The little internet chatter I saw about this recommends changing the svn:mergeinfo to replace any mention of revision 0 with revision 1.

Write an SvnDumpMutator that will replace svn:mergeinfo mentions of a specific revision with another.

TerminatingValidatorTest doesn't check that the TerminatingValidator doesn't terminate when there is no error.

Make sure things still work fine when the --deltas parameter is passed to svnadmin.

NodeAdd has a few problems:

• It never calls endNode(SvnNode) for the newly added node.
• It never does anything with file content chunks (call consume(FileContentChunk) on each one, and also endChunks() after all of them).

PR #9 tried to fix loading of large files into memory. It broke up the parsing of file contents into chunks.

That's fine, however FastCharStream has its own buffer that tries to store the file content in memory. Therefore, parsing large files still doesn't work.

Fix FastCharStream and SvnDumpFileParser to not buffer file content in the FastCharStream buffer. This will require adding some shortcutting methods to FastCharStream.

Right now, SvnPropertyChange will accept a name matcher and a value transform function. The transform function will transform the property value if the name of the property matches the criteria.

In order to match on another property name, you have to create a new SvnPropertyChange consumer. The problem with this is that the SvnPropertyChange consumer rebuilds every properties map in the SvnDump object.

To do that once each time you add a new SvnPropertyChange seems very inefficient. We can just give SvnPropertyChange a list of (name matcher, value transformer) pairs and it will only need to rebuild the properties map once.

SVN allows users to commit extremely large files to a repository. Currently svndumpgui cannot support parsing an SVN dump that contains large files (>12gb) or a single revision of many smaller files that add up to a large size.

The problem is that when we read the file contents for SvnNodes, we read them into memory to create the SvnRevision object, and then pass the SvnRevision on to the SvnDumpConsumers. This is a problem because to load large files into RAM we need to increase the max heap size unnecessarily high. For sufficiently large files, we won't have enough RAM to store the files in memory.

Even if you have a ton of ram, the parser defined in svndump.jj expects a single file's content length to be less than Integer.MAX_SIZE. This won't be the case with files larger than 2^31 - 1 in size.

Reading of file contents should be handled specially, separate from the machinery of JavaCC. File contents should be read in chunks, and the chunks passed on to the SvnConsumer. It's very rare that an SvnConsumers will need to operate on the actual file content. The SvnConsumers should know how to handle file content chunks appropriately for whatever they're doing.

For example, in the case of an SvnDumpWriterImpl, the file chunks should be written directly back out to the output stream. For an SvnDumpSummary writer, the file content chunks can be ignored.

NodeRemove should fail earlier, so that svnadmin load does not commit the transaction.

PathCollisionValidator will choke quickly on a large repository, because revisionSnapshots stores a map with every full path in the repository for each revision. With many tags this becomes quite a lot quite fast, and Java runs out of memory :)

I solved that same issue (I'm doing something very similar to help me refactor a giant repository, but for work so I can't share code ;) ) by instead using a tree structure, with each node storing it's 'file' name and it's revision. The big trick is that for each new revision only the root node is replaced, the sub nodes are the nodes from the previous repository. On each add/delete/replace/modify the nodes in the path are also replaced, but no others. When adding a copy, again only the top node is new, and links to the nodes from the copied revision.
This creates a structure which holds every revision completely, in a easily traverse able format, so you can perform the checks, but which is far more memory use friendly.

Hope this makes sense, hope this helps ;)

SvnDumpInMemory is currently not covered by any tests. Write tests around SvnDumpInMemory.

Based on the Java documentation, PrintStream never throws an IOException. This means that if there is a problem writing to the output stream, the SvnDumpWriter will never know, and won't terminate early.

This is a problem when piping input to CliConsumer and there is an error in the output stream (for example svnadmin load encounters unknown input), SvnDumpWriter will continue "writing" to the bad stream as if nothing ever happened.

We should check for the exception and throw it anew, or use another Stream that doesn't swallow exceptions.

When clearing a revision, ClearRevision will filter out the consume(SvnNode) and consume(FileContentChunk) calls and not call them on the next consumer. However, there's no logic to keep endChunks() and endNode(SvnNode) from being called on the next consumer.

This will cause a follow up consumer to receive endChunks() and endNode(SvnNode) without the matching consume(...) calls, and will lead to undesired behavior.

PathChange only updates the first line of an svn:mergeinfo property. It should update every line, if there's a match.

Currently, to create a consumer chain that operates on an SVN Dump stream you must write Java code, compile it into your JAR and then run it on your SVN dump file. It is a tedious process.

To make this easier, we can create a DSL that describes a consumer chain. The file that describes the consumer chain can be read at runtime and the consumer chain be generated dynamically, requiring no recompilation.

This will make running the program from the CLI much easier.

The DSL should be implemented in JavaCC.

Allow file content to be replaced. New file content should be provided from an InputStream, so that it can come from anywhere.

SvnDumpImplWriter adds extra new lines in some places, and misses newlines in other places. This will be easier to describe with a failing test, so will submit failing test to the branch, and then make it pass.

Ya, so that's bad. Make SvnPropertyChange operate on revision properties too!

As mentioned in this mergeinfo internals article, the merge info ranges can be marked non-inheritable which is encoded as a star * after the merge info range.

For example:

/trunk/file:1234*,1255*,1265*,1266*,1267-1357*,1359*,2001*

Notice the *s. Make MergeInfoParser be able to parse these ranges, and modify the merge info api to deal with these.

Add more tests around NodeHeaderChange and suss out any hidden bugs.

If you try to rebuild a SVN 1.6 repo with an svnadmin command from a newer version of SVN (in my case 1.8.8), you may encounter the following:

svnadmin: E125005: Invalid property value found in dumpstream; consider repairing the source or using --bypass-prop-validation while loading.
svnadmin: E125005: Cannot accept non-LF line endings in 'svn:log' property

So it seems like in Subversion 1.6 you could have a svn:log property that didn't end in a line feed character, but in Subversion 1.8 that's no longer allowed.

It seems like a trivial exercise to make a property mutator that will guarantee that all svn:log properties end with a line feed character. Maybe do this?

So, the current implementation of SvnDumpFileCharStream has done away with proper reading of characters. It just reads bytes one at a time, and then converts the byte to a char via casting.

My assumption is that the casting breaks encoding. Make UTF-8 characters in log messages parse properly.

endNode(SvnNode) should not be chained if the node is removed