It just came to me that I could have done this in a different and much easier manner, but I am not sure how efficient it would be compared to current approach. There would be a question as to whether partitioning the graph w.r.t nodes vs elements (and then handling ghosted nodes) will provide equally high-quality partitioning.

Approach

1. Read in Mesh from Exodus-II
2. Convert directly to nodal matrix (two nodes are adjacent if they share an element in the mesh) [Tpetra::Map::createOneToOne will give it a default meh distribution]
3. Use Zoltan2's MatrixInput adapter to partition the matrix

This results in a partitioning being done based on nodes, and not elements. Also, ghosted IDs are handled prior to partitioning the matrix. Overall, I am skeptical that this will result in as high a quality of a partitioning scheme as one that is based on elements.

Command

exec/ExodusIODecomposeTest --input=data/CreateSupportStructure1.ref.exo --output=output.exo --partitions=4

Inspect output.exo under Cubit and Paraview; missing features and in Paraview missing parts of the mesh. Need to investigate.

Leveraging Locality Information for Domain Decomposition

Locality of individual processors can be better leveraged by distributing data based on how close processors are in locality (i.e. hosted on same NUMA socket, same PE, etc.) This results in a lot more local communication at the boundary of each partition of the mesh, which will definitely speed things up a bunch.

For example, in the below mesh, it would be terrible if you had processors that owned far remote corners from each other that were on the same PE, as it would result in a lot of unnecessary communication; instead you would want the processors to cluster together in some way.

Sharing of the Mesh between Local Processors

Given that Exodus-II API reads the entire file into memory at times, and given that you can have multiple processes on a single PE, it would be greatly beneficial if you had groups of processors that shared memory between them, i.e. for the node mappings, the mappings from one Tpetra::Map to another Tpetra::Map (used when redistributing the graph when export/import is not enough), and on sharing computation with close neighbors, and of course eliding unnecessary communication. This is an extreme optimization that no-doubt will increase the maximum size of the mesh that can be processed.

Exodus-II Improvements: Streaming the Mesh

It would be useful if the Exodus-II API could possibly be streamed in a way that you can read in a portion of the mesh rather than all of it at once.