llnl / lwgrp Goto Github PK

The light-weight group library defines data structures and collective
operations to group MPI processes as an ordered set.  Such groups are
useful as substitutes for MPI communicators when the overhead of
communicator creation is too costly.  For example, certain sorting
algorithms recursively divide processes into subgroups as the sort
algorithm progresses.  These groups may be different with each
invocation, so that it is inefficient to create and destroy
communicators during the sort routine.

Data structures:
  chain    - each member records addresses of left and right members
             first and last rank set boundary to MPI_PROC_NULL
  ring     - like the chain, except first and last rank wrap around 
  logchain - each member records addresses of each member 2^d hops
             to left and right d=0..log(N)-1 (MPI_PROC_NULL at ends)
  logring  - each member records addresses of each member 2^d hops
             to left and right with wrap at ends

We represent groups of processes using a doubly-linked list called
a "chain".  This is a very simple struct that records the number
of processes in the group, the rank of the local process within the
group, the address of the local process, and the addresses of the
processes having ranks one less (left) and one more (right) than the
local process.  We implement the LWGRP library on top of MPI, so for
addresses we record a parent communicator and ranks within that
communicator.  To be light-weight, the reference to the communicator
is a literal copy of the handle value, not a full dup.

Since each member only stores the addresses for a few other group
members, it is the responsibility of higher level software to exchange
address information for more general point-to-point communication.
Additionally, the caller is responsible for choosing tag values as a
group context.

There is also a "ring", which is like a chain, but it wraps around
at the ends.

There are two structures, called a logchain and logring, used to cache
adddresses of more group members.  These structures must be used in
conjunction with either a chain or ring, respectively.  They cache
addresses for processes that are 2^d hops to the left and right sides
where d ranges from 0 to log(N)-1 inclusive.  This pattern enables
one to construct trees, and so it's useful to cache this information
for use in collective operations.