LB boundary slice and node setters are broken about espresso HOT 3 CLOSED

Update: the flag field is actually properly communicated, but we never wrote a communicator for the bounce-back velocity map, and the slice update function doesn't automatically recompute the IndexInfo vectors when only the ghost layer of the flag field is modified.

from espresso.

The node setters have the exact same bug. This is an issue for circular Couette flow, which are set up by node setters.

from espresso.

Here is my understanding so far: we must always run a full communication after each call to a node setter or slice setter. This is already the case for the velocity, force, flag and population fields, but not for the UBB map. We cannot introduce a flag to track pending changes introduced by setter functions and trigger a ghost communication whenever a getter function is called, because this would break the constness of the getters, and would also require a mechanism to synchronize the state of the flag between all MPI ranks, which is tedious since we don't include MPI header files. In addition, there are valid use cases for getting the state of the ghost layer before the ghost update, for example during particle coupling where the particle force is interpolated on the fluid while the fluid force is interpolated on the particle.

For completeness, here is how the pending status flag would have worked:

class LBWalberlaImpl {
  /** Flags for pending ghost layer communication. */
  enum PendingGhostLayerCommunicationFlag : unsigned int {
    /** No pending field update. */
    PENDING_COMM_NONE = 0u,
    /** Pending changes to pdf/velocity/force fields. */
    PENDING_COMM_GHOST = 1u,
    /** Pending changes to boundary fields. */
    PENDING_COMM_BOUNDARY = 2u,
  };

  unsigned int m_pending_changes = PENDING_COMM_NONE;

public:
  void ghost_communication() override {
    if (m_pending_changes & PENDING_COMM_BOUNDARY) {
      reallocate_ubb_field();
      ghost_communication_boundary();
      ghost_communication_pdfs();
    } else if (m_pending_changes & PENDING_COMM_GHOST) {
      ghost_communication_pdfs();
    }
  }

  void ghost_communication_boundary() {
    m_boundary_communicator->communicate();
    m_pending_changes &= ~PENDING_COMM_BOUNDARY;
  }

  void ghost_communication_pdfs() {
    m_pdfs_communicator->communicate();
    m_pending_changes &= ~PENDING_COMM_GHOST;
  }

  bool set_node_velocity(Utils::Vector3i const &node, Utils::Vector3d const &vel) {
    m_pending_changes |= PENDING_COMM_GHOST;
    auto bc = get_block_and_cell(get_lattice(), node, false);
    if (bc) {
      /* ... */
    }
    return bc.has_value();
  }
};

from espresso.