Sierra Toolkit Version of the Day
Sierra Toolkit Rebalance Use Cases
Collaboration diagram for Sierra Toolkit Rebalance Use Cases:

Use Case 1: Unequal element weights

This use case demonstrates unequal element weights. A single-element-thick column of hex elements is constructed in the x-direction. Weights are assigned to elements such that a perfect rebalance is possible with Proc_Rank+1 elements placed on each processor. This is achieved when running on #procs < 4, but Zoltan does not produce the optimal new distribution for 4 or more procs.

The following hex8 mesh is used in this use case:

Global node and element numbering

              3       7      11      15      19                 
              +-------+-------+-------+-------+                +-------+
             /       /       /       /       /|               /       /|
           4/      8/     12/     16/     20/ |              /       / |    Z  Y
           +-------+-------+-------+-------+  |     ......  +-------+  |    | / 
           |       |       |       |       |  +18           |       |  /    |/  
           |  e1   |  e2   |  e3   |  e4   | /              |  eN   | /     *--X       
           |       |       |       |       |/               |       |/         
           +-------+-------+-------+-------+                +-------+
           1       5      9       13      17                  
         

where N = #elements = (#procs)(#procs+1)/2

Local node numbering

              8       7
              +-------+
             /       /|
           5/      6/ |
           +-------+  |
           |       |  +3
           |  e1   | /
           |       |/
           +-------+
           1       2
         

The mesh is constructed on proc 0 using the HexFixture class and unequal element weights are assigned as follows:

  // Assign weights so that a perfect rebalance is possible so long as the rebalancer can figure out
  // to put p_rank+1 elements on proc = p_rank based on these weights.
  unsigned nslabs = 0;
  if( 0 == p_rank ) {
    for ( unsigned l = 1 ; l <= p_size ; ++l ) {
      for ( unsigned k = 0 ; k < nz ; ++k ) {
        for ( unsigned j = 0 ; j < ny ; ++j ) {
          for ( unsigned i = 0 ; i < l ; ++i ) {
            const stk::mesh::EntityId elem_id = 1 + nslabs + i + j*ngx + k*ngx*ny; 
            stk::mesh::Entity * elem = bulk.get_entity(element_rank, elem_id);
            double * const e_weight = stk::mesh::field_data( weight_field , *elem );
            *e_weight = double(ngx) / double(l);
          }
        }
      }
      nslabs += l;
    }
  }
  // end assign weights

See UseCase_Rebal_1.cpp for the complete source listing.

Use Case 2: Node, Edge, Face and Element weights on subset

This use case demonstrates element weights comprised of contributions from nodes, edges, faces and elements over a subset of the mesh. A 3x3x3 cube of hex8 elements is constructed on proc 0 and weights are assigned to a subset of mesh entities. The mesh and weights are assigned as follows:

Global node and element numbering

                   +-------+-------+-------+               +                   +                 +
                  /       /       /       /|              /|                  /|         
                 /       /       /       / |             / |                 / |        
                +-------+-------+-------+  |            +  |                +  |    
               /       /       /       /|  +           /|  +               /   +   
              /       /       /       / | /|          / | /|              /    |  
             +-------+-------+-------+  |/ |         +  |/ |             +     | 
            /       /       /       /|  +  |        /|  +  |            /      |
           /       /       /       / | /|  +       / | /|  +           /       +
          +-------+-------+-------+  |/ | /|      +  |/ | /|          +        |        +
          |       |       |       |  +  |/ |      |  +  |/ |          |        |    
          |  e1   |  e2   |  e3   | /|  +  |      | /|  +  |          |        |        
          |       |       |       |/ | /|  +      |/ | /|  +          |        +                 +
          +-------+-------+-------+  |/ | /       +  |/ | /           +       /         
          |       |       |       |  +  |/        |  +  |/            |      /      
          |  e1   |  e2   |  e3   | /|  +         | /|  +             |     +      
          |       |       |       |/ | /          |/ | /              |    /      
          +-------+-------+-------+  |/           +  |/               +   /      
          |       |       |       |  +            |  +                |  +      
          |  e1   |  e2   |  e3   | /             | /                 | /      
          |       |       |       |/              |/                  |/      
          +-------+-------+-------+               +                   +                 +
        x = 0
         
      Weight_elems = 1.0                 Z  Y      Local node numbering
      Weight_faces = 10.0                | /      
      Weight_edges = 100.0               |/            8       7       
      Weight_nodes = 1000.0              *--X          +-------+              
                                                      /       /|       
                                                    5/      6/ |       
                                                    +-------+  |   
                                                    |       |  +3   
                                                    |  e1   | /
                                                    |       |/
                                                    +-------+
                                                    1       2
      

where all 27 elements are assigned weights along with the 9 faces, 12 edges and 4 nodes on the plane at x = 0.

  bulk.modification_begin();

  // Assign entity weights
  if( 0 == p_rank ) 
  {
    // Get the faces on the x=0 plane and give them a characteristic weight
    stk::mesh::EntityVector selected_nodes;
    stk::mesh::EntityVector selected_faces;
    stk::mesh::EntityVector one_face;
    for ( unsigned j = 0 ; j < ny; ++j ) 
      for ( unsigned k = 0 ; k < nz; ++k ) 
      {
        selected_nodes.clear();
        selected_nodes.push_back( fixture.node(0, j,   k  ) );
        selected_nodes.push_back( fixture.node(0, j+1, k  ) );
        selected_nodes.push_back( fixture.node(0, j,   k+1) );
        selected_nodes.push_back( fixture.node(0, j+1, k+1) );
        stk::mesh::get_entities_through_relations(selected_nodes, face_rank, one_face);
        selected_faces.push_back(one_face[0]);
      }

    for( size_t iface = 0; iface < selected_faces.size(); ++iface )
    {
      stk::mesh::Entity * face = selected_faces[iface];
      double * const weight = stk::mesh::field_data( weight_field, *face );
      weight[0] = 10.0;
    }

    // Get the edges on the boundary of the x=0 plane and give them a characteristic weight
    stk::mesh::EntityVector selected_edges;
    stk::mesh::EntityVector one_edge;
    for ( unsigned j = 0 ; j < ny; ++j ) 
    {
      selected_nodes.clear();
      selected_nodes.push_back( fixture.node(0, j,   0) );
      selected_nodes.push_back( fixture.node(0, j+1, 0) );
      stk::mesh::get_entities_through_relations(selected_nodes, edge_rank, one_edge);
      selected_edges.push_back(one_edge[0]);
      selected_nodes.clear();
      selected_nodes.push_back( fixture.node(0, j,   nz) );
      selected_nodes.push_back( fixture.node(0, j+1, nz) );
      stk::mesh::get_entities_through_relations(selected_nodes, edge_rank, one_edge);
      selected_edges.push_back(one_edge[0]);
    }
    for ( unsigned k = 0 ; k < nz; ++k ) 
    {
      selected_nodes.clear();
      selected_nodes.push_back( fixture.node(0, 0, k) );
      selected_nodes.push_back( fixture.node(0, 0, k+1) );
      stk::mesh::get_entities_through_relations(selected_nodes, edge_rank, one_edge);
      selected_edges.push_back(one_edge[0]);
      selected_nodes.clear();
      selected_nodes.push_back( fixture.node(0, ny, k) );
      selected_nodes.push_back( fixture.node(0, ny, k+1) );
      stk::mesh::get_entities_through_relations(selected_nodes, edge_rank, one_edge);
      selected_edges.push_back(one_edge[0]);
    }
    for( size_t iedge = 0; iedge < selected_edges.size(); ++iedge )
    {
      stk::mesh::Entity * edge = selected_edges[iedge];
      double * const weight = stk::mesh::field_data( weight_field, *edge );
      weight[0] = 100.0;
    }

    // Finally, give the corner nodes of the x=0 plane a characteristic weight
    selected_nodes.clear();
    double * weight = stk::mesh::field_data( weight_field, *fixture.node(0, 0, 0) );
    weight[0] = 1000.0;
    weight = stk::mesh::field_data( weight_field, *fixture.node(0, ny, 0) );
    weight[0] = 1000.0;
    weight = stk::mesh::field_data( weight_field, *fixture.node(0, 0, nz) );
    weight[0] = 1000.0;
    weight = stk::mesh::field_data( weight_field, *fixture.node(0, ny, nz) );
    weight[0] = 1000.0;

    // Assign element weights
    for( size_t i = 0; i < my_element_ids.size(); ++i )
    {
      stk::mesh::Entity * elem = bulk.get_entity(element_rank, my_element_ids[i]);
      double * const e_weight = stk::mesh::field_data( weight_field , *elem );
      *e_weight = 1.0;
    }
    //
    // Get the elements on the x=0 plane and sum in weights from relations
    selected_nodes.clear();
    for ( unsigned j = 0 ; j < ny+1; ++j )
      for ( unsigned k = 0 ; k < nz+1; ++k ) 
        selected_nodes.push_back( fixture.node(0, j, k) );

    std::vector<stk::mesh::EntityRank> ranks;
    ranks.push_back(face_rank);
    ranks.push_back(edge_rank);
    ranks.push_back(node_rank);
    stk::mesh::EntityVector selected_elems;
    for ( unsigned j = 0 ; j < ny; ++j ) 
      for ( unsigned k = 0 ; k < nz; ++k ) 
      {
        selected_nodes.clear();
        selected_nodes.push_back( fixture.node(0, j,   k  ) );
        selected_nodes.push_back( fixture.node(0, j+1, k  ) );
        selected_nodes.push_back( fixture.node(0, j,   k+1) );
        selected_nodes.push_back( fixture.node(0, j+1, k+1) );
        stk::mesh::get_entities_through_relations(selected_nodes, element_rank, one_face);
        selected_elems.push_back(one_face[0]);
      }
    sum_element_weights_through_relations(selected_elems, weight_field, ranks);
  }

  bulk.modification_end();

The use case passes if the amount of imbalance following a rebalance is below 1.45 for 3 procs and below 1.1 for 2 or 4 procs.

See UseCase_Rebal_2.cpp for the complete source listing.

Use Case 3: Periodic Boundary via Constraint Relations

This use case sets up a 2D mesh of quad4 elements and then establishes a constraint realtion between the top and bottom of the mesh as would be needed to enforce periodic boundary conditions.

The following quad4 mesh is manually constructed on proc 0:

Global node and element numbering

           21      22      23      24      25
           +-------+-------+-------+-------+   y = top
           |       |       |       |       |
           |  e13  |  e14  |  e15  |  e16  |
           |       |       |       |5      |
        16 +-------+-------+-------+-------+ 20
           |       |       |       |       |      
           |  e9   |  e10  |  e11  |  e12  |      
           |       |       |       |       |      
        11 +-------+-------+-------+-------+ 15
           |       |       |       |       |
           |  e5   |  e6   |  e7   |  e8   |
           |       |5      |       |       |  
         6 +-------+-------+-------+-------+ 10
           |       |       |       |       |  
           |  e1   |  e2   |  e3   |  e4   |    
           |       |       |       |       | 
           +-------+-------+-------+-------+   y = bottom
           1       2       3       4       5
         

Local node numbering:

           3       4
           +-------+     Y
           |       |     |
           |  e1   |     |
           |       |     *--> X
           +-------+
           1       2
         

and the two sets of nodes at y=bottom and y=top are related through constraint relations as follows:

    // Assign constraint relations between nodes at top and bottom of mesh
    {
      const unsigned iy_bottom  =  0;
      const unsigned iy_top = ny;
      stk::mesh::PartVector add(1, &fem_meta.locally_owned_part());
      for ( unsigned ix = 0 ; ix <= nx ; ++ix ) {
        stk::mesh::EntityId nid_bottom  = 1 + ix + iy_bottom  * nnx ;
        stk::mesh::EntityId nid_top = 1 + ix + iy_top * nnx ;
        stk::mesh::Entity * n_bottom  = bulk_data.get_entity( node_rank, nid_bottom  );
        stk::mesh::Entity * n_top = bulk_data.get_entity( node_rank, nid_top );
        const stk::mesh::EntityId constraint_entity_id =  1 + ix + nny * nnx;
        stk::mesh::Entity & c = bulk_data.declare_entity( constraint_rank, constraint_entity_id, add );
        bulk_data.declare_relation( c , *n_bottom  , 0 );
        bulk_data.declare_relation( c , *n_top , 1 );
      }
    } // end snippet

The use case passes if the load imbalance of the new partition is below the nominal value of 1.5.

See UseCase_Rebal_3.cpp for the complete source listing.

Use Case 4: User-customization following default rebalance

This use case demonstrates additional user customization following a default rebalance in order to enforce constraints for new partitions. In this case, the constraint is that two quad4 elements sharing edge #7 be collocated on the same proc following rebalance. This is enforced using a greedy sideset class which inherits the determine_new_partition method.

The following quad4 mesh is used in this use case:

Global node and element numbering

        13      14      15      16
        +-------+-------+-------+
        |       |       |       |
        |  e7   |  e8   |  e9   |
        |       |       |       |
      9 +-------+-------+-------+ 12    Y
        |       |       |       |       |
        |  e4   |  e5   |  e6   |       |
        |       |       |       |       *--> X
      5 +-------+-------+-------+ 8
        |       |       |       | 
        |  e1   |  e2   |  e3   |
        |       |       |       | 
        +-------+-------+-------+ 
        1       2       3       4
    

Local node numbering

              3       4
              +-------+
              |       |
              |  e1   |
              |       |
              +-------+
              1       2
    

See UseCase_Rebal_4.cpp for the complete source listing.

 All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Friends