Sierra Toolkit Version of the Day
Sierra Toolkit Rebalance Use Cases
Collaboration diagram for Sierra Toolkit Rebalance Use Cases:

## Use Case 1: Unequal element weights

This use case demonstrates unequal element weights. A single-element-thick column of hex elements is constructed in the x-direction. Weights are assigned to elements such that a perfect rebalance is possible with Proc_Rank+1 elements placed on each processor. This is achieved when running on #procs < 4, but Zoltan does not produce the optimal new distribution for 4 or more procs.

The following hex8 mesh is used in this use case:

Global node and element numbering

```              3       7      11      15      19
+-------+-------+-------+-------+                +-------+
/       /       /       /       /|               /       /|
4/      8/     12/     16/     20/ |              /       / |    Z  Y
+-------+-------+-------+-------+  |     ......  +-------+  |    | /
|       |       |       |       |  +18           |       |  /    |/
|  e1   |  e2   |  e3   |  e4   | /              |  eN   | /     *--X
|       |       |       |       |/               |       |/
+-------+-------+-------+-------+                +-------+
1       5      9       13      17
```

where N = #elements = (#procs)(#procs+1)/2

Local node numbering

```              8       7
+-------+
/       /|
5/      6/ |
+-------+  |
|       |  +3
|  e1   | /
|       |/
+-------+
1       2
```

The mesh is constructed on proc 0 using the HexFixture class and unequal element weights are assigned as follows:

```  // Assign weights so that a perfect rebalance is possible so long as the rebalancer can figure out
// to put p_rank+1 elements on proc = p_rank based on these weights.
unsigned nslabs = 0;
if( 0 == p_rank ) {
for ( unsigned l = 1 ; l <= p_size ; ++l ) {
for ( unsigned k = 0 ; k < nz ; ++k ) {
for ( unsigned j = 0 ; j < ny ; ++j ) {
for ( unsigned i = 0 ; i < l ; ++i ) {
const stk::mesh::EntityId elem_id = 1 + nslabs + i + j*ngx + k*ngx*ny;
stk::mesh::Entity * elem = bulk.get_entity(element_rank, elem_id);
double * const e_weight = stk::mesh::field_data( weight_field , *elem );
*e_weight = double(ngx) / double(l);
}
}
}
nslabs += l;
}
}
// end assign weights
```

See UseCase_Rebal_1.cpp for the complete source listing.

## Use Case 2: Node, Edge, Face and Element weights on subset

This use case demonstrates element weights comprised of contributions from nodes, edges, faces and elements over a subset of the mesh. A 3x3x3 cube of hex8 elements is constructed on proc 0 and weights are assigned to a subset of mesh entities. The mesh and weights are assigned as follows:

Global node and element numbering

```                   +-------+-------+-------+               +                   +                 +
/       /       /       /|              /|                  /|
/       /       /       / |             / |                 / |
+-------+-------+-------+  |            +  |                +  |
/       /       /       /|  +           /|  +               /   +
/       /       /       / | /|          / | /|              /    |
+-------+-------+-------+  |/ |         +  |/ |             +     |
/       /       /       /|  +  |        /|  +  |            /      |
/       /       /       / | /|  +       / | /|  +           /       +
+-------+-------+-------+  |/ | /|      +  |/ | /|          +        |        +
|       |       |       |  +  |/ |      |  +  |/ |          |        |
|  e1   |  e2   |  e3   | /|  +  |      | /|  +  |          |        |
|       |       |       |/ | /|  +      |/ | /|  +          |        +                 +
+-------+-------+-------+  |/ | /       +  |/ | /           +       /
|       |       |       |  +  |/        |  +  |/            |      /
|  e1   |  e2   |  e3   | /|  +         | /|  +             |     +
|       |       |       |/ | /          |/ | /              |    /
+-------+-------+-------+  |/           +  |/               +   /
|       |       |       |  +            |  +                |  +
|  e1   |  e2   |  e3   | /             | /                 | /
|       |       |       |/              |/                  |/
+-------+-------+-------+               +                   +                 +
x = 0
```
```      Weight_elems = 1.0                 Z  Y      Local node numbering
Weight_faces = 10.0                | /
Weight_edges = 100.0               |/            8       7
Weight_nodes = 1000.0              *--X          +-------+
/       /|
5/      6/ |
+-------+  |
|       |  +3
|  e1   | /
|       |/
+-------+
1       2
```

where all 27 elements are assigned weights along with the 9 faces, 12 edges and 4 nodes on the plane at x = 0.

```  bulk.modification_begin();

// Assign entity weights
if( 0 == p_rank )
{
// Get the faces on the x=0 plane and give them a characteristic weight
stk::mesh::EntityVector selected_nodes;
stk::mesh::EntityVector selected_faces;
stk::mesh::EntityVector one_face;
for ( unsigned j = 0 ; j < ny; ++j )
for ( unsigned k = 0 ; k < nz; ++k )
{
selected_nodes.clear();
selected_nodes.push_back( fixture.node(0, j,   k  ) );
selected_nodes.push_back( fixture.node(0, j+1, k  ) );
selected_nodes.push_back( fixture.node(0, j,   k+1) );
selected_nodes.push_back( fixture.node(0, j+1, k+1) );
stk::mesh::get_entities_through_relations(selected_nodes, face_rank, one_face);
selected_faces.push_back(one_face[0]);
}

for( size_t iface = 0; iface < selected_faces.size(); ++iface )
{
stk::mesh::Entity * face = selected_faces[iface];
double * const weight = stk::mesh::field_data( weight_field, *face );
weight[0] = 10.0;
}

// Get the edges on the boundary of the x=0 plane and give them a characteristic weight
stk::mesh::EntityVector selected_edges;
stk::mesh::EntityVector one_edge;
for ( unsigned j = 0 ; j < ny; ++j )
{
selected_nodes.clear();
selected_nodes.push_back( fixture.node(0, j,   0) );
selected_nodes.push_back( fixture.node(0, j+1, 0) );
stk::mesh::get_entities_through_relations(selected_nodes, edge_rank, one_edge);
selected_edges.push_back(one_edge[0]);
selected_nodes.clear();
selected_nodes.push_back( fixture.node(0, j,   nz) );
selected_nodes.push_back( fixture.node(0, j+1, nz) );
stk::mesh::get_entities_through_relations(selected_nodes, edge_rank, one_edge);
selected_edges.push_back(one_edge[0]);
}
for ( unsigned k = 0 ; k < nz; ++k )
{
selected_nodes.clear();
selected_nodes.push_back( fixture.node(0, 0, k) );
selected_nodes.push_back( fixture.node(0, 0, k+1) );
stk::mesh::get_entities_through_relations(selected_nodes, edge_rank, one_edge);
selected_edges.push_back(one_edge[0]);
selected_nodes.clear();
selected_nodes.push_back( fixture.node(0, ny, k) );
selected_nodes.push_back( fixture.node(0, ny, k+1) );
stk::mesh::get_entities_through_relations(selected_nodes, edge_rank, one_edge);
selected_edges.push_back(one_edge[0]);
}
for( size_t iedge = 0; iedge < selected_edges.size(); ++iedge )
{
stk::mesh::Entity * edge = selected_edges[iedge];
double * const weight = stk::mesh::field_data( weight_field, *edge );
weight[0] = 100.0;
}

// Finally, give the corner nodes of the x=0 plane a characteristic weight
selected_nodes.clear();
double * weight = stk::mesh::field_data( weight_field, *fixture.node(0, 0, 0) );
weight[0] = 1000.0;
weight = stk::mesh::field_data( weight_field, *fixture.node(0, ny, 0) );
weight[0] = 1000.0;
weight = stk::mesh::field_data( weight_field, *fixture.node(0, 0, nz) );
weight[0] = 1000.0;
weight = stk::mesh::field_data( weight_field, *fixture.node(0, ny, nz) );
weight[0] = 1000.0;

// Assign element weights
for( size_t i = 0; i < my_element_ids.size(); ++i )
{
stk::mesh::Entity * elem = bulk.get_entity(element_rank, my_element_ids[i]);
double * const e_weight = stk::mesh::field_data( weight_field , *elem );
*e_weight = 1.0;
}
//
// Get the elements on the x=0 plane and sum in weights from relations
selected_nodes.clear();
for ( unsigned j = 0 ; j < ny+1; ++j )
for ( unsigned k = 0 ; k < nz+1; ++k )
selected_nodes.push_back( fixture.node(0, j, k) );

std::vector<stk::mesh::EntityRank> ranks;
ranks.push_back(face_rank);
ranks.push_back(edge_rank);
ranks.push_back(node_rank);
stk::mesh::EntityVector selected_elems;
for ( unsigned j = 0 ; j < ny; ++j )
for ( unsigned k = 0 ; k < nz; ++k )
{
selected_nodes.clear();
selected_nodes.push_back( fixture.node(0, j,   k  ) );
selected_nodes.push_back( fixture.node(0, j+1, k  ) );
selected_nodes.push_back( fixture.node(0, j,   k+1) );
selected_nodes.push_back( fixture.node(0, j+1, k+1) );
stk::mesh::get_entities_through_relations(selected_nodes, element_rank, one_face);
selected_elems.push_back(one_face[0]);
}
sum_element_weights_through_relations(selected_elems, weight_field, ranks);
}

bulk.modification_end();
```

The use case passes if the amount of imbalance following a rebalance is below 1.45 for 3 procs and below 1.1 for 2 or 4 procs.

See UseCase_Rebal_2.cpp for the complete source listing.

## Use Case 3: Periodic Boundary via Constraint Relations

This use case sets up a 2D mesh of quad4 elements and then establishes a constraint realtion between the top and bottom of the mesh as would be needed to enforce periodic boundary conditions.

The following quad4 mesh is manually constructed on proc 0:

Global node and element numbering

```           21      22      23      24      25
+-------+-------+-------+-------+   y = top
|       |       |       |       |
|  e13  |  e14  |  e15  |  e16  |
|       |       |       |5      |
16 +-------+-------+-------+-------+ 20
|       |       |       |       |
|  e9   |  e10  |  e11  |  e12  |
|       |       |       |       |
11 +-------+-------+-------+-------+ 15
|       |       |       |       |
|  e5   |  e6   |  e7   |  e8   |
|       |5      |       |       |
6 +-------+-------+-------+-------+ 10
|       |       |       |       |
|  e1   |  e2   |  e3   |  e4   |
|       |       |       |       |
+-------+-------+-------+-------+   y = bottom
1       2       3       4       5
```

Local node numbering:

```           3       4
+-------+     Y
|       |     |
|  e1   |     |
|       |     *--> X
+-------+
1       2
```

and the two sets of nodes at y=bottom and y=top are related through constraint relations as follows:

```    // Assign constraint relations between nodes at top and bottom of mesh
{
const unsigned iy_bottom  =  0;
const unsigned iy_top = ny;
for ( unsigned ix = 0 ; ix <= nx ; ++ix ) {
stk::mesh::EntityId nid_bottom  = 1 + ix + iy_bottom  * nnx ;
stk::mesh::EntityId nid_top = 1 + ix + iy_top * nnx ;
stk::mesh::Entity * n_bottom  = bulk_data.get_entity( node_rank, nid_bottom  );
stk::mesh::Entity * n_top = bulk_data.get_entity( node_rank, nid_top );
const stk::mesh::EntityId constraint_entity_id =  1 + ix + nny * nnx;
stk::mesh::Entity & c = bulk_data.declare_entity( constraint_rank, constraint_entity_id, add );
bulk_data.declare_relation( c , *n_bottom  , 0 );
bulk_data.declare_relation( c , *n_top , 1 );
}
} // end snippet
```

The use case passes if the load imbalance of the new partition is below the nominal value of 1.5.

See UseCase_Rebal_3.cpp for the complete source listing.

## Use Case 4: User-customization following default rebalance

This use case demonstrates additional user customization following a default rebalance in order to enforce constraints for new partitions. In this case, the constraint is that two quad4 elements sharing edge #7 be collocated on the same proc following rebalance. This is enforced using a greedy sideset class which inherits the determine_new_partition method.

The following quad4 mesh is used in this use case:

Global node and element numbering

```        13      14      15      16
+-------+-------+-------+
|       |       |       |
|  e7   |  e8   |  e9   |
|       |       |       |
9 +-------+-------+-------+ 12    Y
|       |       |       |       |
|  e4   |  e5   |  e6   |       |
|       |       |       |       *--> X
5 +-------+-------+-------+ 8
|       |       |       |
|  e1   |  e2   |  e3   |
|       |       |       |
+-------+-------+-------+
1       2       3       4
```

Local node numbering

```              3       4
+-------+
|       |
|  e1   |
|       |
+-------+
1       2
```

See UseCase_Rebal_4.cpp for the complete source listing.