Utility code for working in SPMD mode with MPI

Functions
void	RTOp_parallel_calc_overlap (RTOp_index_type global_dim, RTOp_index_type local_sub_dim, RTOp_index_type local_offset, const RTOp_index_type first_ele, const RTOp_index_type sub_dim, const RTOp_index_type global_offset, RTOp_index_type *overlap_first_local_ele, RTOp_index_type *overalap_local_sub_dim, RTOp_index_type *overlap_global_offset)
	This function helps to implement vector method apply_op(...) for any type of parallel vector.
template<class primitive_value_type>
void	RTOpPack::MPI_type_signature (const int num_values, const int num_indexes, const int num_chars, int *num_entries, int block_lengths[], MPI_Aint displacements[], MPI_Datatype datatypes[])
	Initialize MPI compatible type signature arrays for reduction/transformation operator object instance data and reduction target object data.
template<class primitive_value_type>
int	RTOpPack::reduct_obj_ext_size (int num_values, int num_indexes, int num_chars)
	Return the size in bytes of an external representation of reduct_obj.
template<class Scalar>
void	RTOpPack::extract_reduct_obj_ext_state (const RTOpT< Scalar > &op, const ReductTarget &reduct_obj, int num_values, int num_indexes, int num_chars, void *reduct_obj_ext)
template<class Scalar>
void	RTOpPack::load_reduct_obj_ext_state (const RTOpT< Scalar > &op, const void *reduct_obj_ext, ReductTarget *reduct_obj)
template<class Scalar>
void	RTOpPack::MPI_apply_op (MPI_Comm comm, const RTOpT< Scalar > &op, const int root_rank, const int num_vecs, const SubVectorT< Scalar > sub_vecs[], const int num_targ_vecs, const MutableSubVectorT< Scalar > targ_sub_vecs[], ReductTarget *reduct_obj)
	Apply an RTOp in SMPD mode with MPI to a set of vectors with contiguous storage per processor.
template<class Scalar>
void	RTOpPack::MPI_apply_op (MPI_Comm comm, const RTOpT< Scalar > &op, const int root_rank, const int num_cols, const int num_multi_vecs, const SubMultiVectorT< Scalar > sub_multi_vecs[], const int num_targ_multi_vecs, const MutableSubMultiVectorT< Scalar > targ_sub_multi_vecs[], ReductTarget *reduct_objs[])
	Apply an RTOp in SMPD mode with MPI to a set of columns to a set of multi-vectors with contiguous storage per processor.
template<class Scalar>
void	RTOpPack::MPI_all_reduce (MPI_Comm comm, const RTOpT< Scalar > &op, const int root_rank, const int num_cols, const ReductTarget *i_reduct_objs[], ReductTarget *reduct_objs[])
	Perform global reduction of reduction target objects.
template<class Scalar>
void	RTOpPack::MPI_apply_op (MPI_Comm comm, const RTOpT< Scalar > &op, const int root_rank, const int num_cols, const int num_vecs, const SubVectorT< Scalar > sub_vecs[], const int num_targ_vecs, const MutableSubVectorT< Scalar > sub_targ_vecs[], ReductTarget *reduct_objs[])
	Apply an RTOp in SMPD mode with MPI to a set of columns to a set of multi-vectors with contiguous storage per processor.

---

Detailed Description

These functions are used by implementations of RTOp-compatible vector classes to use RTOp objects and performed global reductions correctly using MPI.

---

Function Documentation

void RTOp_parallel_calc_overlap (  RTOp_index_type  global_dim, RTOp_index_type  local_sub_dim, RTOp_index_type  local_offset, const RTOp_index_type  first_ele, const RTOp_index_type  sub_dim, const RTOp_index_type  global_offset, RTOp_index_type *  overlap_first_local_ele, RTOp_index_type *  overalap_local_sub_dim, RTOp_index_type *  overlap_global_offset )

This function helps to implement vector method apply_op(...) for any type of parallel vector. Parameters: global_dim  [in] Dimension of the original parallel vector 'v' (see above). local_sub_dim  [in] Dimension of the local subvector 'u' (see above). local_offset  [in] Gives the offset of the first element in the local sub-vector 'u' into the global vector 'v' (see above). first_ele  [in] Determines the first element in 'v' which is used to define the logical sub-vector 'g' (see above). sub_dim  [in] Determines the length of the logical sub-vector 'g' (see above). If sub_dim == 0 then sub_dim = global_dim - (first_ele-1) is used in its place. global_offset  [in] Determines the offset of the logical subvector 'g' into the logical global vector 'p' (see above). overlap_first_local_ele  [out] If *overlap_first_local_ele == 0 on output, then this means that there is no overlap of 'u' with 'g' (see above). Otherwise, there is overlap and *overlap_first_local_ele gives the first element in 'u' that overlaps with 'g' which defines 'w' (see above). overlap_local_sub_dim  [out] If *overlap_first_local_ele == 0 on output then this argument is not set and should be ignored. Otherwise, *overlap_local_sub_dim gives number of elements in 'u' that overlaps with 'g' that defines 'w' (see above). overlap_global_offset  [out] If *overlap_first_local_ele == 0 on output then this argument is not set and should be ignored. Otherwise, *overlap_global_offset gives the placement of 'w' into 'p' (see above). Preconditions: global_dim > 0 local_sub_dim > 0 0 <= local_offset <= global_dim - local_sub_dim 1 <= first_ele <= global_dim [sub_dim != 0] 0 < sub_dim <= global_dim - (first_ele-1) 0 <= global_offset Postconditions: 0 <= overlap_first_local_ele <= local_sub_dim [overlap_first_local_ele == 0] There is no overlap of 'g' with 'u' [overlap_first_local_ele != 0] 0 <= overlap_local_sub_dim <= local_sub_dim - (overlap_first_local_ele-1) To understand what this function computes first consider the what an apply_op(...) method might look like from a vector class. This method would take a list of non-mutable vectors: v_1, v_2, ..., v_p and a list of mutable vectors z_1, z_2, ..., z_q and then apply the reduction/transformation operator op over some subset of the elements in these vectors according to their placement as a set of sub-vectors in some other logical vector. Let's consider how things are treated for a single vector argument v_i or z_i which we will call 'v'. This global vector 'v' is the first vector that we identity. One must understand that there are five distict vectors (or sub-vectors) being refered to here. The first vector (call it 'v') and is one of the parallel vectors that apply_op() is called on that we have already discussed. The second vector (call it 'g') is the logical sub-vector that the client wants to represent using the elements in 'v'. This logical sub-vector is specified by the input arguments first_ele, sub_dim and global_offset. If for the time being we ignore global_offset, and then 'g' is defined in terms of 'v' as: g(k) = v((first_ele-1)+k), for k = 1...sub_dim However, for greater flexibility, the client can specify that the logical vector 'g' is really a sub-vector in a larger vector (a third vector, call it 'p') and can therefore specify where 'g' exists in 'p' using global_offset as: p(k+global_offset) = g(k) = v((first_ele-1)+k), for k = 1...sub_dim In order to apply a reduction/transformation operator over the sub-vector 'g' in 'p' each process can only work with the elements of 'v' stored in the local process. Specifically, the local elements of 'v' stored in this process (the fourth vector, all it 'u') are: u(k) = v(local_offset+k), for k = 1...local_sub_dim The tricky part of implementing this function is is determining how much of 'u' overlaps with 'g' and then getting the offset into 'p' correct. If the local elements 'u' overlaps with 'g' then this defines the fifth sub-vector (call it 'w') that defines the overlap and is specified by the return arguments as: w(k) = p(overlap_global_offset+k) = u((overlap_first_local_ele-1)+k), for k = 1...overalap_local_sub_dim Definition at line 36 of file RTOp_parallel_helpers.c.

template<class primitive_value_type> void RTOpPack::MPI_type_signature (  const int  num_values, const int  num_indexes, const int  num_chars, int *  num_entries, int  block_lengths[], MPI_Aint  displacements[], MPI_Datatype  datatypes[] )

Initialize MPI compatible type signature arrays for reduction/transformation operator object instance data and reduction target object data. Parameters: num_values  [in] Number of primitive_value members num_indexes  [in] Number of index members num_chars  [in] Number of character members num_entries  [out] Number of entries in output arrays set block_lengths  [out] array (length >= RTOp_NUM_DATA_TYPES) displacements  [out] array (length >= RTOp_NUM_DATA_TYPES) datatypes  [out] array (length >= RTOp_NUM_DATA_TYPES) See the MPI function MPI_Type_struct(...) for a description of these arrays. Definition at line 135 of file RTOpPack_MPI_apply_op.hpp.

template<class primitive_value_type> int reduct_obj_ext_size (  int  num_values, int  num_indexes, int  num_chars )

Return the size in bytes of an external representation of reduct_obj. Definition at line 68 of file RTOpPack_MPI_apply_op_decl.hpp.

template<class Scalar> void RTOpPack::extract_reduct_obj_ext_state (  const RTOpT< Scalar > &  op, const ReductTarget &  reduct_obj, int  num_values, int  num_indexes, int  num_chars, void *  reduct_obj_ext )

Definition at line 182 of file RTOpPack_MPI_apply_op.hpp.

template<class Scalar> void RTOpPack::load_reduct_obj_ext_state (  const RTOpT< Scalar > &  op, const void *  reduct_obj_ext, ReductTarget *  reduct_obj )

Definition at line 212 of file RTOpPack_MPI_apply_op.hpp.

template<class Scalar> void RTOpPack::MPI_apply_op (  MPI_Comm  comm, const RTOpT< Scalar > &  op, const int  root_rank, const int  num_vecs, const SubVectorT< Scalar >  sub_vecs[], const int  num_targ_vecs, const MutableSubVectorT< Scalar >  targ_sub_vecs[], ReductTarget *  reduct_obj )

Apply an RTOp in SMPD mode with MPI to a set of vectors with contiguous storage per processor. ToDo: Finish documentation! Definition at line 239 of file RTOpPack_MPI_apply_op.hpp.

template<class Scalar> void RTOpPack::MPI_apply_op (  MPI_Comm  comm, const RTOpT< Scalar > &  op, const int  root_rank, const int  num_cols, const int  num_multi_vecs, const SubMultiVectorT< Scalar >  sub_multi_vecs[], const int  num_targ_multi_vecs, const MutableSubMultiVectorT< Scalar >  targ_sub_multi_vecs[], RTOpPack::ReductTarget *  reduct_objs[] )

Apply an RTOp in SMPD mode with MPI to a set of columns to a set of multi-vectors with contiguous storage per processor. ToDo: Finish documentation! Definition at line 259 of file RTOpPack_MPI_apply_op.hpp.

template<class Scalar> void RTOpPack::MPI_all_reduce (  MPI_Comm  comm, const RTOpT< Scalar > &  op, const int  root_rank, const int  num_cols, const ReductTarget *  i_reduct_objs[], ReductTarget *  reduct_objs[] )

Perform global reduction of reduction target objects. Parameters: comm  [in] MPI communicator op  [in] Reduction operator object associated with reduction objects. This operator defines how the global reductions are performed. root_rank  [in] The rank of the root processor num_cols  [in] The number of vector sets (i.e. multi-vector columns) that these reduction objects where collected over. i_reduct_objs  [in] Array (length num_cols) of local intermediate reduction objects. reduct_objs  [in/out] Array (length num_cols) of global reduction objects that each i_reduct_objs[i] from each processor will be reduced into. ToDo: Finish documentation! Definition at line 301 of file RTOpPack_MPI_apply_op.hpp.

template<class Scalar> void RTOpPack::MPI_apply_op (  MPI_Comm  comm, const RTOpT< Scalar > &  op, const int  root_rank, const int  num_cols, const int  num_vecs, const SubVectorT< Scalar >  sub_vecs[], const int  num_targ_vecs, const MutableSubVectorT< Scalar >  sub_targ_vecs[], ReductTarget *  reduct_objs[] )

Apply an RTOp in SMPD mode with MPI to a set of columns to a set of multi-vectors with contiguous storage per processor. ToDo: Finish documentation! Definition at line 436 of file RTOpPack_MPI_apply_op.hpp.

---