Tpetra Matrix/Vector Services Version of the Day
Tpetra::Distributor Class Reference

Sets up and executes a communication plan for a Tpetra DistObject. More...

#include <Tpetra_Distributor.hpp>

Inheritance diagram for Tpetra::Distributor:
Inheritance graph
[legend]

List of all members.

Public Member Functions

Constructors and destructor
 Distributor (const Teuchos::RCP< const Teuchos::Comm< int > > &comm)
 Construct using the specified communicator and default parameters.
 Distributor (const Teuchos::RCP< const Teuchos::Comm< int > > &comm, const Teuchos::RCP< Teuchos::ParameterList > &plist)
 Construct using the specified communicator and ParameterList.
 Distributor (const Distributor &distributor)
 Copy constructor.
virtual ~Distributor ()
 Destructor (virtual for memory safety).
Implementation of ParameterListAcceptorDefaultBase
void setParameterList (const Teuchos::RCP< Teuchos::ParameterList > &plist)
 Set Distributor parameters.
Teuchos::RCP< const
Teuchos::ParameterList
getValidParameters () const
 List of valid Distributor parameters.
Gather / scatter "constructors"
size_t createFromSends (const ArrayView< const int > &exportNodeIDs)
 Set up Distributor using list of process ranks to which this process will send.
template<class Ordinal >
void createFromRecvs (const ArrayView< const Ordinal > &remoteIDs, const ArrayView< const int > &remoteNodeIDs, ArrayRCP< Ordinal > &exportIDs, ArrayRCP< int > &exportNodeIDs)
 Set up Distributor using list of process ranks from which to receive.
Attribute accessor methods
size_t getNumReceives () const
 The number of processes from which we will receive data.
size_t getNumSends () const
 The number of processes to which we will send data.
bool hasSelfMessage () const
 Whether the calling process will send or receive messages to itself.
size_t getMaxSendLength () const
 Maximum number of values this process will send to another single process.
size_t getTotalReceiveLength () const
 Total number of values this process will receive from other processes.
ArrayView< const int > getImagesFrom () const
 Ranks of the processes sending values to this process.
ArrayView< const int > getImagesTo () const
 Ranks of the processes to which this process will send values.
ArrayView< const size_t > getLengthsFrom () const
 Number of values this process will receive from each process.
ArrayView< const size_t > getLengthsTo () const
 Number of values this process will send to each process.
Reverse communication methods
const RCP< Distributor > & getReverse () const
 A reverse communication plan Distributor.
Methods for executing a communication plan
template<class Packet >
void doPostsAndWaits (const ArrayView< const Packet > &exports, size_t numPackets, const ArrayView< Packet > &imports)
 Execute the (forward) communication plan.
template<class Packet >
void doPostsAndWaits (const ArrayView< const Packet > &exports, const ArrayView< size_t > &numExportPacketsPerLID, const ArrayView< Packet > &imports, const ArrayView< size_t > &numImportPacketsPerLID)
 Execute the (forward) communication plan.
template<class Packet >
void doPosts (const ArrayRCP< const Packet > &exports, size_t numPackets, const ArrayRCP< Packet > &imports)
 Post the data for a forward plan, but do not execute the waits yet.
template<class Packet >
void doPosts (const ArrayRCP< const Packet > &exports, const ArrayView< size_t > &numExportPacketsPerLID, const ArrayRCP< Packet > &imports, const ArrayView< size_t > &numImportPacketsPerLID)
 Post the data for a forward plan, but do not execute the waits yet.
void doWaits ()
template<class Packet >
void doReversePostsAndWaits (const ArrayView< const Packet > &exports, size_t numPackets, const ArrayView< Packet > &imports)
 Execute the reverse communication plan.
template<class Packet >
void doReversePostsAndWaits (const ArrayView< const Packet > &exports, const ArrayView< size_t > &numExportPacketsPerLID, const ArrayView< Packet > &imports, const ArrayView< size_t > &numImportPacketsPerLID)
 Execute the reverse communication plan.
template<class Packet >
void doReversePosts (const ArrayRCP< const Packet > &exports, size_t numPackets, const ArrayRCP< Packet > &imports)
 Post the data for a reverse plan, but do not execute the waits yet.
template<class Packet >
void doReversePosts (const ArrayRCP< const Packet > &exports, const ArrayView< size_t > &numExportPacketsPerLID, const ArrayRCP< Packet > &imports, const ArrayView< size_t > &numImportPacketsPerLID)
 Post the data for a reverse plan, but do not execute the waits yet.
void doReverseWaits ()
Implementation of Teuchos::Describable
std::string description () const
 A simple one-line description of this object.
void describe (Teuchos::FancyOStream &out, const Teuchos::EVerbosityLevel verbLevel=Teuchos::Describable::verbLevel_default) const
 Print the object with some verbosity level to an FancyOStream.

Detailed Description

Sets up and executes a communication plan for a Tpetra DistObject.

Note:
Most Tpetra users do not need to know about this class.

This class encapsulates the general information and communication services needed for subclasses of DistObject (such as CrsMatrix and MultiVector) to do data redistribution (Import and Export) operations. It is an implementation detail of Import and Export; in particular; it actually does the communication.

Here is the typical way to use this class: 1. Create a Distributor. (The constructor is inexpensive.) 2. Set up the Distributor once, using one of the two "plan creation" methods: either createFromSends(), or createFromRecvs(). This may be more expensive and communication-intensive than Step 3. 3. Communicate the data by calling doPostsAndWaits() (forward mode), or doReversePostsAndWaits() (reverse mode). You may do this multiple times with the same Distributor instance.

Step 2 is expensive, but you can amortize its cost over multiple uses of the Distributor for communication (Step 3). You may also separate out "posts" (invoking nonblocking communication) and "waits" (waiting for that communication to complete), by calling doPosts() (resp. doReversePosts()), then doWaits() (resp. doReverseWaits()). This is useful if you have local work to do between the posts and waits, because it may overlap communication with computation. Whether it actually does overlap, depends on both the MPI implementation and your choice of parameters for the Distributor.

Instances of Distributor take the following parameters that control communication and debug output:

Definition at line 147 of file Tpetra_Distributor.hpp.


Constructor & Destructor Documentation

Tpetra::Distributor::Distributor ( const Teuchos::RCP< const Teuchos::Comm< int > > &  comm) [explicit]

Construct using the specified communicator and default parameters.

Parameters:
comm[in] Communicator used by the Distributor.

The constructor doesn't actually set up the distribution pattern. You need to call one of the "gather / scatter 'constructors'" to do that.

Definition at line 59 of file Tpetra_Distributor.cpp.

Tpetra::Distributor::Distributor ( const Teuchos::RCP< const Teuchos::Comm< int > > &  comm,
const Teuchos::RCP< Teuchos::ParameterList > &  plist 
) [explicit]

Construct using the specified communicator and ParameterList.

Parameters:
comm[in] Communicator used by the Distributor.
plist[in/out] List of parameters controlling how the Distributor performs communication. Must be nonnull. Please see the class documentation for a list of all accepted parameters and their default values.

The constructor doesn't actually set up the distribution pattern. You need to call one of the "gather / scatter 'constructors'" to do that.

Definition at line 79 of file Tpetra_Distributor.cpp.

Tpetra::Distributor::Distributor ( const Distributor distributor)

Copy constructor.

Definition at line 109 of file Tpetra_Distributor.cpp.

Tpetra::Distributor::~Distributor ( ) [virtual]

Destructor (virtual for memory safety).

Definition at line 143 of file Tpetra_Distributor.cpp.


Member Function Documentation

void Tpetra::Distributor::setParameterList ( const Teuchos::RCP< Teuchos::ParameterList > &  plist) [virtual]

Set Distributor parameters.

Please see the class documentation for a list of all accepted parameters and their default values.

Implements Teuchos::ParameterListAcceptor.

Definition at line 155 of file Tpetra_Distributor.cpp.

Teuchos::RCP< const Teuchos::ParameterList > Tpetra::Distributor::getValidParameters ( ) const [virtual]

List of valid Distributor parameters.

Please see the class documentation for a list of all accepted parameters and their default values.

Reimplemented from Teuchos::ParameterListAcceptor.

Definition at line 218 of file Tpetra_Distributor.cpp.

size_t Tpetra::Distributor::createFromSends ( const ArrayView< const int > &  exportNodeIDs)

Set up Distributor using list of process ranks to which this process will send.

Take a list of process ranks and construct a plan for efficiently scattering to those processes. Return the number of processes which will send me (the calling process) data.

Parameters:
exportNodeIDs[in] List of ranks of the processes that will get the exported data. If there is a process rank greater than or equal to the number of processes, all processes will throw an std::runtime_error exception. Process ranks less than zero are ignored; their placement corresponds to null sends in any future exports. That is, if exportNodeIDs[0] == -1, then the corresponding position in the export array is ignored during a call to doPosts() or doPostsAndWaits(). For this reason, a negative entry is sufficient to break contiguity.
Returns:
Number of imports this process will be receiving.

Definition at line 670 of file Tpetra_Distributor.cpp.

template<class Ordinal >
void Tpetra::Distributor::createFromRecvs ( const ArrayView< const Ordinal > &  remoteIDs,
const ArrayView< const int > &  remoteNodeIDs,
ArrayRCP< Ordinal > &  exportIDs,
ArrayRCP< int > &  exportNodeIDs 
)

Set up Distributor using list of process ranks from which to receive.

Take a list of process ranks and construct a plan for efficiently scattering to those processes. Return the number and list of IDs being sent by me (the calling process).

Import invokes this method in order to create a Distributor from a list of receive neighbors and IDs. A common use case for this process is setting up sends and receives for the remote entries of the source vector in a distributed sparse matrix-vector multiply. The Mantevo HPCCG miniapp shows an annotated and simplified version of this process for that special case.

Parameters:
remoteIDs[in] List of remote IDs wanted.
remoteNodeIDs[in] The ranks of the process that will send the remote IDs listed in remoteIDs. Process ranks less than zero are ignored; their placement corresponds to null sends in any future exports. If there is a process rank greater than or equal to the number of processes, all processes will throw an std::runtime_error exception.
exportIDs[out] List of IDs that need to be sent from this process.
exportNodeIDs[out] The ranks of the processes that will get the exported IDs in exportIDs.

The exportGIDs and exportNodeIDs arrays are allocated by the Distributor, which is why they are passed in as a nonconst reference to an ArrayRCP. They may be null on entry.

size_t Tpetra::Distributor::getNumReceives ( ) const

The number of processes from which we will receive data.

The count does not include the calling process.

Definition at line 253 of file Tpetra_Distributor.cpp.

size_t Tpetra::Distributor::getNumSends ( ) const

The number of processes to which we will send data.

The count does not include the calling process.

Definition at line 259 of file Tpetra_Distributor.cpp.

bool Tpetra::Distributor::hasSelfMessage ( ) const

Whether the calling process will send or receive messages to itself.

Definition at line 256 of file Tpetra_Distributor.cpp.

size_t Tpetra::Distributor::getMaxSendLength ( ) const

Maximum number of values this process will send to another single process.

Definition at line 262 of file Tpetra_Distributor.cpp.

size_t Tpetra::Distributor::getTotalReceiveLength ( ) const

Total number of values this process will receive from other processes.

Definition at line 250 of file Tpetra_Distributor.cpp.

Teuchos::ArrayView< const int > Tpetra::Distributor::getImagesFrom ( ) const

Ranks of the processes sending values to this process.

This is a nonpersisting view. It will last only as long as this Distributor instance does.

Definition at line 265 of file Tpetra_Distributor.cpp.

Teuchos::ArrayView< const int > Tpetra::Distributor::getImagesTo ( ) const

Ranks of the processes to which this process will send values.

This is a nonpersisting view. It will last only as long as this Distributor instance does.

Definition at line 271 of file Tpetra_Distributor.cpp.

Teuchos::ArrayView< const size_t > Tpetra::Distributor::getLengthsFrom ( ) const

Number of values this process will receive from each process.

This process will receive getLengthsFrom[i] values from process getImagesFrom[i].

This is a nonpersisting view. It will last only as long as this Distributor instance does.

Definition at line 268 of file Tpetra_Distributor.cpp.

Teuchos::ArrayView< const size_t > Tpetra::Distributor::getLengthsTo ( ) const

Number of values this process will send to each process.

This process will send getLengthsTo[i] values to process getImagesTo[i].

This is a nonpersisting view. It will last only as long as this Distributor instance does.

Definition at line 274 of file Tpetra_Distributor.cpp.

const Teuchos::RCP< Distributor > & Tpetra::Distributor::getReverse ( ) const

A reverse communication plan Distributor.

The first time this method is called, it creates a Distributor with the reverse communication plan of *this. On subsequent calls, it returns the cached reverse Distributor.

Most users do not need to call this method. If you invoke doReversePosts() or doReversePostsAndWaits(), the reverse Distributor will be created automatically if it does not yet exist.

Definition at line 278 of file Tpetra_Distributor.cpp.

template<class Packet >
void Tpetra::Distributor::doPostsAndWaits ( const ArrayView< const Packet > &  exports,
size_t  numPackets,
const ArrayView< Packet > &  imports 
)

Execute the (forward) communication plan.

Call this version of the method when you have the same number of Packets for each LID (local ID) to send or receive.

Template Parameters:
PacketThe type of data to send and receive.
Parameters:
exports[in] Contains the values to be sent by this process. On exit from this method, it's OK to modify the entries of this buffer.
numPackets[in] The number of Packets per export / import. This version of the routine assumes that each LID has the same number of Packets associated with it. (MultiVector is an example of a DistObject subclass satisfying this property.)
imports[out] On entry, buffer must be large enough to accomodate the data exported (sent) to us. On exit, contains the values exported to us.

Definition at line 683 of file Tpetra_Distributor.hpp.

template<class Packet >
void Tpetra::Distributor::doPostsAndWaits ( const ArrayView< const Packet > &  exports,
const ArrayView< size_t > &  numExportPacketsPerLID,
const ArrayView< Packet > &  imports,
const ArrayView< size_t > &  numImportPacketsPerLID 
)

Execute the (forward) communication plan.

Call this version of the method when you have possibly different numbers of Packets for each LID (local ID) to send or receive.

Template Parameters:
PacketThe type of data to send and receive.
Parameters:
exports[in] Contains the values to be sent by this process. On exit from this method, it's OK to modify the entries of this buffer.
numExportPacketsPerLID[in] The number of packets for each export LID (i.e., each LID to be sent).
imports[out] On entry, buffer must be large enough to accomodate the data exported (sent) to us. On exit, contains the values exported to us.
numImportPacketsPerLID[in] The number of packets for each import LID (i.e., each LID to be received).

Definition at line 728 of file Tpetra_Distributor.hpp.

template<class Packet >
void Tpetra::Distributor::doPosts ( const ArrayRCP< const Packet > &  exports,
size_t  numPackets,
const ArrayRCP< Packet > &  imports 
)

Post the data for a forward plan, but do not execute the waits yet.

Call this overload when you have the same number of Packets for each LID to send or receive.

Template Parameters:
PacketThe type of data to send and receive.
Parameters:
exports[in] Contains the values to be sent by this process. This is an ArrayRCP and not an ArrayView so that we have the freedom to use nonblocking sends if we wish. Do not modify the data in this array until doWaits() has completed.
numPackets[in] The number of Packets per export / import. (Same as the three-argument version of doPostsAndWaits().)
imports[out] On entry, buffer must be large enough to accomodate the data exported (sent) to us. This is an ArrayRCP and not an ArrayView so that we have the freedom to use nonblocking sends if we wish. Do not modify the data in this array until doWaits() has completed. Upon completion of doWaits(), this buffer contains the values exported to us.

Definition at line 770 of file Tpetra_Distributor.hpp.

template<class Packet >
void Tpetra::Distributor::doPosts ( const ArrayRCP< const Packet > &  exports,
const ArrayView< size_t > &  numExportPacketsPerLID,
const ArrayRCP< Packet > &  imports,
const ArrayView< size_t > &  numImportPacketsPerLID 
)

Post the data for a forward plan, but do not execute the waits yet.

Call this overload when you have possibly different numbers of Packets for each LID to send or receive.

Template Parameters:
PacketThe type of data to send and receive.
Parameters:
exports[in] Same as in the three-argument version of doPosts().
numExportPacketsPerLID[in] Same as in the four-argument version of doPostsAndWaits().
imports[out] Same as in the three-argument version of doPosts().
numImportPacketsPerLID[in] Same as in the four-argument version of doPostsAndWaits().

Definition at line 1018 of file Tpetra_Distributor.hpp.

void Tpetra::Distributor::doWaits ( )

Wait on any outstanding nonblocking message requests to complete.

This method is for forward mode communication only, that is, after calling doPosts(). For reverse mode communication (after calling doReversePosts()), call doReverseWaits() instead.

Definition at line 333 of file Tpetra_Distributor.cpp.

template<class Packet >
void Tpetra::Distributor::doReversePostsAndWaits ( const ArrayView< const Packet > &  exports,
size_t  numPackets,
const ArrayView< Packet > &  imports 
)

Execute the reverse communication plan.

This method takes the same arguments as the three-argument version of doPostsAndWaits().

Definition at line 1277 of file Tpetra_Distributor.hpp.

template<class Packet >
void Tpetra::Distributor::doReversePostsAndWaits ( const ArrayView< const Packet > &  exports,
const ArrayView< size_t > &  numExportPacketsPerLID,
const ArrayView< Packet > &  imports,
const ArrayView< size_t > &  numImportPacketsPerLID 
)

Execute the reverse communication plan.

This method takes the same arguments as the four-argument version of doPostsAndWaits().

Definition at line 1308 of file Tpetra_Distributor.hpp.

template<class Packet >
void Tpetra::Distributor::doReversePosts ( const ArrayRCP< const Packet > &  exports,
size_t  numPackets,
const ArrayRCP< Packet > &  imports 
)

Post the data for a reverse plan, but do not execute the waits yet.

This method takes the same arguments as the three-argument version of doPosts().

Definition at line 1345 of file Tpetra_Distributor.hpp.

template<class Packet >
void Tpetra::Distributor::doReversePosts ( const ArrayRCP< const Packet > &  exports,
const ArrayView< size_t > &  numExportPacketsPerLID,
const ArrayRCP< Packet > &  imports,
const ArrayView< size_t > &  numImportPacketsPerLID 
)

Post the data for a reverse plan, but do not execute the waits yet.

This method takes the same arguments as the four-argument version of doPosts().

Definition at line 1361 of file Tpetra_Distributor.hpp.

void Tpetra::Distributor::doReverseWaits ( )

Wait on any outstanding nonblocking message requests to complete.

This method is for reverse mode communication only, that is, after calling doReversePosts(). For forward mode communication (after calling doPosts()), call doWaits() instead.

Definition at line 385 of file Tpetra_Distributor.cpp.

std::string Tpetra::Distributor::description ( ) const [virtual]

A simple one-line description of this object.

Reimplemented from Teuchos::Describable.

Definition at line 393 of file Tpetra_Distributor.cpp.

void Tpetra::Distributor::describe ( Teuchos::FancyOStream out,
const Teuchos::EVerbosityLevel  verbLevel = Teuchos::Describable::verbLevel_default 
) const [virtual]

Print the object with some verbosity level to an FancyOStream.

Reimplemented from Teuchos::Describable.

Definition at line 400 of file Tpetra_Distributor.cpp.


The documentation for this class was generated from the following files:
 All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Friends Defines