Tpetra Matrix/Vector Services Version of the Day
Tpetra::Distributor Class Reference

Sets up and executes a communication plan for a Tpetra DistObject. More...

#include <Tpetra_Distributor.hpp>

Inheritance diagram for Tpetra::Distributor:
Inheritance graph
[legend]

List of all members.

Public Member Functions

Constructors and destructor
 Distributor (const Teuchos::RCP< const Teuchos::Comm< int > > &comm)
 Construct using the specified communicator and default parameters.
 Distributor (const Teuchos::RCP< const Teuchos::Comm< int > > &comm, const Teuchos::RCP< Teuchos::FancyOStream > &out)
 Construct using the specified communicator and default parameters, with an output stream.
 Distributor (const Teuchos::RCP< const Teuchos::Comm< int > > &comm, const Teuchos::RCP< Teuchos::ParameterList > &plist)
 Construct using the specified communicator and ParameterList.
 Distributor (const Teuchos::RCP< const Teuchos::Comm< int > > &comm, const Teuchos::RCP< Teuchos::FancyOStream > &out, const Teuchos::RCP< Teuchos::ParameterList > &plist)
 Construct using the specified communicator and ParameterList, with an output stream.
 Distributor (const Distributor &distributor)
 Copy constructor.
virtual ~Distributor ()
 Destructor (virtual for memory safety).
void swap (Distributor &rhs)
 Swap the contents of rhs with those of *this.
Implementation of ParameterListAcceptorDefaultBase
void setParameterList (const Teuchos::RCP< Teuchos::ParameterList > &plist)
 Set Distributor parameters.
Teuchos::RCP< const
Teuchos::ParameterList
getValidParameters () const
 List of valid Distributor parameters.
Gather / scatter "constructors"
size_t createFromSends (const ArrayView< const int > &exportNodeIDs)
 Set up Distributor using list of process ranks to which this process will send.
template<class Ordinal >
void createFromRecvs (const ArrayView< const Ordinal > &remoteIDs, const ArrayView< const int > &remoteNodeIDs, Array< Ordinal > &exportIDs, Array< int > &exportNodeIDs)
 Set up Distributor using list of process ranks from which to receive.
Attribute accessor methods
size_t getNumReceives () const
 The number of processes from which we will receive data.
size_t getNumSends () const
 The number of processes to which we will send data.
bool hasSelfMessage () const
 Whether the calling process will send or receive messages to itself.
size_t getMaxSendLength () const
 Maximum number of values this process will send to another single process.
size_t getTotalReceiveLength () const
 Total number of values this process will receive from other processes.
ArrayView< const int > getImagesFrom () const
 Ranks of the processes sending values to this process.
ArrayView< const int > getImagesTo () const
 Ranks of the processes to which this process will send values.
ArrayView< const size_t > getLengthsFrom () const
 Number of values this process will receive from each process.
ArrayView< const size_t > getLengthsTo () const
 Number of values this process will send to each process.
Details::EDistributorHowInitialized howInitialized () const
 Return an enum indicating whether and how a Distributor was initialized.
Reverse communication methods
RCP< DistributorgetReverse () const
 A reverse communication plan Distributor.
Methods for executing a communication plan
template<class Packet >
void doPostsAndWaits (const ArrayView< const Packet > &exports, size_t numPackets, const ArrayView< Packet > &imports)
 Execute the (forward) communication plan.
template<class Packet >
void doPostsAndWaits (const ArrayView< const Packet > &exports, const ArrayView< size_t > &numExportPacketsPerLID, const ArrayView< Packet > &imports, const ArrayView< size_t > &numImportPacketsPerLID)
 Execute the (forward) communication plan.
template<class Packet >
void doPosts (const ArrayRCP< const Packet > &exports, size_t numPackets, const ArrayRCP< Packet > &imports)
 Post the data for a forward plan, but do not execute the waits yet.
template<class Packet >
void doPosts (const ArrayRCP< const Packet > &exports, const ArrayView< size_t > &numExportPacketsPerLID, const ArrayRCP< Packet > &imports, const ArrayView< size_t > &numImportPacketsPerLID)
 Post the data for a forward plan, but do not execute the waits yet.
void doWaits ()
template<class Packet >
void doReversePostsAndWaits (const ArrayView< const Packet > &exports, size_t numPackets, const ArrayView< Packet > &imports)
 Execute the reverse communication plan.
template<class Packet >
void doReversePostsAndWaits (const ArrayView< const Packet > &exports, const ArrayView< size_t > &numExportPacketsPerLID, const ArrayView< Packet > &imports, const ArrayView< size_t > &numImportPacketsPerLID)
 Execute the reverse communication plan.
template<class Packet >
void doReversePosts (const ArrayRCP< const Packet > &exports, size_t numPackets, const ArrayRCP< Packet > &imports)
 Post the data for a reverse plan, but do not execute the waits yet.
template<class Packet >
void doReversePosts (const ArrayRCP< const Packet > &exports, const ArrayView< size_t > &numExportPacketsPerLID, const ArrayRCP< Packet > &imports, const ArrayView< size_t > &numImportPacketsPerLID)
 Post the data for a reverse plan, but do not execute the waits yet.
void doReverseWaits ()
Implementation of Teuchos::Describable
std::string description () const
 A simple one-line description of this object.
void describe (Teuchos::FancyOStream &out, const Teuchos::EVerbosityLevel verbLevel=Teuchos::Describable::verbLevel_default) const
 Print the object with some verbosity level to an FancyOStream.

Detailed Description

Sets up and executes a communication plan for a Tpetra DistObject.

Note:
Most Tpetra users do not need to know about this class.

This class encapsulates the general information and communication services needed for subclasses of DistObject (such as CrsMatrix and MultiVector) to do data redistribution (Import and Export) operations. It is an implementation detail of Import and Export; in particular; it actually does the communication.

Here is the typical way to use this class: 1. Create a Distributor. (The constructor is inexpensive.) 2. Set up the Distributor once, using one of the two "plan creation" methods: either createFromSends(), or createFromRecvs(). This may be more expensive and communication-intensive than Step 3. 3. Communicate the data by calling doPostsAndWaits() (forward mode), or doReversePostsAndWaits() (reverse mode). You may do this multiple times with the same Distributor instance.

Step 2 is expensive, but you can amortize its cost over multiple uses of the Distributor for communication (Step 3). You may also separate out "posts" (invoking nonblocking communication) and "waits" (waiting for that communication to complete), by calling doPosts() (resp. doReversePosts()), then doWaits() (resp. doReverseWaits()). This is useful if you have local work to do between the posts and waits, because it may overlap communication with computation. Whether it actually does overlap, depends on both the MPI implementation and your choice of parameters for the Distributor.

Instances of Distributor take the following parameters that control communication and debug output:

Definition at line 188 of file Tpetra_Distributor.hpp.


Constructor & Destructor Documentation

Tpetra::Distributor::Distributor ( const Teuchos::RCP< const Teuchos::Comm< int > > &  comm) [explicit]

Construct using the specified communicator and default parameters.

Parameters:
comm[in] Communicator used by the Distributor.

The constructor doesn't actually set up the distribution pattern. You need to call one of the "gather / scatter 'constructors'" to do that.

Definition at line 174 of file Tpetra_Distributor.cpp.

Tpetra::Distributor::Distributor ( const Teuchos::RCP< const Teuchos::Comm< int > > &  comm,
const Teuchos::RCP< Teuchos::FancyOStream > &  out 
)

Construct using the specified communicator and default parameters, with an output stream.

Parameters:
comm[in] Communicator used by the Distributor.
out[in/out] Output stream (for debugging output).

The constructor doesn't actually set up the distribution pattern. You need to call one of the "gather / scatter 'constructors'" to do that.

Definition at line 193 of file Tpetra_Distributor.cpp.

Tpetra::Distributor::Distributor ( const Teuchos::RCP< const Teuchos::Comm< int > > &  comm,
const Teuchos::RCP< Teuchos::ParameterList > &  plist 
)

Construct using the specified communicator and ParameterList.

Parameters:
comm[in] Communicator used by the Distributor.
plist[in/out] List of parameters controlling how the Distributor performs communication. Must be nonnull. Please see the class documentation for a list of all accepted parameters and their default values.

The constructor doesn't actually set up the distribution pattern. You need to call one of the "gather / scatter 'constructors'" to do that.

Definition at line 213 of file Tpetra_Distributor.cpp.

Tpetra::Distributor::Distributor ( const Teuchos::RCP< const Teuchos::Comm< int > > &  comm,
const Teuchos::RCP< Teuchos::FancyOStream > &  out,
const Teuchos::RCP< Teuchos::ParameterList > &  plist 
)

Construct using the specified communicator and ParameterList, with an output stream.

Parameters:
comm[in] Communicator used by the Distributor.
out[in/out] Output stream (for debugging output).
plist[in/out] List of parameters controlling how the Distributor performs communication. Must be nonnull. Please see the class documentation for a list of all accepted parameters and their default values.

The constructor doesn't actually set up the distribution pattern. You need to call one of the "gather / scatter 'constructors'" to do that.

Definition at line 233 of file Tpetra_Distributor.cpp.

Tpetra::Distributor::Distributor ( const Distributor distributor)

Copy constructor.

Definition at line 254 of file Tpetra_Distributor.cpp.

Tpetra::Distributor::~Distributor ( ) [virtual]

Destructor (virtual for memory safety).

Definition at line 371 of file Tpetra_Distributor.cpp.


Member Function Documentation

void Tpetra::Distributor::swap ( Distributor rhs)

Swap the contents of rhs with those of *this.

This is useful in Import's setUnion() method. It avoids the overhead of copying arrays, since it can use std::swap on the arrays.

Definition at line 313 of file Tpetra_Distributor.cpp.

void Tpetra::Distributor::setParameterList ( const Teuchos::RCP< Teuchos::ParameterList > &  plist) [virtual]

Set Distributor parameters.

Please see the class documentation for a list of all accepted parameters and their default values.

Implements Teuchos::ParameterListAcceptor.

Definition at line 383 of file Tpetra_Distributor.cpp.

Teuchos::RCP< const Teuchos::ParameterList > Tpetra::Distributor::getValidParameters ( ) const [virtual]

List of valid Distributor parameters.

Please see the class documentation for a list of all accepted parameters and their default values.

Reimplemented from Teuchos::ParameterListAcceptor.

Definition at line 438 of file Tpetra_Distributor.cpp.

size_t Tpetra::Distributor::createFromSends ( const ArrayView< const int > &  exportNodeIDs)

Set up Distributor using list of process ranks to which this process will send.

Take a list of process ranks and construct a plan for efficiently scattering to those processes. Return the number of processes which will send me (the calling process) data.

Parameters:
exportNodeIDs[in] List of ranks of the processes that will get the exported data. If there is a process rank greater than or equal to the number of processes, all processes will throw an std::runtime_error exception. Process ranks less than zero are ignored; their placement corresponds to null sends in any future exports. That is, if exportNodeIDs[0] == -1, then the corresponding position in the export array is ignored during a call to doPosts() or doPostsAndWaits(). For this reason, a negative entry is sufficient to break contiguity.
Returns:
Number of imports this process will be receiving.

Definition at line 1030 of file Tpetra_Distributor.cpp.

template<class Ordinal >
void Tpetra::Distributor::createFromRecvs ( const ArrayView< const Ordinal > &  remoteIDs,
const ArrayView< const int > &  remoteNodeIDs,
Array< Ordinal > &  exportIDs,
Array< int > &  exportNodeIDs 
)

Set up Distributor using list of process ranks from which to receive.

Take a list of process ranks and construct a plan for efficiently scattering to those processes. Return the number and list of IDs being sent by me (the calling process).

Import invokes this method in order to create a Distributor from a list of receive neighbors and IDs. A common use case for this process is setting up sends and receives for the remote entries of the source vector in a distributed sparse matrix-vector multiply. The Mantevo HPCCG miniapp shows an annotated and simplified version of this process for that special case.

Parameters:
remoteIDs[in] List of remote IDs wanted.
remoteNodeIDs[in] The ranks of the process that will send the remote IDs listed in remoteIDs. Process ranks less than zero are ignored; their placement corresponds to null sends in any future exports. If there is a process rank greater than or equal to the number of processes, all processes will throw an std::runtime_error exception.
exportIDs[out] List of IDs that need to be sent from this process.
exportNodeIDs[out] The ranks of the processes that will get the exported IDs in exportIDs.

The exportGIDs and exportNodeIDs arrays are resized by the Distributor, which is why they are passed in as a nonconst Array reference.

size_t Tpetra::Distributor::getNumReceives ( ) const

The number of processes from which we will receive data.

The count does not include the calling process.

Definition at line 484 of file Tpetra_Distributor.cpp.

size_t Tpetra::Distributor::getNumSends ( ) const

The number of processes to which we will send data.

The count does not include the calling process.

Definition at line 490 of file Tpetra_Distributor.cpp.

bool Tpetra::Distributor::hasSelfMessage ( ) const

Whether the calling process will send or receive messages to itself.

Definition at line 487 of file Tpetra_Distributor.cpp.

size_t Tpetra::Distributor::getMaxSendLength ( ) const

Maximum number of values this process will send to another single process.

Definition at line 493 of file Tpetra_Distributor.cpp.

size_t Tpetra::Distributor::getTotalReceiveLength ( ) const

Total number of values this process will receive from other processes.

Definition at line 481 of file Tpetra_Distributor.cpp.

Teuchos::ArrayView< const int > Tpetra::Distributor::getImagesFrom ( ) const

Ranks of the processes sending values to this process.

This is a nonpersisting view. It will last only as long as this Distributor instance does.

Definition at line 496 of file Tpetra_Distributor.cpp.

Teuchos::ArrayView< const int > Tpetra::Distributor::getImagesTo ( ) const

Ranks of the processes to which this process will send values.

This is a nonpersisting view. It will last only as long as this Distributor instance does.

Definition at line 502 of file Tpetra_Distributor.cpp.

Teuchos::ArrayView< const size_t > Tpetra::Distributor::getLengthsFrom ( ) const

Number of values this process will receive from each process.

This process will receive getLengthsFrom[i] values from process getImagesFrom[i].

This is a nonpersisting view. It will last only as long as this Distributor instance does.

Definition at line 499 of file Tpetra_Distributor.cpp.

Teuchos::ArrayView< const size_t > Tpetra::Distributor::getLengthsTo ( ) const

Number of values this process will send to each process.

This process will send getLengthsTo[i] values to process getImagesTo[i].

This is a nonpersisting view. It will last only as long as this Distributor instance does.

Definition at line 505 of file Tpetra_Distributor.cpp.

Details::EDistributorHowInitialized Tpetra::Distributor::howInitialized ( ) const [inline]

Return an enum indicating whether and how a Distributor was initialized.

This is an implementation detail of Tpetra. Please do not call this method or rely on it existing in your code.

Definition at line 401 of file Tpetra_Distributor.hpp.

Teuchos::RCP< Distributor > Tpetra::Distributor::getReverse ( ) const

A reverse communication plan Distributor.

The first time this method is called, it creates a Distributor with the reverse communication plan of *this. On subsequent calls, it returns the cached reverse Distributor.

Most users do not need to call this method. If you invoke doReversePosts() or doReversePostsAndWaits(), the reverse Distributor will be created automatically if it does not yet exist.

Definition at line 509 of file Tpetra_Distributor.cpp.

template<class Packet >
void Tpetra::Distributor::doPostsAndWaits ( const ArrayView< const Packet > &  exports,
size_t  numPackets,
const ArrayView< Packet > &  imports 
)

Execute the (forward) communication plan.

Call this version of the method when you have the same number of Packets for each LID (local ID) to send or receive.

Template Parameters:
PacketThe type of data to send and receive.
Parameters:
exports[in] Contains the values to be sent by this process. On exit from this method, it's OK to modify the entries of this buffer.
numPackets[in] The number of Packets per export / import. This version of the routine assumes that each LID has the same number of Packets associated with it. (MultiVector is an example of a DistObject subclass satisfying this property.)
imports[out] On entry, buffer must be large enough to accomodate the data exported (sent) to us. On exit, contains the values exported to us.

Definition at line 1018 of file Tpetra_Distributor.hpp.

template<class Packet >
void Tpetra::Distributor::doPostsAndWaits ( const ArrayView< const Packet > &  exports,
const ArrayView< size_t > &  numExportPacketsPerLID,
const ArrayView< Packet > &  imports,
const ArrayView< size_t > &  numImportPacketsPerLID 
)

Execute the (forward) communication plan.

Call this version of the method when you have possibly different numbers of Packets for each LID (local ID) to send or receive.

Template Parameters:
PacketThe type of data to send and receive.
Parameters:
exports[in] Contains the values to be sent by this process. On exit from this method, it's OK to modify the entries of this buffer.
numExportPacketsPerLID[in] The number of packets for each export LID (i.e., each LID to be sent).
imports[out] On entry, buffer must be large enough to accomodate the data exported (sent) to us. On exit, contains the values exported to us.
numImportPacketsPerLID[in] The number of packets for each import LID (i.e., each LID to be received).

Definition at line 1063 of file Tpetra_Distributor.hpp.

template<class Packet >
void Tpetra::Distributor::doPosts ( const ArrayRCP< const Packet > &  exports,
size_t  numPackets,
const ArrayRCP< Packet > &  imports 
)

Post the data for a forward plan, but do not execute the waits yet.

Call this overload when you have the same number of Packets for each LID to send or receive.

Template Parameters:
PacketThe type of data to send and receive.
Parameters:
exports[in] Contains the values to be sent by this process. This is an ArrayRCP and not an ArrayView so that we have the freedom to use nonblocking sends if we wish. Do not modify the data in this array until doWaits() has completed.
numPackets[in] The number of Packets per export / import. (Same as the three-argument version of doPostsAndWaits().)
imports[out] On entry, buffer must be large enough to accomodate the data exported (sent) to us. This is an ArrayRCP and not an ArrayView so that we have the freedom to use nonblocking sends if we wish. Do not modify the data in this array until doWaits() has completed. Upon completion of doWaits(), this buffer contains the values exported to us.

Definition at line 1105 of file Tpetra_Distributor.hpp.

template<class Packet >
void Tpetra::Distributor::doPosts ( const ArrayRCP< const Packet > &  exports,
const ArrayView< size_t > &  numExportPacketsPerLID,
const ArrayRCP< Packet > &  imports,
const ArrayView< size_t > &  numImportPacketsPerLID 
)

Post the data for a forward plan, but do not execute the waits yet.

Call this overload when you have possibly different numbers of Packets for each LID to send or receive.

Template Parameters:
PacketThe type of data to send and receive.
Parameters:
exports[in] Same as in the three-argument version of doPosts().
numExportPacketsPerLID[in] Same as in the four-argument version of doPostsAndWaits().
imports[out] Same as in the three-argument version of doPosts().
numImportPacketsPerLID[in] Same as in the four-argument version of doPostsAndWaits().

Definition at line 1462 of file Tpetra_Distributor.hpp.

void Tpetra::Distributor::doWaits ( )

Wait on any outstanding nonblocking message requests to complete.

This method is for forward mode communication only, that is, after calling doPosts(). For reverse mode communication (after calling doReversePosts()), call doReverseWaits() instead.

Definition at line 567 of file Tpetra_Distributor.cpp.

template<class Packet >
void Tpetra::Distributor::doReversePostsAndWaits ( const ArrayView< const Packet > &  exports,
size_t  numPackets,
const ArrayView< Packet > &  imports 
)

Execute the reverse communication plan.

This method takes the same arguments as the three-argument version of doPostsAndWaits().

Definition at line 1813 of file Tpetra_Distributor.hpp.

template<class Packet >
void Tpetra::Distributor::doReversePostsAndWaits ( const ArrayView< const Packet > &  exports,
const ArrayView< size_t > &  numExportPacketsPerLID,
const ArrayView< Packet > &  imports,
const ArrayView< size_t > &  numImportPacketsPerLID 
)

Execute the reverse communication plan.

This method takes the same arguments as the four-argument version of doPostsAndWaits().

Definition at line 1844 of file Tpetra_Distributor.hpp.

template<class Packet >
void Tpetra::Distributor::doReversePosts ( const ArrayRCP< const Packet > &  exports,
size_t  numPackets,
const ArrayRCP< Packet > &  imports 
)

Post the data for a reverse plan, but do not execute the waits yet.

This method takes the same arguments as the three-argument version of doPosts().

Definition at line 1881 of file Tpetra_Distributor.hpp.

template<class Packet >
void Tpetra::Distributor::doReversePosts ( const ArrayRCP< const Packet > &  exports,
const ArrayView< size_t > &  numExportPacketsPerLID,
const ArrayRCP< Packet > &  imports,
const ArrayView< size_t > &  numImportPacketsPerLID 
)

Post the data for a reverse plan, but do not execute the waits yet.

This method takes the same arguments as the four-argument version of doPosts().

Definition at line 1897 of file Tpetra_Distributor.hpp.

void Tpetra::Distributor::doReverseWaits ( )

Wait on any outstanding nonblocking message requests to complete.

This method is for reverse mode communication only, that is, after calling doReversePosts(). For forward mode communication (after calling doPosts()), call doWaits() instead.

Definition at line 634 of file Tpetra_Distributor.cpp.

std::string Tpetra::Distributor::description ( ) const [virtual]

A simple one-line description of this object.

Reimplemented from Teuchos::Describable.

Definition at line 641 of file Tpetra_Distributor.cpp.

void Tpetra::Distributor::describe ( Teuchos::FancyOStream out,
const Teuchos::EVerbosityLevel  verbLevel = Teuchos::Describable::verbLevel_default 
) const [virtual]

Print the object with some verbosity level to an FancyOStream.

Reimplemented from Teuchos::Describable.

Definition at line 666 of file Tpetra_Distributor.cpp.


The documentation for this class was generated from the following files:
 All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Friends Defines