Tpetra Matrix/Vector Services Version of the Day
Tpetra::Distributor Class Reference

Sets up and executes a communication plan for a Tpetra DistObject. More...

#include <Tpetra_Distributor.hpp>

Inheritance diagram for Tpetra::Distributor:
Inheritance graph
[legend]

List of all members.

Public Member Functions

Constructor/Destructor
 Distributor (const Teuchos::RCP< const Teuchos::Comm< int > > &comm)
 Construct using the specified communicator and default parameters.
 Distributor (const Teuchos::RCP< const Teuchos::Comm< int > > &comm, const Teuchos::RCP< Teuchos::ParameterList > &plist)
 Construct using the specified communicator and ParameterList.
 Distributor (const Distributor &distributor)
 Copy constructor.
virtual ~Distributor ()
 Destructor (virtual for memory safety).
Implementation of ParameterListAcceptorDefaultBase
void setParameterList (const Teuchos::RCP< Teuchos::ParameterList > &plist)
 Set Distributor parameters.
Teuchos::RCP< const
Teuchos::ParameterList
getValidParameters () const
 List of valid Distributor parameters.
Gather/Scatter Constructors
size_t createFromSends (const ArrayView< const int > &exportNodeIDs)
 Set up Distributor using list of node IDs to which this node will send.
template<class Ordinal >
void createFromRecvs (const ArrayView< const Ordinal > &remoteIDs, const ArrayView< const int > &remoteNodeIDs, ArrayRCP< Ordinal > &exportIDs, ArrayRCP< int > &exportNodeIDs)
 Set up Distributor using list of node IDs from which to receive.
Attribute Accessor Methods
size_t getNumReceives () const
 The number of nodes from which we will receive data, not include this node ("myself").
size_t getNumSends () const
 The number of nodes to which we will send data, not include this node ("myself").
bool hasSelfMessage () const
 Indicates whether values are being sent to/recieved from this node.
size_t getMaxSendLength () const
 Maximum number of values that this node is sending to another single node.
size_t getTotalReceiveLength () const
 Total number of values that this nodes is receiving from other nodes.
ArrayView< const int > getImagesFrom () const
 A list of images sending values to this node. (non-persisting view)
ArrayView< const int > getImagesTo () const
 A list of images to which this node is sending values. (non-persisting view)
ArrayView< const size_t > getLengthsFrom () const
 Number of values we're receiving from each node. (non-persisting view)
ArrayView< const size_t > getLengthsTo () const
 Number of values we're sending to each node. (non-persisting view)
Reverse Communication Methods
const RCP< Distributor > & getReverse () const
 Returns a Distributor with a reverse plan of this Distributor's plan.
Execute Distributor Plan Methods
template<class Packet >
void doPostsAndWaits (const ArrayView< const Packet > &exports, size_t numPackets, const ArrayView< Packet > &imports)
 Execute the (forward) communication plan.
template<class Packet >
void doPostsAndWaits (const ArrayView< const Packet > &exports, const ArrayView< size_t > &numExportPacketsPerLID, const ArrayView< Packet > &imports, const ArrayView< size_t > &numImportPacketsPerLID)
 Execute the (forward) communication plan.
template<class Packet >
void doPosts (const ArrayRCP< const Packet > &exports, size_t numPackets, const ArrayRCP< Packet > &imports)
 Post the data for a forward plan, but do not execute the waits yet.
template<class Packet >
void doPosts (const ArrayRCP< const Packet > &exports, const ArrayView< size_t > &numExportPacketsPerLID, const ArrayRCP< Packet > &imports, const ArrayView< size_t > &numImportPacketsPerLID)
 Post the data for a forward plan, but do not execute the waits yet.
void doWaits ()
 Wait on any outstanding nonblocking message requests to complete.
template<class Packet >
void doReversePostsAndWaits (const ArrayView< const Packet > &exports, size_t numPackets, const ArrayView< Packet > &imports)
 Execute the reverse communication plan.
template<class Packet >
void doReversePostsAndWaits (const ArrayView< const Packet > &exports, const ArrayView< size_t > &numExportPacketsPerLID, const ArrayView< Packet > &imports, const ArrayView< size_t > &numImportPacketsPerLID)
 Execute the reverse communication plan.
template<class Packet >
void doReversePosts (const ArrayRCP< const Packet > &exports, size_t numPackets, const ArrayRCP< Packet > &imports)
 Post the data for a reverse plan, but do not execute the waits yet.
template<class Packet >
void doReversePosts (const ArrayRCP< const Packet > &exports, const ArrayView< size_t > &numExportPacketsPerLID, const ArrayRCP< Packet > &imports, const ArrayView< size_t > &numImportPacketsPerLID)
 Post the data for a reverse plan, but do not execute the waits yet.
void doReverseWaits ()
 Wait on any outstanding reverse message requests to complete.
Overridden from Teuchos::Describable
std::string description () const
 A simple one-line description of this object.
void describe (Teuchos::FancyOStream &out, const Teuchos::EVerbosityLevel verbLevel=Teuchos::Describable::verbLevel_default) const
 Print the object with some verbosity level to an FancyOStream.

Detailed Description

Sets up and executes a communication plan for a Tpetra DistObject.

This class encapsulates the general information and services needed for subclasses of DistObject (such as CrsMatrix and MultiVector) to do data redistribution (Import and Export) operations.

Definition at line 98 of file Tpetra_Distributor.hpp.


Constructor & Destructor Documentation

Tpetra::Distributor::Distributor ( const Teuchos::RCP< const Teuchos::Comm< int > > &  comm) [explicit]

Construct using the specified communicator and default parameters.

Parameters:
comm[in] Communicator used by the Distributor.

The constructor doesn't actually set up the distribution pattern. You need to call one of the "gather / scatter 'constructors'" to do that.

Definition at line 59 of file Tpetra_Distributor.cpp.

Tpetra::Distributor::Distributor ( const Teuchos::RCP< const Teuchos::Comm< int > > &  comm,
const Teuchos::RCP< Teuchos::ParameterList > &  plist 
) [explicit]

Construct using the specified communicator and ParameterList.

Parameters:
comm[in] Communicator used by the Distributor.
plist[in/out] List of parameters controlling how the Distributor performs communication. Must be nonnull.

The constructor doesn't actually set up the distribution pattern. You need to call one of the "gather / scatter 'constructors'" to do that.

Definition at line 79 of file Tpetra_Distributor.cpp.

Tpetra::Distributor::Distributor ( const Distributor distributor)

Copy constructor.

Definition at line 109 of file Tpetra_Distributor.cpp.

Tpetra::Distributor::~Distributor ( ) [virtual]

Destructor (virtual for memory safety).

Definition at line 143 of file Tpetra_Distributor.cpp.


Member Function Documentation

void Tpetra::Distributor::setParameterList ( const Teuchos::RCP< Teuchos::ParameterList > &  plist) [virtual]

Set Distributor parameters.

See the documentation of getValidParameters() for the current list of parameters understood by Distributor.

Implements Teuchos::ParameterListAcceptor.

Definition at line 155 of file Tpetra_Distributor.cpp.

Teuchos::RCP< const Teuchos::ParameterList > Tpetra::Distributor::getValidParameters ( ) const [virtual]

List of valid Distributor parameters.

Current parameters include:

  • "Barrier between receives and sends" (bool): Whether to execute a barrier between receives and sends in do[Reverse]Posts(). n A barrier is required for correctness when the "Send type" parameter is "Rsend". Otherwise, a barrier is correct and may be useful for debugging, but not recommended, since it introduces useless synchronization.
  • "Send type" (std::string): When using MPI, the variant of MPI_Send to use in do[Reverse]Posts(). Valid values include "Isend", "Rsend", "Send", and "Ssend".

Reimplemented from Teuchos::ParameterListAcceptor.

Definition at line 218 of file Tpetra_Distributor.cpp.

size_t Tpetra::Distributor::createFromSends ( const ArrayView< const int > &  exportNodeIDs)

Set up Distributor using list of node IDs to which this node will send.

Take a list of node IDs and construct a plan for efficiently scattering to those nodes. Return the number of nodes which will send me data.

Parameters:
exportNodeIDs[in] List of nodes that will get the exported data. A node ID greater than or equal to the number of nodes will result in a std::runtime_error on all nodes. Node IDs less than zero are ignored; their placement corresponds to null sends in any future exports. That is, if exportNodeIDs[0] == -1, then the corresponding position in the export array is ignored during a call to doPosts() or doPostsAndWaits(). For this reason, a negative entry is sufficient to break contiguity.
Returns:
Number of imports this node will be receiving.

Definition at line 643 of file Tpetra_Distributor.cpp.

template<class Ordinal >
void Tpetra::Distributor::createFromRecvs ( const ArrayView< const Ordinal > &  remoteIDs,
const ArrayView< const int > &  remoteNodeIDs,
ArrayRCP< Ordinal > &  exportIDs,
ArrayRCP< int > &  exportNodeIDs 
)

Set up Distributor using list of node IDs from which to receive.

Take a list of node IDs and construct a plan for efficiently scattering to those nodes. Return the number and list of IDs being sent by me.

Import invokes this method in order to creating a Distributor from a list of receive neighbors and IDs. A common use case for this process is setting up sends and receives for the remote entries of the source vector in a distributed sparse matrix-vector multiply. The Mantevo HPCCG miniapp shows an annotated and simplified version of this process for that special case.

Parameters:
remoteIDs[in] List of remote IDs wanted.
remoteNodeIDs[in] List of the nodes that will send the remote IDs listed in . Node IDs less than zero are ignored; their placement corresponds to null sends in any future exports. A node ID greater than or equal to the number of nodes will result in an std::runtime_error on all nodes.
exportIDs[out] List of IDs that need to be sent from this node.
exportNodeIDs[out] List of nodes that will get the exported IDs in exportIDs.

The exportGIDs and exportNodeIDs arrays are allocated by the Distributor, which is why they are passed in a nonconst reference to an ArrayRCP. They may be null on entry.

size_t Tpetra::Distributor::getNumReceives ( ) const

The number of nodes from which we will receive data, not include this node ("myself").

Definition at line 253 of file Tpetra_Distributor.cpp.

size_t Tpetra::Distributor::getNumSends ( ) const

The number of nodes to which we will send data, not include this node ("myself").

Definition at line 259 of file Tpetra_Distributor.cpp.

bool Tpetra::Distributor::hasSelfMessage ( ) const

Indicates whether values are being sent to/recieved from this node.

If we are sending any elements to ourself, returns true. If we aren't, returns false.

Definition at line 256 of file Tpetra_Distributor.cpp.

size_t Tpetra::Distributor::getMaxSendLength ( ) const

Maximum number of values that this node is sending to another single node.

Definition at line 262 of file Tpetra_Distributor.cpp.

size_t Tpetra::Distributor::getTotalReceiveLength ( ) const

Total number of values that this nodes is receiving from other nodes.

Definition at line 250 of file Tpetra_Distributor.cpp.

Teuchos::ArrayView< const int > Tpetra::Distributor::getImagesFrom ( ) const

A list of images sending values to this node. (non-persisting view)

Definition at line 265 of file Tpetra_Distributor.cpp.

Teuchos::ArrayView< const int > Tpetra::Distributor::getImagesTo ( ) const

A list of images to which this node is sending values. (non-persisting view)

Definition at line 271 of file Tpetra_Distributor.cpp.

Teuchos::ArrayView< const size_t > Tpetra::Distributor::getLengthsFrom ( ) const

Number of values we're receiving from each node. (non-persisting view)

We will receive getLengthsFrom[i] values from node getImagesFrom[i].

Definition at line 268 of file Tpetra_Distributor.cpp.

Teuchos::ArrayView< const size_t > Tpetra::Distributor::getLengthsTo ( ) const

Number of values we're sending to each node. (non-persisting view)

We will send getLengthsTo[i] values to image getImagesTo[i].

Definition at line 274 of file Tpetra_Distributor.cpp.

const Teuchos::RCP< Distributor > & Tpetra::Distributor::getReverse ( ) const

Returns a Distributor with a reverse plan of this Distributor's plan.

This method creates the reverse Distributor the first time the function is called.

Definition at line 278 of file Tpetra_Distributor.cpp.

template<class Packet >
void Tpetra::Distributor::doPostsAndWaits ( const ArrayView< const Packet > &  exports,
size_t  numPackets,
const ArrayView< Packet > &  imports 
)

Execute the (forward) communication plan.

Call this overload when you have the same number of Packets for each LID to send or receive.

Parameters:
exports[in] Contains the values to be sent by this node. On exit from this method, it's OK to modify the entries of this buffer.
numPackets[in] The number of Packets per export / import. This version of the routine assumes that each LID has the same number of Packets associated with it. (MultiVector is an example of a DistObject subclass satisfying this property.)
imports[out] On entry, buffer must be large enough to accomodate the data exported (sent) to us. On exit, contains the values exported to us.

Definition at line 600 of file Tpetra_Distributor.hpp.

template<class Packet >
void Tpetra::Distributor::doPostsAndWaits ( const ArrayView< const Packet > &  exports,
const ArrayView< size_t > &  numExportPacketsPerLID,
const ArrayView< Packet > &  imports,
const ArrayView< size_t > &  numImportPacketsPerLID 
)

Execute the (forward) communication plan.

Call this overload when you have possibly different numbers of Packets for each LID to send or receive.

Parameters:
exports[in] Contains the values to be sent by this node. On exit from this method, it's OK to modify the entries of this buffer.
numExportPacketsPerLID[in] The number of packets for each export LID (i.e., each LID to be sent).
imports[out] On entry, buffer must be large enough to accomodate the data exported (sent) to us. On exit, contains the values exported to us.
numImportPacketsPerLID[in] The number of packets for each import LID (i.e., each LID to be received).

Definition at line 645 of file Tpetra_Distributor.hpp.

template<class Packet >
void Tpetra::Distributor::doPosts ( const ArrayRCP< const Packet > &  exports,
size_t  numPackets,
const ArrayRCP< Packet > &  imports 
)

Post the data for a forward plan, but do not execute the waits yet.

Call this overload when you have the same number of Packets for each LID to send or receive.

Parameters:
exports[in] Contains the values to be sent by this node. This is an ArrayRCP and not an ArrayView so that we have the freedom to use nonblocking sends if we wish. Do not modify the data in this array until doWaits() has completed.
numPackets[in] The number of Packets per export / import. (Same as the three-argument version of doPostsAndWaits().)
imports[out] On entry, buffer must be large enough to accomodate the data exported (sent) to us. This is an ArrayRCP and not an ArrayView so that we have the freedom to use nonblocking sends if we wish. Do not modify the data in this array until doWaits() has completed. Upon completion of doWaits(), this buffer contains the values exported to us.

Definition at line 687 of file Tpetra_Distributor.hpp.

template<class Packet >
void Tpetra::Distributor::doPosts ( const ArrayRCP< const Packet > &  exports,
const ArrayView< size_t > &  numExportPacketsPerLID,
const ArrayRCP< Packet > &  imports,
const ArrayView< size_t > &  numImportPacketsPerLID 
)

Post the data for a forward plan, but do not execute the waits yet.

Call this overload when you have possibly different numbers of Packets for each LID to send or receive.

Parameters:
exports[in] Same as in the three-argument version of doPosts().
numExportPacketsPerLID[in] Same as in the four-argument version of doPostsAndWaits().
imports[out] Same as in the three-argument version of doPosts().
numImportPacketsPerLID[in] Same as in the four-argument version of doPostsAndWaits().

Definition at line 919 of file Tpetra_Distributor.hpp.

void Tpetra::Distributor::doWaits ( )

Wait on any outstanding nonblocking message requests to complete.

Definition at line 333 of file Tpetra_Distributor.cpp.

template<class Packet >
void Tpetra::Distributor::doReversePostsAndWaits ( const ArrayView< const Packet > &  exports,
size_t  numPackets,
const ArrayView< Packet > &  imports 
)

Execute the reverse communication plan.

This method takes the same arguments as the three-argument version of doPostsAndWaits().

Definition at line 1178 of file Tpetra_Distributor.hpp.

template<class Packet >
void Tpetra::Distributor::doReversePostsAndWaits ( const ArrayView< const Packet > &  exports,
const ArrayView< size_t > &  numExportPacketsPerLID,
const ArrayView< Packet > &  imports,
const ArrayView< size_t > &  numImportPacketsPerLID 
)

Execute the reverse communication plan.

This method takes the same arguments as the four-argument version of doPostsAndWaits().

Definition at line 1209 of file Tpetra_Distributor.hpp.

template<class Packet >
void Tpetra::Distributor::doReversePosts ( const ArrayRCP< const Packet > &  exports,
size_t  numPackets,
const ArrayRCP< Packet > &  imports 
)

Post the data for a reverse plan, but do not execute the waits yet.

This method takes the same arguments as the three-argument version of doPosts().

Definition at line 1246 of file Tpetra_Distributor.hpp.

template<class Packet >
void Tpetra::Distributor::doReversePosts ( const ArrayRCP< const Packet > &  exports,
const ArrayView< size_t > &  numExportPacketsPerLID,
const ArrayRCP< Packet > &  imports,
const ArrayView< size_t > &  numImportPacketsPerLID 
)

Post the data for a reverse plan, but do not execute the waits yet.

This method takes the same arguments as the four-argument version of doPosts().

Definition at line 1262 of file Tpetra_Distributor.hpp.

void Tpetra::Distributor::doReverseWaits ( )

Wait on any outstanding reverse message requests to complete.

Definition at line 385 of file Tpetra_Distributor.cpp.

std::string Tpetra::Distributor::description ( ) const [virtual]

A simple one-line description of this object.

Reimplemented from Teuchos::Describable.

Definition at line 393 of file Tpetra_Distributor.cpp.

void Tpetra::Distributor::describe ( Teuchos::FancyOStream out,
const Teuchos::EVerbosityLevel  verbLevel = Teuchos::Describable::verbLevel_default 
) const [virtual]

Print the object with some verbosity level to an FancyOStream.

Reimplemented from Teuchos::Describable.

Definition at line 400 of file Tpetra_Distributor.cpp.


The documentation for this class was generated from the following files:
 All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Friends Defines