Tpetra Matrix/Vector Services Version of the Day
Protected Types | Protected Member Functions | Protected Attributes
Tpetra::DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node > Class Template Reference

Base class for distributed Tpetra objects that support data redistribution. More...

#include <Tpetra_DistObject.hpp>

Inheritance diagram for Tpetra::DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node >:
Inheritance graph
[legend]

List of all members.

Public Member Functions

Constructors and destructor
 DistObject (const Teuchos::RCP< const Map< LocalOrdinal, GlobalOrdinal, Node > > &map)
 Constructor.
 DistObject (const DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node > &source)
 Copy constructor.
virtual ~DistObject ()
 Destructor (virtual for memory safety of derived classes).
Public methods for redistributing data
void doImport (const DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node > &source, const Import< LocalOrdinal, GlobalOrdinal, Node > &importer, CombineMode CM)
 Import using an Import object ("forward mode").
void doExport (const DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node > &dest, const Export< LocalOrdinal, GlobalOrdinal, Node > &exporter, CombineMode CM)
 Export using an Export object ("forward mode").
void doImport (const DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node > &source, const Export< LocalOrdinal, GlobalOrdinal, Node > &exporter, CombineMode CM)
 Import using an Export object ("reverse mode").
void doExport (const DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node > &dest, const Import< LocalOrdinal, GlobalOrdinal, Node > &importer, CombineMode CM)
 Export using an Import object ("reverse mode").
Attribute accessor methods
bool isDistributed () const
 Whether this is a globally distributed object.
const Teuchos::RCP< const Map
< LocalOrdinal, GlobalOrdinal,
Node > > & 
getMap () const
 The Map with which this DistObject was constructed.
I/O methods
void print (std::ostream &os) const
 Print this object to the given output stream.
Implementation of \c Teuchos::Describable
virtual std::string description () const
 One-line descriptiion of this object.
virtual void describe (Teuchos::FancyOStream &out, const Teuchos::EVerbosityLevel verbLevel=Teuchos::Describable::verbLevel_default) const
 Print a descriptiion of this object to the given output stream.

Protected Types

enum  ReverseOption
 Whether the data transfer should be performed in forward or reverse mode. More...

Protected Member Functions

virtual void doTransfer (const DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node > &source, CombineMode CM, size_t numSameIDs, const Teuchos::ArrayView< const LocalOrdinal > &permuteToLIDs, const Teuchos::ArrayView< const LocalOrdinal > &permuteFromLIDs, const Teuchos::ArrayView< const LocalOrdinal > &remoteLIDs, const Teuchos::ArrayView< const LocalOrdinal > &exportLIDs, Distributor &distor, ReverseOption revOp)
 Redistribute data across memory images.
virtual void createViews () const
 Hook for creating a const view.
virtual void createViewsNonConst (Kokkos::ReadWriteOption rwo)
 Hook for creating a nonconst view.
virtual void releaseViews () const
 Hook for releasing views.
Methods implemented by subclasses and used by \c doTransfer().

The doTransfer() method uses the subclass' implementations of these methods to implement data transfer. Subclasses of DistObject must implement these methods. This is an instance of the Template Method Pattern. ("Template" here doesn't mean "C++ template"; it means "pattern with holes that are filled in by the subclass' method implementations.")

virtual bool checkSizes (const DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node > &source)=0
 Compare the source and target (this) objects for compatibility.
virtual void copyAndPermute (const DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node > &source, size_t numSameIDs, const Teuchos::ArrayView< const LocalOrdinal > &permuteToLIDs, const Teuchos::ArrayView< const LocalOrdinal > &permuteFromLIDs)=0
 Perform copies and permutations that are local to this process.
virtual void packAndPrepare (const DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node > &source, const Teuchos::ArrayView< const LocalOrdinal > &exportLIDs, Teuchos::Array< Packet > &exports, const Teuchos::ArrayView< size_t > &numPacketsPerLID, size_t &constantNumPackets, Distributor &distor)=0
 Perform any packing or preparation required for communication.
virtual void unpackAndCombine (const Teuchos::ArrayView< const LocalOrdinal > &importLIDs, const Teuchos::ArrayView< const Packet > &imports, const Teuchos::ArrayView< size_t > &numPacketsPerLID, size_t constantNumPackets, Distributor &distor, CombineMode CM)=0
 Perform any unpacking and combining after communication.

Protected Attributes

Teuchos::RCP< const Map
< LocalOrdinal, GlobalOrdinal,
Node > > 
map_
 The Map over which this object is distributed.

Detailed Description

template<class Packet, class LocalOrdinal = int, class GlobalOrdinal = LocalOrdinal, class Node = Kokkos::DefaultNode::DefaultNodeType>
class Tpetra::DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node >

Base class for distributed Tpetra objects that support data redistribution.

DistObject is a base class for all Tpetra distributed global objects, including CrsMatrix and MultiVector. It provides the basic mechanisms and interface specifications for importing and exporting operations using Import and Export objects.

Template Parameters:
LocalOrdinalThe type of local IDs. Same as Map's LocalOrdinal template parameter. This should be an integer type, preferably signed.
GlobalOrdinalThe type of global IDs. Same as Map's GlobalOrdinal template parameter. Defaults to the same type as LocalOrdinal. This should also be an integer type, preferably signed.
NodeSame as Map's Node template parameter. Defaults to the default Kokkos Node type.

Most Tpetra users will create subclasses of DistObject (like CrsMatrix or MultiVector), describe data distribution via Map instances, create data redistribution plans via Import or Export, and invoke the doImport() or doExport() methods of DistObject with an Import or Export object to redistribute data. Thus, the only methods of DistObject of interest to most Tpetra users are doImport() and doExport().

If you want to implement your own DistObject subclass, you should start by implementing the four pure virtual methods: checkSizes(), copyAndPermute(), packAndPrepare(), and unpackAndCombine(). The implementation of doTransfer() includes documentation that explains how DistObject uses those methods to do data redistribution.

If you are writing a DistObject class that uses Kokkos compute buffers and aims to work for any Kokkos Node type, you should also implement the three hooks that create and release views: createViews(), createViewsNonConst(), and releaseViews(). The default implementation of these hooks does nothing. The documentation of these methods explains different ways you might choose to implement them.

Distributed Tpetra objects may be either "distributed global" or "replicated local." Distributed global objects are partitioned across multiple processes in a communicator. Each process owns at least one element in the object's Map that is not owned by another process. For replicated local objects, each element in the object's Map is owned redundantly by all processes in the object's communicator. Some algorithms use objects that are too small to be distributed across all processes. The upper Hessenberg matrix in a GMRES iterative solve is a good example. In other cases, such as with block iterative methods, block dot product functions produce small dense matrices that are required by all images. Replicated local objects handle these situations.

Definition at line 117 of file Tpetra_DistObject.hpp.


Member Enumeration Documentation

template<class Packet, class LocalOrdinal = int, class GlobalOrdinal = LocalOrdinal, class Node = Kokkos::DefaultNode::DefaultNodeType>
enum Tpetra::DistObject::ReverseOption [protected]

Whether the data transfer should be performed in forward or reverse mode.

"Reverse mode" means calling doExport() with an Import object, or calling doImport() with an Export object. "Forward mode" means calling doExport() with an Export object, or calling doImport() with an Import object.

Definition at line 216 of file Tpetra_DistObject.hpp.


Constructor & Destructor Documentation

template<class Packet , class LocalOrdinal, class GlobalOrdinal, class Node>
Tpetra::DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node >::DistObject ( const Teuchos::RCP< const Map< LocalOrdinal, GlobalOrdinal, Node > > &  map) [explicit]

Constructor.

Definition at line 426 of file Tpetra_DistObject.hpp.

template<class Packet, class LocalOrdinal, class GlobalOrdinal, class Node>
Tpetra::DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node >::DistObject ( const DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node > &  source)

Copy constructor.

Definition at line 432 of file Tpetra_DistObject.hpp.

template<class Packet , class LocalOrdinal , class GlobalOrdinal , class Node >
Tpetra::DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node >::~DistObject ( ) [virtual]

Destructor (virtual for memory safety of derived classes).

Definition at line 437 of file Tpetra_DistObject.hpp.


Member Function Documentation

template<class Packet, class LocalOrdinal, class GlobalOrdinal, class Node>
void Tpetra::DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node >::doImport ( const DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node > &  source,
const Import< LocalOrdinal, GlobalOrdinal, Node > &  importer,
CombineMode  CM 
)

Import using an Import object ("forward mode").

Definition at line 481 of file Tpetra_DistObject.hpp.

template<class Packet, class LocalOrdinal, class GlobalOrdinal, class Node>
void Tpetra::DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node >::doExport ( const DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node > &  dest,
const Export< LocalOrdinal, GlobalOrdinal, Node > &  exporter,
CombineMode  CM 
)

Export using an Export object ("forward mode").

Definition at line 506 of file Tpetra_DistObject.hpp.

template<class Packet, class LocalOrdinal, class GlobalOrdinal, class Node>
void Tpetra::DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node >::doImport ( const DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node > &  source,
const Export< LocalOrdinal, GlobalOrdinal, Node > &  exporter,
CombineMode  CM 
)

Import using an Export object ("reverse mode").

Definition at line 528 of file Tpetra_DistObject.hpp.

template<class Packet, class LocalOrdinal, class GlobalOrdinal, class Node>
void Tpetra::DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node >::doExport ( const DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node > &  dest,
const Import< LocalOrdinal, GlobalOrdinal, Node > &  importer,
CombineMode  CM 
)

Export using an Import object ("reverse mode").

Definition at line 550 of file Tpetra_DistObject.hpp.

template<class Packet , class LocalOrdinal , class GlobalOrdinal , class Node >
bool Tpetra::DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node >::isDistributed ( ) const [inline]

Whether this is a globally distributed object.

For a definition of "globally distributed" (and its opposite, "locally replicated"), see the documentation of Map's isDistributed() method.

Definition at line 573 of file Tpetra_DistObject.hpp.

template<class Packet, class LocalOrdinal = int, class GlobalOrdinal = LocalOrdinal, class Node = Kokkos::DefaultNode::DefaultNodeType>
const Teuchos::RCP<const Map<LocalOrdinal,GlobalOrdinal,Node> >& Tpetra::DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node >::getMap ( ) const [inline]

The Map with which this DistObject was constructed.

Definition at line 172 of file Tpetra_DistObject.hpp.

template<class Packet , class LocalOrdinal , class GlobalOrdinal , class Node >
void Tpetra::DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node >::print ( std::ostream &  os) const

Print this object to the given output stream.

We generally assume that all MPI processes can print to the given stream.

Definition at line 719 of file Tpetra_DistObject.hpp.

template<class Packet , class LocalOrdinal , class GlobalOrdinal , class Node >
std::string Tpetra::DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node >::description ( ) const [virtual]
template<class Packet , class LocalOrdinal , class GlobalOrdinal , class Node >
void Tpetra::DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node >::describe ( Teuchos::FancyOStream out,
const Teuchos::EVerbosityLevel  verbLevel = Teuchos::Describable::verbLevel_default 
) const [virtual]
template<class Packet, class LocalOrdinal, class GlobalOrdinal, class Node>
void Tpetra::DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node >::doTransfer ( const DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node > &  source,
CombineMode  CM,
size_t  numSameIDs,
const Teuchos::ArrayView< const LocalOrdinal > &  permuteToLIDs,
const Teuchos::ArrayView< const LocalOrdinal > &  permuteFromLIDs,
const Teuchos::ArrayView< const LocalOrdinal > &  remoteLIDs,
const Teuchos::ArrayView< const LocalOrdinal > &  exportLIDs,
Distributor distor,
ReverseOption  revOp 
) [protected, virtual]

Redistribute data across memory images.

Parameters:
source[in] The source object, to redistribute into the destination object, which is *this object.
CM[in] The combine mode that describes how to combine values that map to the same global ID on the same process.
permuteToLIDs[in] See copyAndPermute().
permuteFromLIDs[in] See copyAndPermute().
remoteLIDs[in] List of entries (as local IDs) in the destination object to receive from other processes.
exportLIDs[in] See packAndPrepare().
distor[in/out] The Distributor object that knows how to redistribute data.
revOp[in] Whether to do a forward or reverse mode redistribution.

Definition at line 580 of file Tpetra_DistObject.hpp.

template<class Packet, class LocalOrdinal = int, class GlobalOrdinal = LocalOrdinal, class Node = Kokkos::DefaultNode::DefaultNodeType>
virtual bool Tpetra::DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node >::checkSizes ( const DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node > &  source) [protected, pure virtual]
template<class Packet, class LocalOrdinal = int, class GlobalOrdinal = LocalOrdinal, class Node = Kokkos::DefaultNode::DefaultNodeType>
virtual void Tpetra::DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node >::copyAndPermute ( const DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node > &  source,
size_t  numSameIDs,
const Teuchos::ArrayView< const LocalOrdinal > &  permuteToLIDs,
const Teuchos::ArrayView< const LocalOrdinal > &  permuteFromLIDs 
) [protected, pure virtual]

Perform copies and permutations that are local to this process.

Parameters:
source[in] On entry, the source object, from which we are distributing. We distribute to the destination object, which is *this object.
numSameIDs[in] The umber of elements that are the same on the source and destination (this) objects. These elements are owned by the same process in both the source and destination objects. No permutation occurs.
numPermuteIDs[in] The number of elements that are locally permuted between the source and destination objects.
permuteToLIDs[in] List of the elements that are permuted. They are listed by their LID in the destination object.
permuteFromLIDs[in] List of the elements that are permuted. They are listed by their LID in the source object.

Implemented in Tpetra::CrsGraph< LocalOrdinal, GlobalOrdinal, Node, LocalMatOps >, Tpetra::CrsMatrix< Scalar, LocalOrdinal, GlobalOrdinal, Node, LocalMatOps >, Tpetra::MultiVector< Scalar, LocalOrdinal, GlobalOrdinal, Node >, Tpetra::VbrMatrix< Scalar, LocalOrdinal, GlobalOrdinal, Node, LocalMatOps >, Tpetra::CrsMatrix< Scalar, LocalOrdinal, GlobalOrdinal, Node, SpMatOps >, and Tpetra::CrsMatrix< Scalar, LocalOrdinal, GlobalOrdinal, Node >.

template<class Packet, class LocalOrdinal = int, class GlobalOrdinal = LocalOrdinal, class Node = Kokkos::DefaultNode::DefaultNodeType>
virtual void Tpetra::DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node >::packAndPrepare ( const DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node > &  source,
const Teuchos::ArrayView< const LocalOrdinal > &  exportLIDs,
Teuchos::Array< Packet > &  exports,
const Teuchos::ArrayView< size_t > &  numPacketsPerLID,
size_t &  constantNumPackets,
Distributor distor 
) [protected, pure virtual]

Perform any packing or preparation required for communication.

Parameters:
source[in] Source object for the redistribution.
exportLIDs[in] List of the entries (as local IDs in the source object) we will be sending to other images.
exports[out] On exit, the buffer for data to send.
numPacketsPerLID[out] On exit, numPacketsPerLID[i] contains the number of packets to be exported for exportLIDs[i]. If constantNumPackets is nonzero, you should use that instead, and not rely on numPacketsPerLID[i] being filled.
constantNumPackets[out] On exit, 0 if numPacketsPerLID has variable contents (different size for each LID). If nonzero, then it is expected that num-packets-per-LID is constant, and constantNumPackets holds that value.
distor[in] The Distributor object we are using.

Implemented in Tpetra::CrsGraph< LocalOrdinal, GlobalOrdinal, Node, LocalMatOps >, Tpetra::CrsMatrix< Scalar, LocalOrdinal, GlobalOrdinal, Node, LocalMatOps >, Tpetra::MultiVector< Scalar, LocalOrdinal, GlobalOrdinal, Node >, Tpetra::VbrMatrix< Scalar, LocalOrdinal, GlobalOrdinal, Node, LocalMatOps >, Tpetra::CrsMatrix< Scalar, LocalOrdinal, GlobalOrdinal, Node, SpMatOps >, and Tpetra::CrsMatrix< Scalar, LocalOrdinal, GlobalOrdinal, Node >.

template<class Packet, class LocalOrdinal = int, class GlobalOrdinal = LocalOrdinal, class Node = Kokkos::DefaultNode::DefaultNodeType>
virtual void Tpetra::DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node >::unpackAndCombine ( const Teuchos::ArrayView< const LocalOrdinal > &  importLIDs,
const Teuchos::ArrayView< const Packet > &  imports,
const Teuchos::ArrayView< size_t > &  numPacketsPerLID,
size_t  constantNumPackets,
Distributor distor,
CombineMode  CM 
) [protected, pure virtual]

Perform any unpacking and combining after communication.

Parameters:
importLIDs[in] List of the entries (as LIDs in the destination object) we received from other images.
imports[in] Buffer containing data we received.
numPacketsPerLID[in] numPacketsPerLID[i] contains the number of packets imported for importLIDs[i].
constantNumPackets[in] If nonzero, then numPacketsPerLID is constant (same value in all entries) and constantNumPackets is that value. If zero, use numPacketsPerLID[i] instead.
distor[in] The Distributor object we are using.
CM[in] The combine mode to use when combining the imported entries with existing entries.

Implemented in Tpetra::CrsGraph< LocalOrdinal, GlobalOrdinal, Node, LocalMatOps >, Tpetra::CrsMatrix< Scalar, LocalOrdinal, GlobalOrdinal, Node, LocalMatOps >, Tpetra::MultiVector< Scalar, LocalOrdinal, GlobalOrdinal, Node >, Tpetra::VbrMatrix< Scalar, LocalOrdinal, GlobalOrdinal, Node, LocalMatOps >, Tpetra::CrsMatrix< Scalar, LocalOrdinal, GlobalOrdinal, Node, SpMatOps >, and Tpetra::CrsMatrix< Scalar, LocalOrdinal, GlobalOrdinal, Node >.

template<class Packet, class LocalOrdinal = int, class GlobalOrdinal = LocalOrdinal, class Node = Kokkos::DefaultNode::DefaultNodeType>
virtual void Tpetra::DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node >::createViews ( ) const [inline, protected, virtual]

Hook for creating a const view.

doTransfer() calls this on the source object. By default, it does nothing, but the source object can use this as a hint to fetch data from a compute buffer on an off-CPU device (such as a GPU) into host memory.

Reimplemented in Tpetra::MultiVector< Scalar, LocalOrdinal, GlobalOrdinal, Node >.

Definition at line 358 of file Tpetra_DistObject.hpp.

template<class Packet, class LocalOrdinal = int, class GlobalOrdinal = LocalOrdinal, class Node = Kokkos::DefaultNode::DefaultNodeType>
virtual void Tpetra::DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node >::createViewsNonConst ( Kokkos::ReadWriteOption  rwo) [inline, protected, virtual]

Hook for creating a nonconst view.

doTransfer() calls this on the destination (*this) object. By default, it does nothing, but the destination object can use this as a hint to fetch data from a compute buffer on an off-CPU device (such as a GPU) into host memory.

Parameters:
rwo[in] Whether to create a write-only or a read-and-write view. For Kokkos Node types where compute buffers live in a separate memory space (e.g., in the device memory of a discrete accelerator like a GPU), a write-only view only requires copying from host memory to the compute buffer, whereas a read-and-write view requires copying both ways (once to read, from the compute buffer to host memory, and once to write, back to the compute buffer).

Reimplemented in Tpetra::MultiVector< Scalar, LocalOrdinal, GlobalOrdinal, Node >.

Definition at line 375 of file Tpetra_DistObject.hpp.

template<class Packet, class LocalOrdinal = int, class GlobalOrdinal = LocalOrdinal, class Node = Kokkos::DefaultNode::DefaultNodeType>
virtual void Tpetra::DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node >::releaseViews ( ) const [inline, protected, virtual]

Hook for releasing views.

doTransfer() calls this on both the source and destination objects, once it no longer needs to access that object's data. By default, this method does nothing. Implementations may use this as a hint to free host memory which is a view of a compute buffer, once the host memory view is no longer needed. Some implementations may prefer to mirror compute buffers in host memory; for these implementations, releaseViews() may do nothing.

Reimplemented in Tpetra::MultiVector< Scalar, LocalOrdinal, GlobalOrdinal, Node >.

Definition at line 387 of file Tpetra_DistObject.hpp.


Member Data Documentation

template<class Packet, class LocalOrdinal = int, class GlobalOrdinal = LocalOrdinal, class Node = Kokkos::DefaultNode::DefaultNodeType>
Teuchos::RCP<const Map<LocalOrdinal,GlobalOrdinal,Node> > Tpetra::DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node >::map_ [protected]

The Map over which this object is distributed.

Definition at line 390 of file Tpetra_DistObject.hpp.


The documentation for this class was generated from the following file:
 All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Friends Defines