Communications

QUESTIONS	XPRESS	TG X-Stack	DEGAS	D-TEC	DynAX	X-TUNE	GVR	CORVETTE	SLEEC	PIPER
PI	Ron Brightwell	Shekhar Borkar	Katherine Yelick	Daniel Quinlan	Guang Gao	Mary Hall	Andrew Chien	Koushik Sen	Milind Kulkarni	Martin Schulz
What are the communication "primitives" that you expect to emphasize within your project? (e.g. two-sided vs one-sided, collectives, topologies, groups) Do we need to define extensions to the traditional application level interfaces which now emphasize only data transfers and collective operations? Do we need atomics, remote invocation interfaces, or should these be provided ad-hoc by clients?	The communication primitive is based on the “parcel” protocol that is an expanded form of active-message and that operates within a global address space distributed across “localities” (approx. nodes). Logical destinations are hierarchical global names, actions include instantiations of threads and ParalleX processes (spanning multiple localities), data movement, compound atomic operations, and OS calls. Continuations determine follow-on actions. Payload conveys data operands and block data for moves. Parcels are an integral component of the semantics of the ParalleX execution model providing symmetric semantics in the domain of asynchronous distributed processing to local synchronous processing on localities (nodes).	(TG)	The GASNet-EX communications library will provide Active Messages, one-sided data-movement and collectives as its primary communications primitives, with atomic operations being secondary (over AM initially, and natively as time permits). These are used in various ways by the multiple programming models under study by the DEGAS project. The Active Message primitives in particular support the asynchronous/remote invocation operations present in both the Habanero and UPC++ efforts, while atomics will provide efficient point-to-point synchronization.	The primary focus is computation via asynchronous tasks. The primary communication primitive is (reliably delivered) fire-and-forget active messages. Higher level behavior (finish, at) is synthesized by the APGAS runtime ontop of the active message primitive. However, for performance at scale we recognize the importance of additional primitives: both non-blocking collectives and one-sided asynchronous RDMA's for point-to-point bulk data transfer.	(DynAX)	(X-TUNE)	(GVR)	(CORVETTE)	SLEEC	Communication will be out of band and need to be isolated, emphasis on streaming communication
Traditional communication libraries (e.g. MPI or GASNet) have been developed without tight integration with the computation "model". What is your strategy for integrating communication and computation to address the needs of non SPMD execution?	It is important for performance portability that most communication related codes comprise invariants that will always be true. Aggregation, routing, order sensitivity, time to arrival, and error management should be transparent to the user. Destinations, in most cases, should be relative to placement of first class objects for adaptive placement and routing. Scheduling should tolerate asynchrony uncertainty of message delivery without forfeit of performance assuming sufficient parallelism.	(TG)	(DEGAS)	It is not clear to us that tight integration of communication libraries and the computation model is needed to support non SPMD execution. The X10/APGAS runtime supports non-SPMD execution at scale while maintaining a fairly strict separation between the communication layer (X10RT) and the computational model. X10RT provides basic active message facilities, but all higher-level computational model concerns are handled above the X10RT layer of the runtime. However, there are certainly opportunities to optimize some of the small "control" messages sent over the X10RT transport by the APGAS runtime layer by off-loading pieces of runtime logic into message handlers that could run directly within the network software/hardware. Pushing this function down simply requires the network layer to allow execution of user-provided handlers, not a true integration of the computation model into the communication library.	(DynAX)	(X-TUNE)	(GVR)	(CORVETTE)	N/A	N/A
What type of optimizations should be transparently provided by a communication layer and what should be delegated to compilers or application developers? What is the primary performance metric for your runtime?	Time to solution of application workload, with minimum energy cost within that scope.	(TG)	(DEGAS)	Under the assumption that the communication layer is not tightly integrated with the computation model, the scope of transparent optimization seems limited to optimizing the flow of traffic within the network. Perhaps also providing performance diagnostics and control points to higher-levels of the runtime to enable them to optimize communication behavior. Optimizations need to be planned/managed at a level of the stack that has sufficient scope to make good decisions.	(DynAX)	(X-TUNE)	(GVR)	(CORVETTE)	N/A	unclear
What is your strategy towards resilient communication libraries?	To first order runtime system assumes correct operation of communication libraries as being pursued by Portals-4 and the experimental Photon communication fabric. Under NNSA PSAAP-2 the Micro-checkpoint Compute-Validate-Commit cycle will detect errors including those due to communication failures.	(TG)	(DEGAS)	This is not an area of research for D-TEC. We are assuming the low-level communication libraries will (at least as visible by our layer) operate correctly or report faults when it does not. Any faults reported by the underlying communication library will be reflected up to higher-levels of the runtime stack.	(DynAX)	(X-TUNE)	(GVR)	(CORVETTE)	N/A	ability to drop and reroute around failed processes
What and how can a communication layer help in power and energy optimizations?	Energy waste on unused channels needs to be prevented. Delays due to contention for hotspots need to be mitigated through dynamic routing. Information on message traffic, granularity, and power needs to be provided to OSR.	(TG)	(DEGAS)	Not a topic of research for D-TEC.	(DynAX)	(X-TUNE)	(GVR)	(CORVETTE)	N/A	N/A
Congestion management and flow control mechanisms are of particular concern at very large scale. How much can we rely on "vendor" mechanisms and how much do we need to address in higher level layers?	Vendor systems can help with redundant paths and dynamic routing. Runtime system data and task placement can attempt to maximize locality for reduced message traffic contention.	(TG)	(DEGAS)	Higher levels should focus on task & data placement to increase locality and reduce redundant communication. Micro-optimization of routing, congestion management, and flow control are probably most effectively handled by system/vendor mechanisms since it may require detailed understanding of the internals of network software/hardware and the available dynamic information.	(DynAX)	(X-TUNE)	(GVR)	(CORVETTE)	N/A	N/A