Resilience - Revision history

imported>Milind at 16:43, May 20, 2014

2014-05-20T16:43:13Z

← Older revision		Revision as of 16:43, May 20, 2014
Line 34:		Line 34:
	\|corvetteanswer=		\|corvetteanswer=

	\|sleecanswer=		\|sleecanswer=We do not create any new mechanisms. However, we might expect some information from lower level mechanisms (e.g., whether a particular library method might be re-executed for resilience purposes) to inform our cost models and optimizations.

	\|piperanswer=Resilience is important for the data gathering and aggregation mechanism. When layered on top of new RTS we require proper notification and controllable fault isolation (e.g., for process failures in aggregation trees). Infrastructures like MRNet already provide basic support for this.		\|piperanswer=Resilience is important for the data gathering and aggregation mechanism. When layered on top of new RTS we require proper notification and controllable fault isolation (e.g., for process failures in aggregation trees). Infrastructures like MRNet already provide basic support for this.
Line 96:		Line 96:
	\|corvetteanswer=		\|corvetteanswer=

	\|sleecanswer=~~N/A~~		\|sleecanswer=We could integrate resilience or accuracy information into our cost models to drive transformations (e.g., to tune library parameters to hit an overall accuracy target)

	\|piperanswer=The aggregation runtime will likely be highly structured (e.g., use tree overlay networks), which could be exploited for resilience		\|piperanswer=The aggregation runtime will likely be highly structured (e.g., use tree overlay networks), which could be exploited for resilience

imported>Admin at 06:23, May 20, 2014

2014-05-20T06:23:17Z

Show changes

imported>Admin at 06:21, May 20, 2014

2014-05-20T06:21:51Z

Show changes

imported>Admin at 06:06, May 20, 2014

2014-05-20T06:06:26Z

Show changes

imported>Admin at 05:58, May 20, 2014

2014-05-20T05:58:27Z

Show changes

imported>Sriram at 07:56, May 13, 2014

2014-05-13T07:56:06Z

← Older revision		Revision as of 07:56, May 13, 2014
Line 14:		Line 14:
	! style="width: 200;" \| PIPER		! style="width: 200;" \| PIPER
	! style="width: 200;" \| Charm++		! style="width: 200;" \| Charm++
			! style="width: 200;" \| EarlyCareer-SriramK
	\|-		\|-
	\| '''PI''' \|\| Ron Brightwell \|\| Shekhar Borkar \|\| Katherine Yelick \|\| Daniel Quinlan \|\| Guang Gao \|\| Mary Hall \|\| Andrew Chien \|\| Koushik Sen \|\| Milind Kulkarni \|\| Martin Schulz \|\| Sanjay Kale		\| '''PI''' \|\| Ron Brightwell \|\| Shekhar Borkar \|\| Katherine Yelick \|\| Daniel Quinlan \|\| Guang Gao \|\| Mary Hall \|\| Andrew Chien \|\| Koushik Sen \|\| Milind Kulkarni \|\| Martin Schulz \|\| Sanjay Kale \|\| Sriram Krishnamoorthy
	\|- style="vertical-align:top;"		\|- style="vertical-align:top;"
	\|'''Describe how your approach to resilience and its dependence on other programming, runtime, or resilience technologies? (i.e. uses lower-level mechanisms from hardware or lower level software, depends on higher level management, creates new mechanisms)'''		\|'''Describe how your approach to resilience and its dependence on other programming, runtime, or resilience technologies? (i.e. uses lower-level mechanisms from hardware or lower level software, depends on higher level management, creates new mechanisms)'''
Line 83:		Line 84:
	\|Resilience is important for the data gathering and aggregation mechanism. When layered on top of new RTS we require proper notification and controllable fault isolation (e.g., for process failures in aggregation trees). Infrastructures like MRNet already provide basic support for this.		\|Resilience is important for the data gathering and aggregation mechanism. When layered on top of new RTS we require proper notification and controllable fault isolation (e.g., for process failures in aggregation trees). Infrastructures like MRNet already provide basic support for this.
	\|The Charm++ resilience support in production use includes in-remote-memory checkpoints and buddy-based automatic failure detection for fail-stop failures. Proactive schemes evacuate a processor when notified of impending failures. The protocols depend on the runtime for monitoring and object migration. Scalable message-logging scheme is also supported where appropriate. The soft error recovery is a work in progress: experimental versions of replication-based strategies are available, while other methods are planned. These depend on over-decomposition provided by the programming models and supported by the RTS.		\|The Charm++ resilience support in production use includes in-remote-memory checkpoints and buddy-based automatic failure detection for fail-stop failures. Proactive schemes evacuate a processor when notified of impending failures. The protocols depend on the runtime for monitoring and object migration. Scalable message-logging scheme is also supported where appropriate. The soft error recovery is a work in progress: experimental versions of replication-based strategies are available, while other methods are planned. These depend on over-decomposition provided by the programming models and supported by the RTS.
			\|We focus on selective localized recovery from faults. This involves tracking ongoing execution progress and identifying the state to be recovered and tasks re-executed to tolerate faults. Dynamic load balancing adapts the execution around faults. We assume the presence of a failure detector, either in hardware or in software. While the recovery techniques do not rely on frequent collective checkpoints, checkpoints act as backstop when the information tracked is insufficient to effect a localized recovery.
	\|- style="vertical-align:top;"		\|- style="vertical-align:top;"
	\|'''One challenging problem for Exascale systems is that projections of soft error rates and hardware lifetimes (wearout) span a wide range from a modest increase over current systems to as much as a 100-fold increase. How does your system scale in resilience to ensure effective exascale capabilities on both the varied systems that are likely to exist and varied operating points (power, error rate)?'''		\|'''One challenging problem for Exascale systems is that projections of soft error rates and hardware lifetimes (wearout) span a wide range from a modest increase over current systems to as much as a 100-fold increase. How does your system scale in resilience to ensure effective exascale capabilities on both the varied systems that are likely to exist and varied operating points (power, error rate)?'''
Line 119:		Line 121:
	\|N/A		\|N/A
	\|The control system observes the error rates and tunes its behavior (parameters for now, and in future, choice of strategies) accordingly. The simplest example is checkpoint periodicity, controlled by observations of fail-stop failures. Message-logging can be used if failures are frequent, because it avoids rolling back all the processors. Object based checksums are used to contain corruption while the object is passive.		\|The control system observes the error rates and tunes its behavior (parameters for now, and in future, choice of strategies) accordingly. The simplest example is checkpoint periodicity, controlled by observations of fail-stop failures. Message-logging can be used if failures are frequent, because it avoids rolling back all the processors. Object based checksums are used to contain corruption while the object is passive.
			\|The schemes can be tuned to meet the resilience needs. In particular, the overhead during normal execution can be decreased at the expense of increased penalty incurred on a failure. The approach can also reduce to checkpointing when the task-parallel phases are too small compared to node/system MTBF.
	\|- style="vertical-align:top;"		\|- style="vertical-align:top;"
	\|'''What opportunities are there to improve resilience or efficiency of resilience by exporting/exploiting runtime or application semantics information in your system?'''		\|'''What opportunities are there to improve resilience or efficiency of resilience by exporting/exploiting runtime or application semantics information in your system?'''
Line 166:		Line 169:
	\|The aggregation runtime will likely be highly structured (e.g., use tree overlay networks), which could be exploited for resilience		\|The aggregation runtime will likely be highly structured (e.g., use tree overlay networks), which could be exploited for resilience
	\|Our system relies on over-decomposition provided by the application, and runtime instrumentation provided by the runtime, to optimize its resiliency protocols. In addition, work is under way to separate application object memory into different compartments requiring different degree of resilience. This requires some support for application and/or compiler.		\|Our system relies on over-decomposition provided by the application, and runtime instrumentation provided by the runtime, to optimize its resiliency protocols. In addition, work is under way to separate application object memory into different compartments requiring different degree of resilience. This requires some support for application and/or compiler.
			\|The resilience techniques are closely related to the runtime and application semantics exposed by the user. In particular, the task-parallel program specification and properties of the task-parallel abstraction allow us to constrain the inter-task and task-data relationships to be tracked and recovered.
	\|- style="vertical-align:top;"		\|- style="vertical-align:top;"
	\|'''What capabilities of provided by resilience researchers (software or hardware) could have a significant impact on the capabilities or efficiency of resilience? Where does resilience fit into the X-stack runtime abstract architecture?'''		\|'''What capabilities of provided by resilience researchers (software or hardware) could have a significant impact on the capabilities or efficiency of resilience? Where does resilience fit into the X-stack runtime abstract architecture?'''
Line 215:		Line 219:

	The need for (and the complexity of) resilience will depend on the direction taken by the exascale hardware (viz. how much reliability gets sacrificed for power-performance). It is desirable, but challenging, to separate the resilience protocols modularly from the rest of the runtime.		The need for (and the complexity of) resilience will depend on the direction taken by the exascale hardware (viz. how much reliability gets sacrificed for power-performance). It is desirable, but challenging, to separate the resilience protocols modularly from the rest of the runtime.
			\|Efficient low-overhead error detection can enable recovery to be performed at various layers of the software stack. Runtime support for resilience is a non-trivial challenge and, if desired, needs to be planned from the beginning.
	\|}		\|}

imported>NikhilJain: Charm++ answers

2014-05-13T04:55:39Z

Charm++ answers

← Older revision		Revision as of 04:55, May 13, 2014
Line 13:		Line 13:
	! style="width: 200;" \| SLEEC		! style="width: 200;" \| SLEEC
	! style="width: 200;" \| PIPER		! style="width: 200;" \| PIPER
			! style="width: 200;" \| Charm++
	\|-		\|-
	\| '''PI''' \|\| Ron Brightwell \|\| Shekhar Borkar \|\| Katherine Yelick \|\| Daniel Quinlan \|\| Guang Gao \|\| Mary Hall \|\| Andrew Chien \|\| Koushik Sen \|\| Milind Kulkarni \|\| Martin Schulz		\| '''PI''' \|\| Ron Brightwell \|\| Shekhar Borkar \|\| Katherine Yelick \|\| Daniel Quinlan \|\| Guang Gao \|\| Mary Hall \|\| Andrew Chien \|\| Koushik Sen \|\| Milind Kulkarni \|\| Martin Schulz \|\| Sanjay Kale
	\|- style="vertical-align:top;"		\|- style="vertical-align:top;"
	\|'''Describe how your approach to resilience and its dependence on other programming, runtime, or resilience technologies? (i.e. uses lower-level mechanisms from hardware or lower level software, depends on higher level management, creates new mechanisms)'''		\|'''Describe how your approach to resilience and its dependence on other programming, runtime, or resilience technologies? (i.e. uses lower-level mechanisms from hardware or lower level software, depends on higher level management, creates new mechanisms)'''
Line 81:		Line 82:
	\|SLEEC		\|SLEEC
	\|Resilience is important for the data gathering and aggregation mechanism. When layered on top of new RTS we require proper notification and controllable fault isolation (e.g., for process failures in aggregation trees). Infrastructures like MRNet already provide basic support for this.		\|Resilience is important for the data gathering and aggregation mechanism. When layered on top of new RTS we require proper notification and controllable fault isolation (e.g., for process failures in aggregation trees). Infrastructures like MRNet already provide basic support for this.
			\|The Charm++ resilience support in production use includes in-remote-memory checkpoints and buddy-based automatic failure detection for fail-stop failures. Proactive schemes evacuate a processor when notified of impending failures. The protocols depend on the runtime for monitoring and object migration. Scalable message-logging scheme is also supported where appropriate. The soft error recovery is a work in progress: experimental versions of replication-based strategies are available, while other methods are planned. These depend on over-decomposition provided by the programming models and supported by the RTS.
	\|- style="vertical-align:top;"		\|- style="vertical-align:top;"
	\|'''One challenging problem for Exascale systems is that projections of soft error rates and hardware lifetimes (wearout) span a wide range from a modest increase over current systems to as much as a 100-fold increase. How does your system scale in resilience to ensure effective exascale capabilities on both the varied systems that are likely to exist and varied operating points (power, error rate)?'''		\|'''One challenging problem for Exascale systems is that projections of soft error rates and hardware lifetimes (wearout) span a wide range from a modest increase over current systems to as much as a 100-fold increase. How does your system scale in resilience to ensure effective exascale capabilities on both the varied systems that are likely to exist and varied operating points (power, error rate)?'''
Line 116:		Line 118:
	\|N/A		\|N/A
	\|N/A		\|N/A
			\|The control system observes the error rates and tunes its behavior (parameters for now, and in future, choice of strategies) accordingly. The simplest example is checkpoint periodicity, controlled by observations of fail-stop failures. Message-logging can be used if failures are frequent, because it avoids rolling back all the processors. Object based checksums are used to contain corruption while the object is passive.
	\|- style="vertical-align:top;"		\|- style="vertical-align:top;"
	\|'''What opportunities are there to improve resilience or efficiency of resilience by exporting/exploiting runtime or application semantics information in your system?'''		\|'''What opportunities are there to improve resilience or efficiency of resilience by exporting/exploiting runtime or application semantics information in your system?'''
Line 162:		Line 165:
	\|N/A		\|N/A
	\|The aggregation runtime will likely be highly structured (e.g., use tree overlay networks), which could be exploited for resilience		\|The aggregation runtime will likely be highly structured (e.g., use tree overlay networks), which could be exploited for resilience
			\|Our system relies on over-decomposition provided by the application, and runtime instrumentation provided by the runtime, to optimize its resiliency protocols. In addition, work is under way to separate application object memory into different compartments requiring different degree of resilience. This requires some support for application and/or compiler.
	\|- style="vertical-align:top;"		\|- style="vertical-align:top;"
	\|'''What capabilities of provided by resilience researchers (software or hardware) could have a significant impact on the capabilities or efficiency of resilience? Where does resilience fit into the X-stack runtime abstract architecture?'''		\|'''What capabilities of provided by resilience researchers (software or hardware) could have a significant impact on the capabilities or efficiency of resilience? Where does resilience fit into the X-stack runtime abstract architecture?'''
Line 208:		Line 212:
	\|N/A		\|N/A
	\|N/A		\|N/A
			\| The first question is unclear, for software. For hardware, some useful capabilities are: quick detection and flexible notification of errors, more robust memory regions for control variables, sensors for early notification of impending component failures.

			The need for (and the complexity of) resilience will depend on the direction taken by the exascale hardware (viz. how much reliability gets sacrificed for power-performance). It is desirable, but challenging, to separate the resilience protocols modularly from the rest of the runtime.
	\|}		\|}

imported>Jsstone1 at 04:07, May 12, 2014

2014-05-12T04:07:42Z

← Older revision		Revision as of 04:07, May 12, 2014
Line 16:		Line 16:
	\| '''PI''' \|\| Ron Brightwell \|\| Shekhar Borkar \|\| Katherine Yelick \|\| Daniel Quinlan \|\| Guang Gao \|\| Mary Hall \|\| Andrew Chien \|\| Koushik Sen \|\| Milind Kulkarni \|\| Martin Schulz		\| '''PI''' \|\| Ron Brightwell \|\| Shekhar Borkar \|\| Katherine Yelick \|\| Daniel Quinlan \|\| Guang Gao \|\| Mary Hall \|\| Andrew Chien \|\| Koushik Sen \|\| Milind Kulkarni \|\| Martin Schulz
	\|- style="vertical-align:top;"		\|- style="vertical-align:top;"
	\|Describe how your approach to resilience and its dependence on other programming, runtime, or resilience technologies? (i.e. uses lower-level mechanisms from hardware or lower level software, depends on higher level management, creates new mechanisms)		\|'''Describe how your approach to resilience and its dependence on other programming, runtime, or resilience technologies? (i.e. uses lower-level mechanisms from hardware or lower level software, depends on higher level management, creates new mechanisms)'''
	\|''XPRESS will employ Micro-checkpointing, which employs a Compute-Validate-Commit cycle bounded by Error-Zones for localized support of error detection and isolation, also diagnosis, correction, and recovery.''		\|XPRESS will employ Micro-checkpointing, which employs a Compute-Validate-Commit cycle bounded by Error-Zones for localized support of error detection and isolation, also diagnosis, correction, and recovery.
	\|(TG)		\|Not in TG scope.
	\|(DEGAS)		\|(DEGAS)
	Our approach to resilience comprises three principal technologies. First,		Our approach to resilience comprises three principal technologies. First,
Line 82:		Line 82:
	\|Resilience is important for the data gathering and aggregation mechanism. When layered on top of new RTS we require proper notification and controllable fault isolation (e.g., for process failures in aggregation trees). Infrastructures like MRNet already provide basic support for this.		\|Resilience is important for the data gathering and aggregation mechanism. When layered on top of new RTS we require proper notification and controllable fault isolation (e.g., for process failures in aggregation trees). Infrastructures like MRNet already provide basic support for this.
	\|- style="vertical-align:top;"		\|- style="vertical-align:top;"
	\|'One challenging problem for Exascale systems is that projections of soft error rates and hardware lifetimes (wearout) span a wide range from a modest increase over current systems to as much as a 100-fold increase. How does your system scale in resilience to ensure effective exascale capabilities on both the varied systems that are likely to exist and varied operating points (power, error rate)?		\|'''One challenging problem for Exascale systems is that projections of soft error rates and hardware lifetimes (wearout) span a wide range from a modest increase over current systems to as much as a 100-fold increase. How does your system scale in resilience to ensure effective exascale capabilities on both the varied systems that are likely to exist and varied operating points (power, error rate)?'''
	\|''Both hardware and software errors are treated in the same manner for error detection, isolation, and diagnosis. Recovery differs if the error is transient, a software bug, or a hardware error. Because the method is localized as in (1), it is believed to be scalable into the billion-way parallelism era. However, this has to be demonstrated through experience. The methods assume a Poisson error distribution.''		\|Both hardware and software errors are treated in the same manner for error detection, isolation, and diagnosis. Recovery differs if the error is transient, a software bug, or a hardware error. Because the method is localized as in (1), it is believed to be scalable into the billion-way parallelism era. However, this has to be demonstrated through experience. The methods assume a Poisson error distribution.''
	\|(TG)		\|Not in TG scope.
	\|(DEGAS)		\|(DEGAS)
	Our resilience technologies provide tremendous flexibility in handling faults.		Our resilience technologies provide tremendous flexibility in handling faults.
Line 117:		Line 117:
	\|N/A		\|N/A
	\|- style="vertical-align:top;"		\|- style="vertical-align:top;"
	\|''What opportunities are there to improve resilience or efficiency of resilience by exporting/exploiting runtime or application semantics information in your system?"		\|'''What opportunities are there to improve resilience or efficiency of resilience by exporting/exploiting runtime or application semantics information in your system?'''
	\|''Application semantics can provide myriad simple tests of correctness that will detect errors early and limit their propagation throughout the computation through better isolation.''		\|Application semantics can provide myriad simple tests of correctness that will detect errors early and limit their propagation throughout the computation through better isolation.
	\|(TG)		\|Not in TG scope.
	\|(DEGAS)		\|(DEGAS)
	In DEGAS, applications express semantic information required for resilience		In DEGAS, applications express semantic information required for resilience
Line 163:		Line 163:
	\|The aggregation runtime will likely be highly structured (e.g., use tree overlay networks), which could be exploited for resilience		\|The aggregation runtime will likely be highly structured (e.g., use tree overlay networks), which could be exploited for resilience
	\|- style="vertical-align:top;"		\|- style="vertical-align:top;"
	\|'''What capabilities of provided by resilience researchers (software or hardware) could have a significant impact on the capabilities or efficiency of resilience? Where does resilience fit into the X-stack runtime abstract architecture?"		\|'''What capabilities of provided by resilience researchers (software or hardware) could have a significant impact on the capabilities or efficiency of resilience? Where does resilience fit into the X-stack runtime abstract architecture?'''
	\|''Numerical correctness of calculations in a non-deterministic scheduling context would benefit from knowledge about floating-point corruption due to order of actions.''		\|Numerical correctness of calculations in a non-deterministic scheduling context would benefit from knowledge about floating-point corruption due to order of actions.
	\|(TG)		\|Not in TG scope.
	\|(DEGAS)		\|(DEGAS)
	Fast, durable storage is a key technology for increasing the efficiency of		Fast, durable storage is a key technology for increasing the efficiency of

imported>Saman at 21:51, May 9, 2014

2014-05-09T21:51:01Z

@@ Line 95: / Line 95: @@
 local CDs for isolated recovery limits the global restart rate
 |(D-TEC)
 |In a recursive tree hierarchy of containment domains, each level will have successively finer grained resilience, with fewer affected tasks, and less cost of redundant computation if the domain restarts due to failure.  Containment domains are isolated from each other, so there is no extra communication / synchronization necessary between them.  The 3 dependencies mentioned above (identification of containment domains, identification of failures, and the cost function) should be equally applicable to any exascale system.  The only hardware dependence we anticipate is the hardware's ability to identify failures as it occurs, and the details of how that failure is reported.  The system can be tuned to act more conservatively, or less conservatively, by ignoring some containment domains and enforcing others.  The most conservative approach is to back up the necessary data for, and check results of, every containment domain in the application.  The least conservative case is to only act on the outer-most containment domain, i.e. simply retain the program image and the parameters it was launched with, and if a failure occurs anywhere in the application, the entire thing is thrown out and starts again from the beginning.  Reality is usually somewhere between those two extremes... the runtime can pick and choose which containment domains to act on, based on cost/benefit analysis (which I will discuss further in the question below on application semantics information).
 |(X-TUNE)
@@ Line 133: / Line 148: @@
 etc. to ensure good efficiency.
 |(D-TEC)
 |The runtime decides whether or not to act on each individual containment domain.  It chooses which containment domains to act upon, based on tuning parameters from the user and based on the data size and the estimated amount of parallel computation involved in completing the work associated with a containment domain.  The data size is easy to determine at runtime, and the execution information can be provided by runtime analysis of previous tasks, or by hints from the application developer or smart compiler.  Using that information, and resilience tuning parameters provided by the user, the runtime can do simple cost/benefit analysis to determine which containment domains are the best ones to act upon to achieve the user's performance/resilience goal.
 |(X-TUNE)
@@ Line 167: / Line 190: @@
 this teardown requirement.
 |(D-TEC)
 |If the hardware can reliably detect and report soft failures (such as errors in floating point instructions), this would avoid running extra application code to check the results, which are sometimes rather costly.  Reliable, slower memory for storing snapshot data would also be beneficial.  This memory would be written with snapshot data every time a containment domain was entered, but only read while recovering from a failure, so the cost of the write is much more important than the cost of the read.
 |(X-TUNE)

imported>Saman at 21:49, May 9, 2014

2014-05-09T21:49:50Z

@@ Line 43: / Line 43: @@
 any form of persistent storage technology is suitable.
 |(D-TEC)
 |The DynAX project will focus on Resilience in year three. The general approach will be to integrate Containment Domains into the runtime and adapt proxy applications to use them.  This will depend on the application developer (or a smart compiler) identifying key points in the control flow where resumption can occur after failure, and identifying the set of data which must be saved and restored in order to do so.  It also relies on some mechanism to determine when a failure has occurred and which tasks were effected. This will vary from one system and type of failure to the next, examples include a hardware ECC failure notification, or a software data verification step detects invalid output, or maybe even a simple timeout occurs.  Finally, it will require a way for the user to provide some criteria for how often to save resilience information off; the runtime may choose to snapshot at every opportunity it is given, or it might only do a subset of those, to meet some power/performance goal.
 |(X-TUNE)