Correctness Tools - Revision history

imported>Admin at 02:53, May 20, 2014

2014-05-20T02:53:17Z

imported>Admin at 02:51, May 20, 2014

2014-05-20T02:51:16Z

← Older revision		Revision as of 02:51, May 20, 2014
Line 35:		Line 35:
	\|sleecanswer=		\|sleecanswer=

	\|~~piperanwser~~=		\|piperanswer=N/A
	}}		}}

Line 62:		Line 62:
	\|sleecanswer=		\|sleecanswer=

	\|~~piperanwser~~=		\|piperanswer=N/A
	}}		}}

Line 87:		Line 87:
	\|sleecanswer=		\|sleecanswer=

	\|~~piperanwser~~=		\|piperanswer=N/A
	}}		}}

Line 113:		Line 113:
	\|sleecanswer=		\|sleecanswer=

	\|~~piperanwser~~=		\|piperanswer=N/A
	}}		}}

Line 138:		Line 138:
	\|sleecanswer=		\|sleecanswer=

	\|~~piperanwser~~=tbd.		\|piperanswer=tbd.
	}}		}}

Line 163:		Line 163:
	\|sleecanswer=		\|sleecanswer=

	\|~~piperanwser~~=		\|piperanswer=N/A
	}}		}}

Line 188:		Line 188:
	\|sleecanswer=		\|sleecanswer=

	\|~~piperanwser~~=		\|piperanswer=N/A
	}}		}}

Line 213:		Line 213:
	\|sleecanswer=		\|sleecanswer=

	\|~~piperanwser~~=Integration with performance analysis tools (similar domains to represent, don't want to burden users with multiple different tools)		\|piperanswer=Integration with performance analysis tools (similar domains to represent, don't want to burden users with multiple different tools)

	}}		}}
Line 239:		Line 239:
	\|sleecanswer=		\|sleecanswer=

	\|~~piperanwser~~=N/A		\|piperanswer=N/A

	}}		}}
Line 265:		Line 265:
	\|sleecanswer=		\|sleecanswer=

	\|~~piperanwser~~=N/A		\|piperanswer=N/A

	}}		}}
Line 293:		Line 293:
	\|sleecanswer=		\|sleecanswer=

	\|~~piperanwser~~=Energy/power monitoring is a performance problem, in particular in power constrained scenarios		\|piperanswer=Energy/power monitoring is a performance problem, in particular in power constrained scenarios

	}}		}}

imported>Admin at 02:47, May 20, 2014

2014-05-20T02:47:33Z

← Older revision		Revision as of 02:47, May 20, 2014
Line 138:		Line 138:
	\|sleecanswer=		\|sleecanswer=

	\|piperanwser=		\|piperanwser=tbd.
	}}		}}

imported>Admin at 02:45, May 20, 2014

2014-05-20T02:45:41Z

← Older revision		Revision as of 02:45, May 20, 2014
Line 240:		Line 240:

	\|piperanwser=N/A		\|piperanwser=N/A

			}}

			<div id="table10"></div>{{
			CorrectnessToolsTable
			\|question=List the testing and debugging challenges that you think the next generation programming languages and models would face?

			\|xpressanswer=The principal challenges are detection and isolation.

			\|tganswer=The challenges will necessarily be on providing information about data placement and data movement. This exist to a certain extent already in analysis for performance, but additional information about energy usage will be very valuable.

			\|degasanswer=

			\|dtecanswer=For the next generation of programming languages (and resiliency issues of next gen exascale hardware) we expect that avoiding errors by verifying properties at compile time will become increasingly important because debugging at run-time will become even more challenging due to resiliency complicating the debugging process in future.

			\|dynaxanswer=We expect the main challenges to be related to tractability of debugging and testing applications that run on millions of cores, and reproducibility of non-deterministic applications on a non-reliable hardware platform.

			\|xtuneanswer=Scalability and determining what is an error seem like the biggest challenges.

			\|gvranswer=

			\|corvetteanswer=

			\|sleecanswer=

			\|piperanwser=N/A

			}}


			<div id="table11"></div>{{
			CorrectnessToolsTable
			\|question=How can correctness tools help with reasoning about energy? How can correctness tools help with resilience?

			\|xpressanswer=With respect to energy, tools that can determine the critical path of execution would provide the basis for energy/power scheduling. As above for reliability, see 10) and other answers.

			\|tganswer=Please see the note above. Related to resilience: simulations will not be deterministic. Tools that allow the develop to distinguish between differing results that stem, from different operation ordering and results that indicate an actual failure would be useful.

			\|degasanswer=

			\|dtecanswer=Correctness tool can identify erroneous paths and can therefore help to identify and remove those paths which otherwise could have a severe impact on the power-reasoning. Any violation of a property at run-time that has been established by a correctness tool to hold for a given execution path (before at compile-time) can be identified to be the consequence of resilience and trigger appropriate recovery actions at run-time.

			\|dynaxanswer=Correctness tools may be able to help the programmer make design choices which impact energy consumption, such as enabling non-determinism in parts of their application.
			Lightweight tools that detect faults are presumably directly applicable to resiliency, for instance by coupling them with checkpointing.

			\|xtuneanswer=Correctness tools can certainly help with resilience, if they have a concept of what is tolerable vs. an error. I don't see a connection to energy optimization.

			\|gvranswer=

			\|corvetteanswer=

			\|sleecanswer=

			\|piperanwser=Energy/power monitoring is a performance problem, in particular in power constrained scenarios

	}}		}}

imported>Admin at 02:40, May 20, 2014

2014-05-20T02:40:49Z

← Older revision		Revision as of 02:40, May 20, 2014
Line 164:		Line 164:

	\|piperanwser=		\|piperanwser=
			}}

			<div id="table7"></div>{{
			CorrectnessToolsTable
			\|question=What is your strategy to debug a DSL and its runtime? What kind of multi-level debugging support do you want for DSLs?

			\|xpressanswer=XPRESS provides the HPX runtime that will incorporate its own set of test and correctness mechanisms under the Microcheckpointing methodology using the Compute-Validate-Commit cycle and built in testing. It is expected that DSL will support the generation of reverse test cases to support this.

			\|tganswer=Very similar strategy to debugging other codes. There would be nothing fundamentally different.

			\|degasanswer=

			\|dtecanswer=For DSLs that are translated and/or lowered to a general purpose language we expect assertions present in the DSL also to be generated in lower-level code, and allowing to relate any failing assertions to points in the original DSL. We expect DSLs to allow for domain-specific assertions in combination with user-provided domain-specific knowledge in form of properties that can then be checked in the generated lower-level code.

			\|dynaxanswer=

			\|xtuneanswer=Do we trust the DSL to translate code correctly? That seems like the fundamental question. The DSL developer should be able to debug the translation, while the DSL user should just be debugging the code that they added.

			\|gvranswer=

			\|corvetteanswer=

			\|sleecanswer=

			\|piperanwser=
			}}

			<div id="table8"></div>{{
			CorrectnessToolsTable
			\|question=What kind of visualization and presentation support for bugs do you want from correctness tools? What kind of IDE integration do you want for the correctness tools?

			\|xpressanswer=Visualization and presentation support to correlate a detected error in terms of physical location, point in code, virtual user thread instantiation, and exact instruction causing the error will be of great value, especially if combined with a control framework for manipulating the execution stream at the offending point for diagnosis and correction.

			\|tganswer=Visualization can be quite simple. A usefule example is the display used in Intel's performance tuning tools like Vtune. IDEs are personal choices as is the choice not to use one, but I would at a minimum select Eclipse for integration.

			\|degasanswer=

			\|dtecanswer=We want a query language that allows to mine any collected data as well as its scalable visualization showing dependencies and attached properties. This could support debugging at multiple levels.

			\|dynaxanswer=Any tool that allows big-picture and detailed views, as in ETI's tracer for SWARM. Tools that don't integrate with a particular IDE end up being compatible with all IDEs, including vi and emacs, which is desirable.

			\|xtuneanswer=Pinpointing code or data that are involved in an bug/bottleneck is helpful in debugging/performance tools.

			\|gvranswer=

			\|corvetteanswer=

			\|sleecanswer=

			\|piperanwser=Integration with performance analysis tools (similar domains to represent, don't want to burden users with multiple different tools)

			}}

			<div id="table9"></div>{{
			CorrectnessToolsTable
			\|question=When combining various languages/runtimes, can your language/runtime be debugged in isolation from the rest of the system?

			\|xpressanswer=Yes.

			\|tganswer=In practice, but this is a goal. It could be called an aspect of separation of concerns.

			\|degasanswer=

			\|dtecanswer=This depends if the DSL is integrated into a general purpose language or a separate language that interfaces with the language run-time of the host language. In the latter case 'yes', otherwise 'no'.

			\|dynaxanswer=The entire execution stack is visible to the programmer at debug time, including the SWARM parts. We do not think that opacity across system parts would bring clarity in the debugging process.

			\|xtuneanswer=N/A -- We use standard languages and run-time support.

			\|gvranswer=

			\|corvetteanswer=

			\|sleecanswer=

			\|piperanwser=N/A

	}}		}}

imported>Admin at 02:33, May 20, 2014

2014-05-20T02:33:37Z

Show changes

imported>Admin at 02:13, May 20, 2014

2014-05-20T02:13:15Z

← Older revision		Revision as of 02:13, May 20, 2014
Line 15:		Line 15:
	<div id="table1"></div>{{		<div id="table1"></div>{{
	CorrectnessToolsTable		CorrectnessToolsTable
	\|question=		\|question=What kind of runtime overhead can you accept when running your application with a dynamic analysis tools such as a data race detector? 1.1X, 01.5X, 2X, 5X, 10X, 100X.

	\|xpressanswer=		\|xpressanswer=Runtime overhead determines task granularity and therefore available parallelism (strong scaling) determining time to solution. Only fractional overheads can be tolerated at runtime.

	\|tganswer=		\|tganswer=That depends upon how much of the application - which parts - can be excluded from analysis during the run, because that portion is trusted. If the problem is already isolated to particular section then a very significant hit to performance is acceptable to pinpoint it. If a single loop nest was already identified and that nest normally ran for a few seconds, the even 100x would be acceptable if it did not delay turnaround for more than a few minutes.

	\|degasanswer=		\|degasanswer=

imported>Admin at 02:10, May 20, 2014

2014-05-20T02:10:46Z

← Older revision		Revision as of 02:10, May 20, 2014
Line 1:		Line 1:
	'''Questions:'''		'''Questions:'''

	* [[#table1\|What ~~policies and/or mechanisms will~~ your ~~runtime use to schedule code and place~~ data ~~for 100M objects (executing code~~, ~~data elements~~, ~~etc~~.~~) in a scalable fashion?~~]]		* [[#table1\|What kind of runtime overhead can you accept when running your application with a dynamic analysis tools such as a data race detector? 1.1X, 01.5X, 2X, 5X, 10X, 100X.]]
	* [[#table2\|What ~~features will allow your runtime~~ to ~~dynamically adapt the schedule and placement for 100K sockets to improve the metrics of code-data affinity~~, ~~power consumption~~, ~~migration cost and resiliency?~~]]		* [[#table2\|What types of bugs do you want correctness tools to detect? Data races, deadlocks, non-determinism.]]
	* [[#table3\|~~How will the runtime manage resources (compute, memory, power, bandwidth)~~ for ~~100K sockets to meet a power, energy and performance objective~~?]]		* [[#table3\|What kind of correctness tools can help you with interactive debugging in a debugger? Do you want those tools to discover bugs for you?]]
	* [[#table4\|~~How does~~ the ~~runtime software itself scale to 100K sockets? Specifically, how does it distribute, monitor~~ and ~~balance itself and how is it resilient to failures~~?]]		* [[#table4\|Current auto-tuning and performance/precision debugging tools treat the program as a black-box. What kind of white-box program analysis techniques can help in auto-tuning your application with floating-point precision and alternative algorithms?]]
	* [[#table5\|What ~~is the efficiency~~ of ~~the runtime~~? ~~Specifically, how much impact does~~ the ~~runtime have on a) the total execution time of the application and b) resources taken from algorithmic computations~~? What ~~are your plans~~ to ~~maximize efficiency~~? ~~How will runtime overhead scale~~ to ~~100K sockets~~?]]		* [[#table5\|What kind of testing strategy do you normally apply while developing your software component? Do you write tests before or after the development process? What kind of coverage do you try to achieve? Do you write small unit tests or large systems tests to achieve coverage? Are your tests non-deterministic?]]
	* [[#table6\|Do you support ~~isolation of the runtime code from~~ the ~~user code to avoid violations and contamination~~?]]		* [[#table6\|After a bug is discovered, do you want automated support to create a simplified test that will reproduce the bug?]]
	* [[#table7\|What ~~specific hardware features~~ do you ~~require~~ for ~~proper or efficient runtime operation (atomics, DMA, F/E bits, etc.)~~?]]		* [[#table7\|What is your strategy to debug a DSL and its runtime? What kind of multi-level debugging support do you want for DSLs?]]
	* [[#table8\|What ~~is your model~~ for ~~execution~~ of ~~multiple different programs (ie: a single mention would be doing more than one thing) in terms of division, isolation, containment and protection~~?]]		* [[#table8\|What kind of visualization and presentation support for bugs do you want from correctness tools? What kind of IDE integration do you want for the correctness tools?]]
	* [[#table9\|~~What is~~ your ~~model for execution of multiple different programs (ie: a single mention would~~ be ~~doing more than one thing)~~ in ~~terms~~ of ~~division, isolation, containment and protection~~?]]		* [[#table9\|When combining various languages/runtimes, can your language/runtime be debugged in isolation from the rest of the system?]]
	* [[#table10\|~~What is your model for execution of multiple different programs (ie: a single mention~~ would ~~be doing more than one thing) in terms of division, isolation, containment and protection~~?]]		* [[#table10\|List the testing and debugging challenges that you think the next generation programming languages and models would face?]]
	* [[#table11\|~~What is your model for execution of multiple different programs (ie: a single mention would be doing more than one thing) in terms of division, isolation, containment and protection~~?]]		* [[#table11\|How can correctness tools help with reasoning about energy? How can correctness tools help with resilience?]]

			<div id="table1"></div>{{
			CorrectnessToolsTable
			\|question=

	~~{{CorrectnessToolsTable~~}}		\|xpressanswer=

			\|tganswer=

			\|degasanswer=

			\|dtecanswer=

			\|dynaxanswer=

			\|xtuneanswer=

			\|gvranswer=

			\|corvetteanswer=

			\|sleecanswer=

			\|piperanwser=
			}}

	{\| class="wikitable"		{\| class="wikitable"

imported>Admin at 02:00, May 20, 2014

2014-05-20T02:00:38Z

← Older revision		Revision as of 02:00, May 20, 2014
Line 1:		Line 1:
			'''Questions:'''

			* [[#table1\|What policies and/or mechanisms will your runtime use to schedule code and place data for 100M objects (executing code, data elements, etc.) in a scalable fashion?]]
			* [[#table2\|What features will allow your runtime to dynamically adapt the schedule and placement for 100K sockets to improve the metrics of code-data affinity, power consumption, migration cost and resiliency?]]
			* [[#table3\|How will the runtime manage resources (compute, memory, power, bandwidth) for 100K sockets to meet a power, energy and performance objective?]]
			* [[#table4\|How does the runtime software itself scale to 100K sockets? Specifically, how does it distribute, monitor and balance itself and how is it resilient to failures?]]
			* [[#table5\|What is the efficiency of the runtime? Specifically, how much impact does the runtime have on a) the total execution time of the application and b) resources taken from algorithmic computations? What are your plans to maximize efficiency? How will runtime overhead scale to 100K sockets?]]
			* [[#table6\|Do you support isolation of the runtime code from the user code to avoid violations and contamination?]]
			* [[#table7\|What specific hardware features do you require for proper or efficient runtime operation (atomics, DMA, F/E bits, etc.)?]]
			* [[#table8\|What is your model for execution of multiple different programs (ie: a single mention would be doing more than one thing) in terms of division, isolation, containment and protection?]]
			* [[#table9\|What is your model for execution of multiple different programs (ie: a single mention would be doing more than one thing) in terms of division, isolation, containment and protection?]]
			* [[#table10\|What is your model for execution of multiple different programs (ie: a single mention would be doing more than one thing) in terms of division, isolation, containment and protection?]]
			* [[#table11\|What is your model for execution of multiple different programs (ie: a single mention would be doing more than one thing) in terms of division, isolation, containment and protection?]]


	{{CorrectnessToolsTable}}		{{CorrectnessToolsTable}}

imported>Admin at 01:56, May 20, 2014

2014-05-20T01:56:26Z

← Older revision		Revision as of 01:56, May 20, 2014
Line 1:		Line 1:

			{{CorrectnessToolsTable}}

	{\| class="wikitable"		{\| class="wikitable"

← Older revision		Revision as of 02:10, May 20, 2014
Line 1:		Line 1:
	'''Questions:'''		'''Questions:'''

	* [[#table1\|What ~~policies and/or mechanisms will~~ your ~~runtime use to schedule code and place~~ data ~~for 100M objects (executing code~~, ~~data elements~~, ~~etc~~.~~) in a scalable fashion?~~]]		* [[#table1\|What kind of runtime overhead can you accept when running your application with a dynamic analysis tools such as a data race detector? 1.1X, 01.5X, 2X, 5X, 10X, 100X.]]
	* [[#table2\|What ~~features will allow your runtime~~ to ~~dynamically adapt the schedule and placement for 100K sockets to improve the metrics of code-data affinity~~, ~~power consumption~~, ~~migration cost and resiliency?~~]]		* [[#table2\|What types of bugs do you want correctness tools to detect? Data races, deadlocks, non-determinism.]]
	* [[#table3\|~~How will the runtime manage resources (compute, memory, power, bandwidth)~~ for ~~100K sockets to meet a power, energy and performance objective~~?]]		* [[#table3\|What kind of correctness tools can help you with interactive debugging in a debugger? Do you want those tools to discover bugs for you?]]
	* [[#table4\|~~How does~~ the ~~runtime software itself scale to 100K sockets? Specifically, how does it distribute, monitor~~ and ~~balance itself and how is it resilient to failures~~?]]		* [[#table4\|Current auto-tuning and performance/precision debugging tools treat the program as a black-box. What kind of white-box program analysis techniques can help in auto-tuning your application with floating-point precision and alternative algorithms?]]
	* [[#table5\|What ~~is the efficiency~~ of ~~the runtime~~? ~~Specifically, how much impact does~~ the ~~runtime have on a) the total execution time of the application and b) resources taken from algorithmic computations~~? What ~~are your plans~~ to ~~maximize efficiency~~? ~~How will runtime overhead scale~~ to ~~100K sockets~~?]]		* [[#table5\|What kind of testing strategy do you normally apply while developing your software component? Do you write tests before or after the development process? What kind of coverage do you try to achieve? Do you write small unit tests or large systems tests to achieve coverage? Are your tests non-deterministic?]]
	* [[#table6\|Do you support ~~isolation of the runtime code from~~ the ~~user code to avoid violations and contamination~~?]]		* [[#table6\|After a bug is discovered, do you want automated support to create a simplified test that will reproduce the bug?]]
	* [[#table7\|What ~~specific hardware features~~ do you ~~require~~ for ~~proper or efficient runtime operation (atomics, DMA, F/E bits, etc.)~~?]]		* [[#table7\|What is your strategy to debug a DSL and its runtime? What kind of multi-level debugging support do you want for DSLs?]]
	* [[#table8\|What ~~is your model~~ for ~~execution~~ of ~~multiple different programs (ie: a single mention would be doing more than one thing) in terms of division, isolation, containment and protection~~?]]		* [[#table8\|What kind of visualization and presentation support for bugs do you want from correctness tools? What kind of IDE integration do you want for the correctness tools?]]
	* [[#table9\|~~What is~~ your ~~model for execution of multiple different programs (ie: a single mention would~~ be ~~doing more than one thing)~~ in ~~terms~~ of ~~division, isolation, containment and protection~~?]]		* [[#table9\|When combining various languages/runtimes, can your language/runtime be debugged in isolation from the rest of the system?]]
	* [[#table10\|~~What is your model for execution of multiple different programs (ie: a single mention~~ would ~~be doing more than one thing) in terms of division, isolation, containment and protection~~?]]		* [[#table10\|List the testing and debugging challenges that you think the next generation programming languages and models would face?]]
	* [[#table11\|~~What is your model for execution of multiple different programs (ie: a single mention would be doing more than one thing) in terms of division, isolation, containment and protection~~?]]		* [[#table11\|How can correctness tools help with reasoning about energy? How can correctness tools help with resilience?]]

			<div id="table1"></div>{{
			CorrectnessToolsTable
			\|question=

	~~{{CorrectnessToolsTable~~}}		\|xpressanswer=

			\|tganswer=

			\|degasanswer=

			\|dtecanswer=

			\|dynaxanswer=

			\|xtuneanswer=

			\|gvranswer=

			\|corvetteanswer=

			\|sleecanswer=

			\|piperanwser=
			}}

	{\| class="wikitable"		{\| class="wikitable"

← Older revision		Revision as of 02:00, May 20, 2014
Line 1:		Line 1:
			'''Questions:'''

			* [[#table1\|What policies and/or mechanisms will your runtime use to schedule code and place data for 100M objects (executing code, data elements, etc.) in a scalable fashion?]]
			* [[#table2\|What features will allow your runtime to dynamically adapt the schedule and placement for 100K sockets to improve the metrics of code-data affinity, power consumption, migration cost and resiliency?]]
			* [[#table3\|How will the runtime manage resources (compute, memory, power, bandwidth) for 100K sockets to meet a power, energy and performance objective?]]
			* [[#table4\|How does the runtime software itself scale to 100K sockets? Specifically, how does it distribute, monitor and balance itself and how is it resilient to failures?]]
			* [[#table5\|What is the efficiency of the runtime? Specifically, how much impact does the runtime have on a) the total execution time of the application and b) resources taken from algorithmic computations? What are your plans to maximize efficiency? How will runtime overhead scale to 100K sockets?]]
			* [[#table6\|Do you support isolation of the runtime code from the user code to avoid violations and contamination?]]
			* [[#table7\|What specific hardware features do you require for proper or efficient runtime operation (atomics, DMA, F/E bits, etc.)?]]
			* [[#table8\|What is your model for execution of multiple different programs (ie: a single mention would be doing more than one thing) in terms of division, isolation, containment and protection?]]
			* [[#table9\|What is your model for execution of multiple different programs (ie: a single mention would be doing more than one thing) in terms of division, isolation, containment and protection?]]
			* [[#table10\|What is your model for execution of multiple different programs (ie: a single mention would be doing more than one thing) in terms of division, isolation, containment and protection?]]
			* [[#table11\|What is your model for execution of multiple different programs (ie: a single mention would be doing more than one thing) in terms of division, isolation, containment and protection?]]


	{{CorrectnessToolsTable}}		{{CorrectnessToolsTable}}