Thesis
QOS support for scientific workflows using software-defined storage resource enclaves
Washington State University
Master of Science (MS), Washington State University
2018
Handle:
https://hdl.handle.net/2376/102188
Abstract
Data-intensive knowledge discovery requires scientific applications to run concurrently with analytics and visualization codes, executing in situ for timely output inspection and knowledge extraction. Consequently, I/O pipelines for large scientific data analysis can be long and complex because they comprise many "stages" of analytics across different layers of the I/O stack of high-performance computing systems. Performance limitations at any I/O layer or stage can cause an I/O bottleneck resulting in longer than expected end-to-end I/O latency. The causes of such performance issues are missing a performance guarantee (e.g., lower bounds of I/O throughput) across stages of I/O pipelines and across layers of the I/O stacks. In this research, we present the design and implementation of a novel data management infrastructure called Software-defined Storage Resource Enclaves (SIREN) at system levels to enforce end-to-end policies that dictate an I/O pipeline's performance. SIREN provides an I/O performance interface for users to specify the desired storage resources in the context of in-situ analytics. If a suboptimal performance of analytics is caused by an I/O bottleneck when data are transferred between simulations and analytics, schedulers in different layers of the I/O stack automatically provide the guaranteed lower bounds on I/O throughput. Additional resources should be allocated to coupled analytics having higher I/O priority, which can be either specified by users via the interface or determined by the schedulers at runtime. Our results demonstrate that SIREN provides performance isolation among scientific workflows sharing multiple storage servers across two I/O layers while maintaining high system scalability and resource utilization.
Metrics
31 File views/ downloads
27 Record Views
Details
- Title
- QOS support for scientific workflows using software-defined storage resource enclaves
- Creators
- Suman Karki
- Contributors
- Xuechen Zhang (Degree Supervisor)
- Awarding Institution
- Washington State University
- Academic Unit
- Electrical Engineering and Computer Science, School of
- Theses and Dissertations
- Master of Science (MS), Washington State University
- Publisher
- Washington State University; [Pullman, Washington] :
- Identifiers
- 99900524804201842
- Language
- English
- Resource Type
- Thesis