Enforcing End-to-End I/O Policies for Scientific Workflows Using Software-Defined Storage Resource Enclaves

Suman Karki; Bao Nguyen; Joshua Feener; Kei Davis; Xuechen Zhang

doi:10.1109/TMSCS.2018.2879096

Back

Enforcing End-to-End I/O Policies for Scientific Workflows Using Software-Defined Storage Resource Enclaves

Journal article

Open access

Peer reviewed

Enforcing End-to-End I/O Policies for Scientific Workflows Using Software-Defined Storage Resource Enclaves

Suman Karki, Bao Nguyen, Joshua Feener, Kei Davis and Xuechen Zhang

IEEE transactions on multi-scale computing systems, Vol.4(4), pp.662-675

10/2018

DOI: https://doi.org/10.1109/TMSCS.2018.2879096

Handle:

https://hdl.handle.net/2376/107956

Files and links (1)

url

https://doi.org/10.1109/TMSCS.2018.2879096View

Published (Version of record) Open

Abstract

Data analysis

software-defined storage

QoS

Semantics

Quality of service

Throughput

Knowledge discovery

Resource management

Software defined networking

scientific workflows

Data-intensive knowledge discovery requires scientific applications to run concurrently with analytics and visualization codes executing in situ for timely output inspection and knowledge extraction. Consequently, I/O pipelines of scientific workflows can be long and complex because they comprise many stages of analytics across different layers of the I/O stack of high-performance computing systems. Performance limitations at any I/O layer or stage can cause an I/O bottleneck resulting in greater than expected end-to-end I/O latency. In this paper, we present the design and implementation of a novel data management infrastructure called Software-Defined Storage Resource Enclaves (SIREN) at system level to enforce end-to-end policies that dictate an I/O pipeline's performance. SIREN provides an I/O performance interface for users to specify the desired storage resources in the context of in-situ analytics. If suboptimal performance of analytics is caused by an I/O bottleneck when data are transferred between simulations and analytics, schedulers in different layers of the I/O stack automatically provide the guaranteed lower bounds on I/O throughput. Our experimental results demonstrate that SIREN provides performance isolation among scientific workflows sharing multiple storage servers across two I/O layers (burst buffer and parallel file systems) while maintaining high system scalability and resource utilization.

Metrics

8 Record Views

Details

Title: Enforcing End-to-End I/O Policies for Scientific Workflows Using Software-Defined Storage Resource Enclaves
Creators: Suman Karki - School of Engineering and Computer Science, Washington State University Vancouver, Vancouver, WA, USA
Bao Nguyen - School of Engineering and Computer Science, Washington State University Vancouver, Vancouver, WA, USA
Joshua Feener - School of Engineering and Computer Science, Washington State University Vancouver, Vancouver, WA, USA
Kei Davis - Los Alamos National Laboratory, Los Alamos, NM, USA
Xuechen Zhang - School of Engineering and Computer Science, Washington State University Vancouver, Vancouver, WA, USA
Publication Details: IEEE transactions on multi-scale computing systems, Vol.4(4), pp.662-675
Academic Unit: Engineering and Computer Science (VANC), School of
Publisher: IEEE
Grant note: CRII ACI 1565338 / US National Science Foundation U.S. Department of Energy (10.13039/100000015)
Identifiers: 99900547645401842
Language: English
Resource Type: Journal article