Dissertation/Thesis Abstract

Enabling efficient parallel scripting on large-scale computers
by Zhang, Zhao, Ph.D., The University of Chicago, 2014, 125; 3627911
Abstract (Summary)

Many-task computing (MTC) applications assemble existing sequential (or parallel) programs, using POSIX files for intermediate data. The parallelism of such applications often comes from data parallelism. MTC applications can be grouped into stages, and dependencies between tasks in different stages can be in the form of file production and consumption. The computation stage of MTC applications can have a large number of tasks, thus it can have a large amount of I/O traffic (metadata traffic and I/O traffic) which is also highly concurrent. Some MTC applications are iterative, where the computation iterates over a dataset and exit when some condition(s) are reached. Some MTC applications are interactive, where the application requires human action between computation stages.

In this dissertation we develop a complete parallel scripting framework called AMFORA, which has a shared in-RAM file system and task execution engine. It implements the multi-read single-write consistency model, preserves the POSIX interface for original applications, and provides an interface for collective data movement and functional data transformation. It is interoperable with many existing serial scripting languages (e.g., Bash, Python). AMFORA runs on thousands of compute nodes on an IBM BG/P supercomputer. It also runs on cloud environments such as Amazon EC2 and Google Compute Engine. To understand the baseline MTC application performance on large-scale computers, we define MTC Envelope, which is a file system benchmark to measure the capacity of a given software/hardware stack in the context of MTC applications.

The main contributions of this dissertation are: A system independent approach to profile and understand the concurrency of MTC applications' I/O behavior; A benchmark definition that measures the file system's capacity for MTC applications; A theoretical model to estimate the I/O overhead of MTC applications on large-scale computers; A scalable distributed file system design, with no centralized component, that achieves good scalability; A collective file system management toolkit to enable fast data movement; A functional file system management toolkit to enable fast file content transformation; A new parallel scripting programming model that extends a scripting language (e.g., Bash); A novel file system access interface design that combines both POSIX and non-POSIX interfaces to ease programming without loss of efficiency; An automated method for identifying data flow patterns that are amenable to collective optimizations at runtime; The open source implementation of the entire framework to enable MTC applications on large-scale computers. (Abstract shortened by UMI.)

Indexing (document details)
Advisor: Foster, Ian T.
Commitee: Katz, Daniel S., Stevens, Rick L.
School: The University of Chicago
Department: Computer Science
School Location: United States -- Illinois
Source: DAI-B 75/11(E), Dissertation Abstracts International
Subjects: Computer science
Keywords: I/O overload, Many-task computing, Parallel scripting, Scalable file systems
Publication Number: 3627911
ISBN: 978-1-321-03412-7
Copyright © 2020 ProQuest LLC. All rights reserved. Terms and Conditions Privacy Policy Cookie Policy