Optimizing and Debugging Distributed Systems with Declarative Languages
Distributed systems are pervasively adopted in recent years to keep up with the ever-increasing needs of data-oriented applications. The increasingly demanding requirements of scalability, availability, and reliability bring tremendous challenges and opportunities to distributed systems research. Over the decades, people have proposed and implemented numerous distributed protocols and systems to solve emerging challenges. As more components are integrated into new systems, the complexity of these systems increases at a high pace. The inter-components overhead starts to impact the overall system performance as system complexity rises. At the same time, more complex systems tend to fail more frequently. The elevated cost and failure rate call for urgent research attention.In this dissertation, we investigate novel methods to optimize and debug distributed systems with declarative languages. We first design and implement DeDoS, a distributed runtime platform that manages system resources in a fine-granularity fashion. Applications deployed on DeDoS are split into multiple Minimum Splittable Units (MSUs), where each MSU is separately replicable anyplace in the system that has enough resources.With the runtime platform, we then focus on optimizing resource utilization for distributed programs written in Network Datalog (NDlog). NDlog proposes a straightforward approach for engineers to write distributed programs in a declarative language with a remarkably concise yet expressive format. Prior works have shown the effectiveness of NDlog in implementing complex distributed protocols. We propose and implement a compiler that automatically translates NDlog programs into MSUs, and schedules and potentially replicates MSUs according to the workloads observed at runtime such that system resources are prioritized to the portion of the program that has the highest demand. One particular challenge that we solve is to strategically place MSUs such that we can minimize the communication cost. We present an algorithm that solves the optimal MSU placement problem by reducing to a min-cut problem on constructed NDlog execution graphs.With increased scale and more complex protocols, debugging distributed systems also becomes a daunting task. Prior work has shown that network provenance significantly assists network administrators in diagnosing and debugging network systems by explaining system behavior and revealing the dependencies between system states. We extend existing network provenance systems to preserve privacy. We examine the privacy aspect of network provenance by proposing adversary models, designing encryption schemes with Searchable Symmetric Encryption, implementing the privacy-preserving network provenance system, and analytically and experimentally evaluating the security and performance of the proposed scheme.
MetadataShow full item record
Showing items related by title, author, creator and subject.