Collaborative Research: PPoSS: Planning: Cross-layer Coordination and Optimization for Scalable and Sparse Tensor Networks (CROSS)

High-dimensional data computation or analytics are gaining importance in many domains, such as quantum chemistry/physics, quantum circuit simulation, brain processing, social networks, healthcare and machine/deep learning, to name a few. Tensors, a representation of high-dimensional data, are playing an increasingly critical role, and so are tensor methods. Tensor decompositions or factorizations of low-dimensional data (three to five dimensions) have been extensively studied over the past years from a high-performance computing and also compiler and computer architecture angles for their computational core operations, while tensor networks targeting very high-dimensional data (over ten dimensions) and extracting physically meaningful latent variables are underdeveloped because of their complicated mathematical nature, extremely high computational complexity, and more domain-dependent challenges. The project’s novelties are manifold: 1) memory heterogeneity-aware representations with algorithm and system optimizations, which could be adopted to solve other problems such as irregular applications and sparse numerical methods; 2) hardware-software co-design of specialized, sparse-tensor network-accelerator architectures, that are among the first hardware implementations of sparse-tensor networks. The project’s impacts are 1) advancing state-of-the-art tensor decomposition studies to model true higher-order and sparse data; 2) triggering a closer long-term collaboration ranging from academia to research labs to industry by studying solicitous applications; 3) bringing appropriate educational opportunities.This project proposes Cross-layer cooRdination and Optimization for Scalable and Sparse-Tensor Networks (CROSS) for heterogeneous systems that are equipped with various types of accelerators, such as GPUs, TPUs and FPGAs, as well as heterogeneous memories with dynamic and non-volatile random-access memories (DRAM+NVRAM). This research aims to study the sparsity in widely used tensor networks by introducing constraints, regularization, dictionaries, and/or domain knowledge for better data compression, faster computation, lower memory usage and better interpretability. Besides the sparsity challenges, sparse-tensor networks also suffer from the curse of dimensionality, aggravated data randomness and irregular program and memory access behaviors. This planning project conducts preliminary research that aims to address these challenges from four perspectives: (1) memory heterogeneity-aware representations and data (re-)arrangement, (2) balanced sparse tensor contraction (SpTC) algorithms with smart page arrangement, (3) memoization and intelligent allocation to reduce computational cost, and (4) specialized accelerator architectures for sparse-tensor networks. The optimized sparse tensor networks will encompass efforts from high-performance computing, algorithms, compilers, computer architecture and performance modeling and will be tested under multiple application scenarios.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.