Savitha Viswanadh Kandala

Large language models have attracted a significant recent interest, with an overwhelming efforts and emphasis on scaling their parameter size to support general-purpose and emergent capabilities. However, memory and processing requirements for model inference also scales proportionally with the model’s parameter size. This makes it challenging for state-of-the-art, larger models to perform inference locally on edge and mobile devices, even though such models could benefit numerous tasks on these systems. Consequently, these devices are often restricted to accessing larger models through network calls. This approach introduces challenges related to increased inference latency due to network delays and provider capacity. It also raises concerns about sharing private information with third-party vendors. To address these issues, we are developing a system called OTTER. This system tackles the particular problem by enabling the training of smaller yet highly capable foundational language models. As a result of the reduced parameter size, these models can run locally even on constrained edge devices, such as mobile phones and wearables, and are able to provide low-latency responses compared with their larger remotely hosted counterparts. We present our ongoing work describing the framework as applied to training a smaller foundational model for embedded sensing application for tracking a person’s breathing with high accuracy comparable to models orders of magnitude larger in size. Our results demonstrate that carefully pre-trained and fine-tuned smaller sized models outperform much larger counterparts for some tasks, while inferring locally on the constrained edge and mobile devices.