HybridInfer preprint finds routing helps mobile AI handle long queries

An arXiv preprint reports that HybridInfer, a system that sends language-model queries to a phone, a nearby server or the cloud, improved mobile inference in a 210-prompt Android benchmark. It chose where to run each query using the phone's thermal headroom and an estimate of the query's complexity. The system produced higher-quality answers than two hand-tuned routing methods at the lowest cost among the adaptive methods tested. Running every query on the phone matched its answer quality for queries the phone could handle, but took three to six times longer and failed on long queries. The author also observed the phone's GPU software crash or stall after a few consecutive queries, even when the device was cool.