Commercial quantum optimizers have been unable so far to demonstrate scalable performance advantages over standard state-of-the-art optimization algorithms. Multiple independent efforts continue to develop the technology, with the belief that larger optimizers with more coherent and densely connected qubits will be necessary to observe a quantum speedup. We point to two fundamental limitations of physical quantum annealing optimizers that we argue prevents them from functioning as scalable optimizers: their finite temperature and their analog nature. We numerically demonstrate that for quantum annealers to find the minimizing configurations of optimization problems of increasingly larger sizes, their temperature and noise levels must be appropriately scaled down with problem size. We discuss the implications of our results to practical quantum annealers.