In the general framework of kernel machines, the adoption of the hinge loss has become more popular than square loss, also because of computational reasons. Since learning reduces to a linear system of equations, in case of very large tasks for which the number of examples is proportional to the input dimension, the solution of square loss regularization is O(ℓ3), where ℓ is the number of examples, and it has been claimed that learning is unaffordable for large scale problems. However, this is only an upper bound, and in-depth experimental analyses indicate that for linear kernels (or in other cases where the kernel matrix will be sparse or decomposed in a way that is known a priori), regularized least square (RLS) is substantially faster than support vector machine (SVM) both at training and test times. In this paper, we give theoretical results to support those experimental findings by proving that there are conditions under which learning of square loss regularization is Θ(ℓ) even for large input dimensions d for which d ≃ ℓ.