Socially Assistive Robotics (SAR) aims to provide robot-assisted therapy, for physical as well as cognitive rehabilitation. The paper analyzes two distinct use cases of cognitive rehabilitation therapies, one among involving children with Traumatic Brain Injury (TBI); and another one; second among involving individuals with Intellectual Disability (ID), and raises concerns regarding emotional adaptation, personalization, design, and ELS issues of human-robot interaction in such cases. The paper's aim is to provide some guidance on how social robots should be designed in order to accommodate emotions in HRI as well as to respect the rights of the persons with disabilities. We argue that it is critically important to address the concerns highlighted in order to empower robots with empathetic behavior and to deliver effective cognitive rehabilitation therapies.