The steady increase in the number of patients equipped with mechanical heart support implants, such as left ventricular assist devices (LVAD), along with virtually ubiquitous 24/7 internet connectivity coverage is motive to investigate and develop remote patient monitoring. In this study we explore machine learning approaches to infection severity recognition on driveline exit site images. We apply a U-net convolutional neural network (CNN) for driveline tube segmentation, resulting in a Dice score coefficient of 0.95. A classification CNN is trained to predict the membership of one out of three infection classes in photographs. The resulting accuracy of 67% in total is close to the measured expert level performance, which indicates that also for human experts there may not be enough information present in the photographs for accurate assessment. We suggest the inclusion of thermographic image data in order to better resolve mild and severe infections.