When assessing uncertainty in model predictions, it is key to consider potential error patterns in some regions of the feature space. In this paper, we build on quantile regression to propose a new method to produce prediction intervals in regression tasks. It estimates a conditional quantile function of the residual variable given a specific representation. The method then adjusts the regressor’s prediction with an upper and lower conditional quantile prediction in order to produce an adaptive prediction interval for any new input. Further, we suggest an additional layer based on conformal prediction in order to provide coverage guarantees. Lastly, as distribution-free conditional coverage is impossible to achieve, we suggest a tree-based representation which displays patterns of undercoverage. This diagnostic tool aims to reveal which regions of the feature space are significantly less likely to have trustworthy prediction intervals. In order to prove their efficacy, our techniques are tested over various use cases and compared against four main baselines. Our methods empirically achieve the expected coverage and tend to produce shorter intervals.