Atherosclerosis of the carotid artery is a major risk factor for stroke. Current studies analyze cross-sections of 3D MR black-blood images to assess the vessel wall of carotid arteries. To increase the reproducibility of quantitative biomarkers such as vessel wall thickness and radiomic features, a reliable automatic segmentation of the vessel wall in these cross-sections is essential. CNN-based segmentation is well established and has been successfully applied for 2D vessel wall and plaque segmentation. We trained a residual U-Net on sparsely sampled cross-sections that are perpendicular to the vessel’s centerline, making our method invariant to the image plane orientation. Due to the well curated training data and the usage of the vessel’s centerline as anatomical prior we are able to achieve a high mean Dice coefficient of 0.946/0.864 for the vessel’s lumen/wall and low mean average contour distance of 0.100/0.116 mm. To prove the model’s flexibility, we show that it is able to segment regions of the carotid artery that are not incorporated in the training data, achieving a similar Dice coefficient, average contour distance and Hausdorff distance. This validates the potential of the method in accurately automating carotid artery wall segmentation for any vessel cross-section. The model is also evaluated on young, healthy subjects and the 2021 Carotid Artery Vessel Wall Segmentation Challenge test set, proving its versatility.