I was surprised that this function was so difficult to find. I googled with the title of this post and did not find any real solution to what I was looking for. I just want to add some pixels of padding to my image. With this post I hope to make life a bit easier for ppl that are looking to do the same.
The function you are looking for is cv::copyMakeBorder :)
Joel's Blog 
Life became a LOT easier. I just hope this post manages to climb a bit higher on the Google search hits. Thanks anyways.
Doing it by hand-coding kinda takes away the whole point of using an optimized library.