are they a problem per say or is it just when they build up?
No, they aren't a problem.
Unless they build up to a very large amount, in which case you may get a very slightly longer boot-up time. You would have to be an avid install/uninstaller of lots of programmes with poor uninstallers and not reformat/restore for a long time for it to build up to a problem level, and even then the "problem" would be barely noticable.