Yeah.... This problem bothers me, especially when improvising.

When switching strings I often get lost, I mix up all the intervals

... I do know all the notes on the E & A strings, but I have to think a lot until I found some note on the other string....
Any advice? How to get familiar with the fretboard?