i dont see how one can taste that residue, as i see it, every dab you take is that flavor, whats left on the dabber is cooked to the same temperature as most of your last hit was, so a tiny bit more of darker oil is no different than what you will mostly inhale on your next hit. i never wipe it off, but many people think it should be wiped off for flavor, i think its just a mind thing, because of the way it looks.
what kind of dabber you use? metal ones, if you dab completely, leave almost no residue, and will not effect flavor of the next dab to any noticeable degree. glass dabbers however seem to always retain a lot of claim.
but i swear that part of every toke will have the same amount of burntness as that left on your dabber, otherwise, it wouldnt be that color after. a little bit of each toke gets dark dabber oil in it, at the end of the dab, just before you stop dabbing, the last bit of vapor will be coming from dark oil just like that left on the pin, unless one has a dabbing style where you just drop the hit and dont touch the dabber again, but then one is guaranteed a lot of residue on the dabber.