To remedy this, go to Dolphin Controller Configuration Window and disable (untick) Enable Speaker Data and this will mute the Wii Remote speaker, fixing the starbit slowdown problem. Wii Remote slows down when collecting starbits. This issue actually also exists on real hardware, so it's not an emulator bug.