Bit error rate estimation for improving jitter testing of high-speed serial links