fix: latency tests (#685)

Improve logic in latency tests as they might fail on busy machines
as the actual latency is longer than expected.