Tail latency matters more than average latency. A system with a 1-microsecond mean and a 1-millisecond 99th percentile is a system that loses money on the worst 1% of events, which are often the most valuable ones.
Why it matters even without prediction: queue position in the order book is determined by arrival time, so being faster earns better fills on identical forecasts.
The sources: network propagation, serialisation, kernel and application processing, and garbage collection or allocation pauses - which is why hot paths avoid dynamic allocation entirely.