Latency is one of the most critical performance metrics for a wide range of applications. Therefore, it is important to understand the underlying mechanisms that give rise to the observed latency values and diagnose the ones that are unexpectedly high. In this paper, we study the Internet delay space via robust principal component analysis (RPCA). Using RPCA, we show that the delay space, i.e. the matrix of measured round trip times between end hosts, can be decomposed into two components: the estimated latency between end hosts with respect to the current state of the Internet and the inflation on the paths between the end hosts. Using this decomposition, first we study the well-known low-dimensionality phenomena of the delay space and ask what properties of the end hosts define the dimensions. Second, using the decomposition, we develop a filtering method to detect the paths that experience unexpected latencies and identify routing anomalies. We show that our filter successfully identifies an anomalous route even when its observed latency is not obviously high in magnitude.