Calculating Geographic Distance

Any geographical analysis almost invariably involves calculating distance between two points on the surface of the Earth (or two geographic coordinates). Below are four different options/formulas to calculate geographic distance. Each formula makes different assumption about the Earth’s shape and thereby has different accuracy and computational complexity.

Formula 1: Pythagorus’ distance formula

distance = ( (lat2 – lat1) ** 2 + (lng2 – lng1) ** 2) ** 0.5

where
lat1,lat2, lng1, lng2 – lat/lng in decimal degrees

Assumption: Earth is flat.
Notes about error: Although, the Earth is not flat, over a very small distance it can be reasonably assumed to be flat. Thus, Pythagorus formula can be used to calculate geographic distance between two geographic coordinates when two geographic coordinates are close to each other. However, the definition of ‘close’ remains ambiguous. I haven’t found any numerical value to define ‘close’ and identifying that value is an aspect of my next post.

Formula 2: Modified Pythagorus’ distance formula

distance = (
[69.1 * (lat2 – lat1)] ** 2 +
[53.0 * (lng2 – lng1)] ** 2
)** 0.5

Notes: This formula is same as Pythagorus’ formula but includes correction for the spherical surface of the Earth. Again, this is not very accurate formula and should be used only for small distances

Formula 3: Great Circle Distance Formula

distance = R * arccos( [sin(lat1) * sin(lat2)] + [cos(lat1) * cos(lat2) * cos(lon2-lon1)] )

where
R = radius of the Earth (equatorial radius)
lat1, lat2, lon1, lon2 – latitude/longitude in radians.

Assumption: Earth is spherical
Notes about error: One of the problem with Great Circle formula is that cos(x) value tends to be unreliable. For example, see below (Ref)

cos (5 degrees) = 0.996194698
cos (1 degree) = 0.999847695
cos (1 minute) = 0.9999999577
cos (1 second) = 0.9999999999882
cos (0.05 sec) = 0.999999999999971

As can be seen from above values, the difference between cos(1 minute) and cos (1 sec) starts to appear only after 8 decimal places. Thus, make sure to use datatype that has sufficient long to capture these minute differences. In Java, always use “double” (and not float) to perform these calculations.

Formula 4: Haversine Formula

d = R * 2 arcsin ( sqrt[ sin_square(dlat/2) + cos(lat1)*cos(lat2)*sin_square(dlon/2)] )

where
R = radius of the Earth
dlat = lat2 – lat1
dlon = lon2 – lon1
lat1, lat1, lon1, lon2 are in radians

Assumption: Earth is spherical, but includes some correction for flattening around poles
Notes about error: Haversine formula is also based on spherical model of the Earth but is considered more accurate than the Great Circle Formula.

Formula 5: Vincety’s Algorithm

Reference to Vincety’s algorithm

Assumption: Earth is ellipsoidal.
Notes about error: Vincety’s algorithm is the most accurate formula to calculate distance between two geographic coordinates. Based on empirical evidences, it has been shown to be accurate within 0.5 mm. However, Vincety’s algorithm is an iterative distance calculation algorithm and thereby, as compared to any of the above formula, it takes longer time.

References:
1. Vincenty’s Algorithm – http://www.movable-type.co.uk/scripts/latlong-vincenty.html
2. Modified Pythagorus Formula – http://www.meridianworlddata.com/Distance-Calculation.asp