Any geographical analysis almost invariably involves calculating distance between two points on the surface of the Earth (or two geographic coordinates). Below are four different options/formulas to calculate geographic distance. Each formula makes different assumption about the Earth’s shape and thereby has different accuracy and computational complexity.

**Formula 1: Pythagorus’ distance formula**

distance = ( (lat2 – lat1) ** 2 + (lng2 – lng1) ** 2) ** 0.5

where

lat1,lat2, lng1, lng2 – lat/lng in decimal degrees

** Assumption:** Earth is flat.

**Although, the Earth is not flat, over a very small distance it can be reasonably assumed to be flat. Thus, Pythagorus formula can be used to calculate geographic distance between two geographic coordinates when two geographic coordinates are close to each other. However, the definition of ‘close’ remains ambiguous. I haven’t found any numerical value to define ‘close’ and identifying that value is an aspect of my next post.**

*Notes about error:***Formula 2: Modified Pythagorus’ distance formula**

distance = (

[69.1 * (lat2 – lat1)] ** 2 +

[53.0 * (lng2 – lng1)] ** 2

)** 0.5

* Notes:* This formula is same as Pythagorus’ formula but includes correction for the spherical surface of the Earth. Again, this is not very accurate formula and should be used only for small distances

**Formula 3: Great Circle Distance Formula**

distance = R * arccos( [sin(lat1) * sin(lat2)] + [cos(lat1) * cos(lat2) * cos(lon2-lon1)] )

where

R = radius of the Earth (equatorial radius)

lat1, lat2, lon1, lon2 – latitude/longitude in radians.

* Assumption*: Earth is spherical

*One of the problem with Great Circle formula is that cos(x) value tends to be unreliable. For example, see below (Ref)*

**Notes about error:**cos (5 degrees) = 0.996194698

cos (1 degree) = 0.999847695

cos (1 minute) = 0.9999999577

cos (1 second) = 0.9999999999882

cos (0.05 sec) = 0.999999999999971

As can be seen from above values, the difference between cos(1 minute) and cos (1 sec) starts to appear only after 8 decimal places. Thus, make sure to use datatype that has sufficient long to capture these minute differences. In Java, always use “double” (and not float) to perform these calculations**. **

**Formula 4: Haversine Formula**

d = R * 2 arcsin ( sqrt[ sin_square(dlat/2) + cos(lat1)*cos(lat2)*sin_square(dlon/2)] )

where

R = radius of the Earth

dlat = lat2 – lat1

dlon = lon2 – lon1

lat1, lat1, lon1, lon2 are in radians

Assumption: Earth is spherical, but includes some correction for flattening around poles

Notes about error: Haversine formula is also based on spherical model of the Earth but is considered more accurate than the Great Circle Formula.

**Formula 5: Vincety’s Algorithm**

Reference to Vincety’s algorithm

* Assumption:* Earth is ellipsoidal.

*Vincety’s algorithm is the most accurate formula to calculate distance between two geographic coordinates. Based on empirical evidences, it has been shown to be accurate within 0.5 mm. However, Vincety’s algorithm is an iterative distance calculation algorithm and thereby, as compared to any of the above formula, it takes longer time.*

**Notes about error:****References:**

1. Vincenty’s Algorithm – http://www.movable-type.co.uk/scripts/latlong-vincenty.html

2. Modified Pythagorus Formula – http://www.meridianworlddata.com/Distance-Calculation.asp

This is extremely helpful info!!! Especially since you guys are offering it for free!! Very good listing. Everything is true. Thanx.

Great Post

This is all great, and I have actually been using the Haversine formula for calculating distances between points. I simply didn’t need the overhead for the increased accuracy of Vincenty’s. But what I’m now looking for is a way to calculate the distance of a point from a line between 2 geographic points. I have to (lat,lon) coordinates and I need to determine if a given point (late,lonx) is within a given distance of that line. So far, I’m stumped and can find no real answers on the google machine.🙂

Any ideas?

Pingback: Calculating Distance in Miles via SQL Query - Coding Times of a .Net Developer from Yard - Site Home - MSDN Blogs

Pingback: Calculating Distance in Miles via SQL Query | CmdExec Technology News