Why is tracking an IP address Location so Flaky!?

So I know most of the time when you are seeing a location of an IP address that viewed your page that you might not actually be seeing the location of the viewer but (what I am guessing) is the location of your Internet Service Provider (ISP). Am I right? For example, someone views your page who has At&t in Indiana but there IP adress says New York when plugged into a Geographic IP tracker. is there a way to know atleast what city the person is VIEWING from?