Changes between Version 1 and Version 2 of datavali

Timestamp:

Jun 23, 2011, 2:23:17 PM (13 years ago)

Author:

dennisw

Comment:

--

datavali

@@ -1 +1 @@
 = Data validation =
-There is a fairly large amount of data involved in this project. It's not uncommon for data to be incomplete. Possible scenarios will be listed here, together with their possible solutions. Let's assume for every scenario you have measured a 'regular and complete' round.
+There is a fairly large amount of data involved in this project. It's not uncommon for data to be incomplete. Possible scenarios will be listed here, together with their possible solutions. Let's assume for every scenario you have measured a 'regular and complete' round, and that your measurement equipment is fully functional (in other words, no half-broken GPS equip or such).
 
-== Missing coordinate(s) ==
+== Invalid coordinate(s) ==
+=== Detecting ===
+The detection of invalid coordinates doesn't have to be very complicated. Values as '0.000' or 'null' should be easy to catch. The problem lies within 'valid but invalid' values. Say you have:
+{{{
+latitude, longitude
+52.1000, 4.1000
+52.2000, 100.0000
+52.3000, 4.2000
+}}}
+It's obvious the longitude '100.0000' is out of place here. Since this project's focus is Leiden, a boundary could be set. Every value that exceeds that boundary can be marked as invalid.
+
+The other problem could be that a value could be invalid, but still be inside the boundary:
+{{{
+latitude, longitude (range 52-4/55-7)
+52.1000, 4.1000
+54.5000, 4.2000
+52.2000, 4.3000
+}}}
+The second latitude seems invalid, but is in this case still within the valid range.
+It might be possible to spot this value though. You could calculate an average offset in lat/lon, and everything that exceeds the average could be marked as invalid. The catch is that this might be heavy cpu-wise since a lot of calculation is needed.
+
+=== Solutions ===
 Let's say there is a missing coordinate like the following:
 {{{
-'''latitude, longitude'''
+latitude, longitude
 52.1000, 4.1000
-52.2000, null
+52.2000, invalid value
 52.3000, 4.2000
 }}}
@@ -14 +35 @@
 A harder case like this might need some thinking:
 {{{
-'''latitude, longitude'''
+latitude, longitude
 52.1000, 4.1000
-null, null (100 rows)
+invalid values (100 rows)
 53.1000, 5.1000
 }}}
@@ -22 +43 @@
 For large gaps like these, it's hard to calculate an expected route. Even if within these missing values a random value would be measured (like a 10 missing/1 valid ratio), it might be wise to just ignore these values since it's hard to get a correct route on long distances.
 
-Say we still have these 100 missing rows, but your first coordinate is at the start of a street, and the last at the end of that same street. This is more likely to occur when you measure at fair intervals.
+Say we still have these 100 missing rows, but your first coordinate is at the start of a street, and the last at the end of that same street. This is more likely to occur when you measure at fair intervals. In this case, it can't hurt to calculate the estimated route. However, if the street has an 'L' shape which you followed, coordinates are likely to intersect with houses and such.
+
+== Invalid signal strength ==
+You might encounter missing or invalid signal values. Let's assume the following is measured from the same accespoint. (right now, signal strength is '100 + signal_dbm')
+{{{
+signal_dbm, strength %
+-80, 20%
+invalid values
+-50, 50%
+}}}
+Again, it might be wise to calculate an average to replace the invalid values. But say, someone else measures the same accespoint around the same location, and he receives valid values. Our average would be 45%, but he get's a 90%, or maybe a 5%.
+For this, it might be better to look at the history of the accespoint (assuming there is one, if not, there probably will be one in time). You could take the most recent dbms measured at around the same location:
+{{{
+signal_dbm (older), strength % (older)
+-75, 25%
+-90, 10%
+-60, 40%
+}}}
+The first and third value don't show a lot of difference, so it should be fairly safe to take the old second value, and use it for the new measurement.