Where is the Real Money in Location Intelligence ?

As applications of location intelligence continue to appear there seems to be a clear profile that suggests the world expects location capabilities to be free or close to free.

Regardless of the arguments that can be made about anything "free" on the internet not really being free, a simple list of the top 5 location capabilities that most people would think of probably looks a bit like this:

1. Maps with detailed local intelligence - free
2. Routing, complete with imagery and pictures of locations - free
3. Find the nearest - free
4. What's around a location - free
5. Where am I - free

If we add into this competent GIS and spatial toolsets like QGIS and JTS along with PostGIS, the sophistication is far higher and the capabilities impressive. But they are still free.

Apps are regularly released and updated with spatial capabilities and they're usually free or cheap and in most cases have been built to bring customers to the core business involved.

Consultants with spatial skills guide projects using location intelligence and many of these have been very successful but they continue to be sideshows. They don't signal a dramatic uptake of location intelligence in the corporate world, just a passing pain alleviated or a simple improvement in one compartment of the company.

There is still a market for professional GIS engines like MapInfo or ESRI but that is still not the immense market that has been projected for the industry. It feels as though we have been on the brink of the projected monetary success with location intelligence for a very long time.

We have extraordinary cloud processing, big data tools, fascinating computational intelligence tools (like neural nets, fuzzy systems and evolutionary computation), GPU compute infrastructures and powerful mobile platforms and still there are no thoroughly location enabled applications in the wild that use these awesome facilities to make money.

There is reason to be optimistic however. I just saw an announcement that Alibaba, has recognized the importance of location and made an investment. The scale of this Chinese company is impressive. 2 years ago, it had a market value of US$231 billion. It has its own equivalent of Google, Youtube, Amazon, PayPal, cloud computing services and its own SDK that captures location information. It is currently the world's largest retailer and is entering the e-commerce spaces of other countries, starting with India.

Alibaba has invested in a location platform, an announcement that is interesting on the surface but becomes worthy of more careful consideration when the words used are considered. Alibaba refers to location as a "horizontal enabling technology". This is where location needs to be and it seems from the sparse information available that the massive Alibaba group may have plans to use location throughout their entire organization and use it in a powerful manner. They have the data, the tools, the skills and the funds and all under one company.

Perhaps this is one to watch and perhaps this is the group that will lead the way to a massively successful uptake of spatial analytics.

Why World Wide Geocoding should be Free

All of my career I have worked on the side of the vendors - working with folks who build stuff and sell it to customers. That complicated space between the creation of product and the needs of the user has always fascinated me. It always seems like such a complicated space. It is fraught with miscommunication, attitude, ego and downright technical obfuscation.

The "customer" thinks they know what they need but in my experience, mostly do not. Their goals and appetites are biased by their understanding of what is possible. And we in the spatial world have done a very poor job of helping customers understand what is actually possible, what could be done and what the benefits might be. There are so many postings out there espousing the dire need for Location Intelligence but if, as a customer, you would like to know what benefits there may be for your organization, its gets vague very quickly. Or you end up paying very large amounts of money to a consultant for him to sell you something you probably don't need or want.

I am, for the first time for me, now working with a group who surely could use spatial analysis on a grand scale and realize what for me are clearly dramatically useful outcomes. Why doesn't an obviously sophisticated and singularly powerful organisation using advanced analytics on a truly massive data lake include spatial in their analytics ?

After a few weeks here, I think I know the answer. What is the absolute first and basic step before any useful spatial analytics can take place ?

Geocoding.

Pretty much all spatial analytics has to start with geometry information embedded alongside the other attributes. But here's the issue. If you try to justify the expense of geocoding in an IT budget, it literally has no useful direct outcome. Sure, you might correct some addresses and you may find redundant records but that's a feeble justification for an expensive and disruptive process. A process that actually loads more data and poses such ugly problems as keeping versions of addresses, understanding when a record needs to be geocoded again, retiring addresses and spatial indices.

Once the data is geocoded, it seems to me that the path to the inclusion of spatial analytics is still complicated but every addition of software, data or combination of both has a viable business justification.

I suggest then that geocoding should be available to all as a fundamental part of whatever IT mechanisms are incumbent. And it should either be free or bundled such that everybody who could use it, can. Using Pig ? There should be a free geocoding UDF for Pig. Hive ? Same thing. Microsoft SQL Server ? The Data Quality Services extension for geocoding should be included. And so forth.

With a free or at least native geocoder present in standard IT deployments we can remove a barrier that stops spatial analysis creeping (or vaulting ?) into enterprise analytics.

Thoughts on Pokémon Go as a Location Based Game

The news today (11 July 2016) is full of dramatic numbers documenting the success of Nintendo's Pokémon Go. The game was released in Australia and New Zealand on the 6th July and in the US on the 7th. A weekend later and Nintendo was worth US$11 billion more, their servers had been brought down repeatedly by excessive load, the game had a similar daily activity profile to twitter, one player had discovered a dead body and 4 accusations (so far) have been made of armed robbers using the game to lure the vulnerable.

It is that latter item that gives rise to speculation about where this could go wrong (pun intended). This game is not the first AR (Augmented Reality) game, nor is it the first location based game (think of geocaching and urban games like Pac-Manhattan for example). Location based games have been around for at least 11 years. But with the Pokémon Go uptake in the meteoric scale, there is reason to look more closely at the consequences of many people playing a location based game.

1. What if people meet each other ? Games almost inevitably involve some kind of fighting and when they become multi-person games, the players are often competing with one another. Even single player location based games have the unique ability to guide people to places where other gamers are. Games nowadays are played with a persistence and focus that's scary. With aggression fueled by that intensity, it seems inevitable that "bad things" might happen if players physically encountered each other. Perhaps the clever way to do this is to write the game such that the physical interaction that is set up is at least friendly. Imagine a spatial algorithm where the game deliberately keeps players from colliding or competing for the same token but still permits them a friendly recognition as they pass. There would be rules such that players are not competing in the same location but can encounter one another. Its an interesting algorithm when there are 5 players but becomes quite fascinating when there are 100,000 in the same city.
2. Along the same lines as 1, how long before there are first person shooters as an AR game in the real world ? That seems to me to be fraught with danger. Some folks already have difficulty distinguishing between reality and virtual reality. With guns and ammunition readily available one has to think that "bad things" could happen.
3. What happens to the tracking data ? The predecessor to Pokémon Go (Ingress) was built by a Google spin-off and predictably was designed to accumulate and aggregate data. Its not clear whether Pokémon Go is similarly designed but it seems like a pretty safe assumption. With this many users willingly traversing the countryside, the location data accumulated will be very useful. Better still, by strategically placing the beasts to be captured, a clever strategist can fill in doubtfully mapped areas. There's another interesting algorithm to think about - how do you draw your users through areas that need better mapping without detracting from the game experience by sending them to Knockturn Alley or getting them lost. Once collected, this data would set Nintendo up as a rival for foursquare for example. And when the camera is on in the phone for the AR aspect, is this also recorded ? One has to imagine that there are potential privacy issues both with the tracking data and the video imagery.
4. How long before there is advertising ? While the game encourages users to wander, the close range guidance to bring a user to the target has wonderful opportunities for advertisers. Local facilities could pay to draw users by their profile to them as they are passing and in the extreme case could pay Nintendo to direct users to their offerings. And of course with AR readily available, passing users could see into facilities as they pass them (for example to see what a bar looks like inside as they are passing it).

And as a closing thought, how could we as spatial specialists help make the game safer ? We could help Nintendo (or parents ?) understand areas that players at a vulnerable age probably should not frequent. Tracking information for the players could be provided to authorized guardians.

Its a brilliant move by Nintendo and a successful location based game has to be good for our industry but I suggest we also have a duty to do our best to prevent spatial blunders in the design and use of the game.

When Psychology Trumps the Compulsion of Proximity (or why aren't my neighbours coming to my store ?)

Geo-demographers expend considerable energy studying the profile of customer populations based on proximity. This is all based on the principle that "birds of a feather flock together" (BOF). This psychological concept has been extended by geo-demographers using clever clustering technology to predict both regions where the population will behave in a similar fashion (e.g. will buy a similar car, have a similar income, have the same number of children) and to develop profiles of given regions (like postal code polygons).

During these studies there are occasionally awkward anomalies and typically the clever maths of the neural net or machine learning algorithms will gently slide those into the outlier bucket. What is being sought is a generalization that is statistically reliable.

However, in some cases these outliers are evidence of a behaviour pattern that should be included in the profile.

There is a very tight connection between psychological behaviour patterns and spatial proximity. As a simple example, consider the case of a person deciding which retail outlet to go to. Lets assume that all the facilities within reach have what I'm looking for. Between my location and closest facility is a railway but there is a bridge. The next closest facility is quite a bit further away but does not require me to cross the railway.

If the route to a facility was longer because of available crossings of the railway then this would be a physical impedance and the spatial model, clustering algorithms and predictions would incorporate this. However, in the case above, not only is the facility across the railway but there are several other bridges also potentially offering a shorter travel time. This is where the spatial model should be influenced by the psychology of the situation. There is a psychological impedance to crossing the railway tracks. There is probably some very good explanation of why this happens but it definitely does occur and our spatial models need to be aware of this. The evidence of course - if we are training a machine learning engine - is that a statistically significant number of real routes will go the longer route to stay on the same side of the railway line.

This is a simple example that skilled geo-demographers are aware of and take into consideration when the data supports it. However, there are greater depths of psychological factors that may need to be considered. Physically intimidating facilities for example can influence customers to go right past and travel a longer time to find a more "friendly" environment. Factors such as acres of pristine floor space make service desks "remote". Overwhelming lights and signage can deter visitors. Narrow or pokey entrances or entrances that go too close to a service desk can dissuade at least the casual visitor.

Ideally a clever spatial model, well trained would be able to surface anomalies that might be psychological but as spatial specialists, we're not really qualified to analyze these results. Perhaps it is time to sit down with the anthropologists and psychologists and make those outlier results as valuable as possible.

What if self drive cars developed road rage ?

Google has a patent that explains that autonomous vehicles should be able to decide on behaviour based on the proximity of other vehicles in traffic. For example, a vehicle on a highway with a large truck on one side and a car on the other may decide to move to one side of its lane for safety sake. Or it may decide to accelerate past the truck to a safer area ahead.

This seems like an inevitable evolution of the simulation of existing traffic behaviour.

Humans piloting vehicles understand proximity in a different way - even when they drive into each other or cut each other off in traffic. One might hypothesize that a contributing factor to road rage is that the humans driving are autonomous and there is no overseeing power that prevents them from developing conflicts.

Consider now an entire environment of autonomous vehicles in close proximity. In the above example, the vehicle in the middle lane moves over to allow a safer zone around the truck. But the car on the other side sees the middle vehicle move closer and compensates by also changing position in its lane. The middle car sees that and reacts. By the dynamic proximity variations, there is now an unstable situation that is exacerbated by the very fast reflexes of the controlling computers. If this were to happen with many autonomous vehicles near each other, there could be massive disruption of the flow with lane concertinas and swerving.

If we now add in some complications - such as an obstacle in the road, wet roadway, or a human driver in the midst of the traffic then the potential problem is even larger.

Perhaps the right way to think about this spatial problem is to rethink the autonomous vehicle approach. With the present designs we are perpetuating the idea that each vehicle is a unit. The V2V communications work is well developed but there doesn't seem to be much thought about the conflict/feedback situation mentioned above. The intense research is focussed on the actual vehicle being smart enough to figure out how to deal with the dynamics of a complex and unpredictable traffic situation.

Perhaps the real time oversight of many autonomous vehicles attempting to cooperate in a limited space is a spatial problem that deserves attention from the spatial professionals ?

The Frontiers of Spatial Research

It is surely a much observed fact that the community of folk involved in spatial and spatial related fields is a small but diverse group. People know each other across international boundaries. Its a shame therefore that research in this community is not internationally coordinated.

A privileged few (on review boards and committees) around the world are permitted regular insight into proposed and contemplated research projects internationally. This is a wonderful way to see just how the industry is shaping itself and in a small way, permits the steering of the industry to attempt the "really big problems". 

However, with this insight comes a realization of two problems that seem to persist. The first of these is the admirably enthusiastic determination to solve a problem that has been solved already. As an example, there is almost always a project or two in a research portfolio aimed at conquering the problem of displaying points on a map. Firstly, this is not a problem that needs solving again. And secondly, we certainly do not need to build another map viewer to do this. There are plenty of brilliant, cheap or free map viewers available for a simple search. If suddenly a researcher comes up with a clever way of displaying half a billion points in a tenth of a second on a web viewer and making them usefully visible to the user, well then lets add that to an existing viewer and perhaps that research is worthwhile. Another favourite is map styling. Once the points are on the map then wouldn't it be great if they all looked like little robots with shadows behind them ? Maybe so but there is no need to build a style engine to allow the little robots to be drawn, scaled, coloured, haloed, shadowed and so forth. Its been done and done well - many times over. If however, a research project figures out how to emulate the wonderful artistic generalization we have seen in hand created maps then that's another matter.

The second problem is that very often, smart people in our community follow a similar path and arrive at a research project very similar to a project being contemplated elsewhere. It happens often that valuable research funds are expended in different places on very similar projects. Even more frustrating is that there are projects that stall at a problem area that another researcher has conquered or is on the verge of conquering. Without visibility into the research, the stalled project may stay that way.

There is one more factor in the spatial research area to consider. It appears that venture capitalists have figured out that location is a key investment point and they are interested in funding projects. There is plenty of evidence to support this. It is also true that there are research funds in many countries that are directed specifically at spatial problems. Unfortunately, many of these are country specific and so the funds are not always optimally deployed for the international success we need. International cooperation originating in a particular country has proven very difficult to foster. 

A small international group has met several times now to consider these factors and has proposed a possible solution. Its called LINC and there is a web site that explains how this group might work.

The general idea is that there is an international group of experts whose responsibility is to keep track of essential directions of spatial research and help local organizations with coordination of projects to avoid repetition and duplication. Also, the group think that it should be possible to provide perspectives in directions that spatial research needs to continue to evolve.

Venture capitalists also seek guidance as to where the technology dollars should be invested for a future payoff and this group would be ideally situated to provide that guidance. It has been suggested that the group would stamp promising original research that pushes spatial forwards with a logo to encourage investment and visibility of the research.

Your thoughts on this endeavour would be much appreciated.

Spatial Data and the Internet of Things (and the Internet of Humans)

During the recent GeoLife Round Table in Gävle, Sweden, it occurred to me as I'm sure it has to many others before me that the use of spatial in the IoT is facing a fairly challenging roadblock.

The premise of the IoT is that some sensing device is attached to the Internet. If the device is physically accessible then the chances are extremely high that the location of the device is known and its readings are location enabled. If it is remote (perhaps a hostile environment or internal to a live entity for example) then the chances are still very high that the location of the device will be known. Finally, a device that is moving is very likely to be location tracked since the very fact that it is mobile will be relevant to the sensor readings. An example could be a personal 2.5PM particulate sensor (see this reference study). The personal sensors are capable of alerting wearers that it is time to vacate an area before damaging inhalation occurs. The readings from the sensor are however also part of a larger data set that can be used to reduce harmful particulates once the dangerous areas are seen and possible sources identified.

Sensor data can be assumed to be temporal in nature.

However, sensors will also have vastly varying periodicity. The above mentioned 2.5PM sensors may emit a reading every 5 minutes and this may well be sufficient granularity for monitoring particulates. A sensor on a vehicle will have to have a very much finer granularity since a change within a second can be critical. An astronomical observation on the other hand may be fine with readings every day.

Sensors will also experience dramatic amplitude variations. Again with the personal 2.5PM sensor, it is reasonable to expect that the wearer experiences very low readings until entering a hostile environment when the values will suddenly peak. There are also sensors that are smart enough to report only when they have something significant to report. For example a home flood sensor need not report until it senses water where there should be none.

Thus sensors have amplitude irregularities, periodicity variations and regularity differences.

Consider now that the same challenges are present in the location information that is recorded with the sensor reading. A pedestrian gets in a car and suddenly records substantial displacements. The wearer goes to sleep. The wearer goes inside a building and the location tracking is more granular than street addresses. The wearer boards a flight, turns off the sensor and turns it back on in another country.

Even so, it could be argued that for single sensors, the peculiarities of the spatio-temporal readings can be managed since they should be well understood. This however does not work at all when multiple disparate sensors are sources (and this of course is where the IoT and IoH really becomes exceptionally useful). This is where correlations are developed, where multiple inputs lead to dramatic insights and the power of the IoT is fully realized.

For the IoT and IoH to establish a data store that is readily accessible, usable without substantial pre-processing, the location and temporal data must be woven into a stream that is useful on both the time and location axes. And this stream has to accommodate but not distort the fundamental value of the sensor readings. Simply dropping the sensor readings onto a location continuum is just not going to work. There has to be understanding of the location significance of the readings and where they are. And the data cannot be averaged. The causal effects would be lost in the rounding process. And this cannot be done divested from the temporal axis.

Finally, in the spatial world it has become absolutely imperative that spatial data processing use the time axis as an integral element of the process.

The ideal of an ad hoc collection of sensors all contributing readings to a spatio-temporal domain to give a multi-dimensional view of a particular environment is truly exciting but the spatial professionals have an essential role to play in realizing this.

Is it possible that some of today's analytics results are fundamentally flawed ?

There are numerous authorities stridently claiming that analysts working today are missing a vital dimension in that they are ignoring the spatial aspect of the data. The neighbourhood relationship in other words. The claims indicate that spatial data included in the analysis will give better "insights" and add value to the results.

While this is provably true given numerous real and usable examples, the claims have to be coloured to some extent by the vested interests of the claimants. They are selling something.

However, what if the lack of use of neighbourhood in routine (and even extreme) analysis was giving rise to fundamentally flawed results but not because the spatial aspect was failing to contribute to the result but because its absence was actually invalidating the results ? What if the missing spatial variables were forcing a bias in the analytics simply because their presence is essential to a correct or at least predictably reliable result ?

There is a factor in statistical analysis called omitted-variable bias. Coarsely defined, it arises when the analytical model, missing a key variable set, compensates for that missing variable set by changing the influence of other variables. And thus the model is broken. Conventional literature suggests that if there is a risk of OVB then all the variables should be added as control variables. But that's scarcely a safe choice since now the model becomes so constrained that the results, while most likely trustworthy, reveal very little of compelling interest (meaning the return on investment is minimized). Kevin A. Clarke writes of the dangers of excessive inclusion of control variables (referring specifically to regression analysis).

There are extremely effective methods that permit an analyst to identify which variables are most strongly characteristic of a particular data set and in some previous work, I have seen practical and provably useful results from these methods. This is of course a different view of the analytical approach but the fact that neighbourhood was almost always a dominant variable in the various real life examples processed, suggests that the omission of neighbourhood from an analysis of these data sets could lead to results polluted by OVB.

In selecting the variables that will be used for analytics, proximity (neighbourhood) should be considered as at least as important to the model as any other (income, age gender etc). Without proximity, the results could be incomplete or inaccurate and if OVB has occurred due to the absence of proximity, the results could be quite seriously misleading.

Spatial Analytics is Simple - Really

Think of spatial information, location intelligence, GIS and the various other terms used with respect to the storage and use of spatial data and we think of the three basics :

1. spatial data type (for storage),
2. spatial indices and 
3. spatial operators.

These however are only the mechanics of the handling of the data and obscure the elegant simplicity of the application of the data. While these three elements are fundamental to and essential for the successful use of spatial data, once the data is used in an analytical sense, there is really only one asset that spatial data yields. Only one but it is a big one and at least as important as any other factor considered in data analytics.

Spatially enabled data stores permit analysis using neighbourhood.

Location intelligent engines have all kinds of fascinating and complex operators with very clever and refined algorithms but if we look at a few of them, its revealing just how they may be considered when analytics are being used.

• Buffer. Arguably the most used operator in the GIS world; the operator gives you a region around some feature you might be interested in. A truck carrying something toxic has driven down a certain road and has leaked the substance as it goes. A driver wants to know where there are gas stations near the road being driven. But here's the thing. The buffer is cool to look at on a map but it is never the result that a user wants. Its nothing more than a means to help a user understand what's near something interesting. Neighbourhood in other words.
• Topology management. The assembly of a topologically consistent fabric in a data store is a lovely and engaging problem for all spatial systems. There are the famous sliver polygons, the overshoots, the undershoots, the invalid connections, the missing connections and so forth. The purpose of all this fine and expensive processing is to prepare the data such that the maximum value can be extracted during analysis. Meaning that the neighbourhood factors derived have a reasonable chance of being accurate.
• Overlaps. Perhaps the second most used operator and fraught with all kinds of nuances for which spatial professionals have worked out all kinds of semantics. Why do I as an analyst care ? As an example, I may want to know which regions of my township are like to flood in the next 10 years. That's useful but its not the end result of the process. The analyst will want a result that can be acted on (such as the calculation of insurance risk) and now we are back to neighbourhoods again.

We could argue that GIS engines permit the measurement of distance (to see how far things are from other things), area (because the user wants to know whether he can practice the bagpipes in the house without being shot at), slope (for calculations to indicate what neighbourhoods will flood). But it seems to me that these are still the mechanics that lead to an analytics system being able to use neighbourhood effectively.

For at least 25 years, there has been a prevalent assertion that 80% of the world's data has a spatial component. There's a good discussion here that traces where this came from. While the number might be argued, it is very likely to be too low rather than ambitious given the sources of our data (phones, cars etc).

With that density of data concentrated around the single analytical factor of neighbourhood and the riveting actions that can be derived from combining that factor with the analytical factors already in play, it is fundamentally important that the spatial practitioners look after the plumbing such that analysts can add neighbourhood to their portfolio of power analytics.

Right now there is a big gap and the use of spatial data in analytics is being held up by concern and worry about the complexities of GIS, the difficulties of managing the data and consternation about things like "projection systems", "Hausdorff distances", "WFS filters" and so forth. We should be able to enable skilled analysts to make dramatic and lucrative progress with neighbourhood factors in traditional analytics without making their heads hurt.