Exploiting spatio-temporal patterns for event response using reinforcement learning in a synthetically-augmented environment
Date of Award
Doctor of Philosophy (PhD)
Mikel D. Petty
Peter J. Slater
Ramazan S. Aygun
Machine learning (Machine learning), Pheromes., Data mining., Spatial analysis (Statistics), Temporal databases.
Events with detrimental effects caused by adversarial human actors, such as crimes and terrorist acts, require the convergence of three factors: a suitable target, a motivated offender, and the absence of a capable guardian. Positioning guardian agents in the spatio-temporal proximity of anticipated events in order to prevent or respond to them can be a difficult task. The most common approaches to this problem include models of the adversarial actors to which guardian agents respond. Such model-based approaches can be problematic when there is insufficient information to construct a reliable model, when the number of factors governing the actor's behavior is too numerous to yield a practical model, or when actor models are inaccurate. Conversely, finding an effective model-free approach is challenging because without an actor model little more than the time and location of events are available for agents to react to. This research introduces novel methods to rectify the insufficiency of time and location of events as the sole information to support model-free reinforcement learning, and to boost learning from the limited experience agents may gain from the event sequence. To offset the dearth of information in an environment consisting solely of the time and location of a sequence of events, we synthetically augment the information in the environment with digital pheromones, and other information augmenters derived from the event sequence, to define informative states. These augmenters may reveal regularities in the timing and location of events that, through reinforcement learning, can be detected and exploited to position agents in spatio-temporal proximity of anticipated events. To improve agent learning from limited experience and to reduce the effects of the environment's non-stationarity, we enhance standard reinforcement learning in two ways. The first, adapted from existing methodologies, does so by allowing agents to learn from actions not taken and from the experiences of others, a concept known as fictive learning. The second approach is a novel learning boosting method that complements synthetic-augmentation and enables learning from partial-state information. The new techniques were applied to two difficult real-world problems, responding to Somali maritime piracy attacks and to business robberies in Denver Colorado, and analyzed for effectiveness. The combination of enhanced reinforcement learning and information augmenters outperformed the historical record, or a domain-representative heuristic, in both applications.
Barbosa, Salvador E., "Exploiting spatio-temporal patterns for event response using reinforcement learning in a synthetically-augmented environment" (2014). Dissertations. 40.