Computing Agent Competency in First Order Markov Processes

• Main Author: Cao, Xuan
• Format: Others
• Published: BYU ScholarsArchive 2021
• Subjects: artificial agents; first-order Markov process; state-dependent competency; total competency; Monte Carlo approach; Bayesian network; Physical Sciences and Mathematics
• Online Access: https://scholarsarchive.byu.edu/etd/9336; https://scholarsarchive.byu.edu/cgi/viewcontent.cgi?article=10345&context=etd

Artificial agents are usually designed to achieve specific goals. An agent's competency can be defined as its ability to accomplish its goals under different conditions. This thesis restricts attention to a specific type of goal, namely reaching a desired state without exceeding a tolerance threshold of undesirable events in a first-order Markov process. For such goals, the state-dependent competency for an agent can be defined as the probability of reaching the desired state without exceeding the threshold and within a time limit given an initial state. The thesis further defines total competency as the set of state-dependent competency relationships over all possible initial states. The thesis uses a Monte Carlo approach to establish a baseline for estimating state-dependent competency. The Monte Carlo approach (a) uses trajectories sampled from an agent behaving in the environment, and then (b) uses nonlinear regression over the trajectory samples to estimate the competency curve. The thesis further presents an equation demonstrating recurrent relations for total competency and an algorithm based on that equation for computing total competency whose worst case computation time grows quadratically with the size of the state space. Simple maze-based Markov chains show that the Monte Carlo approach to estimating the competency agrees with the results computed by the proposed algorithm. Lastly, the thesis explores a special case where there are multiple sequential atomic goals that make up a complex goal. The thesis models a set of sequential goals as a Bayesian network and presents an equation based on the chain rule for deriving the competency for the complex goal from the competency for atomic goals. Experiments for the canonical taxi problem with sequential goals show the correctness of the Bayesian network-based decomposition approach.