This course will present an overview of program anomaly detection, which
analyzes normal program behaviors and discovers aberrant executions
caused by attacks, misconfigurations, program bugs, and unusual usage
patterns. Advanced models have been developed in the last decade and
comprehensive techniques have been adopted such as hidden Markov model
and machine learning. We will introduce the audience to the problem of
program attacks and the anomaly detection approach against threats. We
will give a general definition for program anomaly detection and derive
model abstractions from the definition. We will cover the development of
program anomaly detection methods from early-age n-gram approaches to
complicated pushdown automata and probabilistic models. This course will
help students understand the objectives and challenges in designing
program anomaly detection models. We will discuss the attacks that
subvert anomaly detection mechanisms.