With high-end systems featuring multicore/multithreaded processors and
high component density, power-aware high-performance multithreading
libraries become a critical element of the system software
stack. Online power and performance adaptation of multithreaded code
from within user-level runtime libraries is a relatively new and
unexplored area of research. We present a user-level library framework
for nearly optimal online adaptation of multithreaded codes for
low-power, high-performance execution. Our framework operates by
regulating concurrency and changing the processors/threads
configuration as the program executes. It is innovative in that it
uses fast, runtime performance prediction derived from hardware
event-driven profiling, to select thread granularities that achieve
nearly optimal energy-efficiency points. The use of predictors
substantially reduces the runtime cost of granularity control and
program adaptation. Our framework achieves performance and $ED^{2}$
(energy-delay-squared) levels which are: i) comparable to or better
than those of oracle-derived offline predictors; ii) significantly
better than those of online predictors using exhaustive or localized
linear search. The complete prediction and adaptation framework is
implemented on a real multi-SMT system with Intel Hyperthreaded
processors and embeds adaptation capabilities in OpenMP programs.