Installation Instructions

The preferable version is the recent beta version of framework which requires patching of three system components: Xen 4.2, Linux 3.2.30 guest, and perfctr 2.6.42.2 (my modified version of 2.6.42 which can work with Linux 3.2.30). Note that this is a beta version (which may still contain bugs). Patches for this version are: perfctr, Linux, and Xen.

Note that the above perfctr tarball contains patches for various (including more recent) versions of Linux. Older versions will most likely not work. More recent versions may or may not work, and they are not tested; it is not recommended to use anything other than 3.2.30.

Previous stable version of framework (which is now obsolete due to its reliance on PvOps 2.6.x kernel) requires patching of:
Xen 4.0.1, PvOps Linux 2.6.32.21 guest, and perfctr 2.6.42 driver. Follow the installation instructions below to properly set up the framework. Also, make sure that Linux guest you use is of 2.6.32-pvops branch. (It may be necessary to switch it if the default branch is 2.6.31 as described here.) Patches for this version are: perfctr, Linux, and Xen.

Instructions below are for the older stable version. For the beta
version, please adjust corresponding names of directories and patches. For the beta version, there may be additional steps mentioned in instructions.

Installation

  1. Download from this website the latest version of patches for the corresponding versions of Linux/Xen/Perfctr mentioned above.
  2. Download and extract Xen, Linux and perfctr. (PvOps Linux 2.6.32.21: it may be necessary to download the Linux archive manually as opposed to automatic download during compilation. Follow the instructions provided here. Make sure to put the downloaded Linux files under standard location xen-4.0.1/linux-2.6-pvops.git ; for this, you need to rename the downloaded directory and move into xen-4.0.1.)  For the newer beta version and Xen 4.2/Linux 3.2.30, Linux is compiled separately from Xen, and you can place Linux elsewhere but make sure to follow integration instructions; please also choose configuration options as specified in step 5.
  3. Patch all of them
  4. cd xen-4.0.1
    patch -p1 < ../xen_r1.patch
    cd linux-2.6-pvops.git
    patch -p1 < ../../linux_r1.patch
    cd ../../perfctr-2.6.42
    patch -p1 < ../perfctr_r1.patch
  5. Integrate perfctr into Linux
  6. cd xen-4.0.1/linux-2.6-pvops.git
    patch -p1 < ../../perfctr-2.6.42/patches/patch-kernel-2.6.32
    cp -r ../../perfctr-2.6.42/linux/* .
  7. For the newer beta version and Xen 4.2 / Linux 3.2.30, Linux is compiled separately from Xen. You need to follow traditional Linux compilation procedure (i.e., make, make menuconfig, make install) choosing the following options: (Disable CONFIG_PERFCTR_DEBUG and CONFIG_PERFCTR_INIT_TESTS!)
  8. CONFIG_PERFCTR=y
    CONFIG_KPERFCTR=y
    CONFIG_PERFCTR_DEBUG=n
    CONFIG_PERFCTR_INIT_TESTS=n
    CONFIG_PERFCTR_VIRTUAL=y
    CONFIG_PERFCTR_GLOBAL=y
    CONFIG_PERFCTR_INTERRUPT_SUPPORT=y
    CONFIG_PERFCTR_CPUS_FORBIDDEN_MASK=y
    CONFIG_PERFCTR_WIDECTR=y
    CONFIG_PERFCTR_XEN=y
  9. Compile & install Xen and Linux guest. (Perfctr support should be enabled automatically after applying patches above. For the newer beta version it is enabled after you select corresponding options in Linux, step 5.)
  10. make
    make install
  11. Follow standard instructions for creating initrd image and adding Xen to GRUB. It can be found at xen-4.0.1/README. Make sure that the hypervisor line you add to the GRUB configuration contains an additional perfctr option, so that perfctr-xen support will be enabled. (e.g. multiboot /boot/xen-4.0.gz dummy=dummy dom0_mem=4096M perfctr)
  12. Copy perfctr-2.6.42 to perfctr-2.6.x (only if you intend to use PAPI).
  13. cp -r perfctr-2.6.42 perfctr-2.6.x
  14. Compile & install perfctr user library
  15. cd perfctr-2.6.42
    make
    make install
  16. If you want to use PAPI or anything that works on top of PAPI (HPCToolkit, PerfExplorer, etc.), download the latest version from here. The code is currently known to work with PAPI 4.1.0 – 4.4.0
  17. Remove PAPI’s copy of perfctr-2.6.x and perfctr-2.7.x and replace it with the patched one
  18. cd papi-4.1.0/src
    rm -r perfctr-2.6.x perfctr-2.7.x
    mv ../../perfctr-2.6.x .
  19. Compile & install PAPI (if desired)
  20. ./configure --prefix=/usr
    make
    make install
  21. The compiled kernel is available at /boot. It will be used by Dom0 when you reboot the computer and load Xen hypervisor. Since domU domains (Dom1, Dom2, etc.) also need to use perfctr-xen, you can just copy the compiled kernel, initrd, modules to the corresponding domain image. (This is possible because the compiled kernel contains both Dom0 and DomU support. It even works with HVM domains.)

Testing
After rebooting, you may want to test the installation. Follow the instructions here.

Caution

Do not use perfctr option (in GRUB configuration) and vpmu simultaneously. The latter one enables hardware counters virtualization in HVM mode, and it should never be used with the modified perfctr.