linux/Documentation/virtual/kvm/timekeeping.txt
<<
>>
Prefs
   1
   2        Timekeeping Virtualization for X86-Based Architectures
   3
   4        Zachary Amsden <zamsden@redhat.com>
   5        Copyright (c) 2010, Red Hat.  All rights reserved.
   6
   71) Overview
   82) Timing Devices
   93) TSC Hardware
  104) Virtualization Problems
  11
  12=========================================================================
  13
  141) Overview
  15
  16One of the most complicated parts of the X86 platform, and specifically,
  17the virtualization of this platform is the plethora of timing devices available
  18and the complexity of emulating those devices.  In addition, virtualization of
  19time introduces a new set of challenges because it introduces a multiplexed
  20division of time beyond the control of the guest CPU.
  21
  22First, we will describe the various timekeeping hardware available, then
  23present some of the problems which arise and solutions available, giving
  24specific recommendations for certain classes of KVM guests.
  25
  26The purpose of this document is to collect data and information relevant to
  27timekeeping which may be difficult to find elsewhere, specifically,
  28information relevant to KVM and hardware-based virtualization.
  29
  30=========================================================================
  31
  322) Timing Devices
  33
  34First we discuss the basic hardware devices available.  TSC and the related
  35KVM clock are special enough to warrant a full exposition and are described in
  36the following section.
  37
  382.1) i8254 - PIT
  39
  40One of the first timer devices available is the programmable interrupt timer,
  41or PIT.  The PIT has a fixed frequency 1.193182 MHz base clock and three
  42channels which can be programmed to deliver periodic or one-shot interrupts.
  43These three channels can be configured in different modes and have individual
  44counters.  Channel 1 and 2 were not available for general use in the original
  45IBM PC, and historically were connected to control RAM refresh and the PC
  46speaker.  Now the PIT is typically integrated as part of an emulated chipset
  47and a separate physical PIT is not used.
  48
  49The PIT uses I/O ports 0x40 - 0x43.  Access to the 16-bit counters is done
  50using single or multiple byte access to the I/O ports.  There are 6 modes
  51available, but not all modes are available to all timers, as only timer 2
  52has a connected gate input, required for modes 1 and 5.  The gate line is
  53controlled by port 61h, bit 0, as illustrated in the following diagram.
  54
  55 --------------             ----------------
  56|              |           |                |
  57|  1.1932 MHz  |---------->| CLOCK      OUT | ---------> IRQ 0
  58|    Clock     |   |       |                |
  59 --------------    |    +->| GATE  TIMER 0  |
  60                   |        ----------------
  61                   |
  62                   |        ----------------
  63                   |       |                |
  64                   |------>| CLOCK      OUT | ---------> 66.3 KHZ DRAM
  65                   |       |                |            (aka /dev/null)
  66                   |    +->| GATE  TIMER 1  |
  67                   |        ----------------
  68                   |
  69                   |        ----------------
  70                   |       |                |
  71                   |------>| CLOCK      OUT | ---------> Port 61h, bit 5
  72                           |                |      |
  73Port 61h, bit 0 ---------->| GATE  TIMER 2  |       \_.----   ____
  74                            ----------------         _|    )--|LPF|---Speaker
  75                                                    / *----   \___/
  76Port 61h, bit 1 -----------------------------------/
  77
  78The timer modes are now described.
  79
  80Mode 0: Single Timeout.   This is a one-shot software timeout that counts down
  81 when the gate is high (always true for timers 0 and 1).  When the count
  82 reaches zero, the output goes high.
  83
  84Mode 1: Triggered One-shot.  The output is initially set high.  When the gate
  85 line is set high, a countdown is initiated (which does not stop if the gate is
  86 lowered), during which the output is set low.  When the count reaches zero,
  87 the output goes high.
  88
  89Mode 2: Rate Generator.  The output is initially set high.  When the countdown
  90 reaches 1, the output goes low for one count and then returns high.  The value
  91 is reloaded and the countdown automatically resumes.  If the gate line goes
  92 low, the count is halted.  If the output is low when the gate is lowered, the
  93 output automatically goes high (this only affects timer 2).
  94
  95Mode 3: Square Wave.   This generates a high / low square wave.  The count
  96 determines the length of the pulse, which alternates between high and low
  97 when zero is reached.  The count only proceeds when gate is high and is
  98 automatically reloaded on reaching zero.  The count is decremented twice at
  99 each clock to generate a full high / low cycle at the full periodic rate.
 100 If the count is even, the clock remains high for N/2 counts and low for N/2
 101 counts; if the clock is odd, the clock is high for (N+1)/2 counts and low
 102 for (N-1)/2 counts.  Only even values are latched by the counter, so odd
 103 values are not observed when reading.  This is the intended mode for timer 2,
 104 which generates sine-like tones by low-pass filtering the square wave output.
 105
 106Mode 4: Software Strobe.  After programming this mode and loading the counter,
 107 the output remains high until the counter reaches zero.  Then the output
 108 goes low for 1 clock cycle and returns high.  The counter is not reloaded.
 109 Counting only occurs when gate is high.
 110
 111Mode 5: Hardware Strobe.  After programming and loading the counter, the
 112 output remains high.  When the gate is raised, a countdown is initiated
 113 (which does not stop if the gate is lowered).  When the counter reaches zero,
 114 the output goes low for 1 clock cycle and then returns high.  The counter is
 115 not reloaded.
 116
 117In addition to normal binary counting, the PIT supports BCD counting.  The
 118command port, 0x43 is used to set the counter and mode for each of the three
 119timers.
 120
 121PIT commands, issued to port 0x43, using the following bit encoding:
 122
 123Bit 7-4: Command (See table below)
 124Bit 3-1: Mode (000 = Mode 0, 101 = Mode 5, 11X = undefined)
 125Bit 0  : Binary (0) / BCD (1)
 126
 127Command table:
 128
 1290000 - Latch Timer 0 count for port 0x40
 130        sample and hold the count to be read in port 0x40;
 131        additional commands ignored until counter is read;
 132        mode bits ignored.
 133
 1340001 - Set Timer 0 LSB mode for port 0x40
 135        set timer to read LSB only and force MSB to zero;
 136        mode bits set timer mode
 137
 1380010 - Set Timer 0 MSB mode for port 0x40
 139        set timer to read MSB only and force LSB to zero;
 140        mode bits set timer mode
 141
 1420011 - Set Timer 0 16-bit mode for port 0x40
 143        set timer to read / write LSB first, then MSB;
 144        mode bits set timer mode
 145
 1460100 - Latch Timer 1 count for port 0x41 - as described above
 1470101 - Set Timer 1 LSB mode for port 0x41 - as described above
 1480110 - Set Timer 1 MSB mode for port 0x41 - as described above
 1490111 - Set Timer 1 16-bit mode for port 0x41 - as described above
 150
 1511000 - Latch Timer 2 count for port 0x42 - as described above
 1521001 - Set Timer 2 LSB mode for port 0x42 - as described above
 1531010 - Set Timer 2 MSB mode for port 0x42 - as described above
 1541011 - Set Timer 2 16-bit mode for port 0x42 as described above
 155
 1561101 - General counter latch
 157        Latch combination of counters into corresponding ports
 158        Bit 3 = Counter 2
 159        Bit 2 = Counter 1
 160        Bit 1 = Counter 0
 161        Bit 0 = Unused
 162
 1631110 - Latch timer status
 164        Latch combination of counter mode into corresponding ports
 165        Bit 3 = Counter 2
 166        Bit 2 = Counter 1
 167        Bit 1 = Counter 0
 168
 169        The output of ports 0x40-0x42 following this command will be:
 170
 171        Bit 7 = Output pin
 172        Bit 6 = Count loaded (0 if timer has expired)
 173        Bit 5-4 = Read / Write mode
 174            01 = MSB only
 175            10 = LSB only
 176            11 = LSB / MSB (16-bit)
 177        Bit 3-1 = Mode
 178        Bit 0 = Binary (0) / BCD mode (1)
 179
 1802.2) RTC
 181
 182The second device which was available in the original PC was the MC146818 real
 183time clock.  The original device is now obsolete, and usually emulated by the
 184system chipset, sometimes by an HPET and some frankenstein IRQ routing.
 185
 186The RTC is accessed through CMOS variables, which uses an index register to
 187control which bytes are read.  Since there is only one index register, read
 188of the CMOS and read of the RTC require lock protection (in addition, it is
 189dangerous to allow userspace utilities such as hwclock to have direct RTC
 190access, as they could corrupt kernel reads and writes of CMOS memory).
 191
 192The RTC generates an interrupt which is usually routed to IRQ 8.  The interrupt
 193can function as a periodic timer, an additional once a day alarm, and can issue
 194interrupts after an update of the CMOS registers by the MC146818 is complete.
 195The type of interrupt is signalled in the RTC status registers.
 196
 197The RTC will update the current time fields by battery power even while the
 198system is off.  The current time fields should not be read while an update is
 199in progress, as indicated in the status register.
 200
 201The clock uses a 32.768kHz crystal, so bits 6-4 of register A should be
 202programmed to a 32kHz divider if the RTC is to count seconds.
 203
 204This is the RAM map originally used for the RTC/CMOS:
 205
 206Location    Size    Description
 207------------------------------------------
 20800h         byte    Current second (BCD)
 20901h         byte    Seconds alarm (BCD)
 21002h         byte    Current minute (BCD)
 21103h         byte    Minutes alarm (BCD)
 21204h         byte    Current hour (BCD)
 21305h         byte    Hours alarm (BCD)
 21406h         byte    Current day of week (BCD)
 21507h         byte    Current day of month (BCD)
 21608h         byte    Current month (BCD)
 21709h         byte    Current year (BCD)
 2180Ah         byte    Register A
 219                       bit 7   = Update in progress
 220                       bit 6-4 = Divider for clock
 221                                  000 = 4.194 MHz
 222                                  001 = 1.049 MHz
 223                                  010 = 32 kHz
 224                                  10X = test modes
 225                                  110 = reset / disable
 226                                  111 = reset / disable
 227                       bit 3-0 = Rate selection for periodic interrupt
 228                                  000 = periodic timer disabled
 229                                  001 = 3.90625 uS
 230                                  010 = 7.8125 uS
 231                                  011 = .122070 mS
 232                                  100 = .244141 mS
 233                                     ...
 234                                 1101 = 125 mS
 235                                 1110 = 250 mS
 236                                 1111 = 500 mS
 2370Bh         byte    Register B
 238                       bit 7   = Run (0) / Halt (1)
 239                       bit 6   = Periodic interrupt enable
 240                       bit 5   = Alarm interrupt enable
 241                       bit 4   = Update-ended interrupt enable
 242                       bit 3   = Square wave interrupt enable
 243                       bit 2   = BCD calendar (0) / Binary (1)
 244                       bit 1   = 12-hour mode (0) / 24-hour mode (1)
 245                       bit 0   = 0 (DST off) / 1 (DST enabled)
 246OCh         byte    Register C (read only)
 247                       bit 7   = interrupt request flag (IRQF)
 248                       bit 6   = periodic interrupt flag (PF)
 249                       bit 5   = alarm interrupt flag (AF)
 250                       bit 4   = update interrupt flag (UF)
 251                       bit 3-0 = reserved
 252ODh         byte    Register D (read only)
 253                       bit 7   = RTC has power
 254                       bit 6-0 = reserved
 25532h         byte    Current century BCD (*)
 256  (*) location vendor specific and now determined from ACPI global tables
 257
 2582.3) APIC
 259
 260On Pentium and later processors, an on-board timer is available to each CPU
 261as part of the Advanced Programmable Interrupt Controller.  The APIC is
 262accessed through memory-mapped registers and provides interrupt service to each
 263CPU, used for IPIs and local timer interrupts.
 264
 265Although in theory the APIC is a safe and stable source for local interrupts,
 266in practice, many bugs and glitches have occurred due to the special nature of
 267the APIC CPU-local memory-mapped hardware.  Beware that CPU errata may affect
 268the use of the APIC and that workarounds may be required.  In addition, some of
 269these workarounds pose unique constraints for virtualization - requiring either
 270extra overhead incurred from extra reads of memory-mapped I/O or additional
 271functionality that may be more computationally expensive to implement.
 272
 273Since the APIC is documented quite well in the Intel and AMD manuals, we will
 274avoid repetition of the detail here.  It should be pointed out that the APIC
 275timer is programmed through the LVT (local vector timer) register, is capable
 276of one-shot or periodic operation, and is based on the bus clock divided down
 277by the programmable divider register.
 278
 2792.4) HPET
 280
 281HPET is quite complex, and was originally intended to replace the PIT / RTC
 282support of the X86 PC.  It remains to be seen whether that will be the case, as
 283the de facto standard of PC hardware is to emulate these older devices.  Some
 284systems designated as legacy free may support only the HPET as a hardware timer
 285device.
 286
 287The HPET spec is rather loose and vague, requiring at least 3 hardware timers,
 288but allowing implementation freedom to support many more.  It also imposes no
 289fixed rate on the timer frequency, but does impose some extremal values on
 290frequency, error and slew.
 291
 292In general, the HPET is recommended as a high precision (compared to PIT /RTC)
 293time source which is independent of local variation (as there is only one HPET
 294in any given system).  The HPET is also memory-mapped, and its presence is
 295indicated through ACPI tables by the BIOS.
 296
 297Detailed specification of the HPET is beyond the current scope of this
 298document, as it is also very well documented elsewhere.
 299
 3002.5) Offboard Timers
 301
 302Several cards, both proprietary (watchdog boards) and commonplace (e1000) have
 303timing chips built into the cards which may have registers which are accessible
 304to kernel or user drivers.  To the author's knowledge, using these to generate
 305a clocksource for a Linux or other kernel has not yet been attempted and is in
 306general frowned upon as not playing by the agreed rules of the game.  Such a
 307timer device would require additional support to be virtualized properly and is
 308not considered important at this time as no known operating system does this.
 309
 310=========================================================================
 311
 3123) TSC Hardware
 313
 314The TSC or time stamp counter is relatively simple in theory; it counts
 315instruction cycles issued by the processor, which can be used as a measure of
 316time.  In practice, due to a number of problems, it is the most complicated
 317timekeeping device to use.
 318
 319The TSC is represented internally as a 64-bit MSR which can be read with the
 320RDMSR, RDTSC, or RDTSCP (when available) instructions.  In the past, hardware
 321limitations made it possible to write the TSC, but generally on old hardware it
 322was only possible to write the low 32-bits of the 64-bit counter, and the upper
 32332-bits of the counter were cleared.  Now, however, on Intel processors family
 3240Fh, for models 3, 4 and 6, and family 06h, models e and f, this restriction
 325has been lifted and all 64-bits are writable.  On AMD systems, the ability to
 326write the TSC MSR is not an architectural guarantee.
 327
 328The TSC is accessible from CPL-0 and conditionally, for CPL > 0 software by
 329means of the CR4.TSD bit, which when enabled, disables CPL > 0 TSC access.
 330
 331Some vendors have implemented an additional instruction, RDTSCP, which returns
 332atomically not just the TSC, but an indicator which corresponds to the
 333processor number.  This can be used to index into an array of TSC variables to
 334determine offset information in SMP systems where TSCs are not synchronized.
 335The presence of this instruction must be determined by consulting CPUID feature
 336bits.
 337
 338Both VMX and SVM provide extension fields in the virtualization hardware which
 339allows the guest visible TSC to be offset by a constant.  Newer implementations
 340promise to allow the TSC to additionally be scaled, but this hardware is not
 341yet widely available.
 342
 3433.1) TSC synchronization
 344
 345The TSC is a CPU-local clock in most implementations.  This means, on SMP
 346platforms, the TSCs of different CPUs may start at different times depending
 347on when the CPUs are powered on.  Generally, CPUs on the same die will share
 348the same clock, however, this is not always the case.
 349
 350The BIOS may attempt to resynchronize the TSCs during the poweron process and
 351the operating system or other system software may attempt to do this as well.
 352Several hardware limitations make the problem worse - if it is not possible to
 353write the full 64-bits of the TSC, it may be impossible to match the TSC in
 354newly arriving CPUs to that of the rest of the system, resulting in
 355unsynchronized TSCs.  This may be done by BIOS or system software, but in
 356practice, getting a perfectly synchronized TSC will not be possible unless all
 357values are read from the same clock, which generally only is possible on single
 358socket systems or those with special hardware support.
 359
 3603.2) TSC and CPU hotplug
 361
 362As touched on already, CPUs which arrive later than the boot time of the system
 363may not have a TSC value that is synchronized with the rest of the system.
 364Either system software, BIOS, or SMM code may actually try to establish the TSC
 365to a value matching the rest of the system, but a perfect match is usually not
 366a guarantee.  This can have the effect of bringing a system from a state where
 367TSC is synchronized back to a state where TSC synchronization flaws, however
 368small, may be exposed to the OS and any virtualization environment.
 369
 3703.3) TSC and multi-socket / NUMA
 371
 372Multi-socket systems, especially large multi-socket systems are likely to have
 373individual clocksources rather than a single, universally distributed clock.
 374Since these clocks are driven by different crystals, they will not have
 375perfectly matched frequency, and temperature and electrical variations will
 376cause the CPU clocks, and thus the TSCs to drift over time.  Depending on the
 377exact clock and bus design, the drift may or may not be fixed in absolute
 378error, and may accumulate over time.
 379
 380In addition, very large systems may deliberately slew the clocks of individual
 381cores.  This technique, known as spread-spectrum clocking, reduces EMI at the
 382clock frequency and harmonics of it, which may be required to pass FCC
 383standards for telecommunications and computer equipment.
 384
 385It is recommended not to trust the TSCs to remain synchronized on NUMA or
 386multiple socket systems for these reasons.
 387
 3883.4) TSC and C-states
 389
 390C-states, or idling states of the processor, especially C1E and deeper sleep
 391states may be problematic for TSC as well.  The TSC may stop advancing in such
 392a state, resulting in a TSC which is behind that of other CPUs when execution
 393is resumed.  Such CPUs must be detected and flagged by the operating system
 394based on CPU and chipset identifications.
 395
 396The TSC in such a case may be corrected by catching it up to a known external
 397clocksource.
 398
 3993.5) TSC frequency change / P-states
 400
 401To make things slightly more interesting, some CPUs may change frequency.  They
 402may or may not run the TSC at the same rate, and because the frequency change
 403may be staggered or slewed, at some points in time, the TSC rate may not be
 404known other than falling within a range of values.  In this case, the TSC will
 405not be a stable time source, and must be calibrated against a known, stable,
 406external clock to be a usable source of time.
 407
 408Whether the TSC runs at a constant rate or scales with the P-state is model
 409dependent and must be determined by inspecting CPUID, chipset or vendor
 410specific MSR fields.
 411
 412In addition, some vendors have known bugs where the P-state is actually
 413compensated for properly during normal operation, but when the processor is
 414inactive, the P-state may be raised temporarily to service cache misses from
 415other processors.  In such cases, the TSC on halted CPUs could advance faster
 416than that of non-halted processors.  AMD Turion processors are known to have
 417this problem.
 418
 4193.6) TSC and STPCLK / T-states
 420
 421External signals given to the processor may also have the effect of stopping
 422the TSC.  This is typically done for thermal emergency power control to prevent
 423an overheating condition, and typically, there is no way to detect that this
 424condition has happened.
 425
 4263.7) TSC virtualization - VMX
 427
 428VMX provides conditional trapping of RDTSC, RDMSR, WRMSR and RDTSCP
 429instructions, which is enough for full virtualization of TSC in any manner.  In
 430addition, VMX allows passing through the host TSC plus an additional TSC_OFFSET
 431field specified in the VMCS.  Special instructions must be used to read and
 432write the VMCS field.
 433
 4343.8) TSC virtualization - SVM
 435
 436SVM provides conditional trapping of RDTSC, RDMSR, WRMSR and RDTSCP
 437instructions, which is enough for full virtualization of TSC in any manner.  In
 438addition, SVM allows passing through the host TSC plus an additional offset
 439field specified in the SVM control block.
 440
 4413.9) TSC feature bits in Linux
 442
 443In summary, there is no way to guarantee the TSC remains in perfect
 444synchronization unless it is explicitly guaranteed by the architecture.  Even
 445if so, the TSCs in multi-sockets or NUMA systems may still run independently
 446despite being locally consistent.
 447
 448The following feature bits are used by Linux to signal various TSC attributes,
 449but they can only be taken to be meaningful for UP or single node systems.
 450
 451X86_FEATURE_TSC                 : The TSC is available in hardware
 452X86_FEATURE_RDTSCP              : The RDTSCP instruction is available
 453X86_FEATURE_CONSTANT_TSC        : The TSC rate is unchanged with P-states
 454X86_FEATURE_NONSTOP_TSC         : The TSC does not stop in C-states
 455X86_FEATURE_TSC_RELIABLE        : TSC sync checks are skipped (VMware)
 456
 4574) Virtualization Problems
 458
 459Timekeeping is especially problematic for virtualization because a number of
 460challenges arise.  The most obvious problem is that time is now shared between
 461the host and, potentially, a number of virtual machines.  Thus the virtual
 462operating system does not run with 100% usage of the CPU, despite the fact that
 463it may very well make that assumption.  It may expect it to remain true to very
 464exacting bounds when interrupt sources are disabled, but in reality only its
 465virtual interrupt sources are disabled, and the machine may still be preempted
 466at any time.  This causes problems as the passage of real time, the injection
 467of machine interrupts and the associated clock sources are no longer completely
 468synchronized with real time.
 469
 470This same problem can occur on native harware to a degree, as SMM mode may
 471steal cycles from the naturally on X86 systems when SMM mode is used by the
 472BIOS, but not in such an extreme fashion.  However, the fact that SMM mode may
 473cause similar problems to virtualization makes it a good justification for
 474solving many of these problems on bare metal.
 475
 4764.1) Interrupt clocking
 477
 478One of the most immediate problems that occurs with legacy operating systems
 479is that the system timekeeping routines are often designed to keep track of
 480time by counting periodic interrupts.  These interrupts may come from the PIT
 481or the RTC, but the problem is the same: the host virtualization engine may not
 482be able to deliver the proper number of interrupts per second, and so guest
 483time may fall behind.  This is especially problematic if a high interrupt rate
 484is selected, such as 1000 HZ, which is unfortunately the default for many Linux
 485guests.
 486
 487There are three approaches to solving this problem; first, it may be possible
 488to simply ignore it.  Guests which have a separate time source for tracking
 489'wall clock' or 'real time' may not need any adjustment of their interrupts to
 490maintain proper time.  If this is not sufficient, it may be necessary to inject
 491additional interrupts into the guest in order to increase the effective
 492interrupt rate.  This approach leads to complications in extreme conditions,
 493where host load or guest lag is too much to compensate for, and thus another
 494solution to the problem has risen: the guest may need to become aware of lost
 495ticks and compensate for them internally.  Although promising in theory, the
 496implementation of this policy in Linux has been extremely error prone, and a
 497number of buggy variants of lost tick compensation are distributed across
 498commonly used Linux systems.
 499
 500Windows uses periodic RTC clocking as a means of keeping time internally, and
 501thus requires interrupt slewing to keep proper time.  It does use a low enough
 502rate (ed: is it 18.2 Hz?) however that it has not yet been a problem in
 503practice.
 504
 5054.2) TSC sampling and serialization
 506
 507As the highest precision time source available, the cycle counter of the CPU
 508has aroused much interest from developers.  As explained above, this timer has
 509many problems unique to its nature as a local, potentially unstable and
 510potentially unsynchronized source.  One issue which is not unique to the TSC,
 511but is highlighted because of its very precise nature is sampling delay.  By
 512definition, the counter, once read is already old.  However, it is also
 513possible for the counter to be read ahead of the actual use of the result.
 514This is a consequence of the superscalar execution of the instruction stream,
 515which may execute instructions out of order.  Such execution is called
 516non-serialized.  Forcing serialized execution is necessary for precise
 517measurement with the TSC, and requires a serializing instruction, such as CPUID
 518or an MSR read.
 519
 520Since CPUID may actually be virtualized by a trap and emulate mechanism, this
 521serialization can pose a performance issue for hardware virtualization.  An
 522accurate time stamp counter reading may therefore not always be available, and
 523it may be necessary for an implementation to guard against "backwards" reads of
 524the TSC as seen from other CPUs, even in an otherwise perfectly synchronized
 525system.
 526
 5274.3) Timespec aliasing
 528
 529Additionally, this lack of serialization from the TSC poses another challenge
 530when using results of the TSC when measured against another time source.  As
 531the TSC is much higher precision, many possible values of the TSC may be read
 532while another clock is still expressing the same value.
 533
 534That is, you may read (T,T+10) while external clock C maintains the same value.
 535Due to non-serialized reads, you may actually end up with a range which
 536fluctuates - from (T-1.. T+10).  Thus, any time calculated from a TSC, but
 537calibrated against an external value may have a range of valid values.
 538Re-calibrating this computation may actually cause time, as computed after the
 539calibration, to go backwards, compared with time computed before the
 540calibration.
 541
 542This problem is particularly pronounced with an internal time source in Linux,
 543the kernel time, which is expressed in the theoretically high resolution
 544timespec - but which advances in much larger granularity intervals, sometimes
 545at the rate of jiffies, and possibly in catchup modes, at a much larger step.
 546
 547This aliasing requires care in the computation and recalibration of kvmclock
 548and any other values derived from TSC computation (such as TSC virtualization
 549itself).
 550
 5514.4) Migration
 552
 553Migration of a virtual machine raises problems for timekeeping in two ways.
 554First, the migration itself may take time, during which interrupts cannot be
 555delivered, and after which, the guest time may need to be caught up.  NTP may
 556be able to help to some degree here, as the clock correction required is
 557typically small enough to fall in the NTP-correctable window.
 558
 559An additional concern is that timers based off the TSC (or HPET, if the raw bus
 560clock is exposed) may now be running at different rates, requiring compensation
 561in some way in the hypervisor by virtualizing these timers.  In addition,
 562migrating to a faster machine may preclude the use of a passthrough TSC, as a
 563faster clock cannot be made visible to a guest without the potential of time
 564advancing faster than usual.  A slower clock is less of a problem, as it can
 565always be caught up to the original rate.  KVM clock avoids these problems by
 566simply storing multipliers and offsets against the TSC for the guest to convert
 567back into nanosecond resolution values.
 568
 5694.5) Scheduling
 570
 571Since scheduling may be based on precise timing and firing of interrupts, the
 572scheduling algorithms of an operating system may be adversely affected by
 573virtualization.  In theory, the effect is random and should be universally
 574distributed, but in contrived as well as real scenarios (guest device access,
 575causes of virtualization exits, possible context switch), this may not always
 576be the case.  The effect of this has not been well studied.
 577
 578In an attempt to work around this, several implementations have provided a
 579paravirtualized scheduler clock, which reveals the true amount of CPU time for
 580which a virtual machine has been running.
 581
 5824.6) Watchdogs
 583
 584Watchdog timers, such as the lock detector in Linux may fire accidentally when
 585running under hardware virtualization due to timer interrupts being delayed or
 586misinterpretation of the passage of real time.  Usually, these warnings are
 587spurious and can be ignored, but in some circumstances it may be necessary to
 588disable such detection.
 589
 5904.7) Delays and precision timing
 591
 592Precise timing and delays may not be possible in a virtualized system.  This
 593can happen if the system is controlling physical hardware, or issues delays to
 594compensate for slower I/O to and from devices.  The first issue is not solvable
 595in general for a virtualized system; hardware control software can't be
 596adequately virtualized without a full real-time operating system, which would
 597require an RT aware virtualization platform.
 598
 599The second issue may cause performance problems, but this is unlikely to be a
 600significant issue.  In many cases these delays may be eliminated through
 601configuration or paravirtualization.
 602
 6034.8) Covert channels and leaks
 604
 605In addition to the above problems, time information will inevitably leak to the
 606guest about the host in anything but a perfect implementation of virtualized
 607time.  This may allow the guest to infer the presence of a hypervisor (as in a
 608red-pill type detection), and it may allow information to leak between guests
 609by using CPU utilization itself as a signalling channel.  Preventing such
 610problems would require completely isolated virtual time which may not track
 611real time any longer.  This may be useful in certain security or QA contexts,
 612but in general isn't recommended for real-world deployment scenarios.
 613
lxr.linux.no kindly hosted by Redpill Linpro AS, provider of Linux consulting and operations services since 1995.