How to Resolve

• It is useful to Identify if the offset value spikes periodically or randomly. This can be done by plotting the system clock offset value against the sfptpd runtime using the system clock output lines from the sfptpd stats_log.
• The stats_log output is updated every second, so over a 60 second runtime period there will be ~60 output lines for each active sync instance and for the system clock.
• Plotting the system clock values over a period of time such as 12 hours or 24 hours, should be sufficient to identify offset outliers and spike patterns.
• Run a system tap script to identify if any user-level process is periodically adjusting or interfering with the system clock.

An example system tap script (sysclock_monitor.stp):

probe nd_syscall.adjtimex, nd_syscall.compat_adjtimex ?
{
  printf("1. %s (pid=%d) %s\n", execname(),pid(),pp())
}
probe nd_syscall.clock_adjtime ?
{
  printf("2. %s (pid=%d) %s\n", execname(),pid(),pp())
}
probe nd_syscall.settimeofday ?
{
  printf("3. %s (pid=%d) %s\n", execname(),pid(),pp())
}
probe nd_syscall.stime ?
{
  printf("4. %s (pid=%d) %s\n", execname(),pid(),pp())
}
probe nd_syscall.clock_settime ?
{
  printf("5. %s (pid=%d) %s\n", execname(),pid(),pp())
}

The script file should be made executable and can then be run on the system (with or without sfptpd running):

# stap -v sysclock_monitor.stp
Pass 1: parsed user script and 93 library script(s) using 201532virt/29960res/3064shr/27464data kb, in 110usr/0sys/120real ms.
Pass 2: analyzed script: 6 probe(s), 3 function(s), 25 embed(s), 0 global(s) using 205180virt/32264res/4012shr/28652data kb, in 10usr/40sys/51real ms.
Pass 3: translated to C into "/tmp/stap8mqN09/stap_db5e04ccc51fad4e17bc414fdb1c3685_15211_src.c" using 205180virt/32500res/4232shr/28652data kb, in 0usr/0sys/0real ms.
Pass 4: compiled C into "stap_db5e04ccc51fad4e17bc414fdb1c3685_15211.ko" in 720usr/130sys/917real ms.
Pass 5: starting run.

Example output when sfptpd and chronyd are running:

1. sfptpd (pid=22803) kprobe.function("sys_adjtimex")?
1. sfptpd (pid=22803) kprobe.function("sys_adjtimex")?
1. sfptpd (pid=22803) kprobe.function("sys_adjtimex")?
1. chronyd (pid=1534) kprobe.function("sys_adjtimex")?
1. chronyd (pid=1534) kprobe.function("sys_adjtimex")?
1. chronyd (pid=1534) kprobe.function("sys_adjtimex")?

The following script will identify any user-level process trying to adjust the system clock, but will not report sfptpd :

probe nd_syscall.adjtimex, nd_syscall.compat_adjtimex ?
{
  if (execname() != "sfptpd"){
    printf("1. %s (pid=%d) %s\n", execname(),pid(),pp())
    exit()
  }
}
probe nd_syscall.clock_adjtime ?
{
  if (execname() != "sfptpd"){
    printf("2. %s (pid=%d) %s\n", execname(),pid(),pp())
    exit()
  }
}
probe nd_syscall.settimeofday ?
{
  if (execname() != "sfptpd"){
    printf("3. %s (pid=%d) %s\n", execname(),pid(),pp())
    exit()
  }
}
probe nd_syscall.stime ?
{
  if (execname() != "sfptpd"){
    printf("4. %s (pid=%d) %s\n", execname(),pid(),pp())
    exit()
  }
}
probe nd_syscall.clock_settime ?
{
  if (execname() != "sfptpd"){
    printf("5. %s (pid=%d) %s\n", execname(),pid(),pp())
    exit()
  }
}

If there is no output from running this example system tap script, then only the sfptpd process is adjusting the system clock.