Measure the resource overhead of NodeProblemDetector

[Random-Liu]

To roll out the NodeProblemDetector and run it as an addon pods of Kubernetes, we need to understand its overhead, at least including cpu and memory overhead.

/cc @kubernetes/sig-node

[Random-Liu]

I've done some measurement. Here is the result:

• VM template: n1-standard-2 (2 vCPUs, 7.5 GB memory)

In the table above:

• Type is the problem type corresponding to the inserted log.
• Frequency is lines of log inserted every second.
• CPU is the stable cpu usage of the NodeProblemDetector.
• Memory is the stable memory usage of the NodeProblemDetector.

@dchen1107

[Random-Liu]

According to the result above #2 (comment), the following configuration should be enough for now:

limits:
  cpu: 100m
  memory: 50Mi
requests:
  cpu: 100m
  memory: 50Mi

Because:

• 100 permanent/temporary problems/s should be rare, even the most frequent problem till now kernel crash after "unregister_netdevice: waiting for lo to become free. Usage count = 3" moby/moby#5618 only produces one line in several seconds.
• Even the problem produces 100 lines of logs per second, the resource limit should also be enough.

@dchen1107 WDYT?

[vishh]

nit: Requests need not be specified if this job is expected to be
Guaranteed.

On Thu, May 19, 2016 at 1:23 AM, Lantao Liu notifications@github.com
wrote:

According to the result above #2 (comment)
#2 (comment),
the following configuration should be enough for now:

limits:
cpu: 100m
memory: 50Mi
requests:
cpu: 100m
memory: 50Mi

Because:

• 100 permanent/temporary problems/s should be rare, even the most
  frequent problem till now kernel crash after "unregister_netdevice: waiting for lo to become free. Usage count = 3" moby/moby#5618
  kernel crash after "unregister_netdevice: waiting for lo to become free. Usage count = 3" moby/moby#5618 only produces one line
  in several seconds.
• Even the problem produces 100 lines of logs per second, the resource
  limit should also be enough.

—
You are receiving this because you are on a team that was mentioned.
Reply to this email directly or view it on GitHub
#2 (comment)

[Random-Liu]

@vishh ACK. Thank you very much~ :)

[Random-Liu]

Close this since this is done in kubernetes/kubernetes#25986.