Python and irqstat

While I was debugging irqbalance issue on a 4 socket Nodes machine, I realized that it is difficult to understand the perf data gathered by mpstat and /proc/interrupts. On this machine, there are 60 logical CPUs, it is very difficult to understand the output without tools helps. That is why I wrote some interrupt tools to help analysis on these outputs.

After I know irqstat, I was really excited, and I thought this is the tool what I need for live monitor and analysis irq workload issues. I decided to integrate this tool into our product. However, irqstat does not provide the license information, which is quite important to us, as our legal team need that information for legal review process. Therefore, I learnt how to file a bug/RFE via Github.

Several weeks later, I found another problem of irqstat. It could not be run at background mode due to its terminal control code. Also the its Python stdout is a kind of buffered IO, which will cause output could not be flushed immediately via the shell pipe. Thus, I worked out some patches to fix the problems. This link could get all my patches to irqstat.

Irqstat is a small tool written by Python, but Python is not my programming language. Now it provided me a good opportunity to learn a new language. Python will be on my script language learning list, as I plan to do more performance related work in next year. With this efficient language, I think I will create more useful tools in my daily work.

Here is the output of irqstat on my 60 CPUs box. You can see that all interrupts are calculated as NUMA nodes, which is pretty cool:

# irqstat
interactive commands -- t: view totals, 0-9: view node, any other key: quit
IRQs / 5 second(s)
IRQ#     TOTAL     NODE0      NODE1      NODE2      NODE3  NAME
 136 337950405 337950405          0          0          0  PCI-MSI-edge ahci
 284 121748888  51769535    3822068   56924905    9232380  PCI-MSI-edge eth7a-fp-0
 294 116510649  15737953   28284037   41159203   31329456  PCI-MSI-edge eth7b-fp-0
 287 116074867   8853954   20284359   57160013   29776541  PCI-MSI-edge eth7a-fp-3
 298 112651243  33515406    5778308   22191092   51166437  PCI-MSI-edge eth7b-fp-4
 286 112159610   9427174    6496883   61879949   34355604  PCI-MSI-edge eth7a-fp-2
 301 110159560  26117566   21492156   17516565   45033273  PCI-MSI-edge eth7b-fp-7
 291 109723482  11233439   15504015   58302343   24683685  PCI-MSI-edge eth7a-fp-7
 290 109533013  25661958    8572641   10357938   64940476  PCI-MSI-edge eth7a-fp-6
Oliver Yang /
Published under (CC) BY-NC-SA in categories English  Software  tagged with irqbalance  perf  python