Performance Data Helper in Python with win32pdh

From CAC Documentation wiki
Jump to: navigation, search

25 June 2009
by Drew Dolgert, Cornell Center for Advanced Computing

Summary: The Python win32pdh module simplifies access to Windows performance counters, the same ones perfmon shows. It will let you watch memory usage and many other statistics of applications and the operating system. The Python module isn't well documented, though, so this article shows some sample code and describes its functions in more detail.

References

This refers to Python 2.5 and pywin32-212. For background on the Performance Data Helper library, see:

Concepts

To use this library, you construct a query, to which you add counters. There are many counters available. You specify a counter with a path:

  • object - These are registered with the operating system. Examples are 'Process', and 'System'.
  • counter - An object contains several counters which can be monitored. For the Process object, this includes "% User Time" and "Working Set." (Note there are two uses of the word counter, one an object you add to a query, and another a string to specify part of the path.)
  • instance - A Process object provides counters for all processes running on the system: csrss, lsass, wininit, cmd. Each of these processes is an example of an instance.
  • instance index - Because instances are listed under a counter by their name, not PID, the instance index distinguishes among several instances with the same name.

The full path of a counter includes the object, instance, index, and counter name:

\Process(svchost#1)\IO Other Bytes/sec

Find Available Counters

The win32pdh library provides a nice method to show a window where you can browse available counters.

from win32pdh import *
def print_counter(counter):
    print counter
BrowseCounters(None, 0, print_counter, PERF_DETAIL_WIZARD, "Counter List")

If you select a counter and click the Add button, this will print the counter's path. Note that counters, as shown here, are not full paths.

To look through counters in a program, use EnumObjects, and EnumObjectItems.

from win32pdh import *
objects = EnumObjects(None, None, -1, 0)
   counter_data = {}
   for obj in objects:
       (counters, instances) = EnumObjectItems(None, None, obj, -1, 0)

The -1 argument is a "detail level." This value lists all members. The counters variable now contains the names of all counters within the given object, and the instances variable shows the names of all instances. Some instance names will be repeated, which means you would need to use their index to query them.

Build a Path to a Counter

You can only retrieve a single counter with any call to win32pdh, so each path must include a specific object, counter, instance, and instance index. A helper function builds a counter path from these.

path = MakeCounterPath((None,obj,instance,None,instance_index,counter),0)
if ValidatePath(path) is 0:
    print "The path is good"

Why is it important to validate the path with win32pdh.ValidatePath? Recall that the list of instances includes duplicate names. Given that you often want to get information on all instances of, say, your web server, you can keep making counter paths, incrementing the instance index, until ValidatePath tells you to stop.

Construct a Query with Counters

In Python, queries and counters are represented by integers which correspond to Windows HANDLEs. You have to remember the counter handle to get data later.

query_handle = OpenQuery()
counter_handle = AddCounter(query_handle, path)
# and later...
CloseQuery(query_handle)

Collect Data

Some performance counters are instantaneous. Others require you collect data for a few seconds. See references for details. A safe bet is to collect data twice before getting values. The amount of time to sleep in between collections can be important to interpreting resulting data.

query_handle = OpenQuery()
counter_handle = AddCounter(query_handle, path)
CollectQueryData(query_handle)
time.sleep(2)
CollectQueryData(query_handle)
(counter_type, value) = GetFormattedCounterValue(counter_handle, PDH_FMT_DOUBLE)
CloseQuery(query_handle)

There are several possible formatting constants to use with GetFormattedCounterValue. Take one of the first three and, optionally, OR it with the last two.

  • PDH_FMT_DOUBLE - Retrieve a floating point value. This is a good default.
  • PDH_FMT_LARGE - Retrieve a 64-bit value. The Python wrapper does not handle this possibility, so don't use it.
  • PDH_FMT_LONG - Store result into a long instead of the default, a DWORD.

The above can be OR'd with

  • PDH_FMT_NOSCALE - The GetFormattedCounterValue method automatically does some scaling, and this turns it off.
  • PDH_FMT_1000 - Multiply the result by 1000.

The return value includes both a counter type and the value itself. If the path no longer exists, which usually means you were monitoring a process that ended, then this method will throw an exception without an error message. After that, you may as well not check this counter again.

That counter type has a lot of information. It is OR'd constants from WinPerf.h:

   PERF_SIZE_DWORD         = 0x00000000
   PERF_SIZE_LARGE         = 0x00000100
   PERF_SIZE_ZERO          = 0x00000200  # for Zero Length fields
   PERF_SIZE_VARIABLE_LEN  = 0x00000300  # length is in CounterLength field
                                         #  of Counter Definition struct
   #
   #  select one of the following values to indicate the counter field usage
   #
   PERF_TYPE_NUMBER        = 0x00000000  # a number (not a counter)
   PERF_TYPE_COUNTER       = 0x00000400  # an increasing numeric value
   PERF_TYPE_TEXT          = 0x00000800  # a text field
   PERF_TYPE_ZERO          = 0x00000C00  # displays a zero
   #
   #  If the PERF_TYPE_NUMBER field was selected, then select one of the
   #  following to describe the Number
   #
   PERF_NUMBER_HEX         = 0x00000000  # display as HEX value
   PERF_NUMBER_DECIMAL     = 0x00010000  # display as a decimal integer
   PERF_NUMBER_DEC_1000    = 0x00020000  # display as a decimal/1000
   #
   #  If the PERF_TYPE_COUNTER value was selected then select one of the
   #  following to indicate the type of counter
   #
   PERF_COUNTER_VALUE      = 0x00000000  # display counter value
   PERF_COUNTER_RATE       = 0x00010000  # divide ctr / delta time
   PERF_COUNTER_FRACTION   = 0x00020000  # divide ctr / base
   PERF_COUNTER_BASE       = 0x00030000  # base value used in fractions
   PERF_COUNTER_ELAPSED    = 0x00040000  # subtract counter from current time
   PERF_COUNTER_QUEUELEN   = 0x00050000  # Use Queuelen processing func.
   PERF_COUNTER_HISTOGRAM  = 0x00060000  # Counter begins or ends a histogram
   #
   #  If the PERF_TYPE_TEXT value was selected, then select one of the
   #  following to indicate the type of TEXT data.
   #
   PERF_TEXT_UNICODE       = 0x00000000  # type of text in text field
   PERF_TEXT_ASCII         = 0x00010000  # ASCII using the CodePage field
   #
   #  Timer SubTypes
   #
   PERF_TIMER_TICK         = 0x00000000  # use system perf. freq for base
   PERF_TIMER_100NS        = 0x00100000  # use 100 NS timer time base units
   PERF_OBJECT_TIMER       = 0x00200000  # use the object timer freq
   #
   #  Any types that have calculations performed can use one or more of
   #  the following calculation modification flags listed here
   #
   PERF_DELTA_COUNTER      = 0x00400000  # compute difference first
   PERF_DELTA_BASE         = 0x00800000  # compute base diff as well
   PERF_INVERSE_COUNTER    = 0x01000000  # show as 1.00-value (assumes:
   PERF_MULTI_COUNTER      = 0x02000000  # sum of multiple instances
   #
   #  Select one of the following values to indicate the display suffix (if any)
   #
   PERF_DISPLAY_NO_SUFFIX  = 0x00000000  # no suffix
   PERF_DISPLAY_PER_SEC    = 0x10000000  # "/sec"
   PERF_DISPLAY_PERCENT    = 0x20000000  # "%"
   PERF_DISPLAY_SECONDS    = 0x30000000  # "secs"
   PERF_DISPLAY_NOSHOW     = 0x40000000  # value is not displayed

If you store the counter type, then you don't have to get it later with a call to GetCounterInfo.

GetCounterInfo

GetCounterInfo(counter_handle, bRetrieveExplainText)
counter_handle - Handle to a counter created with AddCounter.
bRetrieveExplainText - 0 or 1. Whether to get an explanation of what is in the counter.
                       This is always NULL for counters in log files.
returns: (dwType, CVersion, lScale, lDefaultScale, dwUserData, dwQueryUserData,
          szFullPath, (szMachineName, szObjectName, szInstanceName, szParentInstance,
          dwInstanceIndex, szCounterName), szExplainText)

bRetrieveExplainText never seems to have anything relevant and shows random characters, which means there may be a problem in the wrapper code for this function. The return values are described in Microsoft documentation. dwType is the same as counter_type above. It describes what kind of counter this is, including how to format it.

Calling GetCounterInfo immediately after creating the counter can, often as not, lead to a crash in python25.dll. Calling it after retrieving data seems more successful, but, if you look, there is nothing in there you want. Store the object, instance, index, and counter when you create the counter. Store the type when you get the data.

Sample Counter Class

This example makes a Counter class and then uses it. This class retrieves all values as doubles and does no formatting. All the definitions are there to refine formatting.

from time import sleep
from win32pdh import *

class Counter(object):
    # The dwType field from GetCounterInfo returns the following, or'ed.
    # These come from WinPerf.h
    PERF_SIZE_DWORD         = 0x00000000
    PERF_SIZE_LARGE         = 0x00000100
    PERF_SIZE_ZERO          = 0x00000200  # for Zero Length fields
    PERF_SIZE_VARIABLE_LEN  = 0x00000300  # length is in CounterLength field
                                                #  of Counter Definition struct
    #
    #  select one of the following values to indicate the counter field usage
    #
    PERF_TYPE_NUMBER        = 0x00000000  # a number (not a counter)
    PERF_TYPE_COUNTER       = 0x00000400  # an increasing numeric value
    PERF_TYPE_TEXT          = 0x00000800  # a text field
    PERF_TYPE_ZERO          = 0x00000C00  # displays a zero
    #
    #  If the PERF_TYPE_NUMBER field was selected, then select one of the
    #  following to describe the Number
    #
    PERF_NUMBER_HEX         = 0x00000000  # display as HEX value
    PERF_NUMBER_DECIMAL     = 0x00010000  # display as a decimal integer
    PERF_NUMBER_DEC_1000    = 0x00020000  # display as a decimal/1000
    #
    #  If the PERF_TYPE_COUNTER value was selected then select one of the
    #  following to indicate the type of counter
    #
    PERF_COUNTER_VALUE      = 0x00000000  # display counter value
    PERF_COUNTER_RATE       = 0x00010000  # divide ctr / delta time
    PERF_COUNTER_FRACTION   = 0x00020000  # divide ctr / base
    PERF_COUNTER_BASE       = 0x00030000  # base value used in fractions
    PERF_COUNTER_ELAPSED    = 0x00040000  # subtract counter from current time
    PERF_COUNTER_QUEUELEN   = 0x00050000  # Use Queuelen processing func.
    PERF_COUNTER_HISTOGRAM  = 0x00060000  # Counter begins or ends a histogram
    #
    #  If the PERF_TYPE_TEXT value was selected, then select one of the
    #  following to indicate the type of TEXT data.
    #
    PERF_TEXT_UNICODE       = 0x00000000  # type of text in text field
    PERF_TEXT_ASCII         = 0x00010000  # ASCII using the CodePage field
    #
    #  Timer SubTypes
    #
    PERF_TIMER_TICK         = 0x00000000  # use system perf. freq for base
    PERF_TIMER_100NS        = 0x00100000  # use 100 NS timer time base units
    PERF_OBJECT_TIMER       = 0x00200000  # use the object timer freq
    #
    #  Any types that have calculations performed can use one or more of
    #  the following calculation modification flags listed here
    #
    PERF_DELTA_COUNTER      = 0x00400000  # compute difference first
    PERF_DELTA_BASE         = 0x00800000  # compute base diff as well
    PERF_INVERSE_COUNTER    = 0x01000000  # show as 1.00-value (assumes:
    PERF_MULTI_COUNTER      = 0x02000000  # sum of multiple instances
    #
    #  Select one of the following values to indicate the display suffix (if any)
    #
    PERF_DISPLAY_NO_SUFFIX  = 0x00000000  # no suffix
    PERF_DISPLAY_PER_SEC    = 0x10000000  # "/sec"
    PERF_DISPLAY_PERCENT    = 0x20000000  # "%"
    PERF_DISPLAY_SECONDS    = 0x30000000  # "secs"
    PERF_DISPLAY_NOSHOW     = 0x40000000  # value is not displayed

    def BuildCounter(obj, instance, instance_index, counter):
        path=MakeCounterPath((None,obj,instance,None,instance_index,counter),0)
        if ValidatePath(path) is 0:
            return Counter(path,obj,instance, instance_index, counter)
        return None
    BuildCounter = staticmethod(BuildCounter)
    
    def __init__(self,path, obj, instance, index, counter):
        self.path = path
        self.obj = obj
        self.instance = instance
        self.index = index
        self.counter = counter
        self.handle = None
        self.info = None
        self.type = None

    def addToQuery(self,query):
        self.handle = AddCounter(query,self.path)

    def getInfo(self):
        '''GetCounterInfo sometimes crashes in the wrapper code. Fewer crashes if 
        this is called after sampling data.'''
        if not self.info:
            ci = GetCounterInfo( self.handle, 0 )
            self.info = {}
            self.info['type']=ci[0]
            self.info['version']=ci[1]
            self.info['scale']=ci[2]
            self.info['defaultScale']=ci[3]
            self.info['userData']=ci[4]
            self.info['queryUserData']=ci[5]
            self.info['fullPath']=ci[6]
            self.info['machineName']=ci[7][0]
            self.info['objectName']=ci[7][1]
            self.info['instanceName']=ci[7][2]
            self.info['parentInstance']=ci[7][3]
            self.info['instanceIndex']=ci[7][4]
            self.info['counterName']=ci[7][5]
            self.info['explainText']=ci[8]
        return self.info
        
    def value(self):
        (counter_type, value) = GetFormattedCounterValue(self.handle, PDH_FMT_DOUBLE)
        self.type = counter_type
        return value

    def typeString(self):
        '''This string shows which bits are set in the dwType from PdhGetInfo.
        It can be used to format the counter.'''
        type=self.getInfo()['type']
        type_list = []
        type_list.append(str(type))
        for member in self.__class__.__dict__.keys():
            if member.startswith("PERF_"):
                bit = getattr(self,member)
                if bit and bit&type:
                    type_list.append(member[5:])
        return " ".join(type_list)
        
    def __str__(self):
        return self.path

def build_query_paths(object_counter_list, instance_list):
    '''For each (object,counter) iterate through all instances to build a counter path.
    Validate that the path exists.'''
    paths = []
    for (obj, counter_string) in object_counter_list:
        for instance in instance_list:
            instance_index = 0
            counter = Counter.BuildCounter(obj,instance,instance_index,counter_string)
            while counter:
                paths.append(counter)
                instance_index+=1
                counter = Counter.BuildCounter(obj,instance,instance_index,counter_string)
    return paths
    

def build_query_all_counters(obj, instance_list):
    query = OpenQuery()
    (counters, instances) = EnumObjectItems(None, None, obj, -1, 0)
    object_counter_list = zip( [obj]*len(counters), counters )
    counters = build_query_paths(object_counter_list, instance_list)
    for counter in counters:
        counter.addToQuery(query)
    return (query, counters)

def testBuildQueryAllCounters():
    (query,counters)=build_query_all_counters('Process', ('lsass','svchost','csrss'))
    CollectQueryData(query)
    sleep(2)
    CollectQueryData(query)
    for counter in counters:
        print counter,
        print counter.value()
    CloseQuery(query)

if __name__ == "__main__":
    testBuildQueryAllCounters()