Skip to content

Commit 5044175

Browse files
paulrosspaulross
committed
Debug and PyConUK presentation
Added Debug section to the docs. Added presentation made at PyConUK September 2015.
1 parent 94ffa0a commit 5044175

18 files changed

+1315
-9
lines changed

‎.gitignore

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,4 @@
11
src/build/*
22
*.so
33
PythonExtensionPatterns/*
4+
doc/sphinx/build/*

‎Introduction — Python Extension Patterns 0.1.0 documentation.pdf

788 KB
Binary file not shown.

‎doc/presentations/PyConUK_2015_HereBeDragons_v05.pdf

1.27 MB
Binary file not shown.

‎doc/sphinx/source/debugging/debug.rst

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
.. highlight:: python
2+
:linenothreshold: 10
3+
4+
.. toctree::
5+
:maxdepth: 3
6+
7+
*****************
8+
Debugging
9+
*****************
10+
11+
This is very much work in progress. I will add to it/correct it as I develop new techniques.
12+
13+
.. toctree::
14+
:maxdepth: 3
15+
16+
debug_tools
17+
debug_python
18+
valgrind
19+
leak_newrefs_vg
20+
debug_tactics

‎doc/sphinx/source/debugging/debug_python.rst

Lines changed: 348 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,348 @@
1+
.. highlight:: python
2+
:linenothreshold: 10
3+
4+
.. highlight:: c
5+
:linenothreshold: 10
6+
7+
.. toctree::
8+
:maxdepth: 3
9+
10+
.. _debug-version-of-python-label:
11+
12+
===============================================
13+
Building and Using a Debug Version of Python
14+
===============================================
15+
16+
There is a spectrum of debug builds of Python that you can create. This chapter describes how to create them.
17+
18+
19+
--------------------------------------------
20+
Building a Standard Debug Version of Python
21+
--------------------------------------------
22+
23+
Download and unpack the Python source. Then in the source directory create a debug directory for the debug build:
24+
25+
.. code-block:: bash
26+
27+
mkdir debug
28+
cd debug
29+
../configure --with-pydebug
30+
make
31+
make test
32+
33+
-----------------------
34+
Specifying Macros
35+
-----------------------
36+
37+
They can be specified at the configure stage, this works:
38+
39+
.. code-block:: bash
40+
41+
../configure CFLAGS='-DPy_DEBUG -DPy_TRACE_REFS' --with-pydebug
42+
make
43+
44+
45+
However the python documentation suggests the alternative way of specifying them when invoking make:
46+
47+
.. code-block:: bash
48+
49+
../configure --with-pydebug
50+
make EXTRA_CFLAGS="-DPy_REF_DEBUG"
51+
52+
I don't know why one way would be regarded as better than the other.
53+
54+
---------------------------
55+
The Debug Builds
56+
---------------------------
57+
58+
The builds are controlled by the following macros:
59+
60+
61+
=================== ======================================================= ==============
62+
Macro Description Must Rebuild
63+
Extensions?
64+
=================== ======================================================= ==============
65+
``Py_DEBUG`` A standard debug build. ``Py_DEBUG`` implies Yes
66+
``LLTRACE``, ``Py_REF_DEBUG``, ``Py_TRACE_REFS``, and
67+
``PYMALLOC_DEBUG`` (if ``WITH_PYMALLOC`` is enabled).
68+
``Py_REF_DEBUG`` Turn on aggregate reference counting which will be No
69+
displayed in the interactive interpreter when
70+
invoked with ``-X showrefcount`` on the command line.
71+
If you are not keeping references to objects and the
72+
count is increasing there is probably a leak.
73+
Also adds ``sys.gettotalrefcount()`` to the ``sys``
74+
module and this returns the total number of references.
75+
``Py_TRACE_REFS`` Turns on reference tracing. Yes
76+
Implies ``Py_REF_DEBUG``.
77+
``COUNT_ALLOCS`` Keeps track of the number of objects of each type have Yes
78+
been allocated and how many freed.
79+
See: :ref:`debug-version-of-python-COUNT_ALLOCS-label`
80+
``WITH_PYMALLOC`` Enables Pythons small memory allocator. For Valgrind No
81+
this must be disabled, if using Pythons malloc
82+
debugger (using ``PYMALLOC_DEBUG``) this must be
83+
enabled.
84+
See: :ref:`debug-version-of-python-memory_alloc-label`
85+
``PYMALLOC_DEBUG`` Enables Python's malloc debugger that annotates No
86+
memory blocks. Requires ``WITH_PYMALLOC``.
87+
See: :ref:`debug-version-of-python-memory_alloc-label`
88+
=================== ======================================================= ==============
89+
90+
91+
92+
In the source directory:
93+
94+
.. code-block:: bash
95+
96+
mkdir debug
97+
cd debug
98+
../configure --with-pydebug
99+
make
100+
make test
101+
102+
103+
.. _debug-version-of-python-memory_alloc-label:
104+
105+
---------------------------
106+
Python's Memory Allocator
107+
---------------------------
108+
109+
A normal build of Python gives CPython a special memory allocator 'PyMalloc'. When enabled this mallocs largish chunks of memory from the OS and then uses this pool for the actual PyObjects. With PyMalloc active Valgrind can not see all allocations and deallocations.
110+
111+
There are two Python builds of interest to help solve memory problems:
112+
113+
* Disable PyMalloc so that Valgrind can analyse the memory usage.
114+
* Enable PyMalloc in debug mode, this creates memory blocks with special bit patterns and adds debugging information on each end of any dynamically allocated memory. This pattern is checked on every alloc/free and if found to be corrupt a diagnostic is printed and the process terminated.
115+
116+
To make a version of Python with its memory allocator suitable for use with Valgrind:
117+
118+
.. code-block:: bash
119+
120+
../configure --with-pydebug --without-pymalloc
121+
make
122+
123+
See :ref:`valgrind-label` for using Valgrind.
124+
125+
To make a version of Python with its memory allocator using Python's malloc debugger either:
126+
127+
.. code-block:: bash
128+
129+
../configure CFLAGS='-DPYMALLOC_DEBUG' --with-pydebug
130+
make
131+
132+
Or:
133+
134+
.. code-block:: bash
135+
136+
../configure --with-pydebug
137+
make EXTRA_CFLAGS="-DPYMALLOC_DEBUG"
138+
139+
This builds Python with the ``WITH_PYMALLOC`` and ``PYMALLOC_DEBUG`` macros defined.
140+
141+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
142+
Finding Access after Free With ``PYMALLOC_DEBUG``
143+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
144+
145+
Python built with ``PYMALLOC_DEBUG`` is the most effective way of detecting access after free. For example if we have this CPython code:
146+
147+
.. code-block:: c
148+
149+
static PyObject *access_after_free(PyObject *pModule) {
150+
PyObject *pA = PyLong_FromLong(1024L);
151+
Py_DECREF(pA);
152+
PyObject_Print(pA, stdout, 0);
153+
Py_RETURN_NONE;
154+
}
155+
156+
And we call this from the interpreter we get a diagnostic:
157+
158+
.. code-block:: python
159+
160+
Python 3.4.3 (default, Sep 16 2015, 16:56:10)
161+
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.51)] on darwin
162+
Type "help", "copyright", "credits" or "license" for more information.
163+
>>> import cPyRefs
164+
>>> cPyRefs.afterFree()
165+
<refcnt -2604246222170760229 at 0x10a474130>
166+
>>>
167+
168+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
169+
Getting Statistics on PyMalloc
170+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
171+
172+
If the environment variable ``PYTHONMALLOCSTATS`` exists when running Python built with ``WITH_PYMALLOC``+``PYMALLOC_DEBUG`` then a (detailed) report of pymalloc activity is output on stderr whenever a new 'arena' is allocated.
173+
174+
.. code-block:: bash
175+
176+
PYTHONMALLOCSTATS=1 python.exe
177+
178+
I have no special knowledge about the output you see when running Python this way which looks like this::
179+
180+
>>> cPyRefs.leakNewRefs(1000, 10000)
181+
loose_new_reference: value=1000 count=10000
182+
Small block threshold = 512, in 64 size classes.
183+
184+
class size num pools blocks in use avail blocks
185+
----- ---- --------- ------------- ------------
186+
4 40 2 139 63
187+
5 48 1 2 82
188+
...
189+
62 504 3 21 3
190+
63 512 3 18 3
191+
192+
# times object malloc called = 2,042,125
193+
# arenas allocated total = 636
194+
# arenas reclaimed = 1
195+
# arenas highwater mark = 635
196+
# arenas allocated current = 635
197+
635 arenas * 262144 bytes/arena = 166,461,440
198+
199+
# bytes in allocated blocks = 162,432,624
200+
# bytes in available blocks = 116,824
201+
0 unused pools * 4096 bytes = 0
202+
# bytes lost to pool headers = 1,950,720
203+
# bytes lost to quantization = 1,961,272
204+
# bytes lost to arena alignment = 0
205+
Total = 166,461,440
206+
Small block threshold = 512, in 64 size classes.
207+
208+
class size num pools blocks in use avail blocks
209+
----- ---- --------- ------------- ------------
210+
4 40 2 139 63
211+
5 48 1 2 82
212+
...
213+
62 504 3 21 3
214+
63 512 3 18 3
215+
216+
# times object malloc called = 2,045,325
217+
# arenas allocated total = 637
218+
# arenas reclaimed = 1
219+
# arenas highwater mark = 636
220+
# arenas allocated current = 636
221+
636 arenas * 262144 bytes/arena = 166,723,584
222+
223+
# bytes in allocated blocks = 162,688,624
224+
# bytes in available blocks = 116,824
225+
0 unused pools * 4096 bytes = 0
226+
# bytes lost to pool headers = 1,953,792
227+
# bytes lost to quantization = 1,964,344
228+
# bytes lost to arena alignment = 0
229+
Total = 166,723,584
230+
Small block threshold = 512, in 64 size classes.
231+
232+
class size num pools blocks in use avail blocks
233+
----- ---- --------- ------------- ------------
234+
4 40 2 139 63
235+
5 48 1 2 82
236+
...
237+
62 504 3 21 3
238+
63 512 3 18 3
239+
240+
# times object malloc called = 2,048,525
241+
# arenas allocated total = 638
242+
# arenas reclaimed = 1
243+
# arenas highwater mark = 637
244+
# arenas allocated current = 637
245+
637 arenas * 262144 bytes/arena = 166,985,728
246+
247+
# bytes in allocated blocks = 162,944,624
248+
# bytes in available blocks = 116,824
249+
0 unused pools * 4096 bytes = 0
250+
# bytes lost to pool headers = 1,956,864
251+
# bytes lost to quantization = 1,967,416
252+
# bytes lost to arena alignment = 0
253+
Total = 166,985,728
254+
loose_new_reference: DONE
255+
256+
257+
.. _debug-version-of-python-COUNT_ALLOCS-label:
258+
259+
-----------------------------------------------
260+
Python Debug build with ``COUNT_ALLOCS``
261+
-----------------------------------------------
262+
263+
A Python debug build with ``COUNT_ALLOCS`` give some additional information about each object *type* (not the individual objects themselves). A ``PyObject`` grows some extra fields that track the reference counts for that type. The fields are:
264+
265+
=============== ====================================================================
266+
Field Description
267+
=============== ====================================================================
268+
``tp_allocs`` The number of times an object of this type was allocated.
269+
``tp_frees`` The number of times an object of this type was freed.
270+
``tp_maxalloc`` The maximum seen value of ``tp_allocs - tp_frees`` so this is the
271+
maximum count of this type allocated at the same time.
272+
=============== ====================================================================
273+
274+
The ``sys`` module also gets an extra function ``sys.getcounts()`` that returns a list of tuples: ``[(tp_typename, tp_allocs, tp_frees, tp_maxalloc), ...]``.
275+
276+
277+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
278+
Building the Python Executable with ``COUNT_ALLOCS``
279+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
280+
281+
Either:
282+
283+
.. code-block:: bash
284+
285+
../configure CFLAGS='-DCOUNT_ALLOCS' --with-pydebug
286+
make
287+
288+
Or:
289+
290+
.. code-block:: bash
291+
292+
../configure --with-pydebug
293+
make EXTRA_CFLAGS="-DCOUNT_ALLOCS"
294+
295+
.. warning::
296+
297+
When using ``COUNT_ALLOCS`` any Python extensions now need to be rebuilt with this Python executable as it fundementally changes the structure of a ``PyObject``.
298+
299+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
300+
Using the Python Executable with ``COUNT_ALLOCS``
301+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
302+
303+
An example of using this build is here: :ref:`leaked-new-references-usingCOUNT_ALLOCS-label`
304+
305+
-----------------------------------------------------------
306+
Identifying the Python Build Configuration from the Runtime
307+
-----------------------------------------------------------
308+
309+
The module ``sysconfig`` allows you access to the configuration of the Python runtime. At its simplest, and most verbose, this can be used thus:
310+
311+
.. code-block:: bash
312+
313+
$ ./python.exe -m sysconfig
314+
Platform: "macosx-10.9-x86_64"
315+
Python version: "3.4"
316+
Current installation scheme: "posix_prefix"
317+
318+
Paths:
319+
data = "/usr/local"
320+
...
321+
stdlib = "/usr/local/lib/python3.4"
322+
323+
Variables:
324+
ABIFLAGS = "dm"
325+
AC_APPLE_UNIVERSAL_BUILD = "0"
326+
AIX_GENUINE_CPLUSPLUS = "0"
327+
AR = "ar"
328+
...
329+
py_version = "3.4.3"
330+
py_version_nodot = "34"
331+
py_version_short = "3.4"
332+
333+
334+
Importing ``sysconfig`` into an interpreter session gives two useful functions are ``get_config_var(...)`` which gets the setting for a particular macro and ``get_config_vars()`` which gets a dict of ``{macro : value, ...}``. For example:
335+
336+
.. code-block:: python
337+
338+
>>> import sysconfig
339+
>>> sysconfig.get_config_var('Py_DEBUG')
340+
1
341+
342+
For advanced usage you can parse any ``pyconfig.h`` into a dict by opening that file and passing it to ``sysconfig.parse_config_h(f)`` as a file object. ``sysconfig.get_config_h_filename()`` will give you the configuration file for the runtime (assuming it still exists). So:
343+
344+
.. code-block:: python
345+
346+
>>> with open(sysconfig.get_config_h_filename()) as f:
347+
cfg = sysconfig.parse_config_h(f)
348+

0 commit comments

Comments
 (0)