This is Info file /home/pdm/tmp/Python-1.5.2p1/Doc/ext/python-ext.info, produced by Makeinfo version 1.68 from the input file ext.texi. July 6, 1999 1.5.2  File: python-ext.info, Node: Ownership Rules, Next: Thin Ice, Prev: Reference Counting in Python, Up: Reference Counts Ownership Rules --------------- Whenever an object reference is passed into or out of a function, it is part of the function's interface specification whether ownership is transferred with the reference or not. Most functions that return a reference to an object pass on ownership with the reference. In particular, all functions whose function it is to create a new object, e.g. `PyInt_FromLong()' and `Py_BuildValue()', pass ownership to the receiver. Even if in fact, in some cases, you don't receive a reference to a brand new object, you still receive ownership of the reference. For instance, `PyInt_FromLong()' maintains a cache of popular values and can return a reference to a cached item. Many functions that extract objects from other objects also transfer ownership with the reference, for instance `PyObject_GetAttrString()'. The picture is less clear, here, however, since a few common routines are exceptions: `PyTuple_GetItem()', `PyList_GetItem()', `PyDict_GetItem()', and `PyDict_GetItemString()' all return references that you borrow from the tuple, list or dictionary. The function `PyImport_AddModule()' also returns a borrowed reference, even though it may actually create the object it returns: this is possible because an owned reference to the object is stored in `sys.modules'. When you pass an object reference into another function, in general, the function borrows the reference from you -- if it needs to store it, it will use `Py_INCREF()' to become an independent owner. There are exactly two important exceptions to this rule: `PyTuple_SetItem()' and `PyList_SetItem()'. These functions take over ownership of the item passed to them -- even if they fail! (Note that `PyDict_SetItem()' and friends don't take over ownership -- they are "normal.") When a C function is called from Python, it borrows references to its arguments from the caller. The caller owns a reference to the object, so the borrowed reference's lifetime is guaranteed until the function returns. Only when such a borrowed reference must be stored or passed on, it must be turned into an owned reference by calling `Py_INCREF()'. The object reference returned from a C function that is called from Python must be an owned reference -- ownership is tranferred from the function to its caller.  File: python-ext.info, Node: Thin Ice, Next: NULL Pointers, Prev: Ownership Rules, Up: Reference Counts Thin Ice -------- There are a few situations where seemingly harmless use of a borrowed reference can lead to problems. These all have to do with implicit invocations of the interpreter, which can cause the owner of a reference to dispose of it. The first and most important case to know about is using `Py_DECREF()' on an unrelated object while borrowing a reference to a list item. For instance: bug(PyObject *list) { PyObject *item = PyList_GetItem(list, 0); PyList_SetItem(list, 1, PyInt_FromLong(0L)); PyObject_Print(item, stdout, 0); /* BUG! */ } This function first borrows a reference to `list[0]', then replaces `list[1]' with the value `0', and finally prints the borrowed reference. Looks harmless, right? But it's not! Let's follow the control flow into `PyList_SetItem()'. The list owns references to all its items, so when item 1 is replaced, it has to dispose of the original item 1. Now let's suppose the original item 1 was an instance of a user-defined class, and let's further suppose that the class defined a `__del__()' method. If this class instance has a reference count of 1, disposing of it will call its `__del__()' method. Since it is written in Python, the `__del__()' method can execute arbitrary Python code. Could it perhaps do something to invalidate the reference to `item' in `bug()'? You bet! Assuming that the list passed into `bug()' is accessible to the `__del__()' method, it could execute a statement to the effect of `del list[0]', and assuming this was the last reference to that object, it would free the memory associated with it, thereby invalidating `item'. The solution, once you know the source of the problem, is easy: temporarily increment the reference count. The correct version of the function reads: no_bug(PyObject *list) { PyObject *item = PyList_GetItem(list, 0); Py_INCREF(item); PyList_SetItem(list, 1, PyInt_FromLong(0L)); PyObject_Print(item, stdout, 0); Py_DECREF(item); } This is a true story. An older version of Python contained variants of this bug and someone spent a considerable amount of time in a C debugger to figure out why his `__del__()' methods would fail... The second case of problems with a borrowed reference is a variant involving threads. Normally, multiple threads in the Python interpreter can't get in each other's way, because there is a global lock protecting Python's entire object space. However, it is possible to temporarily release this lock using the macro `Py_BEGIN_ALLOW_THREADS', and to re-acquire it using `Py_END_ALLOW_THREADS'. This is common around blocking I/O calls, to let other threads use the CPU while waiting for the I/O to complete. Obviously, the following function has the same problem as the previous one: bug(PyObject *list) { PyObject *item = PyList_GetItem(list, 0); Py_BEGIN_ALLOW_THREADS ...some blocking I/O call... Py_END_ALLOW_THREADS PyObject_Print(item, stdout, 0); /* BUG! */ }  File: python-ext.info, Node: NULL Pointers, Prev: Thin Ice, Up: Reference Counts NULL Pointers ------------- In general, functions that take object references as arguments do not expect you to pass them `NULL' pointers, and will dump core (or cause later core dumps) if you do so. Functions that return object references generally return `NULL' only to indicate that an exception occurred. The reason for not testing for `NULL' arguments is that functions often pass the objects they receive on to other function -- if each function were to test for `NULL', there would be a lot of redundant tests and the code would run slower. It is better to test for `NULL' only at the "source", i.e. when a pointer that may be `NULL' is received, e.g. from `malloc()' or from a function that may raise an exception. The macros `Py_INCREF()' and `Py_DECREF()' do not check for `NULL' pointers -- however, their variants `Py_XINCREF()' and `Py_XDECREF()' do. The macros for checking for a particular object type (`PyTYPE_Check()') don't check for `NULL' pointers -- again, there is much code that calls several of these in a row to test an object against various different expected types, and this would generate redundant tests. There are no variants with `NULL' checking. The C function calling mechanism guarantees that the argument list passed to C functions (`args' in the examples) is never `NULL' -- in fact it guarantees that it is always a tuple.(1) It is a severe error to ever let a `NULL' pointer "escape" to the Python user. ---------- Footnotes ---------- (1) These guarantees don't hold when you use the "old" style calling convention -- this is still found in much existing code.  File: python-ext.info, Node: Writing Extensions in C++, Next: Providing a C API for an Extension Module, Prev: Reference Counts, Up: Extending Python with C or C++ Writing Extensions in C++ ========================= It is possible to write extension modules in C++. Some restrictions apply. If the main program (the Python interpreter) is compiled and linked by the C compiler, global or static objects with constructors cannot be used. This is not a problem if the main program is linked by the C++ compiler. Functions that will be called by the Python interpreter (in particular, module initalization functions) have to be declared using `extern "C"'. It is unnecessary to enclose the Python header files in `extern "C" {...}' -- they use this form already if the symbol `__cplusplus' is defined (all recent C++ compilers define this symbol).  File: python-ext.info, Node: Providing a C API for an Extension Module, Prev: Writing Extensions in C++, Up: Extending Python with C or C++ Providing a C API for an Extension Module ========================================= This section was written by Konrad Hinsen . Many extension modules just provide new functions and types to be used from Python, but sometimes the code in an extension module can be useful for other extension modules. For example, an extension module could implement a type "collection" which works like lists without order. Just like the standard Python list type has a C API which permits extension modules to create and manipulate lists, this new collection type should have a set of C functions for direct manipulation from other extension modules. At first sight this seems easy: just write the functions (without declaring them `static', of course), provide an appropriate header file, and document the C API. And in fact this would work if all extension modules were always linked statically with the Python interpreter. When modules are used as shared libraries, however, the symbols defined in one module may not be visible to another module. The details of visibility depend on the operating system; some systems use one global namespace for the Python interpreter and all extension modules (e.g. Windows), whereas others require an explicit list of imported symbols at module link time (e.g. AIX), or offer a choice of different strategies (most Unices). And even if symbols are globally visible, the module whose functions one wishes to call might not have been loaded yet! Portability therefore requires not to make any assumptions about symbol visibility. This means that all symbols in extension modules should be declared `static', except for the module's initialization function, in order to avoid name clashes with other extension modules (as discussed in section~*Note Module's Method Table and Initialization Function::). And it means that symbols that *should* be accessible from other extension modules must be exported in a different way. Python provides a special mechanism to pass C-level information (i.e. pointers) from one extension module to another one: CObjects. A CObject is a Python data type which stores a pointer (`void *'). CObjects can only be created and accessed via their C API, but they can be passed around like any other Python object. In particular, they can be assigned to a name in an extension module's namespace. Other extension modules can then import this module, retrieve the value of this name, and then retrieve the pointer from the CObject. There are many ways in which CObjects can be used to export the C API of an extension module. Each name could get its own CObject, or all C API pointers could be stored in an array whose address is published in a CObject. And the various tasks of storing and retrieving the pointers can be distributed in different ways between the module providing the code and the client modules. The following example demonstrates an approach that puts most of the burden on the writer of the exporting module, which is appropriate for commonly used library modules. It stores all C API pointers (just one in the example!) in an array of `void' pointers which becomes the value of a CObject. The header file corresponding to the module provides a macro that takes care of importing the module and retrieving its C API pointers; client modules only have to call this macro before accessing the C API. The exporting module is a modification of the `spam' module from section~*Note A Simple Example::. The function `spam.system()' does not call the C library function `system()' directly, but a function `PySpam_System()', which would of course do something more complicated in reality (such as adding "spam" to every command). This function `PySpam_System()' is also exported to other extension modules. The function `PySpam_System()' is a plain C function, declared `static' like everything else: static int PySpam_System(command) char *command; { return system(command); } The function `spam_system()' is modified in a trivial way: static PyObject * spam_system(self, args) PyObject *self; PyObject *args; { char *command; int sts; if (!PyArg_ParseTuple(args, "s", &command)) return NULL; sts = PySpam_System(command); return Py_BuildValue("i", sts); } In the beginning of the module, right after the line #include "Python.h" two more lines must be added: #define SPAM_MODULE #include "spammodule.h" The `#define' is used to tell the header file that it is being included in the exporting module, not a client module. Finally, the module's initialization function must take care of initializing the C API pointer array: void initspam() { PyObject *m, *d; static void *PySpam_API[PySpam_API_pointers]; PyObject *c_api_object; m = Py_InitModule("spam", SpamMethods); /* Initialize the C API pointer array */ PySpam_API[PySpam_System_NUM] = (void *)PySpam_System; /* Create a CObject containing the API pointer array's address */ c_api_object = PyCObject_FromVoidPtr((void *)PySpam_API, NULL); /* Create a name for this object in the module's namespace */ d = PyModule_GetDict(m); PyDict_SetItemString(d, "_C_API", c_api_object); } Note that `PySpam_API' is declared `static'; otherwise the pointer array would disappear when `initspam' terminates! The bulk of the work is in the header file `spammodule.h', which looks like this: #ifndef Py_SPAMMODULE_H #define Py_SPAMMODULE_H #ifdef __cplusplus extern "C" { #endif /* Header file for spammodule */ /* C API functions */ #define PySpam_System_NUM 0 #define PySpam_System_RETURN int #define PySpam_System_PROTO Py_PROTO((char *command)) /* Total number of C API pointers */ #define PySpam_API_pointers 1 #ifdef SPAM_MODULE /* This section is used when compiling spammodule.c */ static PySpam_System_RETURN PySpam_System PySpam_System_PROTO; #else /* This section is used in modules that use spammodule's API */ static void **PySpam_API; #define PySpam_System \ (*(PySpam_System_RETURN (*)PySpam_System_PROTO) PySpam_API[PySpam_System_NUM]) #define import_spam() \ { \ PyObject *module = PyImport_ImportModule("spam"); \ if (module != NULL) { \ PyObject *module_dict = PyModule_GetDict(module); \ PyObject *c_api_object = PyDict_GetItemString(module_dict, "_C_API"); \ if (PyCObject_Check(c_api_object)) { \ PySpam_API = (void **)PyCObject_AsVoidPtr(c_api_object); \ } \ } \ } #endif #ifdef __cplusplus } #endif #endif /* !defined(Py_SPAMMODULE_H */ All that a client module must do in order to have access to the function `PySpam_System()' is to call the function (or rather macro) `import_spam()' in its initialization function: void initclient() { PyObject *m; Py_InitModule("client", ClientMethods); import_spam(); } The main disadvantage of this approach is that the file `spammodule.h' is rather complicated. However, the basic structure is the same for each function that is exported, so it has to be learned only once. Finally it should be mentioned that CObjects offer additional functionality, which is especially useful for memory allocation and deallocation of the pointer stored in a CObject. The details are described in the *Python/C API Reference Manual* in the section "CObjects" and in the implementation of CObjects (files `Include/cobject.h' and `Objects/cobject.c' in the Python source code distribution).  File: python-ext.info, Node: Building C and C++ Extensions on UNIX, Next: Building C and C++ Extensions on Windows, Prev: Extending Python with C or C++, Up: Top Building C and C++ Extensions on UNIX ************************************* This section was written by Jim Fulton . Starting in Python 1.4, Python provides a special make file for building make files for building dynamically-linked extensions and custom interpreters. The make file make file builds a make file that reflects various system variables determined by configure when the Python interpreter was built, so people building module's don't have to resupply these settings. This vastly simplifies the process of building extensions and custom interpreters on Unix systems. The make file make file is distributed as the file `Misc/Makefile.pre.in' in the Python source distribution. The first step in building extensions or custom interpreters is to copy this make file to a development directory containing extension module source. The make file make file, `Makefile.pre.in' uses metadata provided in a file named `Setup'. The format of the `Setup' file is the same as the `Setup' (or `Setup.in') file provided in the `Modules/' directory of the Python source distribution. The `Setup' file contains variable definitions: EC=/projects/ExtensionClass and module description lines. It can also contain blank lines and comment lines that start with `#'. A module description line includes a module name, source files, options, variable references, and other input files, such as libraries or object files. Consider a simple example:: ExtensionClass ExtensionClass.c This is the simplest form of a module definition line. It defines a module, `ExtensionClass', which has a single source file, `ExtensionClass.c'. This slightly more complex example uses an *-I* option to specify an include directory: EC=/projects/ExtensionClass cPersistence cPersistence.c -I$(EC) This example also illustrates the format for variable references. For systems that support dynamic linking, the `Setup' file should begin: *shared* to indicate that the modules defined in `Setup' are to be built as dynamically linked modules. A line containing only `*static*' can be used to indicate the subsequently listed modules should be statically linked. Here is a complete `Setup' file for building a `cPersistent' module: # Set-up file to build the cPersistence module. # Note that the text should begin in the first column. *shared* # We need the path to the directory containing the ExtensionClass # include file. EC=/projects/ExtensionClass cPersistence cPersistence.c -I$(EC) After the `Setup' file has been created, `Makefile.pre.in' is run with the `boot' target to create a make file: make -f Makefile.pre.in boot This creates the file, Makefile. To build the extensions, simply run the created make file: make It's not necessary to re-run `Makefile.pre.in' if the `Setup' file is changed. The make file automatically rebuilds itself if the `Setup' file changes. * Menu: * Building Custom Interpreters:: * Module Definition Options:: * Example:: * Distributing your extension modules::  File: python-ext.info, Node: Building Custom Interpreters, Next: Module Definition Options, Prev: Building C and C++ Extensions on UNIX, Up: Building C and C++ Extensions on UNIX Building Custom Interpreters ============================ The make file built by `Makefile.pre.in' can be run with the `static' target to build an interpreter: make static Any modules defined in the Setup file before the `*shared*' line will be statically linked into the interpreter. Typically, a `*shared*' line is omitted from the Setup file when a custom interpreter is desired.  File: python-ext.info, Node: Module Definition Options, Next: Example, Prev: Building Custom Interpreters, Up: Building C and C++ Extensions on UNIX Module Definition Options ========================= Several compiler options are supported: Option Meaning ------ ----- -C Tell the C pre-processor not to discard comments -DNAME=VALUE Define a macro -IDIR Specify an include directory, DIR -LDIR Specify a link-time library directory, DIR -RDIR Specify a run-time library directory, DIR -lLIB Link a library, LIB -UNAME Undefine a macro Other compiler options can be included (snuck in) by putting them in variables. Source files can include files with `.c', `.C', `.cc', `.cpp', `.cxx', and `.c++' extensions. Other input files include files with `.a', `.o', `.sl', and `.so' extensions.  File: python-ext.info, Node: Example, Next: Distributing your extension modules, Prev: Module Definition Options, Up: Building C and C++ Extensions on UNIX Example ======= Here is a more complicated example from `Modules/Setup.in': GMP=/ufs/guido/src/gmp mpz mpzmodule.c -I$(GMP) $(GMP)/libgmp.a which could also be written as: mpz mpzmodule.c -I$(GMP) -L$(GMP) -lgmp  File: python-ext.info, Node: Distributing your extension modules, Prev: Example, Up: Building C and C++ Extensions on UNIX Distributing your extension modules =================================== When distributing your extension modules in source form, make sure to include a `Setup' file. The `Setup' file should be named `Setup.in' in the distribution. The make file make file, `Makefile.pre.in', will copy `Setup.in' to `Setup'. Distributing a `Setup.in' file makes it easy for people to customize the `Setup' file while keeping the original in `Setup.in'. It is a good idea to include a copy of `Makefile.pre.in' for people who do not have a source distribution of Python. Do not distribute a make file. People building your modules should use `Makefile.pre.in' to build their own make file. A `README' file included in the package should provide simple instructions to perform the build. Work is being done to make building and installing Python extensions easier for all platforms; this work in likely to supplant the current approach at some point in the future. For more information or to participate in the effort, refer to `http://www.python.org/sigs/distutils-sig/' on the Python Web site.  File: python-ext.info, Node: Building C and C++ Extensions on Windows, Next: Embedding Python in Another Application, Prev: Building C and C++ Extensions on UNIX, Up: Top Building C and C++ Extensions on Windows **************************************** This chapter briefly explains how to create a Windows extension module for Python using Microsoft Visual C++, and follows with more detailed background information on how it works. The explanatory material is useful for both the Windows programmer learning to build Python extensions and the UNIX programming interested in producing software which can be successfully built on both UNIX and Windows. * Menu: * A Cookbook Approach:: * Differences Between UNIX and Windows:: * Using DLLs in Practice::  File: python-ext.info, Node: A Cookbook Approach, Next: Differences Between UNIX and Windows, Prev: Building C and C++ Extensions on Windows, Up: Building C and C++ Extensions on Windows A Cookbook Approach =================== This section was written by Neil Schemenauer . This section provides a recipe for building a Python extension on Windows. Grab the binary installer from `http://www.python.org/' and install Python. The binary installer has all of the required header files except for `config.h'. Get the source distribution and extract it into a convenient location. Copy the `config.h' from the `PC/' directory into the `include/' directory created by the installer. Create a `Setup' file for your extension module, as described in Chapter *Note Building C and C++ Extensions on UNIX::. Get David Ascher's `compile.py' script from `http://starship.python.net/crew/da/compile/'. Run the script to create Microsoft Visual C++ project files. Open the DSW file in VC++ and select *Build*. If your module creates a new type, you may have trouble with this line: PyObject_HEAD_INIT(&PyType_Type) Change it to: PyObject_HEAD_INIT(NULL) and add the following to the module initialization function: MyObject_Type.ob_type = &PyType_Type; Refer to section 3 of the Python FAQ (`http://www.python.org/doc/FAQ.html') for details on why you must do this.  File: python-ext.info, Node: Differences Between UNIX and Windows, Next: Using DLLs in Practice, Prev: A Cookbook Approach, Up: Building C and C++ Extensions on Windows Differences Between UNIX and Windows ==================================== This section was written by Chris Phoenix . UNIX and Windows use completely different paradigms for run-time loading of code. Before you try to build a module that can be dynamically loaded, be aware of how your system works. In UNIX, a shared object (.so) file contains code to be used by the program, and also the names of functions and data that it expects to find in the program. When the file is joined to the program, all references to those functions and data in the file's code are changed to point to the actual locations in the program where the functions and data are placed in memory. This is basically a link operation. In Windows, a dynamic-link library (`.dll') file has no dangling references. Instead, an access to functions or data goes through a lookup table. So the DLL code does not have to be fixed up at runtime to refer to the program's memory; instead, the code already uses the DLL's lookup table, and the lookup table is modified at runtime to point to the functions and data. In UNIX, there is only one type of library file (`.a') which contains code from several object files (`.o'). During the link step to create a shared object file (`.so'), the linker may find that it doesn't know where an identifier is defined. The linker will look for it in the object files in the libraries; if it finds it, it will include all the code from that object file. In Windows, there are two types of library, a static library and an import library (both called `.lib'). A static library is like a UNIX `.a' file; it contains code to be included as necessary. An import library is basically used only to reassure the linker that a certain identifier is legal, and will be present in the program when the DLL is loaded. So the linker uses the information from the import library to build the lookup table for using identifiers that are not included in the DLL. When an application or a DLL is linked, an import library may be generated, which will need to be used for all future DLLs that depend on the symbols in the application or DLL. Suppose you are building two dynamic-load modules, B and C, which should share another block of code A. On UNIX, you would *not* pass `A.a' to the linker for `B.so' and `C.so'; that would cause it to be included twice, so that B and C would each have their own copy. In Windows, building `A.dll' will also build `A.lib'. You *do* pass `A.lib' to the linker for B and C. `A.lib' does not contain code; it just contains information which will be used at runtime to access A's code. In Windows, using an import library is sort of like using `import spam'; it gives you access to spam's names, but does not create a separate copy. On UNIX, linking with a library is more like `from spam import *'; it does create a separate copy.  File: python-ext.info, Node: Using DLLs in Practice, Prev: Differences Between UNIX and Windows, Up: Building C and C++ Extensions on Windows Using DLLs in Practice ====================== This section was written by Chris Phoenix . Windows Python is built in Microsoft Visual C++; using other compilers may or may not work (though Borland seems to). The rest of this section is MSVC++ specific. When creating DLLs in Windows, you must pass `python15.lib' to the linker. To build two DLLs, spam and ni (which uses C functions found in spam), you could use these commands: cl /LD /I/python/include spam.c ../libs/python15.lib cl /LD /I/python/include ni.c spam.lib ../libs/python15.lib The first command created three files: `spam.obj', `spam.dll' and `spam.lib'. `Spam.dll' does not contain any Python functions (such as `PyArg_ParseTuple()'), but it does know how to find the Python code thanks to `python15.lib'. The second command created `ni.dll' (and `.obj' and `.lib'), which knows how to find the necessary functions from spam, and also from the Python executable. Not every identifier is exported to the lookup table. If you want any other modules (including Python) to be able to see your identifiers, you have to say `_declspec(dllexport)', as in `void _declspec(dllexport) initspam(void)' or `PyObject _declspec(dllexport) *NiGetSpamData(void)'. Developer Studio will throw in a lot of import libraries that you do not really need, adding about 100K to your executable. To get rid of them, use the Project Settings dialog, Link tab, to specify *ignore default libraries*. Add the correct `msvcrtXX.lib' to the list of libraries.  File: python-ext.info, Node: Embedding Python in Another Application, Next: Miscellaneous Index, Prev: Building C and C++ Extensions on Windows, Up: Top Embedding Python in Another Application *************************************** Embedding Python is similar to extending it, but not quite. The difference is that when you extend Python, the main program of the application is still the Python interpreter, while if you embed Python, the main program may have nothing to do with Python -- instead, some parts of the application occasionally call the Python interpreter to run some Python code. So if you are embedding Python, you are providing your own main program. One of the things this main program has to do is initialize the Python interpreter. At the very least, you have to call the function `Py_Initialize()'. There are optional calls to pass command line arguments to Python. Then later you can call the interpreter from any part of the application. There are several different ways to call the interpreter: you can pass a string containing Python statements to `PyRun_SimpleString()', or you can pass a stdio file pointer and a file name (for identification in error messages only) to `PyRun_SimpleFile()'. You can also call the lower-level operations described in the previous chapters to construct and use Python objects. A simple demo of embedding Python can be found in the directory `Demo/embed/' of the source distribution. * Menu: * Embedding Python in C++::  File: python-ext.info, Node: Embedding Python in C++, Prev: Embedding Python in Another Application, Up: Embedding Python in Another Application Embedding Python in C++ ======================= It is also possible to embed Python in a C++ program; precisely how this is done will depend on the details of the C++ system used; in general you will need to write the main program in C++, and use the C++ compiler to compile and link your program. There is no need to recompile Python itself using C++.  File: python-ext.info, Node: Miscellaneous Index, Prev: Embedding Python in Another Application, Up: Top Miscellaneous Index ******************* * Menu: * Philbrick, Geoff: Keyword Parsing with PyArg_ParseTupleAndKeywords.