;ò @A>@c@s!dZdkZdklZdkZdkZdkZdkZdk Z dk Z dk Z ddddgZ ddgZ hdd<dd<d d<d d>> m = open("memmap.tst","w+",len=48) >>> m In this case, file is initialized to 0. As in C stdio, when a file is opened for 'w+', it is truncated (effectively deleted) if it already exists. The initial contents of a w+ Memmap are undefined. Slice m into the buffers "n" and "p" which will correspond to numarray: >>> n = m[0:16] >>> n >>> p = m[24:48] >>> p NOTE: You *can not* make overlapping slices of a Memmap: >>> q = m[20:28] Traceback (most recent call last): ... IndexError: Slice overlaps previous slice of same file. NOTE: you *can not* make an array directly from a Memmap: >>> import numarrayall as num >>> c = num.NumArray(buffer=m, shape=(len(m)/4,), type=num.Int32) Traceback (most recent call last): ... error: NA_updateDataPtr: error getting read buffer data ptr This fails because m, being the root Memmap and not a MemmapSlice, does not define __buffer__() or resize(). Finally, the good part. Make numarray from the MemmapSlice "buffers": >>> a = num.NumArray(buffer=n, shape=(len(n)/4,), type=num.Int32) >>> a[:] = 0 # Since the initial contents of 'n' are undefined. >>> a += 1 >>> num.explicit_type(a) array([1, 1, 1, 1], type=Int32) >>> b = num.NumArray(buffer=p, shape=(len(p)/8,), type=num.Float64) >>> b[:] = 0 # Since the initial contents of 'p' are undefined. >>> b += 2.0 >>> b array([ 2., 2., 2.]) Here's the other good part about MemmapSlices: they're resizable. >>> _junk = a.resize( 6 ) >>> num.explicit_type(a) array([1, 1, 1, 1, 1, 1], type=Int32) >>> b array([ 2., 2., 2.]) What you should note is that "b" retains the correct contents (i.e., offset within "m") even though "a" grew, effectively moving "b". In reality, "b" stayed where it always was and "a" has moved to a bigger RAM-based buffer. Since we resized "a", "m" is now a different size as well: >>> m After doing resizes, call m.flush() to synchronize the underlying file of "m" with any RAM based slices. This step is required to avoid implicitly shuffling huge amounts of file space for every insert or delete. After calling m.flush(), all slices are once again memory mapped rather than purely RAM based. >>> m.flush() NOTE: Since memory maps don't guarantee when the underlying file will be updated with the values you have written to the map, call m.sync() when you want to be sure your changes are on disk. Note that sync() does not consolidate the mapfile with any purely RAM based slices which have been inserted into the map. >>> m.sync() Now "a" and "b" are both memory mapped on "memmap.tst" again. It is also possible for "a" or "b" to shrink: >>> _junk = a.resize(0) >>> num.explicit_type(a) array([], type=Int32) >>> b array([ 2., 2., 2.]) >>> m Arrays based on MemmapSlices can be pickled: >>> import cPickle >>> c = cPickle.loads(cPickle.dumps(b)) >>> c array([ 2., 2., 2.]) However, when the array is restored by the unpickler, the buffer is restored as an "orphaned" MemmapSlice. There is currently no support for pickling the Memmap. When you're done with the memory map and numarray, call m.close(). m.close() calls m.flush() which will do consolidation if any is needed. >>> m.close() It is an error to use "m" (or slices of m) any further after closing it. >>> m._buffer() Traceback (most recent call last): ... RuntimeError: Memmap no longer valid. (closed?) Slices of a Memmap are MemmapSlice objects. Slices of a MemmapSlice are strings. >>> m = Memmap("memmap.tst",mode="w+",len=100) >>> m1=m[:] >>> m2=m1[:] >>> m3=m1[:10] >>> int(isinstance(m3, types.StringType)) 1 >>> int(isinstance(m2, types.StringType)) 1 >>> m.close() Deletion of a slice of a Memmap "un-registers" the slice, making that region of the Memmap available for reallocation: >>> m = Memmap("memmap.tst",mode="w+",len=100) >>> m1 = m[0:50] Delete directly from the Memmap without referring to the MemmapSlice: >>> del m[0:50] >>> m2 = m[0:70] Note that since the region of m1 was deleted, there is no overlap when m2 is created. However, deleting the region of m1 has invalidated it: >>> m1 Traceback (most recent call last): ... RuntimeError: A deleted MemmapSlice has been used. It is a bad idea to mix operations on a Memmap which modify its data or file structure with slice deletions. DO NOT use a MemmapSlice where it's contents can be modified or resized and then delete the region it refers to later. In this case, the status of the modifications is undefined; the underlying map may or may not reflect the modifications after the deletion. >>> m.close() Copy-on-write memory maps can be opened using either mode="c" or mode="copyonwrite". Copy-on-write maps have writable slices, but cannot be resized, flushed, or synced to the original file. >>> m = Memmap("memmap.tst",mode="c",len=100) >>> n = m[:] >>> n >>> a = num.array(sequence=n, shape=(len(n),), type=num.Int8) >>> a += 1 >>> # it worked! >>> m.close() Try a zero length memmap based on comments from Sebastian Hesse >>> m = Memmap("memmap.tst", mode='w+', len=0) >>> n = m.insert(0,0) >>> a = num.array(sequence=n, type=num.UInt16, shape=(0,0,0)) >>> _junk = a.resize((100,100,100)) >>> m.flush() >>> m.close() Readonly memory maps can be opened using either mode="r" or mode="readonly". Readonly maps have readonly slices as well as all of the restrictions of copy-on-write mappings. >>> m = Memmap("memmap.tst",mode="r",len=100) >>> n = m[:] >>> n >>> a = num.array(sequence=n, shape=(len(n),), type=num.Int8) >>> a += 1 # can't work... buffer is readonly Traceback (most recent call last): ... error: add_Int8_vector_scalar: Problem with write buffer[2]. >>> b = a + 1 # this still works... >>> N(sopensrscsr+sw+sreadonlys copyonwrites readwriteswritecCst||dƒSdS(Nsb(s__opensfilesmode(sfilesmode((sH/mit/python/arch/i386_rh9/lib/python2.3/site-packages/numarray/memmap.pys_openõssMemmapcBsãtZded„Zd„Zd„Zded„Zd„Zd„Zd „Z d „Z dd „Z d „Z d „Z d„Zd„Zd„Zd„Zed„Zed„Zdd„Zd„Zeed„Zd„ZRS(Nsr+cCsì||_g|_t|_|tiƒjot|}n,|t jot dt tiƒƒ‚n|djo |tjot dƒ‚n|dj|_ ||_ t||djodp|ƒ}|iddƒ|iƒ}|tjo |}n|djp|djo ||joF|o5|i|d dƒ|itdƒƒ|iƒn|}n|ob|djo ti}n$|djo ti}n ti}ti|iƒ|d |ƒ|_n t|_|iƒd S( su Valid "mode" values for a Memmap are: "readonly" or "r" "copyonwrite" or "c" "readwrite" or "r+" "write" or "w+" Create a small test file with 100 zeroes: >>> _open("memmap.tst","w+").write(chr(0)*100) Map it as a readwrite mapping: >>> m = Memmap("memmap.tst","r+") smode must be one of %ssw+s"Must specify 'len' if mode is 'w+'srsciisr+isaccessN(sfilenamesselfs _filenames_slicessNones_mmapsmodesmode_equivalentsskeyssvalid_filemodess ValueErrorslens _readonlys_modes_opensfilesseekstellsflenswriteschrsflushsmmaps ACCESS_COPYsaccs ACCESS_READs ACCESS_WRITEsfilenosclose(sselfsfilenamesmodeslensaccsfilesflen((sH/mit/python/arch/i386_rh9/lib/python2.3/site-packages/numarray/memmap.pys__init__ùs@      #   '      % cCs=|iddƒd|i|it|ƒt|iƒfSdS(Nis:(sselfs _chkOverlapss _filenames_modeslens_slices(sself((sH/mit/python/arch/i386_rh9/lib/python2.3/site-packages/numarray/memmap.pys__repr__3scCs)|iƒo|iƒn|iƒdS(së close(self) unites the memory map and any RAM based slices with its underlying file and removes the mapping and all references to its slices. Don't call this till you're done using the Memmap! N(sselfs _mutablefilesflushs _lose_map(sself((sH/mit/python/arch/i386_rh9/lib/python2.3/site-packages/numarray/memmap.pysclose8s icCs€|itjotdƒ‚n|tjot|ƒ}n|iot|i|||ƒSnt i |i|||ƒSdS(sA_buffer(self) returns a buffer object for Memmap 'self'. s"Memmap no longer valid. (closed?)N( sselfs_mmapsNones RuntimeErrorsendslens _readonlysbuffersbeginsmemoryswriteable_buffer(sselfsbeginsend((sH/mit/python/arch/i386_rh9/lib/python2.3/site-packages/numarray/memmap.pys_bufferCs  cCsÐxÉ|iD]¾\}}}||jo |jnp~||jo |jnp`||jo |jnpB||jo |jnp$||jo||jo ||jotdƒ‚q q WdS(s¥_chkOverlaps(self, begin, end) is called to raise an exception if the requested slice will overlap any slices which have already been taken. s+Slice overlaps previous slice of same file.N(sselfs_slicessbsesobjsbeginsends IndexError(sselfsbeginsendsbsesobj((sH/mit/python/arch/i386_rh9/lib/python2.3/site-packages/numarray/memmap.pys _chkOverlapsOs  ŸcCsÇ|iot|iƒ}nd}ttigi}|i D]\}}}|||ƒq=~dƒ}ttigi}|i D]\}}}|t|ƒƒqƒ~dƒ}t|||ƒSdS(s;len(self) is the number of bytes in Memmap 'self'. iN(sselfs_mmapslensmaplensreducesoperatorsaddsappends_[1]s_slicessbsesosorig_lensobj_lensint(sselfsobj_lensorig_lenses_[1]sbsosmaplen((sH/mit/python/arch/i386_rh9/lib/python2.3/site-packages/numarray/memmap.pys__len__\s FHcCs$|itj o |iSndSdS(Ns(sselfs_mmapsNone(sself((sH/mit/python/arch/i386_rh9/lib/python2.3/site-packages/numarray/memmap.pys__str__gs cCs‘t|tiƒo<|itj otdƒ‚n|i|if\}}n |d}|i |dƒ|i |dƒf\}}||fSdS(sx_fix_slice(self, i) converts a __getitem__ 'key' into slice parameters, and returns a tuple (beg, end). s Memmap does not support strides.iN( s isinstancesistypess SliceTypesstepsNones IndexErrorsstopsstartsjsselfs _chkIndex(sselfsisj((sH/mit/python/arch/i386_rh9/lib/python2.3/site-packages/numarray/memmap.pys _fix_slicems *cCs’|tdddƒjot|ƒSn|djo|t|ƒ7}nd|jot|ƒ|jn otd|fƒ‚n|SdS(s^_chkIndex(self, i) raises an IndexError if 'i' is not a valid index of 'self' liiisInvalid Memmap index: %dN(sisintslensselfsisSlices IndexError(sselfsisisSlice((sH/mit/python/arch/i386_rh9/lib/python2.3/site-packages/numarray/memmap.pys _chkIndexzs )cCsg|i|ƒ\}}|i||ƒt|i||ƒd|iƒ}|i i |||fƒ|SdS(s__getitem__(self,i) returns a MemmapSlice corresponding to the index 'i' of the Memmap 'self'. The Memmap keeps a record of the slice so that it can coordinate it with other slices from the same file. Slices of a Memmap are not permitted to overlap. sreadonlyN( sselfs _fix_slicesisjs _chkOverlapss MemmapSlices_buffers _readonlysobjs_slicessappend(sselfsisjsobj((sH/mit/python/arch/i386_rh9/lib/python2.3/site-packages/numarray/memmap.pys __getitem__†s !cCs›|i|ƒ\}}xtt|iƒƒD]R}|i|\}}}||jo ||jo|i ƒ|i|=dSq+q+Wt d||fƒ‚dS(sŽ__delitem__(self,i) deletes a slice from a Memmap, removing the record of the "slice", but not the underlying data footprint. NsCan't find slice (%d,%d)( sselfs _fix_slicesisjsrangeslens_slicessksbsesos _markDeleteds ValueError(sselfsisbsesjsosk((sH/mit/python/arch/i386_rh9/lib/python2.3/site-packages/numarray/memmap.pys __delitem__’s   cCs|iddgjSdS(s™_mutablefile returns 1 iff the file underlying the memory map can be modified. Thus, it returns 0 for readonly and copyonwrite mappings.scsrN(sselfs_mode(sself((sH/mit/python/arch/i386_rh9/lib/python2.3/site-packages/numarray/memmap.pys _mutablefile scCs2|iƒo |itj o|iiƒndS(s¯sync(self) ultimately calls msync, guaranteeing that updates to a MMap are already written to the underlying file system device when the call returns. N(sselfs _mutablefiles_mmapsNonesflush(sself((sH/mit/python/arch/i386_rh9/lib/python2.3/site-packages/numarray/memmap.pyssync¨scCsJttigi}|iD]\}}}||i ƒƒq~dƒSdS(sS_dirty(self) is 1 if any slice of self is "dirty". A slice is dirty if it has been resized in any way, or was not part of the original Memmap. _dirty(self) specifically excludes in-place modification, since this can happen at the C-level and there's no way to know whether it has happened or not. iN( sreducesoperatorsor_sappends_[1]sselfs_slicessbsesosdirty(sselfsbsesos_[1]((sH/mit/python/arch/i386_rh9/lib/python2.3/site-packages/numarray/memmap.pys_dirty°s?cCsXx'|iD]\}}}|itƒq W|itj o|iiƒt|_ndS(s½_lose_map(self) eliminates all references to the underlying mmap, so that it will be deleted. This appeared necessary on Win-NT to be able to re-map the same file. N( sselfs_slicessbsesos _rebuffersNones_mmapsclose(sselfsbseso((sH/mit/python/arch/i386_rh9/lib/python2.3/site-packages/numarray/memmap.pys _lose_mapºs  c Cs$|iƒ o||ijp |tjotdƒ‚nd}t|dƒ}t |ƒ}|i } g|_ | i ƒd}x‹| D]ƒ\}}} ||jo|i|i||ƒƒn|iƒ} |i| iƒƒ|iƒ}|i i| || fƒ|}q…W||jo|i|i||ƒƒn|iƒ|iƒ|tj o ||_ntii|iƒoti|iƒnti||iƒt|idƒ}d|_ d|_!t"i"|i#ƒ|ƒ|_$x3|i D](\}}} | i&|i||ƒƒqêW|iƒdS(sö_consolidate(self) re-writes the memory map file, interspersing RAM based slices with the content of the mmap which has been updated in-place. By default, the new memory mapped file is then mapped in place of the old one.s)Memmap trying to flush onto readonly files memmap.tmpsw+isr+N('sselfs _mutablefilesnew_maps _filenamesNones RuntimeErrorstemp_maps_opensfslensmlens_slicesslssortsmsbsesobjswrites_bufferstellsobs __buffer__soesappendscloses _lose_mapsosspathsexistssremovesrenames _readonlys_modesmmapsfilenos_mmapsos _rebuffer( sselfsnew_mapstemp_mapsbsesmlensfsoesmslsobsobjso((sH/mit/python/arch/i386_rh9/lib/python2.3/site-packages/numarray/memmap.pys _consolidateÅsH+                cCsC|iƒ|iƒp|o |i|jo|i|ƒndS(sŸflush(self) first syncs the memory map with the underlying file, then consolidates it with any RAM based slices in a new file, then remaps the new file. Both slice offsets and buffers change. If there are no RAM based buffers, no consolidation is performed. It is possible to flush a map onto a new file by specifying its name with the filename parameter. N(sselfssyncs_dirtysfilenames _filenames _consolidate(sselfsfilename((sH/mit/python/arch/i386_rh9/lib/python2.3/site-packages/numarray/memmap.pysflushòs  $cCs7|itjotdƒ‚n|ii||ƒSdS(s“find(string, offset=0) returns the first index at which string is found, or -1 on failure. >>> _open("memmap.tst","w+").write("this is a test") >>> Memmap("memmap.tst",len=14).find("is") 2 >>> Memmap("memmap.tst",len=14).find("is", 3) 5 >>> _open("memmap.tst","w+").write("x") >>> Memmap("memmap.tst",len=1).find("is") -1 s_mmap is None; Memmap closed?N(sselfs_mmapsNones RuntimeErrorsfindsstringsoffset(sselfsstringsoffset((sH/mit/python/arch/i386_rh9/lib/python2.3/site-packages/numarray/memmap.pysfinds cCs,|iƒ|||!|iƒ|||+dS(sfmove(dest, src, count) moves 'count' characters from 'src' to 'dest' within a Memmap. N(sselfs_bufferssrcscountsdest(sselfsdestssrcscount((sH/mit/python/arch/i386_rh9/lib/python2.3/site-packages/numarray/memmap.pysmovescCsÇ|i|ddƒ|i||ƒ|tjo"ti|ƒ}ti |ƒ}nC|tjpt |ƒ|jo|}t |ƒ}n t dƒ‚t |ddƒ}|ii|||fƒ|SdS(sá insert(self, offset, size) inserts a new slice of 'size' bytes, possibly between two adjacent slices, at byte 'offset'. It is not legal to insert into the middle of another slice, but pretty much everything else is legal. The resulting MemmapSlice is returned. >>> m = open("memmap.tst",mode="w+",len=100) >>> n = m[0:50] >>> p = m[50:100] >>> q=m.insert(0, 200) >>> r=m.insert(50, 100) >>> s=m.insert(100, 300) >>> t=m.insert(45, 100) Traceback (most recent call last): ... IndexError: Slice overlaps previous slice of same file. >>> m.flush() >>> len(m) 700 >>> m.close() sisSliceisbuffer and size don't match.sdirtyN(sselfs _chkIndexsoffsets _chkOverlapssbuffersNonesmemorys new_memoryssizesmemswriteable_bufferslens ValueErrors MemmapSlicesobjs_slicessappend(sselfsoffsetssizesbuffersobjsmem((sH/mit/python/arch/i386_rh9/lib/python2.3/site-packages/numarray/memmap.pysinserts   cCs|iƒdS(N(sselfsclose(sself((sH/mit/python/arch/i386_rh9/lib/python2.3/site-packages/numarray/memmap.pys__del__?s(s__name__s __module__sNones__init__s__repr__scloses_buffers _chkOverlapss__len__s__str__s _fix_slices _chkIndexs __getitem__s __delitem__s _mutablefilessyncs_dirtys _lose_maps _consolidatesflushsfindsmovesinserts__del__(((sH/mit/python/arch/i386_rh9/lib/python2.3/site-packages/numarray/memmap.pysMemmapøs*:       -   %s MemmapSlicecBsætZddd„Zd„Zd„Zd„Zd„Zd„Zd„Zd „Z d „Z d „Z d „Z d „Z d„Zd„Zedd„Zd„Zd„Zd„Zd„Zd„Zd„Zdd„Zd„ZRS(NicCs(||_||_d|_||_dS(Ni(sbuffersselfs_buffersdirtys_dirtys_deletedsreadonlys _readonly(sselfsbuffersdirtysreadonly((sH/mit/python/arch/i386_rh9/lib/python2.3/site-packages/numarray/memmap.pys__init__Cs   cCs d|_dS(Ni(sselfs_deleted(sself((sH/mit/python/arch/i386_rh9/lib/python2.3/site-packages/numarray/memmap.pys _markDeletedIscCs|io td‚ndS(s_checkDeleted ensures that a deleted Memmap region does not continue to be used. For a map region to be re-used, a new slice must be taken. s$A deleted MemmapSlice has been used.N(sselfs_deleteds RuntimeError(sself((sH/mit/python/arch/i386_rh9/lib/python2.3/site-packages/numarray/memmap.pys _checkDeletedLs cCsK|iƒti|iƒ}t|iƒ|d(sselfs _readonlyssslen(sselfss((sH/mit/python/arch/i386_rh9/lib/python2.3/site-packages/numarray/memmap.pys__repr__gs  cCs&|iot|iƒƒSndSdS(Ni(sselfs_bufferslens __buffer__(sself((sH/mit/python/arch/i386_rh9/lib/python2.3/site-packages/numarray/memmap.pys__len__ns cCs |iSdS(scdirty(self, set=None) is 1 iff 'self' has changed its buffer since it was created. N(sselfs_dirty(sself((sH/mit/python/arch/i386_rh9/lib/python2.3/site-packages/numarray/memmap.pysdirtytscCs|iƒt|ƒtijot|iƒ|ƒSnDt|ƒtijo!t|iƒ|i |i !ƒSn t dƒ‚dS(NsCan't handle index type( sselfs _checkDeletedstypesistypessIntTypesstrs __buffer__s SliceTypesstartsstops TypeError(sselfsi((sH/mit/python/arch/i386_rh9/lib/python2.3/site-packages/numarray/memmap.pys __getitem__zs  !cCsw|iƒt|ƒtijo||iƒ|>> import os >>> os.remove("memmap.tst") N(sdoctestsmemmapstestmod(smemmapsdoctest((sH/mit/python/arch/i386_rh9/lib/python2.3/site-packages/numarray/memmap.pystest ss memmap.datic Cs»dk}dk} t|dƒ}|i|dƒ|idƒ|i ƒt |d|ƒ}|} |id| dt| ƒfdd ƒ}||_|i||ƒ|}||} |SdS( s‡proveit is a diagnostic function which creates a file of size 'N', memory maps it, and then reads one byte at 'pagesize' intervals.Nsw+issmodesbuffersshapestypesInt8(s numarrayallsnumsoss_opensfilenamesfsseeksNswritesclosesMemmapsmodesmsnsNumArrayslensas_mapsarangespagesizeshitssfetch( sNsfilenamespagesizesmodesashitssfsmsnumsnsossfetch((sH/mit/python/arch/i386_rh9/lib/python2.3/site-packages/numarray/memmap.pysproveits    '  s__main__(s__doc__ssyss __builtin__sopens__opensossmmapstypessoperatorsmemoryscopy_regscopysvalid_filemodesswriteable_filemodessmode_equivalentss_opensMemmaps MemmapSlicesNonesclosestestsproveits__name__(swriteable_filemodess__opensopensmmapsclosesMemmaps_opensmode_equivalentsssyss MemmapSlicestestsproveitsmemorysoperatorscopy_regsvalid_filemodesscopysosstypes((sH/mit/python/arch/i386_rh9/lib/python2.3/site-packages/numarray/memmap.pys?ßs,          * ÿKÀ