Lazy loading#
For a large binary mesh, pass lazy=True. polyxios maps the file and,
wherever the file stores an array as one run of bytes in the shape the array
needs, hands back a view of the mapping instead of a copy: nothing is
decoded, the pages come in as they are touched, and the process holds the
mesh once - in the page cache - rather than once there and again in its own
memory.
mesh = px.read("huge.vtu", lazy=True)
mesh.vertices.flags.writeable # False: these are the file's bytes
mesh.vertices.dtype # whatever the file holds - float32 stays float32
mesh.vertices[0] # the first page comes in here
coords = np.asarray(mesh.vertices, dtype=np.float64) # a copy, when one is wanted
What comes back#
A lazy array is read-only, and carries the dtype and byte order the file
holds: a Float32 block stays float32, a big-endian legacy VTK block
stays big-endian. An eager read converts to float64 coordinates and native
integer indices; a lazy read leaves that to you, since the conversion is the
copy you declined. np.asarray(arr, dtype=...) makes one when you need it,
and every NumPy operation reads a swapped or read-only array as it is.
Only an array the file stores in the shape PolyData needs
can be a view. That rules out anything encoded (ASCII, base64, zlib), any
record that interleaves a count or a reference with the values, and any
index that counts from one. Where a format stores some arrays that way and
not others, the ones it does are views and the rest are built in memory -
the offsets of a uniform topology, the element types, a padded third
coordinate. The per-format table below says which is which.
Which formats#
format |
lazy arrays |
notes |
|---|---|---|
|
vertices, connectivity, attributes |
Raw appended section only, the layout VTK writes by default and
|
|
vertices, attributes; cells in v5.1 |
Binary |
|
every array the sidecar stores |
A |
|
everything |
32-byte records: float32 positions and every attribute are strided views. Nothing is decoded. |
|
vertices, every field |
|
|
vertices, every field, extra bytes |
Uncompressed only. The vertices are the file’s |
|
vertices, vertex properties |
Binary only. Coordinates are one strided view when |
|
vertices |
Each record is coordinates then a reference, so the coordinates are one strided view. Elements number vertices from one and are decoded. |
|
none |
A 50-byte record cannot be viewed as |
|
none |
|
the text formats |
none |
|
What a mapping needs#
mmap maps a file descriptor from byte zero, so a lazy read needs a real,
uncompressed file standing at its start: an io.BytesIO, a handle
part-way into a file, or a gzipped one raises
LazyReadError naming the reason rather than
quietly loading eagerly. Binary STL’s lazy mode only skips work, so it takes
a buffer or a compressed file like any other read.
The mapping lives as long as any array viewing it does, and goes when the last one does. Nothing is closed behind your back and nothing needs closing. A mapping does keep its own descriptor open for that long, so a thousand lazy meshes held at once are a thousand open files; drop the arrays to release them.
See also
Each page under Supported formats carries a badge saying what its
codec maps, and the table on the front page has a
lazy column.