qlty

qlty is a Python library designed to handle large 2D or 3D tensors efficiently by splitting them into smaller, manageable chunks. This library is particularly useful for processing large datasets that do not fit into memory, enabling chunked processing for machine learning workflows.

Features

Efficient tensor splitting and stitching
Intelligent border handling to minimize artifacts
Support for both in-memory and disk-cached processing
2D and 3D tensor support
2.5D Quilt - Convert 3D volumetric data to multi-channel 2D by slicing Z dimension
Backend System - Unified interface for multiple data sources (torch.Tensor, Zarr, HDF5, memory-mapped)
Image Stack Utilities - Convert image file stacks to efficient Zarr format
False Color Visualization - UMAP-based false-color visualization of 2D images
Sparse data handling utilities
Patch pair extraction helpers for partially overlapping regions in 2D and 3D
Pre-tokenization utilities (pretokenizer_2d) for preparing patches for sequence models
Numba acceleration for 2D stitching and batch token processing with parallel execution

Quick Start

Installation:

pip install qlty torch zarr numpy einops dask numba

Basic Usage:

import torch
from qlty import NCYXQuilt

# Create a quilt object
quilt = NCYXQuilt(
    Y=128, X=128,
    window=(32, 32),
    step=(16, 16),
    border=(5, 5),
    border_weight=0.1
)

# Split data into patches
data = torch.randn(10, 3, 128, 128)
patches = quilt.unstitch(data)

# Process patches (e.g., with a neural network)
processed = your_model(patches)

# Stitch back together
reconstructed, weights = quilt.stitch(processed)

Documentation

Full documentation is available at https://qlty.readthedocs.io

Modules

In-Memory Classes

NCYXQuilt: 2D tensor splitting and stitching (shape: N, C, Y, X)
NCZYXQuilt: 3D tensor splitting and stitching (shape: N, C, Z, Y, X)

Disk-Cached Classes

LargeNCYXQuilt: 2D with on-disk caching using Zarr
LargeNCZYXQuilt: 3D with on-disk caching using Zarr

License

Free software: BSD license

Credits

This package was created with Cookiecutter and the audreyr/cookiecutter-pypackage project template.