lazyslide.tl.feature_extraction

feature_extraction(wsi, model=None, *, model_path=None, model_name=None, jit=False, token=None, load_kws=None, transform=None, device=None, amp=None, autocast_dtype=None, batch_size=32, num_workers=0, pbar=None, tile_key='tiles', dense=False, pool_mode=None, key_added=None, return_features=False, **kwargs)

Extract features from WSI tiles using a pre-trained vision models.

To list all timm models:

>>> import timm
>>> timm.list_models(pretrained=True)

To list all lazyslide built-in models:

>>> import lazyslide as zs
>>> zs.models.list_models()

Parameters:

wsiWSIData

The whole-slide image object.

modelstr or model object

The model used for image feature extraction. A list of built-in foundation models can be found in Models. Other models can be loaded from Hugging Face, but only models with feature extraction head implemented.

model_pathstr or Path

The path to the model file. Either model or model_path must be provided. If you don’t have internet access, you can download the model file and load it from the local path. You can also load custom models from local files.

model_namestr, optional

If you provide your own model, you can specify the model name for the key_added. Or you can override the model name by providing a new model name.

jitbool, default: False

Whether the model is a JIT model. If True, use torch.jit.load to load the model.

tokenstr, optional

The token for downloading the model from Hugging Face Hub for foundation models.

load_kwsdict, optional

Options to pass to the model creation function.

transformcallable, optional

The transform function for the input image. If not provided, a default ImageNet transform function will be used.

devicestr, optional

The device to use for inference. If not provided, the device will be automatically selected.

ampbool, default: False

Whether to use automatic mixed precision.

autocast_dtypetorch.dtype, default: torch.float16

The dtype for automatic mixed precision.

batch_sizeint, optional

The batch size for inference.

num_workersint, optional

The number of workers for data loading.

pbarbool, default: True

Whether to show progress bar.

tile_keystr, default: ‘tiles’

The key of the tiles dataframe in the spatial data object.

densebool, default: False

Whether to extract dense features for ViT models.

pool_mode{‘cls’, ‘cls_patch_mean’}, optional

The pooling mode for dense features. Only works for ViT models, ‘cls’ uses only CLS tokens, ‘cls_patch_mean’ uses CLS + mean patch tokens. If None, the models before v0.11.0 will keep it’s old behavior.

key_addedstr, optional

The key to store the extracted features.

return_featuresbool, default: False

Whether to return the extracted features. If dense=True, will return a (features, dense_features)

Returns:

numpy.ndarray or None

If return_features is True, return the extracted features.

Note

The feature matrix will be added to {model_name}_{tile_key} in tables slot of WSIData object.

Examples

>>> import lazyslide as zs
>>> wsi = zs.datasets.sample()
>>> zs.pp.find_tissues(wsi)
>>> zs.pp.tile_tissues(wsi, 256, mpp=0.5)
>>> zs.tl.feature_extraction(wsi, "resnet50")
>>> wsi.fetch.features_anndata("resnet50")