lazyslide.tl.text_image_similarity

text_image_similarity(wsi, text_embeddings, model='plip', tile_key='tiles', feature_key=None, key_added=None)

Compute the similarity between text and image.

Note

Prerequisites:

• The image features should be extracted using zs.tl.feature_extraction.

• The text embeddings should be computed using zs.tl.text_embedding.

Parameters:

wsiWSIData

The WSIData object to work on.

text_embeddingspd.DataFrame

The embeddings of the texts, with texts as index.

model{“plip”, “conch”, “omiclip”}, default: “plip”

The text embedding model.

tile_keystr, default: ‘tiles’

The tile key.

feature_keystr

The feature key.

key_addedstr

The key to store the similarity scores. If None, defaults to ‘{feature_key}_text_similarity’.

Returns:

None

Note

The similarity scores will be saved as an to tables slot of the spatial data object.

Examples

>>> import lazyslide as zs
>>> wsi = zs.datasets.sample()
>>> zs.pp.find_tissues(wsi)
>>> zs.pp.tile_tissues(wsi, 256, mpp=0.5, key_added="text_tiles")
>>> zs.tl.feature_extraction(wsi, "plip", tile_key="text_tiles")
>>> terms = ["mucosa", "submucosa", "musclaris", "lymphocyte"]
>>> embeddings = zs.tl.text_embedding(terms, model="plip")
>>> zs.tl.text_image_similarity(wsi, embeddings, model="plip", tile_key="text_tiles")