Create or obtain embeddings
Replace the toy random vectors with real image and text embeddings (for example CLIP image/text vectors) computed only on training/approved content.
Match landmarks with guide text. Judge multimodal relevance.
This project demonstrates a simple multimodal retrieval baseline: represent images and text as embeddings, combine them, and find nearest neighbours to suggest candidate guide entries for a photographed landmark. In real builds you must replace synthetic embeddings with models such as CLIP, ensure the guide text is licensed, and handle ambiguous or multiple landmarks in one photo.
Learning concept · Not a validated service
This project demonstrates a simple multimodal retrieval baseline: represent images and text as embeddings, combine them, and find nearest neighbours to suggest candidate guide entries for a photographed landmark. In real builds you must replace synthetic embeddings with models such as CLIP, ensure the guide text is licensed, and handle ambiguous or multiple landmarks in one photo.
A proposed application is a photo-to-guide assistant that suggests short guide snippets for a photographed landmark; human editors should review suggestions before publishing to avoid mislabelling or misleading content.
Replace the toy random vectors with real image and text embeddings (for example CLIP image/text vectors) computed only on training/approved content.
Concatenate or otherwise fuse image and text embeddings to produce a joint representation for each guide entry.
Compute cosine similarity between a query (photo ± query text) and stored embeddings, then rank and return the top-K matches.
Measure recall@K and human usefulness on a labelled validation set; inspect failure modes such as visually similar but different landmarks.
The supplied starter creates random 'text' and 'image' embeddings, concatenates them into multimodal vectors, forms a noisy query from the first pair, computes cosine similarity to the stored vectors, and prints the top-3 match indices and their scores. It is a toy retrieval demo and does not use CLIP, real images, or guide text.
text_embeddings = rng.normal(size=(5, 8))
image_embeddings = rng.normal(size=(5, 8))
combined_embeddings = np.hstack([text_embeddings, image_embeddings])
query_text_embedding = text_embeddings[0] + rng.normal(0, 0.05, 8)
query_image_embedding = image_embeddings[0] + rng.normal(0, 0.05, 8)
query = np.hstack([query_text_embedding, query_image_embedding]).reshape(1, -1)The code uses randomly generated vectors rather than any real image or text embeddings, so its matches are synthetic and do not reflect real-world landmark-to-guide retrieval. There is no dataset, no grounding of guide text, no indexing for scale, and no evaluation on labelled retrieval pairs.
A proposed extension: replace random vectors with CLIP embeddings, index a small curated dataset of landmark images and short guide snippets, and build a UI that shows top-K suggestions for human editors to accept or correct.
This is a potential application pathway, not a delivered product. The supplied starter does not implement this service or validate it with real users.
Networkers Home runs this topic inside its AI Engineer Course, with placement support and hands-on labs.