Assemble an aligned dataset
Collect image–caption pairs (for example MS COCO or curated website images with human-written alt text) and add edge cases such as diagrams and screenshots.
01 / THE IDEA
Automatic alt text generation in practice.
Automatic alt text generation produces concise, descriptive text that conveys the important visual content of an image for people who use screen readers. Typical failure modes include hallucinated details, omitted salient objects, non-descriptive language, over-long captions and cultural or privacy-sensitive content.
Real-world use case · Proposed extension
A proposed application: generate candidate alt text for a content-management system so an editor can review and approve suggestions. A human reviewer should remain in charge for accessibility and privacy decisions; this starter does not deliver a production-ready system.
02 / THE BUILD
How to build a automatic alt text generation project
step by step.
Choose a baseline model
Start with a multimodal encoder–decoder such as CLIP or an image‑encoder with a transformer decoder, and evaluate greedy and beam decoding variants.
Evaluate with task-specific metrics
Use recall@K for candidate grounding, CIDEr/BLEU/ROUGE for language overlap, and human usefulness/grounding scores to measure harmful hallucinations.
Design a review workflow
Expose candidate captions to human editors, log corrections for continuous training, and include checks for privacy-sensitive content.
03 / SOURCE SNAPSHOT
What the automatic alt text generation starter actually does.
The supplied starter is a small text-classification pipeline: it trains a scikit-learn Pipeline with TfidfVectorizer and LogisticRegression on six short, hand-written English sentences with binary labels, then prints a classification report and a sample prediction. It does not process images, image features or generate textual captions from visual input.
SOURCE EXCERPTPython
model = Pipeline(
steps=[
("tfidf", TfidfVectorizer(ngram_range=(1, 2), min_df=1)),
("classifier", LogisticRegression(max_iter=1000)),
]
)
model.fit(X_train, y_train)
predictions = model.predict(X_test)Important limitation
The starter code is a plain text classifier on six synthetic sentences and does not implement image captioning, multimodal embeddings, or any form of alt text generation. The tiny labelled set and its text-domain labels cannot validate the project task; no image data, visual encoders or decoder-generation logic are present.
04 / BEYOND THE CLASSROOM
Where a automatic alt text generation project could go next.
PROPOSED BUSINESS CONCEPT
Automatic Alt Text: a pilot concept
A proposed extension: replace the starter data with image–caption pairs, add an image encoder plus a text decoder (or a multimodal model like an encoder–decoder transformer), and deploy a candidate-suggestion API for human editors to accept or correct.
WHO IT COULD SERVEWeb publishers and CMS teams who need scalable accessibility checks.
FIRST MVPA small web UI that accepts images, returns 3 candidate captions via a pre-trained multimodal model, and logs editor selections for retraining.
KEY RISKHallucinated or privacy-sensitive captions could mislead users; ensure a human reviewer approves alt text before public use and do not use the prototype for critical accessibility decisions without evaluation.
This is a potential application pathway, not a delivered product. The supplied starter does not implement this service or validate it with real users.
05 / WHAT YOU TAKE AWAY
What a automatic alt text generation project puts on your resume.
- Describe the alt-text task and common failure modes such as hallucination and omitted salient objects.
- Prepare and validate an image–caption dataset that includes edge cases like diagrams and screenshots.
- Select and evaluate a multimodal encoder–decoder baseline using automatic metrics and human grounding checks.
- Explain this starter’s limitation: it trains a TF‑IDF + logistic classifier on six short texts and contains no image processing or caption generation.
Want to build this with mentors and a lab?
AI Engineer Course Networkers Home runs this topic inside its AI Engineer Course, with placement support and hands-on labs.
06 / QUESTIONS STUDENTS ASK
Automatic Alt Text Generation project FAQs.
- Is this suitable as a final year project for CSE students?
- Yes — it is an advanced, research-oriented topic that suits final year students comfortable with deep learning, multimodal models and evaluation with accessibility users.
- What skills and tools will I need to complete this idea?
- You should know Python, PyTorch, transformers or multimodal encoders, image preprocessing, dataset curation and evaluation metrics for captioning and accessibility.
- How should I evaluate progress and expected timeline?
- Start with dataset collection and baselines, then build multimodal models and evaluate on caption relevance and avoidance of hallucination; expect several months of work including experiments and user-focused evaluation.