| title | Sign Sarthi |
|---|---|
| emoji | 🤟 |
| colorFrom | purple |
| colorTo | blue |
| sdk | docker |
| pinned | false |
| license | mit |
A state-of-the-art web application for Indian Sign Language (ISL) recognition and translation, empowering deaf and hard-of-hearing communities through real-time gesture detection, text-to-ISL conversion, and multi-language translation capabilities.
- Overview
- Key Features
- System Architecture
- Technology Stack
- Project Structure
- Installation
- Usage Examples
- API Endpoints
- Model Details
- Deployment
- Performance
- Contributing
- License
Sign Sarthi bridges the communication gap between sign language users and non-users by providing:
- Real-time ISL-to-Text Translation: Convert hand gestures into readable text instantly
- Text-to-ISL Animation: Transform written text into animated sign language videos
- Multi-Language Support: Translate between ISL and 9+ Indian regional languages
- Accessibility Features: Text-to-speech and intuitive UI
🎥 Watch our application in action: YouTube Demo
- Live Hand Tracking: MediaPipe-powered 21-point hand landmark detection
- 26 Alphabets Recognition: A-Z static sign gesture recognition
- High Accuracy: 95%+ character recognition accuracy
- Smart Detection:
- 1-second hold time to confirm gesture
- 2-second cooldown to prevent duplicate characters
- Visual feedback with color-coded prediction indicators
- Optimistic UI Updates: Instant display updates without server lag
- Convert any text input into ISL video animations
- Character-by-character video playback
- Support for complete sentences and phrases
- Video library of 26 alphabets + common words
Translate ISL-recognized text into:
- Hindi (हिंदी)
- Marathi (मराठी)
- Bengali (বাংলা)
- Tamil (தமிழ்)
- Telugu (తెలుగు)
- Gujarati (ગુજરાતી)
- Kannada (ಕನ್ನಡ)
- Malayalam (മലയാളം)
- Punjabi (ਪੰਜਾਬੀ)
- Add Space: Insert spaces between words
- Delete Last: Remove the last character
- Clear All: Reset the entire sentence
- Speak: Text-to-speech using browser's Web Speech API
- Real-time Display: Live update of predicted text
- Responsive Design: Works on desktop, tablet, and mobile
- Dark/Light Theme: Comfortable viewing in any environment
- Visual Feedback: Color-coded status indicators
- 🔵 Blue: New sign detected
- 🟠 Orange: Confirming gesture (1-second hold)
- 🟢 Green: Character successfully added
- 🔴 Red: Error or no detection
- Landmark Visualization: Real-time hand skeleton overlay on video feed
- Camera Controls: Start/stop camera with permission management
┌─────────────────────────────────────────────────────────────────┐
│ User Browser │
│ ┌─────────────┐ ┌──────────────┐ ┌─────────────────┐ │
│ │ Webcam │ │ MediaPipe │ │ Web Speech API │ │
│ │ Feed │→ │ Hand Track │→ │ (TTS/STT) │ │
│ └─────────────┘ └──────────────┘ └─────────────────┘ │
│ ↓ ↓ ↓ │
│ ┌──────────────────────────────────────────────────────┐ │
│ │ JavaScript Frontend Controller │ │
│ │ • Frame Processing • Gesture Detection │ │
│ │ • Optimistic Updates • Real-time Display │ │
│ └──────────────────────────────────────────────────────┘ │
└────────────────────────────┬────────────────────────────────────┘
│ HTTPS/WebSocket
↓
┌─────────────────────────────────────────────────────────────────┐
│ Flask Backend Server │
│ ┌───────────────────────────────────────────────────────┐ │
│ │ RESTful API Endpoints │ │
│ │ /process_frame /add_character /clear_sentence │ │
│ └───────────────────────────────────────────────────────┘ │
│ ↓ ↓ ↓ │
│ ┌─────────────┐ ┌─────────────────┐ ┌──────────────┐ │
│ │ MediaPipe │ │ TensorFlow/ │ │ Translation │ │
│ │ Processor │→ │ Keras Model │→ │ Service │ │
│ └─────────────┘ └─────────────────┘ └──────────────┘ │
│ ↓ │
│ ┌──────────────────────────────────────────────────────┐ │
│ │ Global State Management │ │
│ │ • Predicted Text Storage │ │
│ │ • Session Handling │ │
│ └──────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────┘
↓
┌─────────────────────────────────────────────────────────────────┐
│ Docker Container (Hugging Face Spaces) │
│ • Python 3.10 Runtime │
│ • Gunicorn WSGI Server (2 workers, 4 threads) │
│ • Port 7860 Exposed │
└─────────────────────────────────────────────────────────────────┘
1. Real-time ISL Recognition Flow:
Webcam → Canvas Capture → Base64 Encoding → POST /process_frame
→ MediaPipe Hand Detection → Landmark Extraction → Feature Engineering
→ Model Prediction → Character Return → Frontend Display Update
2. Text-to-ISL Conversion Flow:
Text Input → Character Split → Video Mapping → Sequential Playback
→ Animation Display → Completion Callback
3. Multi-Language Translation Flow:
Recognized Text → Google Translate API → Target Language
→ Formatted Output → Display
| Technology | Purpose | Version |
|---|---|---|
| HTML5/CSS3 | UI Structure & Styling | Latest |
| JavaScript (ES6+) | Client-side Logic | ES2020+ |
| MediaPipe Hands | Hand Landmark Detection | 0.10.9 |
| Canvas API | Video Frame Processing | Native |
| Web Speech API | Text-to-Speech & Speech-to-Text | Native |
| Fetch API | Asynchronous HTTP Requests | Native |
| Technology | Purpose | Version |
|---|---|---|
| Python | Core Programming Language | 3.10+ |
| Flask | Web Framework | 3.0.0 |
| TensorFlow | Deep Learning Framework | 2.15.0 |
| Keras | Neural Network API | Bundled |
| MediaPipe | Computer Vision Pipeline | 0.10.9 |
| OpenCV | Image Processing | 4.8.1 |
| NumPy | Numerical Computing | 1.24.3 |
| Gunicorn | WSGI HTTP Server | 21.2.0 |
| Platform | Purpose |
|---|---|
| Hugging Face Spaces | Cloud Hosting |
| Docker | Containerization |
| Git LFS | Large File Storage (model.h5) |
- Git - Version Control
- VS Code - IDE
- Jupyter Notebook - Model Training & Experimentation
ISL/
├── 📄 app.py # Main Flask application (413 lines)
│ ├── Route handlers (/process_frame, /add_character, etc.)
│ ├── MediaPipe initialization
│ ├── Model loading and prediction logic
│ └── Global state management
│
├── 🤖 model.h5 # Trained Keras model (~11.5 MB)
│ └── Sequential CNN for gesture classification
│
├── 📋 requirements.txt # Python dependencies
│ ├── Flask==3.0.0
│ ├── tensorflow==2.15.0
│ ├── mediapipe==0.10.9
│ └── opencv-python==4.8.1.78
│
├── 🐳 Dockerfile # Docker container configuration
│ ├── Base: python:3.10-slim
│ ├── System dependencies (libgl1, ffmpeg, etc.)
│ ├── Python packages installation
│ └── Gunicorn server setup
│
├── 🌐 templates/ # HTML templates
│ ├── index.html # Landing page with features overview
│ ├── ISL.html # Real-time ISL detection interface (1052 lines)
│ ├── text_to_isl.html # Text-to-ISL converter
│ ├── about.html # About page
│ └── community.html # Community resources
│
├── 🎨 static/ # Static assets
│ ├── assets/
│ │ ├── css/ # Stylesheets
│ │ ├── js/ # JavaScript libraries
│ │ ├── images/ # UI images and icons
│ │ └── webfonts/ # Font files
│ ├── images/ # Feature images
│ ├── videos/ # ISL alphabet video library (A-Z)
│ │ ├── a.mp4
│ │ ├── b.mp4
│ │ └── ... (26 videos)
│ └── script/
│ └── script.js # Global JavaScript utilities
│
├── 🧠 Model/ # Model development directory
│ └── Model/
│ ├── dataset_keypoint_generation.py # Feature extraction script
│ ├── ISL.ipynb # Training notebook
│ ├── keypoint.csv # Training dataset
│ ├── main.py # Model training script
│ └── model.h5 # Backup model file
│
├── 🎬 NFSL/ # Non-Formal Sign Language (Action Recognition)
│ ├── NFSL.py # Action detection script
│ ├── Action Detection Refined.ipynb # Development notebook
│ ├── action.h5 # LSTM model for action sequences
│ ├── MP_Data/ # MediaPipe landmark data
│ │ ├── hello/ # Sequence data for "hello"
│ │ ├── iloveyou/ # Sequence data for "I love you"
│ │ ├── namaste/ # Sequence data for "namaste"
│ │ └── thanks/ # Sequence data for "thanks"
│ ├── Logs/ # TensorBoard training logs
│ └── templates/
│ └── index.html # NFSL interface
│
├── 📷 image/ # Application screenshots
│
├── 📝 README.md # This file
├── 📝 README_HF.md # Hugging Face Spaces README
├── 🔐 .gitignore # Git ignore rules
├── 📦 .dockerignore # Docker ignore rules
└── 🗂️ .gitattributes # Git LFS configuration
app.py - Core application logic:
- Flask routes for all API endpoints
- MediaPipe hand detection and landmark processing
- Keras model inference
- Global state management for predicted text
- Image processing and base64 encoding/decoding
templates/ISL.html - Main detection interface:
- Camera initialization and frame capture
- Real-time landmark visualization
- Gesture detection with 1-second hold + 2-second cooldown
- Optimistic UI updates (frontend-first approach)
- Button controls (Space, Delete, Clear, Speak)
model.h5 - Trained neural network:
- Input: 42 features (21 landmarks × 2 coordinates)
- Architecture: Dense layers with dropout
- Output: 26 classes (A-Z)
- Training accuracy: ~95%
- Python 3.10+
- Webcam for real-time gesture recognition
- 4GB+ RAM recommended
- Modern browser (Chrome, Edge, Firefox, or Safari)
- Git for cloning the repository
- Git LFS (for downloading model.h5)
git clone https://github.com/A-01-hub/ISL.git
cd ISL
git lfs install
git lfs pull
This will download the model.h5 file (~11.5 MB).
Windows:
python -m venv venv
venv\Scripts\activate
macOS/Linux:
python3 -m venv venv
source venv/bin/activate
pip install --upgrade pip
pip install -r requirements.txt
Dependencies installed:
- Flask (web framework)
- TensorFlow (deep learning)
- MediaPipe (hand tracking)
- OpenCV (image processing)
- NumPy (numerical operations)
- Gunicorn (production server)
Development mode:
python app.py
Production mode (with Gunicorn):
gunicorn --bind 0.0.0.0:7860 --workers 2 --threads 4 --timeout 120 app:app
Open your browser and navigate to:
- Local:
http://localhost:5000 - With Gunicorn:
http://localhost:7860
# Build the Docker image
docker build -t sign-sarthi .
# Run the container
docker run -p 7860:7860 sign-sarthi
Access at http://localhost:7860
- Navigate to the ISL Detection page
- Click "Start Camera" button
- Allow camera permissions when prompted
- Make a sign gesture (e.g., "A", "B", "C")
- Hold the gesture for 1 second (indicator turns orange → green)
- Watch the predicted text appear in the display box
- Continue making signs to form words
Example Output:
User signs: H → E → L → L → O
Display shows: "HELLO"
- Go to "Text to ISL" page
- Type any text (e.g., "HELLO WORLD")
- Click "Convert to ISL"
- Watch animated videos play for each character
- See H → E → L → L → O → (space) → W → O → R → L → D videos
Scenario: User makes a mistake while signing
1. User signs: H → E → L → P (meant to type "HELLO")
Display: "HELP"
2. User clicks "Delete Last" button
Display: "HEL"
3. User signs: L → O
Display: "HELLO"
4. User clicks "Add Space" button
Display: "HELLO "
5. User signs: W → O → R → L → D
Display: "HELLO WORLD"
6. User clicks "Speak" button
Audio: "Hello World" (spoken by browser TTS)
Workflow:
1. Sign in ISL: H → I
Display: "HI"
2. Select language: "Hindi"
3. Click "Translate"
4. Output: "नमस्ते" (Namaste)
Supported Languages:
- English → Hindi: "HELLO" → "नमस्ते"
- English → Marathi: "HELLO" → "नमस्कार"
- English → Bengali: "HELLO" → "হ্যালো"
- English → Tamil: "HELLO" → "வணக்கம்"
- And 5 more languages...
Process a video frame and detect hand gestures.
Request:
{
"frame": "data:image/jpeg;base64,/9j/4AAQSkZJRg..."
}
Response:
{
"success": true,
"prediction": "A",
"confidence": 0.98,
"processed_image": "data:image/jpeg;base64,..."
}
Error Response:
{
"error": "No hands detected",
"success": false
}
Add a character to the predicted text.
Request:
{
"character": "A",
"current_text": "HELL"
}
Response:
{
"success": true,
"predicted_text": "HELLA"
}
Retrieve the current predicted text.
Response:
{
"predicted_text": "HELLO WORLD"
}
Clear all predicted text.
Response:
{
"success": true,
"predicted_text": ""
}
Remove the last character from predicted text.
Response:
{
"predicted_text": "HELL"
}
Add a space character.
Response:
{
"predicted_text": "HELLO "
}
Trigger text-to-speech (deprecated - now client-side).
Response:
{
"success": true,
"text": "HELLO WORLD"
}
- Process Frame: ~6-7 FPS (150ms intervals)
- Add Character: No limit (optimistic updates)
- Other endpoints: No explicit limits
| Code | Description |
|---|---|
| 200 | Success |
| 400 | Bad Request (invalid frame data) |
| 500 | Server Error (model failure, processing error) |
Model Type: Sequential Keras Neural Network
Input Layer:
- Shape: (42,) - 21 landmarks × 2 coordinates (x, y)
- Normalized hand keypoints from MediaPipe
Hidden Layers:
Dense(256, activation='relu') → Dropout(0.3) →
Dense(128, activation='relu') → Dropout(0.3) →
Dense(64, activation='relu') → Dropout(0.2)
Output Layer:
- Dense(26, activation='softmax')
- 26 classes for A-Z alphabets
Optimizer: Adam Loss: Categorical Crossentropy Metrics: Accuracy
Dataset:
- Custom ISL alphabet dataset
- 21 landmarks per hand (42 features total)
- ~1000+ samples per alphabet
- Train/Test split: 80/20
Data Preprocessing:
- Hand detection using MediaPipe
- Landmark extraction (21 points)
- Normalization to [0, 1] range
- Feature vector creation (42 dimensions)
Training Configuration:
- Epochs: 50-100
- Batch Size: 32
- Validation Split: 20%
- Early Stopping: Patience 10
Performance Metrics:
- Training Accuracy: ~97%
- Validation Accuracy: ~95%
- Test Accuracy: ~93%
- Inference Time: <50ms per frame
MediaPipe Hand Landmarks (21 points):
0: WRIST
1-4: THUMB (CMC, MCP, IP, TIP)
5-8: INDEX (MCP, PIP, DIP, TIP)
9-12: MIDDLE (MCP, PIP, DIP, TIP)
13-16: RING (MCP, PIP, DIP, TIP)
17-20: PINKY (MCP, PIP, DIP, TIP)
Feature Vector Construction:
[x0, y0, x1, y1, ..., x20, y20] = 42 features
Normalization:
- All coordinates scaled to [0, 1]
- Relative to frame dimensions
Planned Enhancements:
- ✅ Static gesture recognition (A-Z)
- 🔄 Dynamic gesture recognition (words, phrases)
- 🔄 Two-hand gesture support
- 🔄 Context-aware predictions
- 🔄 Transfer learning with larger datasets
Live Demo: https://huggingface.co/spaces/adityasuhane01/ISL
Deployment Steps:
- Create Space on Hugging Face
- Configure
README.mdwith metadata:--- title: Sign Sarthi sdk: docker --- - Push code with Git LFS:
git lfs track "*.h5" git add .gitattributes model.h5 git commit -m "Add model with Git LFS" git push - Auto-build triggers on push
- Access via provided URL
Configuration:
- Docker container with Gunicorn
- 2 workers, 4 threads
- Port 7860 exposed
- 16GB RAM allocation
- GPU: Not required (CPU inference)
# Development server (Flask built-in)
python app.py
# Production server (Gunicorn)
gunicorn --bind 0.0.0.0:7860 --workers 2 --threads 4 app:app
# Build image
docker build -t sign-sarthi:latest .
# Run container
docker run -d -p 7860:7860 --name sign-sarthi sign-sarthi:latest
# View logs
docker logs -f sign-sarthi
# Stop container
docker stop sign-sarthi
# Optional configurations
export PORT=7860
export PYTHONUNBUFFERED=1
export SECRET_KEY="your-secret-key"
| Metric | Value |
|---|---|
| Frame Processing | ~150-200ms |
| Model Inference | ~30-50ms |
| End-to-End Latency | ~200-300ms |
| FPS | 6-7 frames/sec |
| Memory Usage | ~1.5-2GB |
| CPU Usage | 30-50% (2 cores) |
- Optimistic UI Updates: Frontend updates display immediately, syncs with backend asynchronously
- Frame Throttling: Process at 6-7 FPS instead of 30 FPS to reduce server load
- Cooldown Mechanism: 2-second cooldown prevents duplicate character detection
- Request Batching: Combine multiple operations when possible
- Caching: Model loaded once at startup
| Browser | Version | Support |
|---|---|---|
| Chrome | 90+ | ✅ Full |
| Edge | 90+ | ✅ Full |
| Firefox | 88+ | ✅ Full |
| Safari | 14+ | ✅ Full (with limitations) |
| Opera | 76+ | ✅ Full |
Required Browser Features:
- WebRTC (camera access)
- Canvas API
- Fetch API
- Web Speech API (for TTS)
- ES6+ JavaScript
Issue 1: Camera Not Working
Error: "Camera permission denied"
Solution:
1. Check browser permissions
2. Use HTTPS (required for camera on non-localhost)
3. Try different browser
4. Check if camera is being used by another application
Issue 2: Model Not Loading
Error: "Failed to load resource: model.h5"
Solution:
1. Ensure Git LFS is installed: git lfs install
2. Pull LFS files: git lfs pull
3. Verify model.h5 exists and is ~11.5 MB
Issue 3: Text Disappearing
Error: Characters vanish after adding
Solution:
1. Hard refresh browser (Ctrl+Shift+R)
2. Clear browser cache
3. Check console for JavaScript errors
4. Ensure latest code is deployed
Issue 4: Poor Detection Accuracy
Error: Wrong characters detected
Solution:
1. Improve lighting conditions
2. Position hand clearly in frame
3. Make distinct, clear gestures
4. Hold gesture for full 1 second
5. Avoid moving hand during detection
Issue 5: High Latency
Error: Slow response time
Solution:
1. Close other browser tabs
2. Check internet connection
3. Reduce camera resolution
4. Use desktop instead of mobile
Enable detailed logging by checking browser console (F12):
// Look for these log messages:
"Page loaded, starting camera..."
"Camera started successfully"
"Adding character: X (last: Y, cooldown: Zms)"
"Server response: {predicted_text: '...', success: true}"
We welcome contributions from the community! Here's how you can help:
- 🐛 Report Bugs: Open an issue with detailed description
- 💡 Suggest Features: Share ideas for new functionality
- 📝 Improve Documentation: Fix typos, add examples
- 🔧 Submit Code: Fix bugs or add features
- 🎨 Design Improvements: Enhance UI/UX
- 🧪 Testing: Test on different devices/browsers
- Fork the repository
- Clone your fork:
git clone https://github.com/YOUR_USERNAME/ISL.git cd ISL - Create branch for your feature:
git checkout -b feature/AmazingFeature - Make changes and test thoroughly
- Commit with descriptive messages:
git commit -m "Add feature: Real-time translation to Hindi" - Push to your fork:
git push origin feature/AmazingFeature - Open Pull Request with description of changes
- Python: Follow PEP 8 style guide
- JavaScript: Use ES6+ features, proper indentation
- Comments: Add clear comments for complex logic
- Testing: Test all changes locally before PR
- Documentation: Update README if adding features
# Install development dependencies
pip install -r requirements.txt
pip install black flake8 pytest
# Run linter
flake8 app.py
# Format code
black app.py
# Run tests (if available)
pytest tests/
- ✅ A-Z alphabet recognition
- ✅ Real-time detection with MediaPipe
- ✅ Text-to-ISL conversion
- ✅ Audio-to-ISL translation
- ✅ Multi-language support (9 languages)
- ✅ Browser-based TTS
- ✅ Optimistic UI updates
- ✅ Docker deployment
High Priority:
- 🔄 Word-level gesture recognition (not just letters)
- 🔄 Two-hand gesture support
- 🔄 Context-aware predictions
- 🔄 User accounts and history
- 🔄 Mobile app (React Native)
Medium Priority:
- 🔄 Custom gesture training interface
- 🔄 Offline mode with PWA
- 🔄 Video recording and sharing
- 🔄 Community gesture database
- 🔄 Gamification and learning mode
Low Priority:
- 🔄 AR/VR integration
- 🔄 3D hand model visualization
- 🔄 Multi-user video chat with ISL translation
- 🔄 Integration with popular video conferencing tools
- Support for international sign languages (ASL, BSL, etc.)
- AI-powered sentence completion
- Real-time conversation translation
- Educational platform for learning ISL
- API for third-party integrations
This project is licensed under the MIT License.
MIT License
Copyright (c) 2024 Sign Sarthi Team
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
- MediaPipe - Google's ML framework for hand landmark detection
- TensorFlow - Deep learning framework for model training
- Keras - High-level neural networks API
- Flask - Python web framework
- OpenCV - Computer vision library
- Hugging Face - ML platform for hosting and deployment
- Indian Sign Language Research Foundation - For ISL gesture standards
- Deaf community members - For feedback and testing
- Open source community - For tools and libraries
- Developers: A-01-hub team
- ML Engineers: Model training and optimization
- UI/UX Designers: Interface design
- Contributors: See Contributors
- 📧 Email: support@signsarthi.com
- 💬 Discussions: GitHub Discussions
- 🐛 Bug Reports: GitHub Issues
- 📚 Documentation: Wiki
- 🌐 Website: www.signsarthi.com
- 🐦 Twitter: @SignSarthi
- 📘 Facebook: SignSarthi
- 📱 Instagram: @signsarthi
- ⭐ Star this repo to show support
- 👀 Watch for updates and releases
- 🔔 Subscribe to notifications
Made with ❤️ for the Deaf and Hard-of-Hearing Community