All the sign language interpreter tutorials on YouTube train models with images, and not with landmarks. Due to this, their models work accurately only in conditions where the lighting and skin-tone is similar to the training data. This project aimed to solve that problem by using Mediapipe's landmark detection model.
I estimated and normalised the distance of each landmark of the hand from the center of the user's wrist. After that, I trained a simple model with scikit-learn. The training process is straightforward and well-documented in the Github repository.
As this solution uses landmarks instead of images, we eliminate chances of wrong predictions due to lighting or skin-tone.
As this project can currently work only with alphabets, I think it can be improved by incorporating Long Short-Term Memory (LSTM) networks to handle complex sequences, allowing to translates words and potentially sentences.
My friends and I spelling my name to also show that it works with varying skin tones and lighting, and with minimal training data.
Technologies used:
- Python
- Scikit-learn
- OpenCV
- Mediapipe