Assistive Artificial Intelligence for Independence: Computer Vision, Speech Recognition, and Multimodal Interaction

Introduction: From Traditional Aids to Smart Assistive Systems

Traditional assistive devices, such as the white cane, guide dogs, and augmentative communication devices, reduce specific barriers but rely on residual perception, user learning, or support from others. Recent reviews describe a transition toward systems capable of capturing environmental information, interpreting it, and producing tailored responses via audio, touch, or alternative interfaces (Abidi et al., 2024; Okolo et al., 2024). Within this framework, assistive artificial intelligence is not a single tool, but rather an architecture that combines sensors, machine learning models, natural language processing, and actuation mechanisms.

Autonomy should be understood as the ability to initiate, direct, and complete activities with less dependence, not as a complete replacement for human decision-making. Current prototypes focus on mobility, object recognition, communication, and environmental control (Giansanti & Pirrera, 2025). This literature shows clear technical advances, although it also reveals that model accuracy alone does not equate to everyday autonomy. The following analysis examines what computer vision and speech recognition can contribute, and what conditions limit their safe use.

Computer Vision for Safe Mobility and Environmental Perception

Assistive computer vision converts images captured by cameras into information about objects, obstacles, text, surfaces, and navigable paths. Its usefulness depends on the output being understandable, timely, and compatible with the decisions the person must make. Reviews distinguish between systems based on images, non-visual data, maps, and three-dimensional sound, along with solutions that combine ultrasonic sensors, LiDAR, smartphones, and location systems (Abidi et al., 2024). This diversity suggests that computer vision works best as part of a perceptual chain rather than as a standalone substitute for a white cane or other assistive devices.

Real-Time Recognition of Obstacles, Objects, and Signals

Recent systems identify objects and obstacles using deep detection models and translate the results into auditory warnings or tactile signals. AI-SenseVision integrates a camera, sensors, and a single-board processor to recognize objects, warn of obstacles, count items, and operate in both indoor and outdoor modes (Joshi et al., 2024). AIris expands on this functionality with scene description, text reading, facial recognition, money counting, barcode scanning, and note-taking via a natural language processing interface (Brilli et al., 2024). In both cases, the main contribution is not merely to detect, but to select information that can be used during an activity.

Obstacle detection has also been incorporated into mobile and portable devices. A YOLOv8-based system installed on a Raspberry Pi identified nine types of obstacles with an average accuracy of 91.70%, in addition to estimating distance, orientation, and surface moisture (Okolo et al., 2025). On a specific set of tactile paths in Dhaka, YOLOv8s achieved an mAP of 0.974 and an F1 score of 0.934; quantification reduced the model size by 49.53% and preserved 98.97% of the original mAP (Chowdhury et al., 2024). These figures pertain to different tasks and datasets, so they do not allow for the establishment of a general hierarchy among devices.

Fall detection represents another application of computer vision in residential settings. DFFCV-FDC combines noise filtering, features from MobileNet, DenseNet, and ResNet, hyperparameter selection, and an autoencoder to classify fall events in a benchmark dataset (ALMUKADI et al., 2024). This approach shifts the focus of assistance from individual mobility to incident detection and caregiver response. Its main limitation is that the described result corresponds to performance on a database, not to a demonstrated reduction in injuries or an improvement in autonomy in real-world nursing homes.

Assisted navigation using spatial reasoning, wearable sensors, and edge computing

Detecting objects in isolation does not necessarily indicate where to go. More advanced systems incorporate spatial reasoning, segmentation of navigable areas, and rules to convert relative positions into instructions. In the tactile path system, a heuristic breadth-first search algorithm generates obstacle-free routes and transmits them via a mobile app with voice output (Chowdhury et al., 2024). SightMate combines YOLOv8, Fast-SCNN semantic segmentation, left-center-right zone analysis, motion filtering, and inter-frame stability assessment (R et al., 2026). The workflow can be represented as follows:

Edge computing reduces reliance on a continuous connection and can minimize the exposure of personal images. AI-SenseVision processes data on a single, low-cost device (Joshi et al., 2024), while SightMate performs local inference using TensorFlow Lite on a smartphone (R et al., 2026). Another portable system based on a Raspberry Pi and a camera performed object detection and optical character recognition without an internet connection, with an average response time of 1.2 to 1.5 seconds and an object recognition accuracy of 88–90% (A et al., 2026). These implementations promote portability, although the described response times still need to be considered in relation to walking speed, pedestrian density, and collision risk.

The trade-off between low cost, algorithmic accuracy, and operation in dynamic environments

Low cost requires balancing computational capacity, battery life, size, connectivity, and accuracy. The model quantification in the Dhaka study shows that reducing size can maintain performance within a defined task (Chowdhury et al., 2024). In contrast, results from prototypes using Raspberry Pi, cameras, and sensors should be interpreted with caution because they typically come from selected scenarios and known obstacle sets (A et al., 2026; Okolo et al., 2025). Reviews note that indoor and outdoor environments require different combinations of sensors, maps, and feedback (Okolo et al., 2024).

Dynamic scenarios introduce pedestrians, changing lighting, uneven surfaces, occlusions, and camera movement. SightMate’s frame stabilization and motion filtering address part of this problem (R et al., 2026), but the review does not demonstrate safety over extended periods or across large populations. Therefore, detection accuracy must be distinguished from accident prevention. Autonomous mobility requires user validation, metrics for false alarms, reaction time, cognitive load, and error recovery—aspects that are addressed by voice analysis and communication.

Speech Recognition and Communication Adapted to Different Disabilities

Automatic speech recognition can generate captions, commands, speech synthesis, or device control, but its performance varies depending on the speaker, the environment, and the communicative purpose. For people with impaired speech, the difference between reading sentences and conversing spontaneously constitutes a specific source of error. In ALS, moreover, progressive deterioration necessitates reconsidering when and how to customize the model. Assistive communication should be evaluated based on the preservation of meaning and participation in interaction, not solely on the rate of recognized words.

Automatic Recognition of Conversational Speech, Dysarthric Speech, and Communication in ALS

In 27 individuals with various speech disorders, personalized and non-adapted models showed a gap between read and conversational speech; linguistic complexity accounted for much of this difference, while atypical speech characteristics also played a role (Tobin et al., 2024). Including conversational samples in training improved recognition, supporting personalization based on real-life situations rather than exclusively on controlled phrases (Tobin et al., 2024). The result calls into question evaluations that rely solely on reading aloud to represent everyday communication.

The experience of three people with ALS who used personalized captioning during conversations lasting up to twelve months was less favorable than technical expectations: all perceived lower-than-expected accuracy and found the system useful only in certain contexts (Cave, 2024). Participants valued real-time captioning but required training on customization, system limitations, and practical use. Erroneously attributing errors to the speaker can affect psychological well-being and conversational participation (Cave, 2024). Thus, customization must include human support, temporary adaptation to functional decline, and procedures for correcting errors without continuously interrupting the conversation.

Skin Sensors, Natural Language Processing, and Sign Language Recognition

Skin sensors offer an alternative approach when ambient noise or vocal production hinders recognition. A flexible acoustic sensor attached to the skin detected vibrations from the vocal organs and skin movements, achieving over 96% accuracy in classifying laryngeal features; in everyday sentences, the system achieved 99.8% accuracy among study participants (Liu et al., 2025). The architecture combined PMUT transducers, flexible encapsulation, and a ResNet network, with applications for human-machine interaction in adverse acoustic environments (Liu et al., 2025). These results are promising, although they come from a technology demonstration and do not yet establish sustained communicative utility for users with speech impairments.

For people with speech and hearing impairments, another system used a convolutional neural network to recognize Indian sign language and personalized signs, and then converted the gestures into voice commands using natural language processing. The convolutional network achieved 90% accuracy, compared to 80% for SVM, 81% for Random Forest, and 76% for XGBoost (Gandhi et al., 2025). The comparison shows an advantage within that experimental dataset, but does not allow us to conclude that the system generalizes to all regional variants, gesture speeds, or users with different manual mobility. Adaptive communication therefore requires models that incorporate individual vocabulary, conversational context, and alternative input modalities.

Multimodal Interaction and Expanding Autonomy Beyond Navigation

Multimodal interaction combines signals that complement one another when a single modality is ambiguous or inaccessible. Voice, vision, posture, facial expression, touch, and movement can distribute the control load between the user and the system. This expansion allows users to move beyond simply detecting obstacles to interpreting intentions, controlling home devices, requesting assistance, and supporting motor tasks. The evidence, however, remains concentrated on prototypes and limited tasks.

Fusion of voice, deictic posture, and computer vision to interpret intent

In human-robot interaction, voice specifies an action, and deictic posture helps identify the referred object. An experimental framework processed the scene using object detection, estimated bounding boxes based on depth, and aligned those boxes with speech-to-text converted commands; a large language model generated action sequences under syntactic constraints designed to reduce hallucinations (Lai et al., 2025). Evaluation using a UR3e manipulator showed better accuracy and robustness than the unimodal approaches described by the authors (Lai et al., 2025). The design illustrates a necessary condition for assistance: generative output must be constrained by spatial perception and verifiable control rules.

This logic can be extended to users who have difficulty producing complex syntax, executing conventional gestures, or selecting visual targets. Vision provides context, speech provides intent, and posture resolves reference. However, multimodal fusion also introduces new points of failure: the system may correctly detect an object and associate it with the wrong command. Evaluation should distinguish between recognition errors, reference errors, and execution errors, as each requires a different intervention.

Facial Interfaces, Exoskeletons, and Intelligent Assistance for People with Motor Impairments

Facial interfaces offer control when hands and voice are not reliable options. In a group of 11 participants with severe motor disabilities, an interface recognized facial gestures with 100% accuracy, responded in less than a second, and allowed control of lights, televisions, and requests for help; evaluation using the System Usability Scale yielded scores above 90 for most participants (Tapia & Rivera, 2025). The same study identified difficulties in customizing commands for people with more limited facial mobility (Tapia & Rivera, 2025). High accuracy under validation conditions does not eliminate the need for individual calibration nor does it demonstrate sustained effects on daily independence.

Exoskeletons and exosuits add physical actuation to the perception of intent. The review of these devices describes uses of machine learning to recognize intent, synchronize movement, assess the patient, and control tasks, as well as reinforcement learning supported by digital twins (van der Kooij et al., 2025). The authors note that patient-specific data and more accurate digital twins are needed to validate clinical use (van der Kooij et al., 2025). This caution is relevant: motor assistance can increase the capacity for action, but it can also lead to fatigue, unintended movements, or dependence on difficult calibration.

Critical Synthesis and Gaps in the Evidence on Assisted Autonomy

The evidence converges on a recurrent architecture: sensor, interpretation model, contextual decision, and accessible output. The following comparison shows that the highest results come from different tasks and populations; therefore, they should not be ranked as a common scale of autonomy.

Methodological approaches: technological prototypes, narrative reviews, and user validation

Technological prototypes evaluated based on accuracy, mAP, F1, response time, or usability predominate. Reviews of navigation and AI integration organize device families and describe opportunities, but they do not replace comparative autonomy trials (Abidi et al., 2024; Giansanti & Pirrera, 2025). User validations do exist, although they typically involve small samples or very specific populations, as is the case with the facial interface and the longitudinal ALS study (Cave, 2024; Tapia & Rivera, 2025). This combination allows for an assessment of feasibility but provides little information on adoption, discontinuation, cumulative safety, and effects on daily life.

Gaps in generalizability, accessibility, personalization, and assessment of real-world autonomy

Generalizability remains limited by local databases, small repertoires of objects, and experimental conditions. Personalization improves the recognition of dysarthric speech when it incorporates real conversation (Tobin et al., 2024), but it requires time, data, and technical support. Economic accessibility improves with smartphones, Raspberry Pi devices, and quantized models, although the total cost also includes maintenance, training, connectivity, and component replacement (A et al., 2026; Chowdhury et al., 2024). Future evaluation should measure completed decisions, avoided errors, cognitive effort, calibrated confidence, and perceived control, in addition to algorithmic performance.

Conclusion: Toward Integrated, Personalized, and Person-Centered Assistive Systems

Computer vision and speech recognition expand access to information that previously depended on visual perception, vocal production, or assistance from another person. Current systems already detect obstacles, describe scenes, read text, convert sign language to speech, provide captions for dysarthric speech, and control devices using facial gestures (Gandhi et al., 2025; Joshi et al., 2024; Tapia & Rivera, 2025). Their assistive value depends on the balance between accuracy, latency, interpretability, cost, and adaptability.

The next stage should prioritize multimodal platforms that operate locally whenever possible, incorporate individual preferences, and allow users to confirm or correct the system’s decisions. True autonomy requires long-term studies with diverse users, everyday scenarios, and outcomes focused on the activity itself, not solely on classification metrics. It also requires addressing privacy, digital literacy, standardization, and regulation—issues identified in the review of AI integration into assistive technologies (Giansanti & Pirrera, 2025). The goal is not to automate a person’s life, but to expand their scope for decision-making with systems that are predictable, adjustable, and subordinate to their needs.

References

• A, G. C., A, A. S., A, G., M, M., Harini, & R, K. (2026). A real-time multi-modal assistive system for the visually impaired using computer vision. In International Conference on Intelligent Computing and Control Systems (ICICCS) (pp. 1771–1774). https://doi.org/10.1109/iciccs67901.2026.11502716
• Abidi, M. H., Siddiquee, A. N., Alkhalefah, H., & Srivastava, V. (2024). A comprehensive review of navigation systems for visually impaired individuals. Heliyon, 10(11), e31825. https://doi.org/10.1016/j.heliyon.2024.e31825
• ALMUKADI, W. S., Alrowais, F., Saeed, M. K., Yahya, A. E., Mahmud, A., & Marzouk, R. (2024). Deep feature fusion with a computer vision-driven fall detection approach for enhanced assisted living safety. Scientific Reports, 14(1), 21537. https://doi.org/10.1038/s41598-024-71545-6
• Brilli, D. D., Georgaras, E., Tsilivaki, S., Melanitis, N., & Nikita, K. S. (2024). AIris: An AI-Powered Wearable Assistive Device for the Visually Impaired. In 2022 9th IEEE RAS/EMBS International Conference on Biomedical Robotics and Biomechatronics (BioRob) (pp. 1236–1241). https://doi.org/10.1109/biorob60516.2024.10719976
• Cave, R. (2024). How people living with amyotrophic lateral sclerosis use personalized automatic speech recognition technology to support communication. Journal of Speech, Language, and Hearing Research, 67(11), 4186–4202. https://doi.org/10.1044/2024_jslhr-24-00097
• Chowdhury, R. I., Anjom, J., & Hossain, Md. I. A. (2024). A novel edge intelligence-based solution for safer footpath navigation for the visually impaired using computer vision. Journal of King Saud University – Computer and Information Sciences, 36(8), 102191. https://doi.org/10.1016/j.jksuci.2024.102191
• Gandhi, K. I., Jawahar, P. K., & Kannan, G. (2025). AI-NLP-powered assistive technology system for individuals with speech and hearing impairments. Assistive Technology, 1–11. https://doi.org/10.1080/10400435.2025.2571782
• Giansanti, D., & Pirrera, A. (2025). Integrating AI and assistive technologies in healthcare: Insights from a narrative review of reviews. Healthcare, 13(5), 556. https://doi.org/10.3390/healthcare13050556
• Joshi, R. C., Singh, N. K., Sharma, A. K., Bürget, R., & Dutta, M. K. (2024). AI-SenseVision: A low-cost, artificial intelligence-based, robust, and real-time assistance system for visually impaired people. IEEE Transactions on Human-Machine Systems, 54(3), 325–336. https://doi.org/10.1109/thms.2024.3375655
• Lai, Y., Yuan, S., Nassar, Y. M., Fan, M., Gopal, A., Yorita, A., Kubota, N., & Rätsch, M. (2025). Natural multimodal fusion-based human–robot interaction: Application with voice and deictic posture via a large language model. IEEE Robotics & Automation Magazine, 33(2), 29–38. https://doi.org/10.1109/mra.2025.3543957
• Liu, T., Zhang, M., Li, Z., Dou, H., Zhang, W., Yang, J., Wu, P., Li, D., & Mu, X. (2025). Machine learning-assisted wearable sensing systems for speech recognition and interaction. *Nature Communications*, 16(1), 2363. https://doi.org/10.1038/s41467-025-57629-5
• Okolo, G. I., Althobaiti, T., & Ramzan, N. (2024). Assistive systems for visually impaired persons: Challenges and opportunities for navigation assistance. Sensors, 24(11), 3572. https://doi.org/10.3390/s24113572
• Okolo, G. I., Althobaiti, T., & Ramzan, N. (2025). Smart assistive navigation system for visually impaired people. Journal of Disability Research, 4(1). https://doi.org/10.57197/jdr-2024-0086
• R, D., Harmithaa, S, G., & Gnanapriya. (2026). SightMate: A vision-based assistive navigation system for the visually impaired. In Zenodo (CERN European Organization for Nuclear Research). https://doi.org/10.5281/zenodo.19533366
• Tapia, C., & Rivera, G. (2025). Human-Machine Interface for the Assistance and Care of Elderly People with Motor Dependence: A Transformative Proposal Based on Computer Vision. In World Physiotherapy Congress Archive. https://doi.org/10.82161/ga4p-a459
• Tobin, J., Nelson, P., MacDonald, B., Heywood, R., Cave, R., Seaver, K., Desjardins, A., Jiang, P.-P., & Green, J. R. (2024). Automatic speech recognition of conversational speech in individuals with speech disorders. Journal of Speech Language and Hearing Research, 67(11), 4176–4185. https://doi.org/10.1044/2024_jslhr-24-00045
• van der Kooij, H., van Asseldonk, E. H. F., Sartori, M., Basla, C., Esser, A., & Riener, R. (2025). AI in therapeutic and assistive exoskeletons and exosuits: Influences on performance and autonomy. Science Robotics, 10(104), eadt7329. https://doi.org/10.1126/scirobotics.adt7329