TWI809301B - Visually impaired voice assistance system and visually impaired voice assistance method - Google Patents

Abstract

The invention provides a voice assistance system for the visually impaired and a voice assistance method for the visually impaired. The speech assistance system for the visually impaired includes a head-mounted device, a portable electronic device and a recognition server. The head-mounted device captures the environment in front of the user to generate a real-time environment image. The portable electronic device receives real-time environmental images and transmits them. The database of the recognition server stores a plurality of reference object features and corresponding reference object information. The learning module of the identification server recognizes a plurality of environmental object features from the real-time environmental images from the portable electronic device, and generates real-time auxiliary information containing corresponding reference object information when comparing the environmental object features with the reference object features, and feeds back to Portable Electronic Devices. Wherein, the portable electronic device generates auxiliary voice messages according to the real-time auxiliary information and controls the head-mounted device to send out.

Description

Visually impaired voice assistance system and visually impaired voice assistance method

The present invention relates to an identification system, in particular to a voice assistance system for the visually impaired and a voice assistance method for the visually impaired.

By the way, due to visual obstacles, the visually impaired people have a lot of inconvenience in the process of action. Thanks to the advancement of technology today, many related companies are committed to developing various visually impaired mobility aids to allow the visually impaired It is more convenient to engage in various activities and helps the visually impaired to live more conveniently at home.

However, although the above-mentioned mobility aids for the visually impaired can achieve the effect of assisting the movement of the visually impaired, in actual operation and use, if the functions are complete, the mobility aids for the visually impaired are not only relatively complicated in overall structural design. Moreover, most of the visually impaired mobility aids will remind the user of obstacles, and there are many variables in the outdoor environment to restrict the user's actions, and even cause the user's danger, which is not comprehensive enough. Or, most of the mobility aids for the visually impaired directly instruct the user to walk, and it lacks the opportunity for the user to choose the walking path by himself or the like.

In view of the foregoing, the inventor of the present invention conceived and designed a voice assistance system and a voice assistance method for the visually impaired, in order to improve the deficiencies of conventional technologies, and further enhance the implementation and utilization in the industry.

In view of the above known problems, the object of the present invention is to provide a voice assistance system and a voice assistance method for the visually impaired to solve the problems faced in the prior art.

Based on the above purpose, the present invention provides a speech assistance system for the visually impaired, which includes a head-mounted device, a portable electronic device, and a recognition server. Wherein, the head-mounted device includes a camera module and a sound output module. The camera module captures the environment in front of the user to generate a real-time environment image. The sound output module sends auxiliary voice messages. The portable electronic device includes a processing module and a wireless transmission module. The wireless transmission module executes multiple execution procedures between the client and the server according to the transmission control protocol, and transmits the real-time environment image to the recognition server. The recognition server system includes a database and a learning module. The recognition server receives the real-time environment image from the portable electronic device; the database stores a plurality of reference object features and corresponding reference object information; the learning module recognizes a plurality of environments from the real-time environment image Object characteristics, and the learning module compares and analyzes the characteristics of each of the environmental objects, and when it matches the characteristics of each of the reference objects, the learning module generates real-time auxiliary information containing the corresponding information of each of the reference objects and feeds it back to the portable electronic device. device. Among them, the wireless transmission module executes multiple execution procedures between the client and the server according to the transmission control protocol, so that the real-time auxiliary information can be stored synchronously and remotely to the cloud server or the local terminal for quick query , capture, deployment and analysis. Wherein, the processing module generates the auxiliary voice message according to the real-time auxiliary information, and controls the sound output module to send out the auxiliary voice message.

Preferably, the auxiliary voice information includes each piece of reference object information and the position and distance of the object corresponding to each piece of reference object information relative to the user in the real-time environment image.

Preferably, the learning module compares the features of the environmental objects, including object and face features, based on the single-shot multi-frame detector module of the tensor flow algorithm and the convolutional neural network algorithm of deep learning , whether it conforms to the characteristics of each reference object.

Preferably, the head-mounted device further includes a radio module, which receives an inquiry message from the user, and the processing module generates the auxiliary voice message according to the inquiry message and the real-time auxiliary information, and controls the sound output module The system sends out the auxiliary voice message.

Preferably, when the real-time environment image contains mobile barcode information, the processing module is connected to the application server according to the mobile barcode information to read the corresponding item information, and generate the auxiliary voice according to the item information message, to control the voice module to send out the auxiliary voice message.

Based on the above purpose, the present invention further provides a voice assistance method for the visually impaired, which includes the following steps: using the camera module of the head-mounted device to capture the environment in front of the user to generate a real-time environment image; receiving the real-time environment image through a portable electronic device And send it to the recognition server; identify a plurality of environmental object features from the real-time environmental image, and compare whether each of the environmental object features conforms to a plurality of reference object features in the database; generate a reference object corresponding to the reference object feature The real-time auxiliary information of the object information is fed back to the portable electronic device; and the auxiliary voice message is generated according to the real-time auxiliary information, and the sound output module of the head-mounted device is controlled to send out the auxiliary voice message.

Preferably, the auxiliary voice message includes the reference object information and the position of the object corresponding to the reference object information in the environment image, and the relative distance between the object corresponding to the reference object information and the user. The location and distance relative to the user in the real-time environment image.

Preferably, the voice assistance method for the visually impaired further includes the following steps: comparing whether the features of the environmental objects match the reference objects according to the tensor flow algorithm for object recognition and the deep learning convolutional neural network algorithm for face recognition feature, and estimate the distance between the recognized environmental object and the user, wherein the environmental object includes a human face.

Preferably, the voice assistance method for the visually impaired further includes the following steps: comparing each of the facial features according to the convolutional neural network of deep learning to see if they match the features of each of the reference objects in the training data set.

Preferably, the voice assistance method for the visually impaired further includes the following steps: receiving an inquiry message from the user; voice message.

Preferably, the voice assistance method for the visually impaired further includes the following steps: judging whether the real-time environment image contains a mobile barcode (QR Code) information; if so, connecting to an application server according to the mobile barcode information to read the corresponding An item information; and generating the auxiliary voice message according to the item information, so as to control the voice output module to send out the auxiliary voice message.

Based on the above, the speech assistance system and speech assistance method for the visually impaired according to the present invention have one or more of the following advantages:

(1) In the voice assistance system and voice assistance method for the visually impaired of the present invention, after the real-time environment image is obtained by the camera module, it can be sent to the recognition server through the portable electronic device for object recognition and distance estimation processing, thereby performing Including pedestrians, vehicles, road signs, potholes, stairs and other object recognition and distance estimation and safety reminder functions, thereby increasing the user experience when using it.

(2) The visually impaired speech assistance system and the visually impaired speech assistance method of the present invention use tensor flow single-shot multi-frame detector modules to perform object recognition processing by the recognition server, so that real-time recognition can be performed quickly and effectively objects and people in the environment.

(3) The visually impaired speech assistance system and the visually impaired speech assistance method of the present invention use the convolutional neural network of deep learning for face recognition processing by the recognition server, so that it can quickly and effectively recognize familiar faces in the environment in real time. characters.

(4) The voice assistance system and voice assistance method for the visually impaired of the present invention can issue safety reminders based on the user's inquiries, thereby increasing the user's active experience in use.

The technical features of the present invention will be described in detail below with specific embodiments and accompanying drawings, so that those skilled in the art can easily understand the purpose, technical features, and advantages of the present invention.

1: Visually impaired voice assistance system

10:Head mounted device

11: Camera module

111: Real-time environment image

12: Sound module

121: Auxiliary voice message

13: Radio module

131: inquiry message

20: Portable Electronic Devices

21: Processing Module

22: Wireless transmission module

30: Identify the server

31: Database

311:Reference object feature

312: Reference object information

32: Learning modules

321: Environmental object characteristics

322: Real-time auxiliary information

91: Mobile Barcode Information

92:Application server

921: Item information

S11~S15: Steps

S21~S22: Steps

S31~S33: steps

是本發明的一些實施例，對於所屬技術領域中具有通常知識者來講，還可以根據這些附圖獲得其他的附圖。 In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the accompanying drawings that need to be used in the description of the embodiments of the present invention. Obviously, the accompanying drawings described below are only

These are some embodiments of the present invention, and those skilled in the art can also obtain other drawings based on these drawings.

FIG. 1 is a schematic block diagram of the first embodiment of the voice assistance system for the visually impaired according to the present invention.

FIG. 2 is a schematic flow chart of the learning module of the first embodiment of the voice assistance system for the visually impaired according to the present invention.

FIG. 3 is a schematic block diagram of the second embodiment of the voice assistance system for the visually impaired according to the present invention.

FIG. 4 is a schematic block diagram of the third embodiment of the voice assistance system for the visually impaired according to the present invention.

Fig. 5 is a schematic diagram of the first step of the voice assistance method for the visually impaired according to the present invention.

Fig. 6 is a schematic diagram of the second step of the voice assistance method for the visually impaired according to the present invention.

Fig. 7 is a schematic diagram of the third step of the voice assistance method for the visually impaired according to the present invention.

The advantages, features and technical methods achieved by the present invention will be described in more detail with reference to exemplary embodiments and accompanying drawings to make it easier to understand, and the present invention can be implemented in different forms, so it should not be understood as being limited to what is shown here The stated embodiments, on the contrary, for those skilled in the art, the provided embodiments will make the present disclosure more thorough and comprehensive and completely convey the scope of the present invention, and the present invention will be only the appended The scope of the patent application is defined.

It should be understood that although the terms "first", "second" and the like may be used in the present invention to describe various elements, components, regions, sections, layers and/or sections, these elements, components, regions, sections , layer and/or section should not be limited by these terms. These terms are only used to distinguish one element, component, region, section, layer and/or section from another element, component, region, section, layer and/or section.

Unless otherwise defined, all terms (including technical and scientific terms) used in this invention have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. It will be further understood that terms such as those defined in commonly used dictionaries should be interpreted as having definitions consistent with their meanings in the context of the relevant art and the present invention, and will not be construed as idealistic or overly formal unless otherwise expressly defined herein.

Please refer to Figures 1 and 2, which are respectively the block diagram of the first embodiment of the voice assistance system for the visually impaired and the flow diagram of the learning module of the present invention. The present invention is mainly applied to visual disabled users, but can also be applied to general users under appropriate circumstances, so it should not be limited to this only. As shown in FIG. 1 , a voice assistance system 1 for the visually impaired according to the present invention includes a head-mounted device 10 , a portable electronic device 20 and a recognition server 30 .

Wherein, the head-mounted device 10 can be smart glasses, a smart hat or a smart helmet, etc., which includes a camera module 11 and a sound output module 12 . Of course, the head-mounted device 10 may further include communication modules, battery modules, and other components that are well known to those skilled in the art, so details will not be repeated here. Wherein, the camera module 11 is preferably a depth camera module, which captures the environment in front of the user to generate a real-time environment image 111 . The voice output module 12 is used to issue auxiliary voice messages 121 such as the name or position of the object in front of the user or its distance relative to the user.

Wherein, the portable electronic device 20 includes a processing module 21 and a wireless transmission module 22 . Wherein, the processing module 21 may include an application program (APP) installed in the portable electronic device 20 , a processor and other elements for coordinated operation. The wireless transmission module 22 can include bluetooth, 2.4G, 5G or WIFI transmission methods, and in the present invention, the USB serial port transmission method is used to electrically connect the camera module 11 to transmit the real-time environment image 111 . Then, the wireless transmission module 22 uses 2.4G, 5G and other transmission methods of WIFI to quickly transmit the real-time environment image 111 to the identification server 30 by executing the client-server multi-execution program according to the transmission control protocol.

Wherein, the recognition server 30 includes a database 31 and a learning module 32 . The database 31 stores a plurality of reference object features 311 and corresponding reference object information 312 . For example, the reference object feature 311 can be the appearance feature of trained objects such as pedestrians, vehicles, potholes, stairs, etc., and the reference object information 312 is the name or title and further information of the aforementioned object.

As shown in FIG. 2, wherein, the identification server 30 receives the real-time environment image 111 from the portable electronic device 20, and the learning module 32 recognizes a plurality of environmental object features 321 from the real-time environment image 111, And the learning module 32 compares whether the environmental object feature 321 matches the reference object feature 311 . Incidentally, the learning module 32 is mainly realized by using the tensor flow algorithm for object recognition and the deep learning convolutional neural network algorithm for face recognition. That is to say, the learning module 32 uses the tensor flow algorithm to identify a plurality of environmental object features 321 from the real-time environmental image 111, and also uses the convolutional neural network of deep learning to compare the human faces in each environmental object feature 321 , whether it matches the face feature 311 of each reference object, and estimate the distance between the recognized environmental object (including the face) and the user.

Among them, the object recognition algorithm can infer the name of the object from the dynamic real-time environment image based on the single-shot multi-frame detector module of the Microsoft common object detection data set, and the frozen inference graph exported by tensorflow; among them, the human face The recognition algorithm is based on Caffe face tracking to detect "where is there a face" (face number, position, size, face features, etc.) from the dynamic real-time image, and based on the convolutional neural network of deep learning The data of positional features and geometric relationships, among which the feature items include the main features of organs such as eyebrows, eyes, nose, mouth, face shape, etc., do feature comparisons from the database to see if there is the same face, arrangement similarity, etc. to further judge the The detected face is "who" to judge whether the image is a face, and if it meets the conditions, the relevant face data will be returned.

When the environmental object characteristics 321 match the reference object characteristics 311 , the learning module 32 generates real-time auxiliary information 322 including corresponding reference object information 312 and feeds back to the portable electronic device 20 . Wherein, the processing module 21 of the portable electronic device 20 can generate the auxiliary voice message 121 according to the real-time auxiliary information 322, and control the sound output module 12 Send out the auxiliary voice message 121 . It is worth mentioning that the auxiliary voice message 121 includes each of the reference object information 312 and the position of the object corresponding to each of the reference object information 312 in the real-time environment image 111 .

For example, after the camera module 11 finishes shooting the front picture, after the above procedure, the processing module 21 will control the sound module 12 to prompt the user what kind of objects (including familiar or unknown characters) and how far away they are from the user, what kind of objects are in the middle environment (including familiar or unknown characters) and how far are they relative to the user, what kind of objects are in the right environment (including familiar or unknown characters) and how far away they are from the user. After learning the information, the user can greet the acquaintances and decide how to walk, such as leaning to the left, walking in the middle, or leaning, instead of just listening to the voice of the assisting device or passively following the assisting device. The device walks. Incidentally, since the user may be constantly walking forward, in the subsequent process, the controllable mode of the processing module 21 is only for new objects or objects that are too close to the user. The sound module 12 prompts the user by voice.

Please refer to FIG. 3 , which is a schematic block diagram of the second embodiment of the voice assistance system for the visually impaired of the present invention. The present invention is mainly applied to visually impaired users, but it can also be applied to ordinary users under appropriate circumstances, so it should not be limited thereto. As shown in the figure, a voice assistance system 1 for the visually impaired according to the present invention includes a head-mounted device 10 , a portable electronic device 20 and a recognition server 30 .

The main difference between this embodiment and the previous embodiments is that the head-mounted device 10 further includes a radio module 13 for receiving an inquiry message 131 from the user. the place The management module 21 generates the auxiliary voice message 121 according to the inquiry message 131 and the real-time auxiliary information 322, and controls the voice output module 12 to send out the auxiliary voice message 121.

Wherein, after the camera module 11 shoots the front picture, the processing module 21 will not actively control the sound module 12 to prompt the user by voice after the above-mentioned procedure. After the processing module 21 receives the query message 131 from the user, the processing module 21 generates the auxiliary voice message 121 according to the query message 131 and the real-time auxiliary information 322, and controls the sound output module 12 to send out the auxiliary voice Message 121.

For example, after the camera module 11 shoots the front frame, the processing module 21 will remain in the standby state after the above procedure. And the user can inquire about the inquiry message 131 of what kind of object is there on the left side of the body through the sound receiving module 13 . At this time, the processing module 21 can control the sound module 12 to remind the user that there are potholes and other objects on the left side in front of them and how far they are relative to each other. In addition, in practical application, at the beginning of walking, the processing module 21 will first prompt all the objects in front, and during the walking process, the processing module 21 will prompt the user according to the query message 131 .

It is worth mentioning that the sound emitting module 12 and the sound receiving module 13 can be of bone conduction type, and the sound is directly transmitted to the receiving end of the brain through bone resonance, so as to avoid the influence of external noise.

Please refer to FIG. 4 , which is a schematic block diagram of a third embodiment of the voice assistance system for the visually impaired of the present invention. The present invention is mainly applied to visually impaired users, but it can also be applied to ordinary users under appropriate circumstances, so it should not be limited thereto. As shown in the figure, a voice assistance system 1 for the visually impaired according to the present invention includes a head-mounted device 10 , a portable electronic device 20 and a recognition server 30 .

The main difference between this embodiment and the preceding embodiments is that when the real-time environment image 111 includes a mobile barcode information 91, the processing module 21 is connected to an application server 92 according to the mobile barcode information 91 to read Corresponding to an item information 921, and generate auxiliary voice message 121 according to the item information 921, so as to control the sound module 12 to send out the auxiliary voice message 121.

That is to say, after the camera module 11 shoots the front picture, and the real-time environment image 111 is transmitted to the portable electronic device 20 in real time, the processing module 21 can first judge whether the real-time environment image 111 contains the action barcode information 91, and if so, based on The mobile barcode information 91 is connected to the corresponding application server 92 to read the corresponding item information 921 . Then, the processing module 21 can generate the auxiliary voice message 121 according to the item information 921 to control the sound module 12 to send out the auxiliary voice message 121 . For example, what is the object corresponding to the mobile barcode information 91 and so on.

In addition, in different embodiments, in order to avoid continuously sending out the information of the object corresponding to the mobile barcode information 91, the user can set the shopping mode through the radio module 13, etc., and in this mode, the processing module 21 Only then will it be judged whether the real-time environment image 111 contains the mobile barcode information 91.

Please refer to Figures 5, 6, and 7, which are schematic views of the first step, second step, and third step of the voice assistance method for the visually impaired according to the present invention.

As shown in FIG. 5, a voice assistance method for the visually impaired of the present invention includes the following steps: (S11) using a camera module 11 of a head-mounted device 10 to capture the environment in front of the user to generate a real-time Environmental image 111; (S12) receiving the real-time environmental image 111 through a portable electronic device 20 and sending it to a recognition server 30; (S13) identifying a plurality of environmental object features 321 from the real-time environmental image 111, and comparing each Whether this environmental object feature 321 accords with a plurality of reference object features 311 in a database 31; (S14) generate and include corresponding to this parameter and (S15) generate an auxiliary voice message 121 according to the real-time auxiliary information 322, and control the head-mounted device A voice module 12 of 10 sends out the auxiliary voice message 121 .

As shown in FIG. 6, a speech assistance method for the visually impaired of the present invention further includes the following steps: (S21) receiving an inquiry message 131 from the user; and (S22) according to the inquiry message 131 and the real-time auxiliary information 322 generates the auxiliary voice message 121, and controls the sound output module 12 to send out the auxiliary voice message 121.

As shown in Figure 7, a voice assistance method for the visually impaired of the present invention further includes the following steps: (S31) judging whether the real-time environment image 111 contains an action barcode information 91; (S32) if so, according to the action barcode The information 91 is connected to an application server 92 to read a corresponding item information 921; (S33) generating the auxiliary voice message 121 according to the item information 921, so as to control the voice output module 12 to send out the auxiliary voice message 121 .

The detailed description and implementation of the voice assistance method for the visually impaired of the present invention have been described above in the speech assistance system for the visually impaired of the present invention, and will not be repeated here for brevity.

The above descriptions are illustrative only, not restrictive. Any equivalent modification or change made without departing from the spirit and scope of the present invention shall be included in the scope of the appended patent application.

1: Visually impaired voice assistance system

10:Head mounted device

11: Camera module

111: Real-time environment image

12: Sound module

121: Auxiliary voice message

20: Portable Electronic Devices

21: Processing Module

22: Wireless transmission module

30: Identify the server

31: Database

311:Reference object feature

312: Reference object information

32: Learning modules

321: Environmental object characteristics

322: Real-time auxiliary information

Claims (6)

A voice assistance system for the visually impaired, which includes: a head-mounted device including a camera module and a sound output module, the camera module captures the environment in front of the user to generate a real-time environment image, the output The audio module sends an auxiliary voice message; a portable electronic device includes a processing module and a wireless transmission module, and the wireless transmission module transmits the real-time environment image; a recognition server includes a database and a learning module, the database stores a plurality of reference object features and corresponding reference object information, the recognition server receives the real-time environment image from the portable electronic device, and the learning module is generated by the real-time environment When the image recognizes a plurality of environmental object features, and the learning module compares the environmental object features with the reference object features, the learning module generates a real-time auxiliary information containing the corresponding reference object information and feeds back to the portable electronic device, and synchronously stored in a cloud server; wherein, the processing module generates the auxiliary voice message according to the real-time auxiliary information, and controls the sound output module to send out the auxiliary voice message, the voice auxiliary message Contains a plurality of orientations, and the reference object information corresponding to the plurality of orientations, and the voice assistance message does not directly indicate the way the user is walking; wherein, when the user maintains the direction of travel, the processing module is only for the reference The new object in the object is used to control the sound module to prompt the user with the auxiliary voice message; wherein, the learning module is based on the tensor flow algorithm of a single multi-frame detector module and the volume of deep learning The product neural network algorithm compares whether the characteristics of each environmental object conform to the characteristics of each reference object. When the face in each environmental object characteristic matches the face characteristics of each reference object, the processing module controls the sound model Group voice prompts have met the reference object The orientation of the facial feature of the piece, the distance from the user and the corresponding facial data. The voice assistance system for the visually impaired as described in claim 1, wherein the head-mounted device further includes a radio module, which receives an inquiry message from the user, and the processing module is based on the inquiry message and the real-time auxiliary information Generate the auxiliary voice message, and control the sound output module to send out the auxiliary voice message. The voice assistance system for the visually impaired as described in claim 1, wherein when the real-time environment image contains a mobile barcode (QR code) information, the processing module is connected to an application server according to the mobile barcode information to read Fetch a corresponding item information, and generate the auxiliary voice message according to the item information, so as to control the sound output module to send out the auxiliary voice message. A voice assistance method for the visually impaired, which includes the following steps: using a camera module of a head-mounted device to capture the environment in front of the user to generate a real-time environment image; receiving and transmitting the real-time environment image through a portable electronic device to a recognition server; identify a plurality of environmental object characteristics from the real-time environmental image, and compare whether each of the environmental object characteristics conforms to a plurality of reference object characteristics in a database; generate an image corresponding to the reference object characteristics Refer to a real-time auxiliary information of the object information, feed it back to the portable electronic device, and store it in a cloud server synchronously; and generate an auxiliary voice message according to the real-time auxiliary information, and control a sound mode of the head-mounted device The group sends out the auxiliary voice message, the voice auxiliary message includes a plurality of directions, and the information of the reference object corresponding to the plurality of directions, and the voice auxiliary message does not directly instruct the user to walk; wherein, when the user keeps walking direction, the processing module controls the sound output module to prompt the user with the auxiliary voice message only for newly-appeared objects; Among them, the learning module compares whether the characteristics of each environmental object conform to the characteristics of each reference object based on the single-shot multi-frame detector module of the tensor flow algorithm and the convolutional neural network algorithm of deep learning. When the face in the environmental object features matches the face features of each of the reference objects, the processing module controls the sound output module to prompt the orientation with the face features in line with the reference object, and the user's distance. The distance between them and the corresponding face data. The voice assistance method for the visually impaired as described in Claim 4, further comprising the following steps: receiving an inquiry message from the user; and generating the auxiliary voice message according to the inquiry message and the real-time auxiliary information, and controlling the sound output mode The system sends out the auxiliary voice message. The voice assistance method for the visually impaired as described in claim 4, further comprising the following steps: judging whether the real-time environment image contains a mobile barcode (QR code) information; if so, connecting to an application server according to the mobile barcode information, to read corresponding item information; and generate the auxiliary voice message according to the item information, so as to control the sound output module to send out the auxiliary voice message.