TWM648987U - Image-to-speech assistive device for visually impaired - Google Patents

Abstract

This new model relates to an image-to-speech assistive device for the visually impaired. The main structure includes a head-mounted device with a camera element, on which a first wireless communication element, an image recognition module, an article generation module, and an The voice conversion module and a playback device also have a guide stick. The guide stick has a first control part for driving the camera component and a second wireless communication component that is wirelessly connected to the first wireless communication component. With this, visually impaired persons only need to use the first control part on the guide cane to drive the camera element to capture the scene in front of them, and then use object recognition technology and chatGPT's artificial intelligence natural language processing technology to convert the image information into text. And through the voice output of the playback device to the visually impaired, it can achieve the functions of easy operation, quick recognition, and automatic spoken explanation of the situation in front of them.

Description

Image-to-speech assistive device for the visually impaired

This new model provides an image-to-speech auxiliary device for the visually impaired that has simple and concealed operation, can quickly identify objects, and automatically explain the situation in front of you in spoken language.

According to the definition, visually impaired people refer to people who have partial or complete impairment in the structure or function of their visual organs. They are unable or have difficulty identifying external objects and often require the assistance of visual aids to complete related behaviors in daily life.

With the advancement of technology in recent years, many functions have been implemented through digital technology and applied to assistive devices to provide users with more assistance. They can also provide a lot of assistance to users other than those with disabilities. At present, there are few applications for digitizing assistive devices, and only a few combine image recognition technology. However, for complex artificial intelligence calculations, relatively large processors and huge databases are needed. These large systems are not suitable for carrying around, and due to manual When processing huge amounts of image information, the response time of smart devices cannot keep up with the needs of users, making it impossible for visually impaired people to have a friendly outdoor assistive device, or making it less convenient to use.

Taking the Republic of China Patent No. M584676 "Digital Assistive Device" as an example, there are still the following problems and deficiencies that need to be improved during use:

First, its camera lens is combined with a wearable accessory to provide front image recognition and voice feedback at any time. However, the user does not always need voice assistance, and cannot control the movement timing of the camera lens, causing the user's ears to receive information that is complicated or even obscured. The other sounds around him were even more dangerous.

Second, the camera lens is a high-definition lens and is constantly taking dynamic shots, which puts a huge burden on the computing device and affects the speed of voice feedback.

Third, its semantic output is based on Tesseract and TTS (Text to Speech) technology, which analyzes the semantics generated by text arrangement and then converts it into speech output. These two technologies do not use artificial intelligence natural language processing technology and can only predict The recorded words are pieced together, and the resulting semantic meaning is rather blunt.

Therefore, how to solve the above-mentioned conventional problems and deficiencies is the direction that the applicant of the present invention and relevant manufacturers engaged in this industry are eager to study and improve.

Therefore, in view of the above shortcomings, the applicant of this new model collected relevant information, evaluated and considered it from many parties, and used his many years of experience in this industry, and through continuous trials and modifications, he designed such a simple and concealed operation. The patentee of a new type of visually impaired assistive device that can quickly identify objects and automatically explain the situation in front of you in spoken language.

The main purpose of this new model is: the article generation module is a language model chatGPT composed of a deep neural network model. It is an upgraded version of the chat robot driven by AI artificial intelligence technology and natural language processing technology. It can learn and improve independently, simulating human processes. A more natural and smooth dialogue, so combined with the image recognition module, it can generate content that describes the image information in smooth sentences, which is easier for users to understand.

Another main purpose of the present invention is to use the first control part on the guide stick to drive the camera element on the head-mounted device, so that identification requests can be issued only when needed, making the operation simpler and more concealed.

In order to achieve the above purpose, the main structure of the present invention includes: a head-mounted device with a camera element, a first wireless communication element, an image recognition module, an article generation module, a voice conversion module, and a playback module. The device also has a guide stick, which has a first control part and a second wireless communication element, wherein the camera element is installed on the head-mounted device, and the photographing direction is consistent with the line of sight of the head-mounted device. The directions are the same to obtain an image information. The first wireless communication element is provided on the head-mounted device and is electrically connected to the camera element. The first control part is used to drive the camera element. The second wireless communication element is electrically connected to the second wireless communication element. A control unit is wirelessly connected to the first wireless communication element. The image recognition module is located in the head-mounted device and is electrically connected to the camera element to identify a plurality of item information from the image information. The article generation module is located in The head-mounted device is electrically connected to the image recognition module and has an image description database to generate text information with clear meaning based on some item information, and the article generation module is a language composed of a deep neural network model. Model chatGPT, the speech conversion module is located in the head-mounted device and is electrically connected to the article generation module to convert the text information into a voice message, and the playback device is located on the head-mounted device and is electrically connected to the voice message Conversion module.

So that when the user uses this new model as a visually impaired assistive device, he only needs to wear a head-mounted device , holding the guide cane, and wirelessly connecting the head-mounted device and the guide cane through the first wireless communication element and the second wireless communication element, so that identification and conversion can be performed at any time. When the user moves to any certain point and wants to confirm the people and things in front of him, he only needs to use the first control part on the guide cane to drive the camera element to capture a static image information. At this time, the image recognition module will automatically The item information in the image information is identified, and the article generation module combines the data in the image description database to integrate the image information and all item information to generate text information with smooth meaning. Finally, the speech conversion module is used to convert the text information into The voice message is played to the user by the playback device. In this way, a simple and concealed operation method can be used to assist users with insufficient visual recognition of the surrounding environment.

Through the above technology, breakthroughs can be made to overcome the problems existing in conventional digital assistive devices such as the inability to control the recognition timing, complicated voice messages affecting normal hearing, dynamic image recognition causing high burden and low efficiency, and semantic output without involving artificial intelligence, which is more blunt. Practical progress to achieve the above advantages.

1:Head mounted device

11:Camera components

111:The first wireless communication component

12:Image recognition module

13: Article generation module

131: Image description database

132: Map real scene database

133:Known character database

14: Voice conversion module

15:Playback device

16: Radio components

17: Positioning components

18: Facial recognition module

2:Guide cane

21:First Control Department

22: Second wireless communication component

23:Second Control Department

The first figure is a three-dimensional perspective view of the preferred embodiment of the present invention.

The second figure is a structural block diagram of a preferred embodiment of the present invention.

The third figure is a schematic diagram of a preferred embodiment of the present invention.

The fourth figure is a schematic diagram of image recognition according to a preferred embodiment of the present invention.

The fifth figure is a schematic diagram of voice playback according to the preferred embodiment of the present invention.

Figure 6 is a schematic diagram of another preferred embodiment of the present invention.

Figure 7 is a structural block diagram of another preferred embodiment of the present invention.

Figure 8 is a schematic diagram of fine positioning of another preferred embodiment of the present invention.

Figure 9 is a structural block diagram of another preferred embodiment of the present invention.

Figure 10 is a schematic diagram of person recognition according to another preferred embodiment of the present invention.

In order to achieve the above-mentioned purposes and effects, the technical means and structures adopted by the present invention are described in detail below with respect to the preferred embodiments of the present invention in order to facilitate a complete understanding.

Please refer to the first and second figures, which are three-dimensional perspective views and structural block diagrams of preferred embodiments of the present invention. From the figures, it can be clearly seen that the present invention includes:

A head-mounted device 1, wherein the head-mounted device 1 is one of glasses, sunglasses or goggles;

A camera element 11 is installed on the head-mounted device 1, and the photographing direction is the same as the line of sight direction of the head-mounted device 1, in order to obtain an image information. In this embodiment, the camera element 11 is a lens. Example;

A first wireless communication component 111 is provided on the head-mounted device 1 and is electrically connected to the camera component 11;

A guide stick 2 has a first control part 21 for driving the camera element 11, and a second wireless communication element 22 that is electrically connected to the first control part 21 and wirelessly connected to the first wireless communication element 111. , in this embodiment, the first control part 21 takes a push switch as an example, and the first and second wireless communication components 111 and 22 take a Bluetooth connection as an example;

An image recognition module 12 is provided in the head-mounted device 1 and is electrically connected to the camera element 11 to identify a plurality of item information from the image information. The image recognition module 12 is based on object detection. Technology (Object Detection) is executed. This embodiment uses one of the softwares such as YOLO or AI DIY Playform as an example;

An article generation module 13 is provided in the head-mounted device 1 and is electrically connected to the image recognition module 12, and has an image description database 131 to generate an article based on the image information and the item information. Smooth text information, and the article generation module 13 is a language model chatGPT composed of a deep neural network model;

A voice conversion module 14 is provided in the head-mounted device 1 and is electrically connected to the article generation module 13 to convert the text information into a voice message. This embodiment uses text-to-speech technology (Text To Speech). Speech, TTS) for conversion; and

A playback device 15 is provided on the head-mounted device 1 and is electrically connected to the voice conversion module 14. In this embodiment, the playback device 15 is a speaker as an example.

Through the above description, we can understand the structure of this technology, and based on the corresponding cooperation of this structure, we can achieve the advantages of simple and concealed operation, and the ability to quickly identify objects and automatically explain the situation at hand. The detailed explanation will be in Instructions below.

Please also refer to Figures 1 to 5, which are three-dimensional perspective views to voice playback schematic diagrams of preferred embodiments of the present invention. When assembled with the above components, it can be clearly seen from the figures that the features of the present invention The only physical equipment is the head-mounted device 1 and the guide cane 2, which are the same as those commonly used by the visually impaired. Therefore, the visually impaired can still perform the operation by wearing the head-mounted device 1 and holding the guide stick 2, and there is no difference in appearance. Then, the head-mounted device is carried out through the first wireless communication element 111 and the second wireless communication element 22. Device 1 and guide cane 2 are connected via Bluetooth to enable identification and conversion at any time.

When the user moves to any certain point and wants to confirm the people and things in front of him (for example, a picture in front of him), he only needs to press the first control part 21 on the guide stick 2 to drive the imaging element 11 to capture the image. A static image information. For the visually impaired, this action is just a movement of the fingers holding the guide stick 2. The action is simple and hidden. Others cannot see this operation action. At this time, the image recognition module 12 will automatically recognize it. To extract item information from image information, the recognition principle includes several parts: image scan, object feature recognition, and output interface. Specifically, the content to be input is first scanned into one or more images through the camera element 11, and then the image is sent to the image recognition module 12 to detect the physical characteristics of the object appearance in the image, and the recognition result is By matching with the object feature database in the cloud, the corresponding object name can be selected from the object feature database, and the object name is passed to the article generation module 13 as item information, for example: a beach, a pavilion, a mountain, Three birds, two boats.

Then the article generation module 13 combines the data in the image description database 131 to integrate all the item information to generate smooth text information. The article generation module 13 is a language model chatGPT composed of a deep neural network model, and The GPT is a transformer-based generative pre-trained model (GPT). It is a large-scale language model and an important framework for generative artificial intelligence. It can process text and image input and then use Humans use natural dialogue to interact with text, and can also be used for very complex language work, including automatic text generation, automatic question and answer, automatic summarization and other tasks. Therefore, as long as some text or images are input, the article generation module 13 can automatically Piece together a piece of text information with smooth semantics. In the GPT-4 model version launched in 2023, it can be further adapted to specific tasks and/or subject areas to form a more targeted system. For example, this new image description database 131 is incorporated into the training model to make the article generation module 13 better at generating a piece of text information for describing the content of the image information. Taking this embodiment as an example, the image information is a painting, and the image recognition model is integrated Group 12 provides item information and generates a paragraph such as "The picture is a landscape painting. The scene in the painting is a beach. There is a mountain at the end of the beach and the sea. There is a pavilion on the beach. There are two sailboats on the sea. There are three birds in the sky." ” text information.

Finally, the speech conversion module 14 is used to convert the text information into a speech message, which is played to the user by the playback device 15. The conversion principle of the speech conversion module 14 includes text processing and There are two steps in speech synthesis. First, the processing of this article involves language analysis, word boundary determination, and sentence segmentation operations. This step can improve the quality and fluency of speech synthesis. Second, speech synthesis is based on the analyzed content of this article. Use specific algorithms and rules to generate voice signals, or use many recorded voices in the database to connect them. Since the input content is a single photo and the input conditions are relatively simple, the processing speed of the image recognition module 12 and the article generation module 13 is faster and the burden is lower, and the recognition conversion action can be completed more quickly. For the user, It can be operated in a simple and concealed manner to assist users with insufficient visual recognition of the surrounding environment.

Please also refer to the sixth figure, which is a schematic diagram of another preferred embodiment of the present invention. It can be clearly seen from the figure that this embodiment is similar to the above-mentioned embodiment, except that the head-mounted device 1 There is a radio component 16 electrically connected to the article generation module 13 for receiving the user's voice command for the article generation module 13 to adjust the content of the text information, and the guide stick 2 is provided with a The second control part 23 is used to drive the sound collecting element 16 , wherein the sound collecting element 16 is a miniature microphone, and the second control part 23 is a push switch provided on one side of the first control part 21 . When the user thinks that the text information content generated by the article generation module 13 is not clear enough, the second control part 23 can be used to activate the radio component 16 to allow the user to input voice commands orally to input updates to the article generation module 13 Specific conditions, such as "What is the positional relationship of those things?" At this time, the article generation module 13 can regenerate a piece of text information based on the positional relationship of the item information in the image information, "The right side of the scene is the beach, and the bottom is the beach." On the left is the sea water, and the mountains extend from the right to the center of the end of the sea water. The pavilion is on the beach at the foot of the mountain. Two sailboats are about close to the stalls. Three birds are at a low altitude between the sailboats and the mountains." In this way, the user can interact with the article generation module 13 through the radio component 16 to obtain a satisfactory reply.

Please refer to the seventh and eighth figures at the same time, which are structural block diagrams and fine positioning diagrams of another preferred embodiment of the present invention. It can be clearly seen from the figures that this embodiment is similar to the above-mentioned embodiment. , there is only a positioning element 17 electrically connected to the article generation module 13 in the head-mounted device 1 for generating location information, and the article generation module 13 has a map real-time database 132 to generate location information. The article generation module 13 integrates the image information, the item information, and the location information to generate fine positioning information, in which the positioning component 17 is a GPS positioning system, and the map real-view database 132 is a database that stores map street view information. . Since the positioning element 17 is based on satellite positioning, even if it can know the approximate location of the user, it can generally only grasp the range of about 100 meters of a certain road section. Especially when the user is at a fixed position in a non-moving state, it cannot Use judgment Therefore, when the second control part 23 is pressed to activate the radio element 16 to receive the position determination instruction, the article generation module 13 will integrate the image information, item information, and location information to search the map real scene database. 132 to determine the precise location of the street view in front of the user and output text information about the precise location.

For example, the positioning element 17 can locate the user within 100 meters of No. 1 XX Road. The image recognition module 12 determines that the user is the XX coffee shop located in the triangular window, and then integrates the map real-view database 132 You can know the specific address of the XX coffee shop located in the triangular window within 100 meters of No. 1 XX Road. Obtaining precise positioning information and the current direction not only helps the user to know the location. Cognition can also provide correct information about the location of relatives and friends when they are lost.

Please refer to Figures 9 and 10 at the same time, which are structural block diagrams and character recognition diagrams of another preferred embodiment of the present invention. It can be clearly seen from the figures that this embodiment is similar to the above-mentioned embodiment. , there is only a facial recognition module 18 in the head-mounted device 1, and a known person database 133 electrically connected to the facial recognition module 18 and the article generation module 13 to provide the article. The generation module 13 adjusts the content of the text information according to the characters in the image information. In addition, the playback device 15 in this embodiment is in the form of headphones as an example, and the facial recognition module 18 is based on face recognition software. For example, take one of FaceMe or FaceMaze. The facial recognition is a type of biometric recognition technology. Its operating principle is to capture facial feature values in a vector manner and compare it with the feature values of previously registered faces. Yes, and then through deep neural networks, algorithms and mathematical formulas are used to measure various variables of the face, turn them into feature values, and then compare them with the database to find out the correct identity of the face. Specifically, the face recognition action includes three steps: face detection, facial feature value extraction and face recognition. First, face detection. Through face detection technology, even if only part of the face is Even if the face appears in the screen, it can still accurately scan, detect and frame the position of the face in the image or video; secondly, extract the facial feature values and use the facial recognition engine to frame the face Divide into n dimensions. For example, when the n value of a high-precision facial recognition engine is 1024, the face can be divided into a matrix of 1024 dimensions, such as: the length and width of the nose, the width of the forehead, the shape of the eyes, etc. The first variable extracts the facial feature values based on the vector; third, facial recognition, compares the facial feature values with the pre-registered faces in the database to identify the correct identity, with a ratio of 1:N For example, the comparison is to compare the facial feature values that appear on the screen with N pre-registered faces in the database to identify the identity.

When the user uses the first control unit 21 to take pictures and the image recognition module 12 recognizes When there is a person in the image information, it will be automatically connected to the facial recognition module 18 to compare the face information of the person with the face information in the known person database 133. When a matching face appears, When providing information, the more general character description (such as "a woman") in the item information is changed to the name or title of the character, so that the article generation module 13 can adjust its text information to the playback device 15, for example, adjust it to " "Sister is coming." If there is no matching face information in the known character database 133, the character is determined to be a stranger, and the output item information will maintain the original unified character narrative. This not only helps users identify the person in front of them, but also avoids redundant text information.

However, the above descriptions are only preferred embodiments of the present invention, and do not limit the patent scope of the present invention. Therefore, any simple modifications and equivalent structural changes made using the contents of the description and drawings of the present invention should be treated in the same way. It is included in the patent scope of this new model and is hereby stated.

Therefore, the key points of this new type of image-to-speech assistive device for the visually impaired that can improve usage are:

First, the article generation module 13 is a language model chatGPT composed of a deep neural network model. It is an upgraded version of the chat robot driven by AI artificial intelligence technology and natural language processing technology. It can learn and improve independently, simulating human beings to speak more naturally and smoothly. dialogue, therefore, combined with the image recognition module 12, content describing the image information in clear sentences can be generated, which is conducive to user understanding.

Second, by using the first control part 21 on the guide stick 2 to drive the camera element 11 on the head-mounted device 1, the recognition request can be issued only when needed, making the operation simpler and more concealed.

Third, the input content is a single static photo taken semi-automatically by the user, and the input conditions are relatively simple, so that the image recognition module 12 and the article generation module 13 have faster processing speed and lower burden, and can complete the recognition more quickly. Convert actions.

Fourth, using the design of the radio element 16, when the user thinks that the text information content generated by the article generation module 13 is not clear enough, the user can enter voice commands orally to enter more specific conditions, and through the radio element 16 16Interact with the article generation module 13 and obtain satisfactory responses.

Fifth, using the design of the positioning element 17 and the map real-time database 132, a fine positioning information and the current direction can be obtained, which not only helps the user to understand the location, but also provides the location of relatives and friends when they are lost. correct information.

Sixth, the design of the facial recognition module 18 and the known person database 133 can not only help the user identify the person in front of him, but also avoid generating redundant text data for strangers. News.

To sum up, the image-to-speech auxiliary device for the visually impaired of this new model can indeed achieve its effect and purpose when used. Therefore, this new model is truly a new model with excellent practicality and meets the application requirements for a new model patent. The application is submitted in accordance with the law, and we hope that the review committee will approve this model as soon as possible to protect the applicant's hard work. If the review committee of the Jun Bureau has any doubts, please feel free to write a letter for instructions. The applicant will do its best to cooperate and it will be greatly appreciated.

1:Head mounted device

11:Camera components

111:The first wireless communication component

12:Image recognition module

13: Article generation module

131: Image description database

14: Voice conversion module

15:Playback device

2:Guide cane

21:First Control Department

22: Second wireless communication component

Claims (7)

An image-to-speech assistive device for the visually impaired, which mainly includes: head-worn device; A camera element is installed on the head-mounted device, and the photographing direction is the same as the line of sight of the head-mounted device, in order to obtain an image information; A first wireless communication component is provided on the head-mounted device and is electrically connected to the camera component; A guide stick has a first control part for driving the camera element, and a second wireless communication element that is electrically connected to the first control part and wirelessly connected to the first wireless communication element; An image recognition module is installed in the head-mounted device and electrically connected to the camera element to identify a plurality of item information from the image information; An article generation module is installed in the head-mounted device and is electrically connected to the image recognition module, and has an image description database to generate a clear text information based on the image information and the item information. , and the article generation module is a language model chatGPT composed of a deep neural network model; A voice conversion module is installed in the head-mounted device and is electrically connected to the article generation module to convert the text information into a voice message; and A playback device is installed on the head-mounted device and electrically connected to the voice conversion module. For example, the image-to-speech assistive device for the visually impaired described in Item 1 of the patent application scope, wherein the head-mounted device is one of glasses, sunglasses or goggles. For example, in the image-to-speech assistive device for the visually impaired described in Item 1 of the patent application, the head-mounted device is provided with a radio component electrically connected to the article generation module for receiving the user's voice commands. Allows the article generation module to adjust the content of the text information. For example, in the image-to-speech assistive device for the visually impaired described in Item 3 of the patent application, the guide stick is provided with a second control part for driving the radio element. For example, the image-to-speech assistive device for the visually impaired described in item 3 of the patent application, wherein the head-mounted device has a positioning element electrically connected to the article generation module for generating a location information, and the article The generation module has a real map database for the article generation module to integrate the image information, the item information, and the location information to generate fine positioning information. For example, the image-to-speech assistive device for the visually impaired described in item 1 of the patent application, wherein the head-mounted device has a facial recognition module and an electrical connection between the facial recognition module and the article generation module. A database of known characters is provided for the article generation module to adjust the content of the text information according to the characters in the image information. For example, in the image-to-speech assistive device for the visually impaired described in Item 1 of the patent application, the playback device is either a speaker or an earphone.

Images