TWI839285B - Image-to-speech assistive device for the visually impaired - Google Patents

Abstract

The present invention is related to a visually impaired assistive device for converting images into speech. The main structure includes a head-mounted device with a camera element, on which a first wireless communication element, an image recognition module, an article generation module, a speech conversion module, and a playback device are provided. In addition, there is a guide stick, which has a first control unit for driving the camera element and a second wireless communication element wirelessly connected to the first wireless communication element. In this way, the visually impaired person only needs to use the first control unit on the guide stick to drive the camera element to shoot the scene in front of them, and then the image information is converted into text through object recognition technology and chatGPT's artificial intelligence natural language processing technology, and the voice is output to the visually impaired person through the playback device, so as to achieve the effect of convenient operation, rapid recognition, and automatic oral explanation of the situation in front of them.

Description

Image-to-speech assistive device for the visually impaired

The present invention provides a visually impaired assistive device that has simple and concealed operating actions, can quickly identify objects and automatically explain the current situation verbally.

According to the law, visually impaired people refer to those who have partial or complete impairments in the structure or function of their visual organs. They are unable to or have difficulty recognizing external objects and often need visual assistive devices to assist them in completing related behaviors in daily life.

With the advancement of technology in recent years, many functions can be applied to assistive devices through digital technology to provide users with more assistance, and can also provide a lot of assistance to users other than those with disabilities. Currently, there are few applications of digital assistive devices, and only a few are combined with image recognition technology. However, for complex artificial intelligence calculations, relatively large processors and large databases are required. These large systems are not suitable for carrying around, and because artificial intelligence cannot keep up with user needs when processing large amounts of image information, the response time is not fast enough for visually impaired people to have a friendly assistive device for going out, or it is less convenient to use.

Taking the Republic of China patent No. M584676 "Digital Assistive Device" as an example, the following problems and deficiencies still exist when using it and need to be improved:

First, the camera is integrated with wearable accessories to identify the image in front and provide voice feedback at any time, but the user does not always need voice assistance, and cannot control the timing of the camera's movement, causing the user's ears to receive complex information and even mask other sounds around them, which is more dangerous.

Second, its camera lens is a high-definition lens, and it continuously shoots dynamic images, which puts a heavy burden on the computing device and affects the speed of voice feedback.

Third, its semantic output is based on Tesseract and TTS (Text to Speech) technology, which analyzes the semantics generated by the text arrangement and then converts it into voice output. These two technologies do not use artificial intelligence natural language processing technology, but can only splice pre-recorded words, and the generated semantics are relatively stiff.

Therefore, how to solve the above-mentioned problems and deficiencies in usage is the direction that the applicant of this invention and related manufacturers engaged in this industry are eager to study and improve.

Therefore, in view of the above-mentioned deficiencies, the applicant of this invention has collected relevant information, evaluated and considered various aspects, and used the years of experience accumulated in this industry, after continuous trials and modifications, to design the invention patent of this image-to-speech assistive device for the visually impaired, which has simple and concealed operation, can quickly identify objects and automatically explain the current situation verbally.

The main purpose of this invention is that the article generation module is a language model chatGPT composed of a deep neural network model. It is an upgraded chat robot that uses AI artificial intelligence technology to drive natural language processing technology. It can self-learn and improve itself, and simulate humans to have a more natural and fluent conversation. Therefore, combined with the image recognition module, it can generate sentences that describe the content of the image information in a fluent manner, which is conducive to user understanding.

Another main purpose of the present invention is to use the first control unit on the guide stick to drive the camera element on the head-mounted device, so that the identification request can be issued only when necessary, which is simpler and more concealed in operation.

To achieve the above-mentioned purpose, the main structure of the present invention includes: a head-mounted device with a camera element, a first wireless communication element, an image recognition module, an article generation module, a voice conversion module, and a playback device, and a guide stick for the blind, the guide stick has a first control unit, and a second wireless communication element, wherein the camera element is arranged on the head-mounted device, and the shooting direction is the same as the sight direction of the head-mounted device to obtain image information, the first wireless communication element is arranged on the head-mounted device and is electrically connected to the camera element, the first control unit is used to drive the camera element, and the second wireless communication element is electrically connected to the first control unit and is wirelessly connected to the first control unit. The first wireless communication element is connected, the image recognition module is arranged in the head-mounted device and is electrically connected to the camera element to recognize multiple item information from the image information, the article generation module is arranged in the head-mounted device and is electrically connected to the image recognition module, and has an image description database to generate a fluent text information according to the item information, and the article generation module is a language model chatGPT composed of a deep neural network model, the voice conversion module is arranged in the head-mounted device and is electrically connected to the article generation module to convert the text information into a voice message, and the playback device is arranged on the head-mounted device and is electrically connected to the voice conversion module.

When the user uses the present invention as a visually impaired assistive device, he only needs to wear a head-mounted device, hold a guide stick, and wirelessly connect the head-mounted device and the guide stick through the first wireless communication element and the second wireless communication element, so that he can identify and convert at any time. When the user moves to any fixed point and wants to confirm the people, things and scenes in front of him, he only needs to use the first control unit on the guide stick to drive the camera element to shoot and obtain a static image information. At this time, the image recognition module will automatically identify the object information in the image information, and the text generation module will combine the data in the image description database to integrate the image information and all the object information to generate text information with fluent meaning. Finally, the voice conversion module will convert the text information into voice messages, which will be played to the user by the playback device. This allows for a simple and concealed operation method to assist users with their limited visual perception of the surrounding environment.

Through the above technology, we can overcome the problems of digital assistive devices such as the inability to control recognition timing, complex voice messages affecting normal hearing, dynamic image recognition leading to heavy burden and low efficiency, and semantic output without artificial intelligence and being relatively stiff, and achieve the practical progress of the above advantages.

1: Head-mounted device

11: Imaging components

111: First wireless communication element

12: Image recognition module

13: Article generation module

131: Image Description Database

132: Map real scene database

133: Database of known characters

14: Voice conversion module

15:Playback device

16: Radio components

17: Positioning element

18: Facial recognition module

2: Guide stick

21: First control unit

22: Second wireless communication component

23: Second control unit

The first figure is a three-dimensional perspective view of a preferred embodiment of the present invention.

The second figure is a structural block diagram of a preferred embodiment of the present invention.

The third figure is a photographic diagram of a preferred embodiment of the present invention.

The fourth figure is a schematic diagram of image recognition of a preferred embodiment of the present invention.

Figure 5 is a schematic diagram of voice playback of a preferred embodiment of the present invention.

Figure 6 is a schematic diagram of another preferred embodiment of the present invention.

Figure 7 is a structural block diagram of another preferred embodiment of the present invention.

Figure 8 is a schematic diagram of the fine positioning of another preferred embodiment of the present invention.

Figure 9 is a structural block diagram of another preferred embodiment of the present invention.

Figure 10 is a schematic diagram of character recognition of another preferred embodiment of the present invention.

In order to achieve the above-mentioned purpose and effect, the technical means and structure adopted by the present invention are described in detail in the following figure for the preferred embodiment of the present invention, and its features and functions are explained for complete understanding.

Please refer to the first and second figures, which are the three-dimensional perspective diagram and structural block diagram of the preferred embodiment of the present invention. It can be clearly seen from the figure that the present invention includes:

A head-mounted device 1, wherein the head-mounted device 1 is one of glasses, sunglasses or goggles;

An imaging element 11 is disposed on the head mounted device 1, and the photographing direction is the same as the sight direction of the head mounted device 1, so as to obtain image information. In this embodiment, the imaging element 11 is taken as an example of a lens;

A first wireless communication element 111 is disposed on the head mounted device 1 and electrically connected to the imaging element 11;

A guide stick 2 has a first control unit 21 for driving the imaging element 11, and a second wireless communication element 22 electrically connected to the first control unit 21 and wirelessly connected to the first wireless communication element 111. In this embodiment, the first control unit 21 is exemplified by a push switch, and the first and second wireless communication elements 111 and 22 are exemplified by Bluetooth connection;

An image recognition module 12 is disposed in the head mounted device 1 and electrically connected to the imaging element 11 to recognize multiple object information from the image information. The image recognition module 12 is implemented by object detection technology. This embodiment uses one of the software such as YOLO or AI DIY Playform as an example;

An article generation module 13 is disposed in the head mounted device 1 and electrically connected to the image recognition module 12, and has an image description database 131 to generate a coherent text message based on the image information and the object information, and the article generation module 13 is a language model chatGPT composed of a deep neural network model;

A speech conversion module 14 is disposed in the head mounted device 1 and electrically connected to the article generation module 13 to convert the text information into a voice message. In this embodiment, the conversion is performed using text to speech technology (Text To Speech, TTS); and

A playback device 15 is disposed on the head mounted device 1 and electrically connected to the voice conversion module 14. In this embodiment, the playback device 15 is exemplified by a speaker.

Through the above explanation, we can understand the structure of this technology. According to the corresponding coordination of this structure, we can achieve the advantages of simple and concealed operation, and can quickly identify objects to automatically explain the current situation. The detailed explanation will be given below.

Please refer to the first to fifth figures at the same time, which are stereoscopic perspective diagrams and voice playback schematic diagrams of the preferred embodiments of the present invention. Through the above-mentioned component assembly, it can be clearly seen from the figure that the physical equipment of the present invention only has a head-mounted device 1 and a guide stick 2, which is the same as the common equipment of the general visually impaired. Therefore, the visually impaired only need to wear a head-mounted device 1 and hold a guide stick 2 to implement it. There is no difference in appearance. Then, the Bluetooth pairing of the head-mounted device 1 and the guide stick 2 is performed through the first wireless communication element 111 and the second wireless communication element 22, and identification and conversion can be performed at any time.

When the user moves to any fixed point and wants to confirm the people, things and scenes in front of him (such as a painting in front of him), he only needs to press the first control part 21 on the guide stick 2 to drive the camera element 11 to shoot and obtain static image information. For the visually impaired, this action is just to move the fingers holding the guide stick 2. The action is simple and hidden, and other people cannot see this operation action. At this time, the image recognition module 12 will automatically recognize the object information in the image information. Its recognition principle includes several parts: image scanning, object feature recognition, and output interface. Specifically, the content to be input is first scanned into one or more images through the camera element 11, and then the image is sent to the image recognition module 12 to detect the physical features of the object in the image, and the recognition result is matched with the object feature database in the cloud. The corresponding object name can be selected from the object feature database, and the object name is transmitted to the article generation module 13 as the object information, for example: a beach, a pavilion, a mountain, three birds, two boats.

The article generation module 13 then combines the data in the image description database 131 to integrate all the object information to generate coherent text information. The article generation module 13 is a language model chatGPT composed of a deep neural network model, and the GPT is a generative pre-trained transformer-based model (GPT), which is a large language model and an important framework for generative artificial intelligence. It can process text and image input, and then use human natural dialogue to interact with text. It can also be used for very complex language work, including automatic text generation, automatic question answering, automatic summarization and other tasks. Therefore, as long as some text or images are input, the article generation module 13 can automatically piece together a piece of coherent text information. In the GPT-4 model version to be launched in 2023, it can be further adapted to specific tasks and / or subject area to form a more targeted system. For example, the image description database 131 of the present invention is incorporated into the training model, so that the article generation module 13 is better at generating a paragraph of text information for describing the image content according to the content of the image information. For example, in this embodiment, the image information is a painting, and the object information provided by the image recognition module 12 is integrated to generate a paragraph of text information such as "the picture is a landscape painting, the scene in the painting is a beach, there is a high mountain at the end of the beach and the sea, there is a pavilion on the beach, there are two sailboats on the sea, and there are three birds in the sky".

Finally, the speech conversion module 14 is used to convert the text information into a speech message, which is played to the user by the playback device 15. The conversion principle of the speech conversion module 14 includes two steps: text processing and speech synthesis. First, text processing is to perform language analysis, determine word boundaries, and perform sentence segmentation operations. This step can improve the quality and fluency of speech synthesis. Second, speech synthesis is to generate a speech signal based on the analyzed text content using specific algorithms and rules, or to connect many recorded voices in the database. Since the input content is a single photo, the input conditions are relatively simple, so the processing speed of the image recognition module 12 and the article generation module 13 is faster and the burden is lower, and the recognition conversion action can be completed more quickly. For the user, a simple and concealed operation method is provided to assist the user's lack of visual recognition of the surrounding environment.

Please refer to FIG. 6 at the same time, which is a schematic diagram of another preferred embodiment of the present invention. It can be clearly seen from the figure that the present embodiment is similar to the above-mentioned embodiment, except that the head-mounted device 1 is provided with a sound receiving element 16 electrically connected to the article generating module 13 for receiving the user's voice command so that the article generating module 13 can adjust the content of the text information, and the guide stick 2 is provided with a second control unit 23 for driving the sound receiving element 16, wherein the sound receiving element 16 is a miniature microphone and the second control unit 23 is a push switch arranged on one side of the first control unit 21. When the user thinks that the text information generated by the article generation module 13 is not clear enough, the second control unit 23 can be used to activate the sound receiving element 16, allowing the user to input voice commands orally to input more specific conditions to the article generation module 13, such as "What is the position relationship of those things?" At this time, the article generation module 13 can regenerate a piece of text information based on the position relationship of each item information in the image information, "The lower right side of the scene is the beach, the lower left side is the sea, the mountain extends from the right to the center of the end of the sea, the pavilion is on the beach at the foot of the mountain, the two sailboats are approximately near the stall, and the three birds are in the low-altitude position between the sailboats and the mountains." In this way, the user can interact with the article generation module 13 through the sound receiving element 16 and obtain a satisfactory response.

Please refer to FIG7 and FIG8 at the same time, which are a structural block diagram and a precise positioning diagram of another preferred embodiment of the present invention. It can be clearly seen from the figure that the present embodiment is similar to the above-mentioned embodiment, except that the head-mounted device 1 has a positioning element 17 electrically connected to the article generation module 13 for generating location information, and the article generation module 13 has a map real scene database 132 for the article generation module 13 to integrate the image information, the object information, and the location information to generate precise positioning information, wherein the positioning element 17 is a GPS positioning system, and the map real scene database 132 is a database storing map street view data. Since the positioning element 17 is based on satellite positioning, even if the user's approximate location can be known, it can generally only grasp the range of about 100 meters of a certain road section. In particular, when the user is in a fixed position without moving, the user's position cannot be determined. Therefore, when the user presses the second control unit 23 to activate the sound receiving element 16 so that it receives the position determination instruction, the article generation module 13 will integrate the image information, object information, and location information, search the street view data in the map real scene database 132, and determine the precise location of the street view in front of the user to output text information about the precise location.

For example, the positioning element 17 can locate the user within 100 meters of No. 1 XX Road, and the image recognition module 12 determines that the user is in front of the XX coffee shop located in the triangular window. Then, the map real scene database 132 is integrated to know the specific address of the XX coffee shop located in the triangular window within 100 meters of No. 1 XX Road, and obtain a precise positioning information and the current direction, which is not only conducive to the user's recognition of the location, but also can provide relatives and friends with correct information about their own location when they are lost.

Please refer to FIG. 9 and FIG. 10 at the same time, which are a structural block diagram and a person recognition schematic diagram of another preferred embodiment of the present invention. It can be clearly seen from the figure that the present embodiment is similar to the above-mentioned embodiment, except that the head mounted device 1 has a face recognition module 18 and a known person database 133 electrically connected to the face recognition module 18 and the article generation module 13, so that the article generation module 13 can adjust the content of the text information according to the person in the image information. In addition, the playback of the present embodiment is The device 15 is in the form of a headset, and the facial recognition module 18 is in the form of a facial recognition software, such as FaceMe or FaceMaze. Facial recognition is a type of biometric recognition technology. Its operating principle is to capture facial feature values in a vector form and compare them with the feature values of pre-registered faces. Then, through a deep neural network, algorithms and mathematical formulas are used to measure various variables of the face, convert them into feature values, and then compare them with the database to find the correct identity of the face. Specifically, the process of face recognition includes three steps: face detection, facial feature value extraction and face recognition. First, face detection. Through facial detection technology, even if only a part of the face appears in the picture, the location of the face can be accurately scanned, detected and framed in the image or video. Second, facial feature value extraction uses the face recognition engine to divide the framed face into n dimensions. For example, the n value of a high-precision face recognition engine is 1024. When the face is divided into a 1024-dimensional matrix, the variables such as the length and width of the nose, the width of the forehead, and the shape of the eyes are extracted into vector-based facial feature values; third, facial recognition, the facial feature values are compared with the pre-registered faces in the database to identify the correct identity. For example, 1:N comparison is to compare the facial feature values appearing on the screen with N pre-registered faces in the database to identify the identity.

When the user uses the first control unit 21 to take a photo, and the image recognition module 12 recognizes that there is a person in the image information, it will automatically connect to the facial recognition module 18 to compare the facial information of the person with the facial information in the known person database 133. When the facial information that matches the match appears, the more general character description in the item information (such as "a woman") is changed to the name or title of the person, so that the article generation module 13 can adjust its text information to the playback device 15, for example, to "sister is coming towards me". If there is no facial information that matches the match in the known person database 133, the person is determined to be a stranger, and the output item information will maintain the original general character description. In this way, it can not only help the user to identify the person in front of him, but also avoid redundant text information.

However, the above is only a preferred embodiment of the present invention, and does not limit the patent scope of the present invention. Therefore, all simple modifications and equivalent structural changes made by using the contents of the present invention's specification and drawings should be included in the patent scope of the present invention and should be stated.

Therefore, the key technology of the image-to-speech assistive device for the visually impaired that can improve learning lies in:

First, the article generation module 13 is a language model chatGPT composed of a deep neural network model. It is an upgraded chatbot that uses AI artificial intelligence technology to drive natural language processing technology. It can self-learn and improve itself, and simulate humans to have a more natural and fluent conversation. Therefore, combined with the image recognition module 12, it can generate sentences that describe the content of the image information in a fluent manner, which is conducive to user understanding.

Second, the first control unit 21 on the guide stick 2 drives the imaging element 11 on the head mounted device 1, and the identification request can be issued only when necessary, which is simpler and more concealed in operation.

Third, the input content is a single still photo taken semi-automatically by the user, and the input conditions are relatively simple, so that the processing speed of the image recognition module 12 and the article generation module 13 is faster and the burden is lower, and the recognition conversion action can be completed more quickly.

Fourth, by utilizing the design of the sound receiving element 16, when the user believes that the text information generated by the article generation module 13 is not clear enough, the user can orally input voice commands to input more specific conditions, and obtain a satisfactory response through the interaction between the sound receiving element 16 and the article generation module 13.

Fifth, by utilizing the design of the positioning element 17 and the map real scene database 132, a precise positioning information and the current direction can be obtained, which is not only helpful for the user to recognize the location, but also can provide accurate information about the location of relatives and friends when they are lost.

Sixth, the design of the facial recognition module 18 and the known person database 133 can not only help the user to identify the person in front of him, but also avoid generating redundant text information for strangers.

In summary, the image-to-speech assistive device for the visually impaired of this invention can truly achieve its efficacy and purpose when used. Therefore, this invention is truly an invention with excellent practicality. In order to meet the application requirements for invention patents, an application is filed in accordance with the law. I hope that the review committee will approve this invention as soon as possible to protect the applicant's hard work. If the review committee of the Jun Bureau has any doubts, please feel free to write to give instructions. The applicant will do his best to cooperate and feel very grateful.

1: Head-mounted device

11: Imaging components

111: First wireless communication element

12: Image recognition module

13: Article generation module

131: Image Description Database

14: Voice conversion module

15:Playback device

2: Guide stick

21: First control unit

22: Second wireless communication component

Claims (7)

A visually impaired assistive device for converting images to speech, which mainly includes: A head mounted device; A camera element is disposed on the head-mounted device, and the camera direction is the same as the sight direction of the head-mounted device, so as to obtain image information; A first wireless communication element is disposed on the head mounted device and electrically connected to the imaging element; A guide stick for the blind, comprising a first control unit for driving the imaging element, and a second wireless communication element electrically connected to the first control unit and wirelessly connected to the first wireless communication element; An image recognition module is disposed in the head mounted device and electrically connected to the imaging element to recognize multiple object information from the image information; An article generation module is disposed in the head mounted device and electrically connected to the image recognition module, and has an image description database to generate a coherent text message based on the image information and the object information, and the article generation module is a language model chatGPT composed of a deep neural network model; A speech conversion module is disposed in the head mounted device and is electrically connected to the article generation module to convert the text information into a speech message; and A playback device is disposed on the head-mounted device and electrically connected to the voice conversion module. As described in Item 1 of the patent application scope, the image-to-speech assistive device for the visually impaired, wherein the head-mounted device is one of glasses, sunglasses or goggles. As described in item 1 of the patent application scope, the visually impaired assistive device for converting images into speech, wherein the head-mounted device is provided with a sound receiving element electrically connected to the article generation module for receiving the user's voice commands so that the article generation module can adjust the content of the text information. As described in item 3 of the patent application scope, the visually impaired assistive device for converting images into speech, wherein the guide stick is provided with a second control unit for driving the sound receiving element. As described in item 3 of the patent application scope, the visually impaired assistive device for converting images into speech, wherein the head-mounted device has a positioning element electrically connected to the article generation module for generating location information, and the article generation module has a map real scene database for the article generation module to integrate the image information, the object information, and the location information to generate precise positioning information. As described in item 1 of the patent application scope, the visually impaired assistive device for converting images into speech, wherein the head-mounted device has a facial recognition module and a known person database electrically connected to the facial recognition module and the article generation module, so that the article generation module can adjust the content of the text information according to the person in the image information. As described in Item 1 of the patent application scope, the visually impaired assistive device for converting images into speech, wherein the playback device is one of a speaker or a headset.

Images