CN214231714U - Intelligent glasses are assisted to visual barrier - Google Patents

Abstract

The embodiment of the utility model discloses supplementary intelligent glasses of visual impairment, intelligent glasses include the glasses body, this internal integration of glasses has microprocessor, auricular bone conduction earphone, wireless communication module, CCD optical module, DSP image processing module, camera lens, memory module, power module, rechargeable battery, IO data port, camera lens establishes at glasses body front end, CCD optical module will convert the optical image of gathering from camera lens into high resolution image compression data transmission to DSP image processing module, microprocessor passes through IO data port and auricular bone conduction earphone, wireless communication module, DSP image processing module electricity is connected, memory module, power module is connected with the microprocessor electricity, rechargeable battery is connected with the power module electricity. The utility model discloses from the visual angle of looking barrier user eyes, can do reading, processing and discernment image signal in real time to look barrier user through voice prompt, can help looking barrier user to differentiate scenery and object in real time, all-roundly.

Description

Intelligent glasses are assisted to visual barrier

Technical Field

The utility model relates to an artificial vision technical field especially relates to a supplementary intelligent glasses of visual impairment.

Background

Currently, users need to transmit video and audio colleagues to a remote party by using a mobile phone or a computer, or by connecting a camcorder to the internet and transmitting the video and audio colleagues in real time. Most of intelligent glasses or internet of things equipment are far from common glasses in appearance and heavy, so that users feel inconvenient or embarrassed. The user may be interfered by environmental noise and the like in receiving sound, so that the receiving is not clear.

The intelligent glasses in the current market only use a simple AI MCU or transmit the AI MCU to the cloud for processing and identification. However, many artificial intelligence systems cannot provide the most necessary assistance for visually impaired people and the elderly, such as accurately recognizing streets and characters, and the cloud processing has a long time delay and cannot solve the problem of interaction with people.

SUMMERY OF THE UTILITY MODEL

The embodiment of the utility model provides a technical problem that will solve provides a look supplementary intelligent glasses of barrier to help looking barrier user identification image.

In order to solve the technical problem, the embodiment of the utility model provides an intelligent glasses are assisted to visual barrier, including the glasses body, this internal integration of glasses has microprocessor, ear bone conduction earphone, wireless communication module, CCD optical module, DSP image processing module, camera lens, memory module, power module, rechargeable battery, IO data port, camera lens establishes at glasses body front end, CCD optical module will be followed camera lens collection's optical image and converted high resolution image compression data transmission to DSP image processing module, microprocessor passes through IO data port and ear bone conduction earphone, wireless communication module, DSP image processing module electricity is connected, memory module, power module and microprocessor electricity are connected, rechargeable battery is connected with power module electricity.

Further, the wireless communication module is one or more of a wifi module, a bluetooth module and a 5G module.

The utility model has the advantages that: the utility model discloses from the visual angle of looking barrier user eyes, can do reading, processing and discernment image signal in real time to look barrier user through voice prompt, can help looking barrier user to differentiate scenery and object in real time, all-roundly.

Drawings

Fig. 1 is the utility model discloses the structure schematic diagram of the supplementary intelligent glasses of visual impairment.

Fig. 2 is a circuit diagram of a power module according to an embodiment of the present invention.

Fig. 3 is a circuit diagram of a CCD optical module according to an embodiment of the present invention.

Fig. 4 is a circuit diagram of a microprocessor according to an embodiment of the present invention.

Fig. 5 is a schematic structural diagram of the auxiliary intelligent glasses system for the visual impairment of the embodiment of the present invention.

Fig. 6 is a schematic flow chart of a control method of the vision-impairment-assisted smart glasses system adopted in the embodiment of the present invention.

Fig. 7 is a model diagram of a convolutional neural network model employed by an embodiment of the present invention.

Fig. 8 is a block diagram of a convolutional neural network model employed by an embodiment of the present invention.

Detailed Description

It should be noted that, in the present application, the embodiments and features of the embodiments may be combined with each other without conflict, and the present invention is further described in detail with reference to the accompanying drawings and specific embodiments.

In the embodiment of the present invention, if there is directional indication (such as upper, lower, left, right, front, and rear … …) only for explaining the relative position relationship between the components and the motion situation under a certain posture (as shown in the drawing), if the certain posture is changed, the directional indication is changed accordingly.

In addition, the descriptions of the first, second, etc. in the present invention are for descriptive purposes only and are not to be construed as indicating or implying any relative importance or implicit indication of the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature.

Referring to fig. 1 to 4, the pair of vision-impairment-assisted intelligent glasses of the embodiment of the present invention includes a glasses body. The glasses body is internally integrated with a microprocessor, an ear bone conduction earphone, a wireless communication module, a CCD optical module, a DSP image processing module, a camera lens, a memory module, a power module, a rechargeable battery and an I/O data port.

The camera lens is arranged at the front end of the glasses body, and the CCD optical module converts an optical image collected from the camera lens into high-resolution compressed data and sends the high-resolution compressed data to the DSP image processing module. The glasses body can be made of light and firm ABS and PC materials. The DSP image processing module uses a ten-bit ADC to convert the simulation image signal into serial digital data. The digital data is a huge four-dimensional numpy array (array), the four dimensions are each data, image gray-scale value, image length and image width, each numerical value in the array is a floating point number (float) between 0 and 1, and the floating point number represents the gray-scale value between 0 and 255 after normalization of each position in different images. The label information of which number each image is stored in another array.

The microprocessor is electrically connected with the ear bone conduction earphone, the wireless communication module and the DSP image processing module through the I/O data port, the memory module and the power supply module are electrically connected with the microprocessor, and the rechargeable battery is electrically connected with the power supply module. The rechargeable battery adopts a rechargeable small-sized high-energy lithium battery. The microprocessor preferably adopts an ESP32 singlechip, and an ESP32 singlechip internally integrates a DSP image processing module and an I/O data port.

As one implementation mode, the wireless communication module is one or more of a wifi module, a Bluetooth module and a 5G module. The utility model discloses a wireless communication module's radio signal is with image transmission to cell-phone end.

In one embodiment, the glasses body is integrated with a built-in antenna connected with the wireless communication module.

The embodiment of the utility model provides a can change the scenery that sees with the user into digital image signal and do reading, processing and discernment in real time, through the voice prompt user to see through wireless signal and 4G 5G network and send the volunteer to backstage service terminal, the volunteer describes the scenery detail in front of the user eye at once and makes the replenishment, carries out the real-time interactive communication between people and the people. The embodiment of the utility model provides a can provide real-time assistance for the sight impaired personage or the impaired person of vision, also can be used for by backstage personnel to assign instruction, supervision and revise in order to accomplish the task to the line worker.

Please refer to fig. 5, the utility model discloses supplementary intelligent glasses system of visual impairment includes supplementary intelligent glasses of visual impairment, cell-phone end, server, service terminal.

The intelligent glasses for assisting the visual impairment are connected and communicated with the mobile phone end through the wireless communication module, so that the image in front of the visual impairment user is collected in real time, the image is sent to the mobile phone end, and meanwhile, information returned by the mobile phone end is output through the ear bone conduction earphones.

The mobile phone end: preprocessing images sent by the auxiliary intelligent glasses for the visual impairment in real time, continuously comparing the preprocessing with a digital target default value, and sending a result to the auxiliary intelligent glasses for the visual impairment when the similarity reaches a threshold value; if the similarity does not reach the threshold value or the user selects to enter the manual mode, the current image is uploaded to the server, and meanwhile, the information returned by the server is sent to the auxiliary intelligent glasses for the visual impairment. Such as humans (men/women, elderly/toddlers/pre-designed relatives), household items (doors/windows/tables/chairs/tv/sand/pots/cups), outdoors (cars/buses, stairways/elevators) or larger road signs etc. The utility model discloses data set accessible can be constantly updated and self-learning and revision in order to improve the accuracy.

A server: distributing the images uploaded by the mobile phone terminal to the service terminal, and transmitting the identification result uploaded by the service terminal back to the corresponding mobile phone terminal; or the vision impairment auxiliary intelligent glasses and the service terminal are connected in real time through a network.

The service terminal: the corresponding volunteers receive the images uploaded by the mobile phone terminal through the service terminal and then send the manual identification results of the volunteers to the server; or the intelligent glasses are connected with the auxiliary visual impairment glasses for communication, so that the volunteers can directly communicate with the visual impairment users in real time, and manual help is provided for the visual impairment users in real time.

The utility model discloses make the user that has the eyesight problem can obtain helping in real time or in the shortest time, the image that automatic identification looked the barrier user and shot. In addition, through a volunteer (i.e., a significant worker) community, the service provided by the utility model can be operated in twenty-four hours all the day, and people all over the world can download the program and become the volunteer (i.e., the significant worker). When the vision-impaired user needs help, the vision-impaired auxiliary intelligent glasses can shoot the scenery or the object and send the scenery or the object to a volunteer (a prosthetist) in real time, the volunteer (the prosthetist) of the service terminal receives the notification, the volunteer (the prosthetist) can select whether to respond, and the volunteer (the prosthetist) with time can describe the picture to the vision-impaired user by voice or text messages.

In addition, to provide real-time assistance, embodiments of the present invention also allow the visually impaired user to issue a real-time support request, having time for the volunteer (prosthetist) to contact the visually impaired user through a form similar to a video conference and to be able for the volunteer (prosthetist) to see the scenery in front of the visually impaired user and to assist the visually impaired user in real-time through voice.

The utility model discloses there is bigger flexibility in the aspect of the received information. For users with visual impairment, the utility model can contact volunteers who voluntarily provide visual instructions, and the volunteers are willing to provide help to facilitate the daily life of users with visual impairment. For the volunteers, the utility model can help people at any time and any place while accumulating the opportunity of the volunteer hours. For advertisers, the present invention provides a wide audience segment so that their information can be received by different people in various regions.

As an implementation manner, the mobile phone side preprocesses the image by using the convolutional neural network model, and the image data sequentially enters a first convolutional layer, a second convolutional layer, a pooling layer, a first complete connection layer and a second complete connection layer of the convolutional neural network model and then outputs a result.

Referring to fig. 7 to 8, the image data is decomposed into layers of different pixel sizes in a convolutional neural network model (CNN), which learns the mapping relationship between the input and output from the input image data without a precise expression process of human intervention features. The input layer is an array converted from pixels; the convolutional neural network model comprises a plurality of convolutional layers (convolutional layers), pooling layers, which are all called sampling layers (fully connected layers), and the like, and can be regarded as a neural network consisting of connected neurons; and the output layer is the result of the discrimination. The input layer, the hidden layer and the full connection layer are composed of three groups of convolution pooling layers. The data entering the input layer is an array converted from a two-dimensional color map (RGB color mode), and the size of the array is determined by multiplying the resolution of the image by the number of RGB bits (width × height × 3). Data from the input layer first enters the first convolutional layer. The convolutional layer has a filter (filter) corresponding to it, which is a matrix of numbers. At the convolutional layer, the input matrix is convolutionally multiplied by the filter matrix to obtain a new matrix, i.e. the feature map.

The convolution layer adjusts the size, step size and filling mode of the filter according to the practical application, wherein the selection of the filter determines the range of each sampling, the step size determines the number of pixels moved by each sampling, and the filling mode includes zero filling and discarding filling (for processing the condition that the size of the filter is not consistent with the size of the image). The nature of the convolution operation allows the feature map to preserve the relationship between pixels in the original image. The feature map is dimensionality reduced at the pooling layer, enters a second convolution layer, and is then pooled again. There are three ways of maximum pooling (max pooling), mean pooling (average pooling), and pooling (sum pooling) to compress image data while preserving the relationship between image pixels.

And the activation layer selects different functions to perform non-linear processing on the data, and the more common mode is a modified linear Unit (ReLU). Convolutional layers with different filters implement a variety of operations on the image, such as edge detection, contour detection, blurring, sharpening, and so on.

And the full connection layer, after convolution and pooling for many times, data reaches the full connection layer. The image data is classified by an excitation function (e.g., a logistic regression function with loss), and the final output result indicates the probability of the input image belonging to a certain class.

In the evaluation, whether the model is over-fitted or not can be judged by respectively comparing the difference values between the accuracy and the loss values of the training sample and the verification sample. If the difference of the accuracy values in the training and verifying stages is large, the model is over-fitted; the lower the accuracy of the training and verification stage is, the less satisfactory the image discrimination effect is. The model uses binary cross entropy (binary entropy) as an objective function, and the model is subjected to updating iteration with the aim of minimizing the loss value. Thus, a smaller loss value means that the trained model fits better to the data. The utility model provides a training and verification result record in the background database, the test result then embodies in the visual analysis chart of following each model.

For example, the preset data set may be an MS COCO data set. The MS COCO data set was released by microsoft team in 2014, and a total of 8 million images for training were collected, wherein 80 common objects in life were labeled, for example: cats, dogs, airplanes, tables, cars …, etc., and provides about 4 million test images. The data set is updated again in 2019, and although different object types are not newly added, the number of images is increased to 30 ten thousand images. The data set provides a completely open content, and the calculation model proposed by each family can have objective comparison basis by taking the same data set as a reference.

As an implementation manner, the mobile phone further includes a text recognition module: and identifying characters in the image, and sending an identification result to the auxiliary intelligent glasses for the visual impairment.

The visually impaired user also can use the utility model discloses a text recognition module, the self-service ability of performance, if need the recognition characters for example be letter or mansion notice he just can oneself one hand hold the cell-phone one hand hold the object take a picture and just release the characters with the characters recognition function, text recognition module also can turn into the characters that the discolour out and read and visually impaired user listens to sound.

These prosthetics support function and artificial intelligence word recognition function are complementary and insufficient, when the simple recognition is carried out in the simple word recognition work, the vision-impaired user can choose to solve the simple problem without depending on the prosthetics, but for the object with more complex surface shape or the nearby environment vision user, the user needs to send the picture to the prosthetics or contact the prosthetics through the real-time function of the vision and get real-time assistance.

Referring to fig. 6, an embodiment of the present invention provides a method for controlling an intelligent glasses system for assisting visual impairment, including:

step 1: the intelligent glasses for assisting the visual impairment collect images in front of the visual impairment user in real time and send the images to the mobile phone end;

step 2: the mobile phone end preprocesses images sent by the auxiliary intelligent glasses for the visual impairment in real time, continuously compares the preprocessing with a digital target default value, and sends a result to the auxiliary intelligent glasses for the visual impairment when the similarity reaches a threshold value; if the similarity does not reach the threshold value through comparison or the user selects to enter a manual mode, uploading the current image to a server;

and step 3: the server distributes the images uploaded by the mobile phone terminals to the service terminals, and transmits the identification results uploaded by the service terminals back to the corresponding mobile phone terminals; or the auxiliary intelligent glasses for the visual impairment and the service terminal are connected in real time through a network;

and 4, step 4: the service terminal receives the images uploaded by the mobile phone terminal and then sends the result of the volunteer manual identification to the server; or the intelligent glasses are connected with the auxiliary visual impairment glasses for communication, so that the volunteers can directly communicate with the visual impairment users in real time, and manual help is provided for the visual impairment users in real time.

As an implementation manner, the convolutional neural network model is used to preprocess the image in step 1, and the image data sequentially enters the first convolutional layer, the second convolutional layer, the pooling layer, the first fully-connected layer and the second fully-connected layer of the convolutional neural network model and then outputs the result.

As an embodiment, step 1 further comprises a text recognition substep: and identifying characters in the image, and sending an identification result to the auxiliary intelligent glasses for the visual impairment.

The embodiment of the utility model provides a provide three kinds of assistance methods for the vision-impaired user: (1) the mobile phone terminal automatically identifies images in front of the auxiliary intelligent glasses for the visual impairment and sends the identification result to the auxiliary intelligent glasses for the visual impairment, and the auxiliary intelligent glasses for the visual impairment are output in a voice mode so that the visual impairment can understand the seen images; (2) for complex situations such as street environments, the visually impaired user can send images to the server through the visually impaired auxiliary intelligent glasses and the mobile phone terminal and send the images to a volunteer, and the volunteer with time can describe the images to the user; (3) the visually impaired user can also select to contact with volunteers in a visual mode, and the volunteers with time can provide visual assistance for real-time communication for the user; (4) the mobile phone end automatically identifies character images in front of the intelligent glasses for assisting the visual impairment, the mobile phone end identifies characters in the intelligent glasses, the identification result is sent to the intelligent glasses for assisting the visual impairment, and the intelligent glasses for assisting the visual impairment are output in a voice mode so that the visually impaired people can understand texts.

The utility model discloses a 4 functions are complementary, and artificial intelligence characters are discerned and are the function of direct and appropriate, not needing the volunteer, and the visual barrier user can exert its self-service ability. The volunteer description picture and the function of the volunteer to visually contact the visually impaired user are more suitable for complex situations. The utility model relates the visually impaired people and the care volunteers, which makes contribution to the social public utilities. The utility model discloses make volunteer can see through sound or characters and directly instruct the personage that looks the barrier, look the barrier personage also can be more convenient receipt characters and image information.

Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that changes, modifications, substitutions and alterations can be made in these embodiments without departing from the principles and spirit of the invention, the scope of which is defined in the appended claims and their equivalents.

Claims (2)

1. The utility model provides a supplementary intelligent glasses of visual impairment, including the glasses body, a serial communication port, this internal integration of glasses has microprocessor, auricular bone conduction earphone, wireless communication module, CCD optical module, DSP image processing module, camera lens, memory module, a power supply module, rechargeable battery, IO data port, camera lens establishes at glasses body front end, CCD optical module converts the optical image of gathering from camera lens into high resolution compression data and sends to DSP image processing module, microprocessor passes through IO data port and auricular bone conduction earphone, wireless communication module, DSP image processing module electricity is connected, memory module, power module is connected with microprocessor electricity, rechargeable battery is connected with power module electricity.

2. A pair of auxiliary smart glasses according to claim 1, wherein the wireless communication module is one or more of a wifi module, a bluetooth module, and a 5G module.

Images