WO2016086438A1

WO2016086438A1 - Head-mounted multimedia terminal auxiliary viewing system for visually impaired person

Info

Publication number: WO2016086438A1
Application number: PCT/CN2014/093530
Authority: WO
Inventors: 杨小康; 翟广涛; 闵雄阔; 李铎
Original assignee: 上海交通大学
Priority date: 2014-12-04
Filing date: 2014-12-11
Publication date: 2016-06-09
Also published as: CN104483754A

Abstract

Provided is a head-mounted multimedia terminal auxiliary viewing system for a visually impaired person, comprising: a head-mounted display for displaying an image to be viewed; an information interaction module for communication between the multimedia terminal and the head-mounted display; an image processing module for processing the image to enhance vision of the visually impaired person; the system obtains the display image of the multimedia terminal in real time, the image processing module processes the obtained display image using image processing and machine vision algorithms, and the head-mounted display displays the processed image in a manner more acceptable to the visually impaired person, thus assisting the visually impaired person in viewing the multimedia terminal.

Description

Head-mounted multimedia terminal assisted viewing system for visually impaired patients

Technical field

The present invention relates to the field of visual aid technology and multimedia, and in particular to a head-mounted multimedia terminal assisted viewing system for a visually impaired patient.

Background technique

Visual impairment, also known as "visual disability", "visual defect", "visual impairment" refers to some or all of the obstacles to the structure or function of the visual organs (including the eyeballs, the ocular nerves) and the cerebral optic nerve center caused by various reasons. It manifests as different degrees of visual impairment or narrowing of vision in both eyes, and it is difficult to achieve the work, study and other activities that ordinary people can do, which limits or hinders the normal effects that can be exerted according to their age, gender, social and cultural conditions.

According to estimates by the World Health Organization, there are 40 to 45 million blind people worldwide, and the low vision population is three times that of the blind, about 140 million people. Among the more than 83 million disabled people in China, there are 12.33 million visually handicapped persons, 5 million blind people and 7.1 million low-vision population. It is one of the countries with the largest number of blind people and low vision population in the world. How to solve their normal learning, work and life issues has become the focus of social attention.

Visual impairment is caused by innate or acquired causes of partial or total loss of function of the visual system, so that it is impossible to accurately identify external things. According to the relevant regulations, visual failure is less than 0.3, or visual field is within 20 °. According to the degree of obstacles, visual impairment can be divided into two categories: the visual acuity of the two eyes is less than 0.3 above 0.03 or the visual field is less than 20°; the visual acuity of the two eyes is less than 0.03 for total blindness. People with amblyopia can usually use vision for daily life, but there are certain obstacles in visual observation. Generally, optical aids or special magnification equipment are needed to observe things. But amblyopia itself is also very different: there is a lack of three-dimensionality, sensitivity to light, unrecognizable color, and too narrow a field of view. Therefore, the optical auxiliary equipment or special amplification equipment required is also very different. The blind population can't rely on vision for daily life at all, and usually can only take an alternative way to conduct daily life. For example, haptic aids, hearing aids, and other types of assistive devices are used in place of vision. It can be said that the visually impaired people need the most assistive technology and equipment. Most information barrier technologies and devices can be applied to people with visual impairments. Almost all visually impaired people can compensate for the quality of life and participate in social life through assistive devices. However, the second sample survey showed that only 7.97% of the visually impaired persons in China were assisted with services; 14.49% of the cities and 5.82% of the rural areas.

Visual impairment aids can be divided into two categories: visual and non-visual appliances:

Visual aids refer to devices or devices that can visually assist visually impaired persons to improve their visual ability. They are also classified into optical, non-optical, and electronic visual aids. Optical aids are more common, including hand-held, vertical magnifiers, glasses, monoculars, and binoculars. The optical aid is easy to use, easy to carry, low in price, and the main visual aid for low vision patients; however, it also has some limitations in use, such as low magnification, small reading range, long-term use. Causes poor posture, will be affected by the light source. Non-optical visual aid refers to improving visual effects by improving the surrounding environment. Such as: enlarged prints to change the size of the readings, high-intensity illuminators to increase contrast, high-contrast objects to facilitate the differentiation of local items. The electronic visual aid is an electronic video device that can read a document, a picture, an observed item, etc. through a photographic lens, and transmit the image to a screen for the user to browse. There are a wide variety of electronic vision aids, including: far-sighted, near-looking (zooming in), near-far, pocket-sized, portable, and so on. It is characterized by easy adjustment of magnification, brightness, contrast, etc., which can avoid glare; it has multiple display modes such as anti-white and half-color; it is ergonomically designed and can provide a large working space.

Non-visual aids refer to auxiliary devices that make full use of functions other than vision, such as hearing and touch, to compensate for the defects of visual functions and improve the quality of life of visually impaired people, including life, work, study, entertainment, etc. Every aspect of it. For example: blind mobile phones and telephones used for daily life, vocal clocks; blind canes and blind roads for travel; Braille for reading communication, dot display, audio books, various screen reading software, etc. Such aids are used more fully by blind patients.

In summary, the assistive technologies and equipment required by different visually impaired people are also different. The existing types of visual obstacle aids have the characteristics of single function and poor expandability, and do not conform to the principles of ease of use, application, simplicity, portability, and scalability. In addition, with the development of multimedia technology, various multimedia terminals such as computers, smart TVs, tablets, mobile phones, etc. have greatly enriched our daily life, and visually impaired patients cannot enjoy the convenience brought by scientific and technological progress due to visual defects. .

Summary of the invention

In view of the deficiencies in the prior art, an object of the present invention is to provide a head-mounted multimedia terminal assisted viewing system for a visually impaired patient, which can acquire a display image of a multimedia terminal in real time and process it by using image processing and machine vision algorithms. Then, the head-mounted display is used to display the processed image in a more acceptable manner for the visually impaired patient, thereby achieving the purpose of assisting the visually impaired patient to view the multimedia terminal.

To achieve the above objective, the present invention provides a head-mounted multimedia terminal assisted viewing system for a visually impaired patient, comprising:

a head mounted display for displaying an image to be viewed;

An information interaction module, configured to communicate between a multimedia terminal and a head mounted display;

An image processing module that processes the image to enhance vision for a visually impaired patient;

The system is capable of acquiring a display image of the multimedia terminal in real time, and the image processing module processes the acquired display image by using image processing and a machine vision algorithm, and the head mounted display makes the processed image more acceptable to a visually impaired patient The way to achieve the purpose of assisting visually impaired patients to view multimedia terminals.

Preferably, the head mounted display includes an imaging system for displaying an image, various types of sensors for detecting external information and the state of the glasses, a control center for overall control, and the like.

Preferably, the image display mode of the imaging system may be a projection type or a non-projection type; if a projection display mode is adopted, the imaging system includes a projection device and a display panel, and the projection device is projected on the display panel in front of the human eye. A virtual image is formed at a certain distance; if a non-projection display mode is employed, the imaging system includes a display panel, and the image is directly displayed on the screen of the display panel.

Further, if a non-projection display mode is employed, a convex lens is further added between the eyes and the display panel to make the imaging clearer.

Preferably, the display panel of the non-projection display mode is a high definition liquid crystal display or an LED dot matrix display.

Preferably, the display panel may be transparent or non-transmissive. If the display is transparent, the image formed on the display panel will be superimposed with the real scene.

Preferably, the display of the imaging system is a binocular display or a monocular display; if it is a single eye, a single channel, the projection device and the display panel only need one way; if it is a binocular or a dual channel, the projection device and the display panel are For the left and right.

Preferably, the various types of sensors include an infrared sensor, an accelerometer, a gyroscope, etc., for detecting external information and the state of the glasses and transmitting the signals to the control center.

Preferably, the information interaction module is partially installed on the multimedia terminal and partially mounted on the head mounted display for communication between the multimedia terminal and the head mounted display. The information interaction can be performed in a wired manner or in a wireless manner. If the wireless method is adopted, the system will be more convenient and will also generate higher energy consumption.

Preferably, the image processing module acquires the display image of the multimedia terminal in real time, processes it by using image processing and machine vision algorithms, and then uses the head mounted display to display the processed image in a more acceptable manner for the visually impaired patient. Image processing and machine vision algorithms have different settings depending on the type of visual impairment; different image processing algorithms correspond to different patients.

Further, for the age-related macular degeneration, central serous retinopathy, cataract, and the like The deformed patient displays the image by pre-distorting the display image of the multimedia terminal, so that the patient produces a normal image; for the color-blind patient, the color of the color-blind patient in the multimedia terminal display image is color-converted or calibrated, so that the patient can distinguish the original Undistinct color; for low vision patients, the multimedia terminal displays images for image enlargement for easy reading; for visual disorders caused by other diseases, the corresponding algorithm is used to improve the patient's viewing experience and enhance their vision.

Preferably, the supporting program and related image processing and machine vision algorithms can be run on the multimedia terminal, and can be transmitted to the head-mounted display by using the computing power of the multimedia terminal; or can be run on the control center of the head-mounted display. The multimedia terminal first transmits the current display image to the head mounted display, and the head mounted display performs processing and display.

Compared with the prior art, the present invention has the following beneficial effects:

Compared with the conventional visual obstacle assisting device, the present invention is a wearable device, which has the advantages of simple portability, easy expansion, and the like, and can make the visually impaired patient better enjoy the convenience brought by the multimedia terminal; further, the present invention discloses The hardware can be used as a visual obstacle assisting device platform. For different types and different degrees of visual obstacles, the matching image processing program can be designed in a targeted manner to achieve the purpose of assisting the visually impaired patient to view the multimedia terminal.

DRAWINGS

Other features, objects, and advantages of the present invention will become apparent from the Detailed Description of Description

1 is a schematic structural view of a preferred embodiment of the present invention;

2 is a block diagram showing the working principle of a preferred embodiment of the present invention;

FIG. 3 is an effect diagram of a weak example of an embodiment of the present invention; wherein: (a) is an image displayed by the current multimedia terminal, and (b) is an image displayed by the head mounted display after image processing (image enlargement).

In the figure: 1 is a multimedia terminal, 2 is a head-mounted display, 3 is an information interaction module, 4 is an imaging system of a head-mounted display; 5 is a sensor of various types of head-mounted displays, and 6 is a control center of a head-mounted display.

detailed description

The invention will now be described in detail in connection with specific embodiments. The following examples are intended to further understand the invention, but are not intended to limit the invention in any way. It should be noted that a number of variations and modifications may be made by those skilled in the art without departing from the inventive concept. These are all within the scope of protection of the present invention.

As shown in FIG. 1, a head-mounted visual aid system for a visually impaired patient according to a preferred embodiment of the present invention Schematic diagram of the structure, the multimedia terminal 1, the head-mounted display 2, the information interaction module 3, the imaging system 4 of the head-mounted display, the various sensors 5 of the head-mounted display, and the control center 6 of the head-mounted display, wherein:

The head mounted display 2 includes an imaging system 4 for displaying an image, various types of sensors 5 for detecting external information and the state of the glasses, a control center 6 for overall control, and the like; the head mounted display The control center 6 of 2 is used for overall control of the head-mounted display, including image acquisition, processing, display of the display panel, and acceptance and processing of the sensor data;

The image display mode of the imaging system 4 of the head mounted display 2 may be a projection type or a non-projection type; if a projection type display mode is adopted, the imaging system includes a projection device and a display panel, and the projection device is projected on the display panel. A virtual image is formed at a certain distance in front of the human eye; if a non-projection display mode is employed, the imaging system includes a display panel, and the image is directly displayed on the screen of the display panel. The display panel may be transmissive or non-transmissive. If the transmissive type is used, the image formed on the display panel will be superimposed on the real scene.

The various types of sensors 5 of the head mounted display 2 include an infrared sensor, an accelerometer, a gyroscope, etc., for detecting external information and the state of the glasses and transmitting the signals to the control center.

The multimedia terminal may be a computer, a smart TV, a tablet computer, a mobile phone, or the like widely used in daily life.

In an embodiment of the invention, the information interaction module 3 is partially installed on the multimedia terminal 1 and partially mounted on the head mounted display 2 for communication between the multimedia terminal 1 and the head mounted display 2. The information interaction can be performed in a wired manner or in a wireless manner. If the wireless method is adopted, the system will be more convenient and will also generate higher energy consumption.

The main function of the system is to provide a multimedia terminal-assisted viewing system for visually impaired patients. The system can not only have different visual enhancement effects on different visual conditions, but also enable the patient to use the aid of the instrument in the multimedia terminal to improve vision and enhance. Its experience.

In addition to the physical components described above, the system further comprises an image processing module designed to enhance the vision of the visually impaired patient, the program acquiring the display image of the multimedia terminal 1 in real time, using image processing and machine vision algorithms. The processing is performed, transmitted to the head mounted display 2 through the information interaction module 3, and the processed image is displayed by the head mounted display 2 in a more acceptable manner for the visually impaired patient.

The image processing and machine vision algorithms have different settings depending on the type of visual impairment; different image processing algorithms correspond to different patients.

For example, in an embodiment of the present invention, a patient who has undergone visual distortion such as age-related macular degeneration, central serous retinopathy, cataract, etc., is displayed by pre-distorting the display image of the multimedia terminal to make the patient Produce a normal image;

For another example, in an embodiment of the present invention, for a color blind patient, a color conversion or calibration of a color that cannot be distinguished by a color blind patient in an image displayed by the multimedia terminal is performed, so that the patient can distinguish a color that cannot be distinguished;

For example, in an embodiment of the present invention, for a low vision patient, the multimedia terminal displays an image for image enlargement, which is convenient for reading;

For visual disorders caused by other conditions, use appropriate algorithms to improve the patient's viewing experience and enhance their vision.

There are other patients and corresponding image processing methods that are not listed here. Changing the image processing algorithm does not affect the essence of the system. The image processing algorithm may use the prior art, or an algorithm that is innovative in order to adapt to the visual aid system.

As shown in FIG. 2, it is a block diagram of a working principle of a system according to an embodiment of the present invention. The workflow includes the following steps:

First, display image acquisition

The part is completed by the multimedia terminal 1, the multimedia terminal collects the current display image in real time, and transmits the collected display image to the control center 6 of the head mounted display 2 through the information interaction module 3 in the form of a video stream;

In this embodiment, the image processing module is located in the control center 6.

Second, image processing

This portion may be completed by the multimedia terminal 1, or may be completed by the control center 6 of the head mounted display 2, the latter being an example. The multimedia terminal 1 transmits the collected display image in the form of a video stream to the control center 6 of the head mounted display 2 through the information interaction module 3, and the control center 6 performs specific processing on the transmitted image, such as: enlargement, enhancement, color Correction, etc.; image processing and machine vision correlation algorithms are needed in this step; the system has high performance requirements for related image processing algorithms, and requires real-time requirements, so real-time performance should be considered when designing image processing algorithms. The problem is that the algorithm with too high complexity and unable to meet the real-time requirements cannot be adopted; if there are some more complicated algorithms, the corresponding system should be simplified or innovated accordingly, so that the algorithm can better cooperate with the system. The image processing algorithm may use the prior art, or an algorithm that is innovative in order to adapt to the visual aid system. The control center 6 then transmits the processed image to the imaging system 4 of the head mounted display;

Third, processing image display

The display mode of the system of the present invention may be a projection type or a non-projection type. If the projection display is used, the imaging system 4 forms a virtual image at a certain distance in front of the human eye; if the non-projection display mode is adopted, the imaging system 4 forms a real image, but a convex lens needs to be added between the eye and the display panel to make the imaging clearer. . Non-projection The display panel of the display can be a high-definition liquid crystal display or an LED dot matrix display, and the selection of the display can be adjusted according to the actual condition of the patient's patient. This does not affect the substance of the system. specific:

1) Projection display (for transmissive display method)

The portion is completed by the projection device and the display panel. The projection device respectively projects the processed image transmitted from the control center 6 to the display panel, so that the image viewed by the user from the display panel is processed by the image processing and the machine vision related algorithm. After the image;

2) Display using the display (for non-projected display mode)

The part of the display panel is completed, and the display panel respectively displays the processed images transmitted by the control center 6 so that the image seen by the user from the display panel is an image processed by the image processing and the machine vision related algorithm.

As shown in FIG. 3, it is an effect diagram which is considered as a weak example, wherein: (a) is a display image of the current multimedia terminal, and (b) is an image which is projected and displayed after the related image processing (image enlargement).

The invention is a wearable device, which has the advantages of simple portability, easy expansion, and the like, and can enable the visually impaired patient to better enjoy the convenience brought by the multimedia terminal; in addition, the hardware disclosed by the present invention can be used as a visual obstacle assisting device platform. For different types and different degrees of visual impairment, a matching image processing program can be designed in a targeted manner to achieve the purpose of assisting visually impaired patients to view multimedia terminals.

The specific embodiments of the present invention have been described above. It is to be understood that the invention is not limited to the specific embodiments described above, and various modifications and changes may be made by those skilled in the art without departing from the scope of the invention.

Claims

A head-mounted multimedia terminal assisted viewing system for a visually impaired patient, comprising:

a head mounted display for displaying an image to be viewed;

An information interaction module, configured to communicate between a multimedia terminal and a head mounted display;

An image processing module that processes the image to enhance vision for a visually impaired patient;

The system is capable of acquiring a display image of the multimedia terminal in real time, and the image processing module processes the acquired display image by using image processing and a machine vision algorithm, and the head mounted display makes the processed image more acceptable to a visually impaired patient The way to achieve the purpose of assisting visually impaired patients to view multimedia terminals.
A head mounted multimedia terminal assisted viewing system for a visually impaired patient according to claim 1, wherein said head mounted display comprises an imaging system for displaying an image, for detecting external information and glasses Various types of sensors in the state, control centers for overall control.
The head-mounted multimedia terminal assisted viewing system for a visually impaired patient according to claim 2, wherein the image display mode of the imaging system is a projection type or a non-projection type;

If the projection display mode is adopted, the imaging system includes a projection device and a display panel, and the projection device is projected on the display panel to form a virtual image at a certain distance in front of the human eye;

If a non-projection display mode is employed, the imaging system includes a display panel, and the image is directly displayed on the screen of the display panel.
A head-mounted multimedia terminal assisted viewing system for a visually impaired patient according to claim 3, wherein if a non-projection display mode is employed, a convex lens is further added between the eye and the display panel to make the imaging clearer.
The head-mounted multimedia terminal assisted viewing system for a visually impaired patient according to claim 3, wherein the display panel of the non-projection display mode is a high definition liquid crystal display or an LED dot matrix display.
The head-mounted multimedia terminal assisted viewing system for a visually impaired patient according to claim 3, wherein the display panel is of a transmissive or non-transmissive type, and if it is transparent, projection The image formed on the display panel will be superimposed on the real scene.
A head-mounted multimedia terminal assisted viewing system for a visually impaired patient according to claim 3, wherein the display of the imaging system is a binocular display or a monocular display;

If it is a single eye or a single channel, the projection device and the display panel only need one way;

If it is binocular or dual channel, the projection device and the display panel are two left and right.
A head-mounted multimedia terminal assisted viewing system for a visually impaired patient according to claim 2, wherein said various types of sensors comprise an infrared sensor, an accelerometer, and a gyroscope for detecting external information and glasses The status is transmitted and the signal is transmitted to the control center.
The head-mounted multimedia terminal assisted viewing system for a visually impaired patient according to any one of claims 1-8, wherein the information interaction module is partially installed on the multimedia terminal and partially mounted on the headset On the display, it is used for communication between the multimedia terminal and the head mounted display; the information interaction is performed by wired or wireless.
The head-mounted multimedia terminal assisted viewing system for a visually impaired patient according to any one of claims 1-8, wherein the image processing module operates on a multimedia terminal, and uses a computing capability of the multimedia terminal to perform correlation After processing, the image is transmitted to the head-mounted display; or the image processing module is operated in the control center of the head-mounted display, and the multimedia terminal first transmits the current display image to the head-mounted display, and the head-mounted display performs processing and display.
A head-mounted multimedia terminal assisted viewing system for a visually impaired patient according to any one of claims 1-8, characterized in that it is directed to age-related macular degeneration, central serous retinopathy, cataract a deformed patient, the image processing module pre-deforms the display image of the multimedia terminal, and causes the patient to generate a normal image;

For a color blind patient, the image processing module performs color conversion or calibration on a color that cannot be distinguished by a color blind patient in a multimedia terminal display image, so that the patient can distinguish a color that cannot be distinguished;

For a low vision patient, the image processing module enlarges the image displayed by the multimedia terminal to facilitate image reading;

For visual impairments caused by other conditions, the image processing module uses a corresponding algorithm to improve the patient's viewing experience and enhance their vision.