WO2019119372A1

WO2019119372A1 - Display method and device, electronic device and computer program product

Info

Publication number: WO2019119372A1
Application number: PCT/CN2017/117837
Authority: WO
Inventors: 崔华坤; 王恺; 廉士国
Original assignee: 深圳前海达闼云端智能科技有限公司
Priority date: 2017-12-21
Filing date: 2017-12-21
Publication date: 2019-06-27
Also published as: CN108140362A; CN108140362B

Abstract

A display method and device, an electronic device, and a computer program product, the method comprising: collecting depth data of an environment; determining mapping between each depth value and a color combination; generating a pseudo color image of the environment according to the depth data and the mapping; and superimposing the pseudo color image with an image of the environment and displaying the same. According to the present application, the depth data of the environment is mapped to obtain a pseudo color image, and the pseudo color image is superimposed with an image of a current environment to enhance display, so that the residual vision of a patient with poor eyesight may be fully utilized without substantially affecting the field of vision of the patient with poor eyesight, and the patient is thus helped to effectively sense the distance of objects in the environment.

Description

Display method, device, electronic device and computer program product

Technical field

The present application relates to the field of guiding blind technology, and in particular to a display method, device, electronic device and computer program product.

Background technique

Visual disability is a serious public health, social and economic problem worldwide. The main causes of blindness in different economic regions are age-related macular degeneration, diabetic retinopathy, etc., while developing countries are older. Cataracts and infectious eye diseases are predominant. Patients with visual disability are usually divided into blind and low-vision patients, of which blind people account for only about a quarter of all visually impaired patients, while low-vision patients account for nearly three-quarters. According to statistics, there are between 40 and 45 million blind people and about 135 million low vision patients worldwide, and about 7 million blind people and 21 million low vision patients are added each year. It can be seen that patients with visual disability, especially those with a large number of low vision patients, face serious challenges. Although some of the patients with low vision can recover or improve vision through surgery and refractive correction, there are still a large number of patients with low vision who need low vision equipment to assist.

Because low vision has not been widely recognized by the society, patients with residual vision are often submerged in the crowd and are considered blind. The visual aids developed in the prior art for low vision patients are mainly low vision aids. . The existing visual aids mainly use the optical principle to volume enlarge or project the image seen by the low vision patient, and adjust the imaging distance or the imaging angle to assist the low vision patient to obtain a clear image.

The shortcomings of the prior art are:

Existing visual aids generally have a large influence on the user's field of view, can not make good use of the user's residual vision, and can not assist the low vision patient to effectively perceive the distance of objects in the environment through the image.

Summary of the invention

Embodiments of the present application provide a display method, apparatus, device, and computer program product, which are mainly used to assist a low vision patient to effectively perceive the distance of an object in an environment.

In one aspect, the embodiment of the present application provides a display method, including: collecting depth data of an environment; determining a mapping relationship between each depth value and a color combination; and according to the depth data and the mapping a relationship, generating a pseudo color image of the environment; superimposing the pseudo color image with an image of the environment.

In another aspect, an embodiment of the present disclosure provides a display device, where the device includes: a data acquisition module, configured to collect depth data of an environment; and a color mapping module, configured to determine each depth value and color combination. a mapping relationship between the pseudo color image generating module for generating a pseudo color image of the environment according to the depth data and the mapping relationship; and a display module, configured to superimpose the pseudo color image and the image of the environment display.

In another aspect, an embodiment of the present application provides an electronic device, including: a memory, one or more processors; and one or more modules, the one or more modules being Stored in the memory and configured to be executed by the one or more processors, the one or more modules including instructions for performing the various steps of the above methods.

In another aspect, embodiments of the present application provide a computer program product for use in conjunction with an electronic device, the computer program product comprising a computer program embedded in a computer readable storage medium, the computer program comprising An instruction to cause the electronic device to perform the various steps in the above methods.

The beneficial effects of the embodiments of the present application are as follows:

In the present application, the depth data of the environment is mapped to obtain a pseudo color image, and the pseudo color image is superimposed with the current environment image for enhanced display, and the low vision patient can be fully utilized without substantially affecting the field of vision of the low vision patient. Residual vision and assist in its effective perception of the distance of objects in the environment.

DRAWINGS

Specific embodiments of the present application will be described below with reference to the accompanying drawings, in which:

1 is a schematic flow chart showing a display method in Embodiment 1 of the present application;

2 is a schematic diagram showing a mapping relationship between depth values and color combinations in the present application;

FIG. 3 is a schematic flowchart diagram of a display method in Embodiment 2 of the present application;

4 is a schematic diagram showing a mapping relationship between depth values and color combinations in the present application;

FIG. 5 is a schematic flowchart diagram of a display method in Embodiment 3 of the present application;

6 is a schematic diagram showing a relationship between a depth value and a gray value and a color combination in the present application;

7a-7d are schematic diagrams showing four implementation scenarios in Embodiment 4 of the present application;

FIG. 8 is a schematic structural diagram of a display device in Embodiment 5 of the present application;

FIG. 9 is a schematic structural diagram of an electronic device in Embodiment 6 of the present application.

Detailed ways

The exemplary embodiments of the present application are further described in detail below with reference to the accompanying drawings, in which the embodiments described are only a part of the embodiments of the present application, but not all embodiments. An exhaustive example. And in the case of no conflict, the features in the embodiments and the embodiments in the description can be combined with each other.

The inventor noticed during the invention that the existing visual aid generally has a large influence on the user's field of view, can not make good use of the user's residual vision, and cannot assist the low vision patient to effectively perceive the distance of the object in the environment through the image.

In view of the above deficiencies, the present application provides a display method for mapping depth data of an environment to obtain a pseudo color image, and superimposing the pseudo color image with the current environment image for enhanced display, which can substantially not affect the field of view of the low vision patient. In this case, make full use of the residual vision of patients with low vision and assist them to effectively perceive the distance of objects in the environment.

The embodiments of the present application are generally implemented in a low vision auxiliary device, such as AR (Augmented Reality) glasses, VR (Virtual Reality) glasses, or a blind helmet, etc., and may also be in a user's portable device such as a mobile phone or a tablet. Computer and other implementations.

The essence of the technical solution of the embodiment of the present invention is further clarified by specific examples below.

Embodiment 1:

FIG. 1 is a schematic flowchart of a display method in Embodiment 1 of the present application. As shown in FIG. 1 , the display method includes:

Step 101: Collect depth data of an environment.

Step 102: Determine a mapping relationship between each depth value and a color combination.

Step 103: Generate a pseudo color image of the environment according to the depth data and the mapping relationship;

Step 104: superimpose and display the pseudo color image and the image of the environment.

Taking the AR glasses as a low-visibility patient as an example, in step 101, the AR glasses worn by the low-vision patients collect the depth data of the environment. The depth data here may be obtained by directly collecting the depth sensor or the depth of field camera mounted on the AR glasses, or by calculating the front environment image by a binocular camera mounted on the AR glasses. The depth data may be separately collected and stored in a form of a table or a matrix corresponding to the current environment image location; the collection of the depth data may also be collected along with the collection of the environment image, and the depth data will be compared with the environment image. Each pixel corresponds.

Due to the limitations of the device, the depth data usually has a limit recognition range. For areas that are beyond the range of the depth sensor, the depth data is usually recorded as 0 or the default value.

In step 102, a mapping relationship between each depth value and a color combination is determined. Taking the depth sensor as an example, the effective working distance can be usually 0.5m to 5m. Therefore, when the unit is in millimeters, the range of the depth data of the front environment is [500, 5000], and the depth values and colors in the range are established. Combined mapping relationship.

FIG. 2 is a schematic diagram showing a relationship between a depth value and a color combination in the present application. As shown in FIG. 2, a combination of a depth value and a combination of three colors, that is, a combination of red, green, and blue, wherein the depth is When the value is between [500, 1500], it is mapped to a single red, and the gray value of the red color component changes with the depth value. When the depth value is between [1500, 2500], it is mapped to red and green. The combination, and the color components of red and green vary with depth values, and so on. The visible depth value is in the full range of [500, 5000], and will correspond to the gradient of "red-green-blue" depending on the depth value. When the depth value has been recorded as 0 or the default value outside the range of [500, 5000], it can be directly mapped to 0, that is, the various color components are all zero, and the gray value is 0, which corresponds to black.

In fact, the depth value may have a linear or non-linear, continuous or discrete mapping relationship with two or more colors, and the various colors are not limited to three colors of blue, red, and green, and may also be orange, yellow, cyan, Purple or pink, etc., or a combination of these colors and black or white.

In addition, the mapping relationship between the depth values and the color combination may be preset or dynamically changed according to environmental conditions such as brightness, light, and the like.

In some embodiments, before the step 102, the method further includes: acquiring current user information, and determining, in the step 102, a mapping relationship between each depth value and a color combination according to the user information;

In general, patients with low vision have obstacles in color resolution. For example, some patients can only distinguish between red and blue, while yellow, cyan and gray are considered the same, while others can feel red, yellow, cyan and blue. But I don't know what color it is; some users can clearly distinguish the gradation of certain colors, while others can only distinguish the color combinations with clear boundaries. Therefore, each low vision patient can be matched in advance with the combination of the applicable colors and the mapping with the depth value, and the combination of colors that can be sensitively distinguished by each user and the mapping manner of the user's habits can be determined. For example, for patients who cannot distinguish red, red is not included in the color combination; for users who can identify the gradient blue very sensitively, the mapping range of blue in the color combination can be appropriately expanded; for the indistinguishable or not Patients who adapt to the gradient color adopt a step-like mapping method, that is, a certain depth value range is mapped to a single color, and there is a clear boundary between colors in each depth value range.

After the AR and the color combination are mapped and set in advance, after the user wears the AR glasses, the user can identify and obtain the user information of the current user, and match the corresponding depth value and color combination. The mapping relationship is performed for subsequent pseudo color conversion processing. The manner of identification and acquisition may be selected by a user's manual menu operation, or by biometric recognition such as user voice, iris, or the like.

It should be noted that the implementation steps of step 101 and step 102 are not limited, that is, as long as the depth data of the environment is collected before step 103, and the mapping relationship between each depth value and the color combination is determined.

In step 103, a pseudo color image of the environment is generated according to the collected environmental depth data and the mapping relationship between the depth values and color combinations.

The depth value in the depth data of the collected environment is converted into the color corresponding to the depth value according to the mapping relationship, and the corresponding color is obtained after the corresponding conversion completes the corresponding color at each position of the current environment image, and the pseudo color image of the current environment is obtained. The various colors in the pseudo-color image correspond to the distance of the object in the environment to the depth sensor of the user. If somewhere in the pseudo-color image is black, it may indicate that the distance has exceeded the range of the depth sensor.

In step 104, a pseudo color image of the environment is superimposed with an image of the environment. When the user wears the AR glasses, the user can see the environment image through the transparent lens. At this time, according to the corresponding relationship between the pseudo color image of the environment and the environment depth data, or the corresponding color image of the environment and the environment image The relationship superimposes the current environmental image on the transparent lens to display the pseudo color image processed in real time. When the user wears VR glasses, a blind guide helmet or other terminals, the current environment image can be acquired synchronously and superimposed with the pseudo color image for display.

Because the human eye has a relatively strong ability to distinguish colors, thousands of colors can be distinguished. Superimposing corresponding pseudo-color images on the basis of the original environment image enables the patient to extract graphic information more effectively, and more easily recognize the details of the environmental image. Sensitive, visual perception and recognition of objects at various distances.

In this embodiment, the depth data of the environment is mapped to obtain a pseudo color image, and the pseudo color image is superimposed with the current environment image for enhanced display, and the low vision patient can be fully utilized without substantially affecting the vision of the low vision patient. Residual vision and assists it in effectively sensing the distance of objects in the environment. For different patient users, different mappings of depth values and color combinations can be debugged for them to generate unique pseudo color images that are more suitable for each user's viewing.

Embodiment 2:

FIG. 3 is a schematic flowchart of a display method in Embodiment 2 of the present application. As shown in FIG. 3, the display method includes:

Step 201: Collect depth data of an environment.

Step 202: Determine a mapping relationship between each depth value and a color combination in a preset range.

Step 203: Generate a pseudo color image of the environment according to the depth data of the environment and the mapping relationship within a preset range.

Step 204: Display the pseudo color image and the image of the environment.

For the implementation of step 201, reference may be made to the description of step 101 in the first embodiment, and the depth data of the environment is collected in step 201.

In step 202, a mapping relationship between each depth value and a color combination within a preset range is determined, where the preset range falls within the limit acquisition range of the depth data and is smaller than the limit collection range of the depth data.

The significance of the preset range of depth values is that the user can perform subsequent pseudo-color processing according to the range of interest. Usually low vision patients are more interested in obstacles within 2m around, so the depth value can be limited to a range of depth values corresponding to 1m to 2m. For example, if the limit range of the depth sensor is 0.5m to 5m, that is, in millimeters, the limit range of the depth data of the front environment is [500, 5000], and the user can customize a preset range. The preset range is in the range of [500, 5000] and is smaller than the range, for example, [1000, 2000] can be selected.

The mapping relationship between the depth values and the color combinations in the preset range is similar to the mapping relationship between the depth values and the color combinations in the entire range. FIG. 4 is a schematic diagram showing a mapping relationship between depth values and color combinations in the present application. As shown in FIG. 4, within a preset range, that is, a combination of depth values and three colors in [1000, 2000], that is, The combination of red, green, and blue has a mapping relationship. The visible depth value is within the preset range of [1000, 2000], and will correspond to the gradient of "red-green-blue" depending on the depth value. When the depth value has been recorded as 0 or the default value outside the range of [1000, 2000], it can be directly mapped to 0, that is, the various color components are all zero, and the gray value is 0, corresponding to black.

Comparing FIG. 2 in the first embodiment, the present embodiment completes the gradation correspondence of “red-green-blue” within a small depth value range of [1000, 2000], and objects of 1 m and 1.5 m from the user. It can be easily identified by the difference between red and green, and in the first embodiment, objects 1m and 1.5m away from the user are basically displayed in red, and the color difference is small. It can be seen that when the mapping manner with the color combination is determined, the smaller the preset range of the depth value is, the more the objects of different distances in the range can be more finely distinguished by different colors. When the user performs close-range fine operations, the depth range can be adjusted even smaller, so that low-vision users can accurately distinguish the subtle distance differences of objects within a certain distance range.

Similarly, each depth value within a preset range may have a linear or non-linear, continuous or discrete mapping relationship with two or more colors, and the various colors are not limited to three colors of blue, red, and green. It may be orange, yellow, cyan, purple or pink, etc., or a combination of these colors and black or white.

In addition, the mapping relationship between the depth values and the color combinations in the preset range may be preset or dynamically changed according to environmental conditions such as brightness, light, and the like.

In some embodiments, before the step 102, the method further includes: acquiring current user information, and determining, in the step 102, a mapping relationship between each depth value and a color combination according to the user information. For the implementation of this step, reference may be made to the related description of step 102 in the first embodiment.

It should be noted that the implementation steps of step 201 and step 202 are not limited, that is, as long as the depth data of the environment is collected before step 203, and the mapping relationship between the depth values and the color combination in the preset range is determined.

In step 203, a pseudo color image of the environment is generated according to the collected depth data of the environment within a preset range, and a mapping relationship between each depth value and color combination within the preset range.

The depth value in the depth data of the collected environment is converted into the color corresponding to the depth value according to the mapping relationship, and the corresponding conversion completes the color corresponding to each position of the current environment image, and the current environment is obtained. a pseudo color image in which the various colors in the pseudo color image correspond to the distance of the object in the environment to the depth sensor of the user, and the color portion in which the pseudo color image is generated is the image portion of the object conforming to the preset depth value range , that is, the image portion of the object within the distance range that the user is most concerned about. At this point, if somewhere in the pseudo color image is black, it may indicate that the distance has exceeded the range of the depth sensor, or the distance is not within the preset range specified by the user.

In step 204, a pseudo color image of the environment is superimposed with an image of the environment. When the user wears the AR glasses, the user can see the environment image through the transparent lens. At this time, according to the corresponding relationship between the pseudo color image of the environment and the environment depth data, or the corresponding color image of the environment and the environment image The relationship superimposes the current environmental image on the transparent lens to display the pseudo color image processed in real time. When the user wears VR glasses, a blind guide helmet or other terminals, the current environment image can be acquired synchronously and superimposed with the pseudo color image for display.

In this embodiment, the depth data of the environment within the preset depth value range is mapped to obtain a pseudo color image, and the pseudo color image is superimposed with the current environment image for enhanced display, which can substantially not affect the vision of the low vision patient. Under the full view of the residual vision of patients with low vision, and assist them to effectively perceive the distance of objects in the environment, for different patient users can debug different mappings of depth values and color combinations to generate more suitable for each user to watch Unique pseudo color image. In addition, the embodiment can focus on the distance range portion that the user is most concerned about. When the mapping manner of the color combination is determined, the smaller the preset depth value range, the more the objects of different distances in the range can be different. Colors are more subdivided.

Embodiment 3:

FIG. 5 is a schematic flowchart of a display method in Embodiment 3 of the present application. As shown in FIG. 5, the display method includes:

Step 301: Collect depth data of the environment, and generate a grayscale image according to the depth data of the environment;

Step 302: Determine a mapping relationship between each depth value and a color combination; and determine a mapping relationship between each gray value and a color combination according to a mapping relationship between the depth values and the color combination;

Step 303: Generate a pseudo color image of the environment according to a grayscale map generated by the depth data and a mapping relationship between the grayscale values and color combinations;

Step 304: superimpose and display the pseudo color image and the image of the environment.

Taking the AR glasses as a low-visibility patient as an example, in step 301, the AR glasses worn by the low vision patient collect the depth data of the environment, and generate a grayscale image of the current environment according to the depth data.

The depth data here may be obtained by directly collecting the depth sensor or the depth of field camera mounted on the AR glasses, or by calculating the front environment image by a binocular camera mounted on the AR glasses. The depth data may be separately collected and stored in the form of a table or a matrix corresponding to the current environment image position; the collection of the depth data may also be accompanied by the collection of the environment image, and the depth data will be associated with each pixel of the environment image. correspond.

The grayscale image divides white and black into several grades by logarithmic relationship, and the grayscale map can be divided into 256 steps or 65536 steps. Taking the 256-level grayscale image as an example, when generating a grayscale image, the mapping relationship between the depth value and the grayscale value is usually established. Due to the limitation of the device, the depth data usually has a limit recognition range, and the depth sensor cannot exceed the range. The identified depth data is usually recorded as 0 or a default value. The limit acquisition range of the depth value is scaled to the gray value of [0, 255], and the gray value corresponding to the depth value outside the limit range corresponds to 0 or 255. At the same time, the depth value and the gray value can be linear or non-linear, continuous or discrete mapping relationship; the short distance can be mapped to white, the long distance to black, or the short distance to black, and the long distance to white.

In step 302, a mapping relationship between each depth value and a color combination is determined, and a mapping relationship between each gray value and a color combination is determined according to a mapping relationship between the depth values and the color combination.

Taking the depth sensor as an example, the effective working distance can be usually 0.5m to 5m. Therefore, when the unit is in millimeters, the range of the depth data of the front environment is [500, 5000], and the depth values in the range can be established. The mapping of color combinations. Moreover, since there is usually a mapping relationship between the depth value and the gray value in step 301, the mapping relationship between the gray value and the color combination can be determined according to the mapping relationship between the depth value and the color combination and the mapping relationship between the depth value and the gray value. .

FIG. 6 is a schematic diagram showing a relationship between a depth value and a gray value and a color combination in the present application. As shown in FIG. 6, a combination of a depth value and three colors, that is, a combination of red, green, and blue has a mapping. Relationship, where the depth value is between [500, 1500], it is mapped to a single red, and the gray value of the red color component changes with the depth value. When the depth value is between [1500, 2500], its mapping It is a combination of red and green, and the color components of red and green vary with depth values, and so on. The visible depth value is in the full range of [500, 5000], and will correspond to the gradient of "red-green-blue" depending on the depth value. When the depth value has been recorded as 0 or the default value outside the range of [500, 5000], it can be directly mapped to 0, that is, the various color components are all zero, and the gray value is 0, which corresponds to black. The mapping relationship between the gray value and the color combination can be further determined by the mapping relationship between the depth value of the abscissa and the gray value. It can be seen that when the gray value changes between [0, 255], it will correspond to the gradient color of "blue-green-red".

In fact, the depth value may have a linear or non-linear, continuous or discrete mapping relationship with two or more colors, and the various colors are not limited to three colors of blue, red, and green, and may also be orange, yellow, cyan, Purple or pink, etc., or a combination of these colors and black or white. Therefore, the gray value can also be combined with different colors to form different mapping modes.

In some embodiments, before the step 302, the method further includes: acquiring current user information, in step 302, determining a mapping relationship between each depth value and a color combination according to the user information, and according to the depth values, The mapping relationship of the color combinations determines a mapping relationship between each gray value and a color combination.

After the AR and the color combination are mapped and set in advance, after the user wears the AR glasses, the user can identify and obtain the user information of the current user, and match the corresponding depth value and color combination. The mapping relationship further determines the mapping relationship between the user-specific gray value and the color combination according to the mapping relationship between the user-specific depth value and the color combination, and performs subsequent pseudo color conversion processing.

It should be noted that the implementation steps of the foregoing

steps

301 and 302 are not limited, that is, as long as the depth data of the environment is collected before step 303, a grayscale image is generated, and the mapping relationship between the depth values and the color combination is determined, and the mapping relationship is also determined. The mapping relationship between each gray value and color combination is sufficient.

In step 303, a pseudo color image of the environment is generated based on a grayscale map generated by the depth data and a mapping relationship between the grayscale values and color combinations.

The gray value in the grayscale image generated according to the depth data of the collected environment is converted into the color corresponding to the grayscale value according to the mapping relationship between the grayscale value and the color combination, and the corresponding conversion completes the grayscale image of the current environment. After the corresponding color at the position, the pseudo color image of the current environment is obtained from the gray image, and the various colors in the pseudo color image correspond to the distance of the object to the depth sensor of the user in the environment, if the pseudo color image Black in somewhere may indicate that the distance is beyond the depth sensor.

In the existing image processing technology, especially in a user wearable device having a binocular camera, there may already be some intermediate steps of environmental image processing to obtain a grayscale image generated based on depth information, this embodiment The mapping relationship between the gray value and the color combination can be obtained by the mapping relationship between the depth value and the color combination, and the gray image generated by these intermediate steps can be directly used for pseudo color processing.

For the implementation of step 304, reference may be made to the descriptions of step 104 and step 204 in the first embodiment and the second embodiment. In step 304, the pseudo color image of the environment is superimposed with the image of the environment.

In some embodiments, step 302 and step 303 of this embodiment may be:

Step 302: Determine a mapping relationship between each depth value and a color combination in a preset range, and determine each gray value and color in the corresponding range according to the mapping relationship between each depth value and the color combination in the preset range. Combined mapping relationship;

Step 303: Generate a pseudo color image of the environment according to a grayscale map generated by the depth data and a mapping relationship between each grayscale value and a color combination in the corresponding range.

That is, in step 302, the mapping relationship between each depth value and the color combination within the preset range is first determined, where the preset range falls within the limit collection range of the depth data and is smaller than the limit collection range of the depth data.

The mapping relationship between the depth values and the color combinations in the preset range is similar to the mapping relationship between the depth values and the color combinations in the entire range. Refer to FIG. 4 . It can be understood that when the mapping manner with the color combination is determined, the smaller the preset range of the depth value is, the more the objects of different distances in the range can be more finely distinguished by different colors. When the user performs close-range fine operations, the depth range can be adjusted even smaller, so that low-vision users can accurately distinguish the subtle distance differences of objects within a certain distance range.

After determining the mapping relationship between the depth values and the color combinations in the preset range, the mapping between the depth values and the gray values when the grayscale image is generated may be determined, and the corresponding mappings of the depth values in the preset range may be determined. Each gray value in the range further determines a mapping relationship between each gray value and a color combination in the corresponding range. For example, when the preset range of the depth value is [1000, 2000], the gray value in the corresponding range is between 220-160.

Similarly, each depth value within a preset range, that is, each gray value in the corresponding range may have a linear or non-linear, continuous or discrete mapping relationship with two or more colors, and various colors are not limited to blue. , red and green, can also be orange, yellow, cyan, purple or pink, or a combination of these colors and black or white.

Correspondingly, in step 303, a pseudo color image of the environment is generated according to a grayscale map generated by the depth data and a mapping relationship between each grayscale value and a color combination in the corresponding range. The gray value in the grayscale image generated according to the depth data of the collected environment is converted into the corresponding grayscale value according to the mapping relationship, and the corresponding conversion completes the grayscale image of the current environment. After the corresponding color at each position, the pseudo color image of the current environment is obtained from the grayscale image. Since there is no correspondence between each gray value outside the corresponding range and the color combination, it is usually recorded as black.

It can be seen that in this embodiment, the image in the gray scale range is actually processed according to the preset depth value range. When the mapping manner with the color combination is determined, the depth value is pre-predicted. The smaller the range is, the smaller the gray value in the corresponding range is, and the more the objects of different distances in the range can be more finely distinguished by different colors when processing the grayscale image. When the user performs close-range fine operations, the depth range can be adjusted even smaller, so that low-vision users can accurately distinguish the subtle distance differences of objects within a certain distance range.

In this embodiment, a grayscale image is generated according to depth data of the environment, and a pseudo color image is obtained based on a mapping relationship between the grayscale value and the color combination, and the pseudo color image is superimposed with the current environment image to be enhanced and displayed. Grayscale images obtained by some intermediate steps of the image processing scheme are easy to combine with existing image processing techniques, and can fully utilize the residual vision of patients with low vision and assist them without affecting the field of vision of patients with low vision. Effectively perceive the distance of objects in the environment; in addition, it can focus on the grayscale part of the distance range part that the user is most concerned about. When the mapping mode of the color combination is determined, the smaller the preset depth value range needs to be processed. The grayscale portion is about small, and the more diverse objects in the range can be more finely distinguished by different colors.

Embodiment 4:

7a-7d are schematic diagrams showing four implementation scenarios in the fourth embodiment of the present application, wherein the user wears AR glasses, and the AR glasses have depth sensors. When the user looks down at the front, the solid line drawn by the depth sensor is The upper and lower field of view of the user AR glasses, the dashed line is the range of distances that the depth sensor can effectively detect. The depth values of objects outside the maximum recognition distance are both 0 or default values. The right side of each drawing is a schematic diagram of a pseudo color image superimposed for the user in the AR glasses in each scene.

Figure 7a shows a schematic diagram of an implementation scenario of the present application. As shown in Figure 7a, the front of the user is flat and unobstructed. Taking the mapping relationship between the depth value and the color combination in FIG. 2 as an example, the patient sees a small amount of green and a small amount of red synthesized color superimposed on the actual environment image at the lower edge A; the following is seen at the B near the middle of the field of view. Blue is superimposed on the actual environment image; above B in the middle of the field of view, the superimposed pseudo color is black because the depth detection is exceeded, that is, the partial area on the AR glasses is not processed and is transparent.

The user can recognize the change in the distance of the front flat ground by the color change between A and B in the field of view.

FIG. 7b shows a schematic diagram of another implementation scenario of the present application. As shown in FIG. 7b, the front of the user is a raised step. Taking the mapping relationship between the depth value and the color combination in FIG. 2 as an example, the patient sees a small amount of green and a small amount of red synthesized color superimposed on the actual environment image at the lower edge A; the following is seen below the upper portion of the field of view B. Blue is superimposed to the actual environment image; between the A and B of the field of view, the user can distinguish the dividing lines C and D, that is, the outline of the step, because the color changes from green to blue between AC and DB, and between CDs The color changes from blue to green. Above B in the field of view, the superimposed pseudo color is black because the depth detection is exceeded, that is, the partial area on the AR glasses is not processed and is transparent.

The user can identify the position and distance of the front obstacle well by the higher color boundary line B in the field of view and the color change between A and B.

FIG. 7c is a schematic diagram showing another implementation scenario of the present application. As shown in FIG. 7c, the front of the user is a descending step. Taking the mapping relationship between the depth value and the color combination in FIG. 2 as an example, the patient sees a small amount of green and a small amount of red synthesized color superimposed on the actual environment image at the lower edge A; the following is seen below the lower portion of the field of view B. A small amount of green and a small amount of blue are superimposed on the actual environment image; since the edge of the step does not reach the farthest distance, the display of B in the field of view has not yet reached the dark blue, and above B, the superposition of the superposition is exceeded because of the depth detection. The color is black, that is, the part of the area on the AR glasses is not processed and is transparent.

It can be seen that in this case, the pseudo color boundary displayed by the superposition is low, and at the boundary, it is directly changed from blue-green to colorless. After the user is trained, it can know that the distance value corresponding to the blue is missing according to the color mapping relationship. The object at the location, so there may be a lower step at this location.

FIG. 7d shows a schematic diagram of another implementation scenario of the present application. As shown in FIG. 7d, the front of the user is a descending step. In this implementation scenario, the user presets an object whose range of interest is 2m-4m, that is, within the distance range marked by the shadow in the figure, and the depth value and color in the preset range in FIG. 4 Taking the combined mapping relationship as an example, the patient sees red at the lower edge A superimposed to the actual environment image; below the lower portion of the field of view, B sees the blue overlay to the actual environment image; the AB basic realization in the field of view The gradient of the full color combination of "red-green-blue"; above B, the superimposed pseudo color is black because the depth detection is exceeded, that is, the partial area on the AR glasses is not processed, and is transparent. ,

It can be seen that in this case, more colors can be displayed in a smaller field of view for the user to distinguish. When the user advances, the blue display is missing because the front step edge does not reach the farthest value of the distance range that the user pays attention to, so that the user can more clearly perceive some missing in the 3 main colors and pay attention to the road ahead.

In the scene shown in FIG. 7c, the pseudo color superimposed in the user's field of view is mainly green, and it is easy to understand that when the distance range of the smaller user is preset, the display can be more abundant in the same field of view. The color is more convenient for patients with low vision to distinguish.

Embodiment 5:

Based on the same inventive concept, a display device is also provided in the embodiment of the present application. Since the principle of solving the problem is similar to the display method, the implementation of the device may refer to the implementation of the method, and the repeated description is not repeated. As shown in FIG. 8, the display device 500 includes:

The data collection module 501 is configured to collect depth data of the environment;

a color mapping module 502, configured to determine a mapping relationship between each depth value and a color combination;

The pseudo color image generating module 503 is configured to generate a pseudo color image of the environment according to the depth data and the mapping relationship;

The display module 504 is configured to superimpose and display the pseudo color image and the image of the environment.

In some implementations, the color mapping module 502 is configured to determine a mapping relationship between each depth value and a color combination within a preset range;

The pseudo color image generating module 503 is configured to generate a pseudo color image of the environment according to the depth data of the environment and the mapping relationship within a preset range.

In some embodiments, the apparatus 500 further includes:

a grayscale map generating module 505, configured to generate a grayscale map according to the depth data of the environment;

The color mapping module 502 is further configured to determine a mapping relationship between each gray value and a color combination according to a mapping relationship between the depth values and the color combination;

The pseudo color image generation module 503 is configured to generate a pseudo color image of the environment according to a grayscale map generated by the depth data and a mapping relationship between the grayscale values and color combinations.

In some embodiments, the color mapping module 502 is configured to determine a mapping relationship between each depth value and a color combination within a preset range; and, according to the depth values and the color within the preset range. The combined mapping relationship determines a mapping relationship between each gray value and a color combination in the corresponding range;

The pseudo color image generation module 503 is configured to generate a pseudo color image of the environment according to a grayscale map generated by the depth data and a mapping relationship between each grayscale value and a color combination in the corresponding range.

In some embodiments, the apparatus 500 further includes:

The user information obtaining module 506 is configured to acquire current user information.

The color mapping module 502 is configured to determine a mapping relationship between each depth value and a color combination according to the user information.

Example 6:

Based on the same inventive concept, an electronic device is also provided in the embodiment of the present application. Since the principle is similar to the display method, the implementation of the method may refer to the implementation of the method, and the repeated description is not repeated. As shown in FIG. 9, the electronic device 600 includes: a memory 601, one or more processors 602; and one or more modules, the one or more modules being stored in the memory and configured to Executed by the one or more processors, the one or more modules include instructions for performing the various steps of any of the above methods.

Example 7:

Based on the same inventive concept, an embodiment of the present application further provides a computer program product for use in combination with an electronic device, the computer program product comprising a computer program embedded in a computer readable storage medium, the computer program comprising An instruction to cause the electronic device to perform each of the steps of any of the above methods.

For the convenience of description, the various parts of the above-described apparatus are separately described by functions into various modules. Of course, the functions of each module or unit may be implemented in the same software or hardware in the implementation of the present application.

Those skilled in the art will appreciate that embodiments of the present application can be provided as a method, system, or computer program product. Thus, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment in combination of software and hardware. Moreover, the application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) including computer usable program code.

The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (system), and computer program products according to embodiments of the present application. It will be understood that each flow and/or block of the flowchart and/or block diagrams, and combinations of flows and/or blocks in the flowcharts and/or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing device to produce a machine for the execution of instructions for execution by a processor of a computer or other programmable data processing device. Means for implementing the functions specified in one or more of the flow or in a block or blocks of the flow chart.

The computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture comprising the instruction device. The apparatus implements the functions specified in one or more blocks of a flow or a flow and/or block diagram of the flowchart.

These computer program instructions can also be loaded onto a computer or other programmable data processing device such that a series of operational steps are performed on a computer or other programmable device to produce computer-implemented processing for execution on a computer or other programmable device. The instructions provide steps for implementing the functions specified in one or more of the flow or in a block or blocks of a flow diagram.

While the preferred embodiment of the present application has been described, it will be apparent that those skilled in the art can make further changes and modifications to the embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and the modifications and

Claims

A display method, characterized in that the method comprises:

Collect depth data of the environment;

Determining a mapping relationship between each depth value and a color combination;

Generating a pseudo color image of the environment according to the depth data and the mapping relationship;

The pseudo color image is superimposed on the image of the environment.
The method of claim 1 wherein

The determining a mapping relationship between each depth value and a color combination includes:

Determining a mapping relationship between each depth value and a color combination within a preset range;

The generating a pseudo color image of the environment according to the depth data and the mapping relationship includes:

A pseudo color image of the environment is generated based on depth data of the environment within a preset range and the mapping relationship.
The method of claim 1 wherein

Before the generating the pseudo color image of the environment according to the depth data and the mapping relationship, the method further includes:

Generating a grayscale map according to depth data of the environment;

Determining, according to a mapping relationship between each depth value and the color combination, a mapping relationship between each gray value and a color combination;

The generating a pseudo color image of the environment according to the depth data and the mapping relationship includes:

A pseudo color image of the environment is generated based on a grayscale map generated by the depth data and a mapping relationship between the respective grayscale values and a color combination.
The method of claim 3 wherein:

The determining a mapping relationship between each depth value and a color combination includes:

Determining a mapping relationship between each depth value and a color combination within a preset range;

And determining a mapping relationship between each gray value and a color combination according to the mapping relationship between the depth values and the color combination;

Determining, according to the mapping relationship between each depth value and the color combination in the preset range, a mapping relationship between each gray value and a color combination in the corresponding range;

And generating, according to the grayscale image generated by the depth data and the mapping relationship between the grayscale values and the color combination, generating the pseudo color image of the environment includes:

A pseudo color image of the environment is generated based on a grayscale map generated by the depth data and a mapping relationship between each grayscale value and color combination in the corresponding range.
The method according to any one of claims 1 to 4, further comprising: before the determining the mapping relationship between the depth values and the color combination, the method further comprising:

Obtain current user information;

The determining a mapping relationship between each depth value and a color combination includes:

A mapping relationship between each depth value and a color combination is determined according to the user information.
A display device, characterized in that the device comprises:

a data acquisition module for collecting depth data of the environment;

a color mapping module, configured to determine a mapping relationship between each depth value and a color combination;

a pseudo color image generating module, configured to generate a pseudo color image of the environment according to the depth data and the mapping relationship;

And a display module, configured to superimpose and display the pseudo color image and the image of the environment.
The device of claim 6 wherein:

The color mapping module is configured to determine a mapping relationship between each depth value and a color combination within a preset range;

The pseudo color image generating module is configured to generate a pseudo color image of the environment according to the depth data of the environment and the mapping relationship within a preset range.
The device of claim 6 wherein said device further comprises:

a grayscale graph generating module, configured to generate a grayscale map according to the depth data of the environment;

The color mapping module is further configured to determine a mapping relationship between each gray value and a color combination according to a mapping relationship between the depth values and the color combination;

The pseudo color image generating module is configured to generate a pseudo color image of the environment according to a grayscale map generated by the depth data and a mapping relationship between the grayscale values and a color combination.
The device of claim 8 wherein:

The color mapping module is configured to determine a mapping relationship between each depth value and a color combination within a preset range; and

Determining, according to the mapping relationship between each depth value and the color combination in the preset range, a mapping relationship between each gray value and a color combination in the corresponding range;

The pseudo color image generating module is configured to generate a pseudo color image of the environment according to a grayscale map generated by the depth data and a mapping relationship between each grayscale value and a color combination in the corresponding range.
The device according to any one of claims 6 to 9, wherein the device further comprises:

a user information obtaining module, configured to acquire current user information;

The color mapping module is configured to determine a mapping relationship between each depth value and a color combination according to the user information.
An electronic device, comprising:

a memory, one or more processors; and one or more modules, the one or more modules being stored in the memory and configured to be executed by the one or more processors, the one or The plurality of modules includes instructions for performing the various steps of the method of any of claims 1 to 5.
A computer program product for use in conjunction with an electronic device, the computer program product comprising a computer program embedded in a computer readable storage medium, the computer program comprising means for causing the electronic device to perform claims 1 to 5 The instructions of the various steps in any of the methods described.