WO2021110035A1

WO2021110035A1 - Eye positioning apparatus and method, and 3d display device, method and terminal

Info

Publication number: WO2021110035A1
Application number: PCT/CN2020/133329
Authority: WO
Inventors: 刁鸿浩; 黄玲溪
Original assignee: 北京芯海视界三维科技有限公司; 视觉技术创投私人有限公司
Priority date: 2019-12-05
Filing date: 2020-12-02
Publication date: 2021-06-10
Also published as: CN112929639A; TW202123693A

Abstract

The present application discloses an eye positioning apparatus, comprising: an eye positioner comprising a black-and-white camera configured to shoot a black-and-white image of a user's face and a depth of field acquisition apparatus configured to acquire depth of field information of the face; and an eye positioning image processor configured to determine a spatial position of an eye on the basis of the black-and-white image and the acquired depth of field information. The eye positioning apparatus can accurately determine the spatial position of user's eyes, thereby improving 3D display quality. The present application further discloses an eye positioning method, a 3D display device, a 3D display method, a 3D display terminal, a computer readable storage medium and a computer program product.

Description

Eye positioning device, method and 3D display device, method and terminal

This application requires the priority of a Chinese patent application filed with the China Intellectual Property Office, the application number is 201911231165.9, and the invention title is "eye tracking device, method, and 3D display device, method, and terminal" on December 5, 2019, all of which The content is incorporated in this application by reference.

Technical field

This application relates to 3D display technology, such as eye positioning devices, methods, and 3D display equipment, methods, and terminals.

Background technique

In some conventional face or eye positioning devices, only the distance between the face and the screen is detected, and a preset or default interpupillary distance is used to determine the eye position. The accuracy of such recognition is not high, which may cause errors in the calculation of the viewpoint, which cannot meet the requirements of high-quality 3D display.

This background technology is only for facilitating the understanding of related technologies in the field, and is not regarded as an acknowledgement of the prior art.

Summary of the invention

In order to have a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. This summary is not a general comment, nor is it intended to determine key/important components or describe the scope of protection of these embodiments, but serves as a prelude to the detailed description that follows.

The embodiments of the present application intend to provide eye positioning devices, methods, 3D display devices, methods and terminals, computer-readable storage media, and computer program products.

In one solution, an eye positioning device is provided, including: an eye locator, including a black and white camera configured to capture a black and white image of a user's face, and a depth acquisition device configured to acquire depth information of the face ; The eye positioning image processor is configured to determine the spatial position of the eye based on the black and white image and the depth of field information.

With this kind of eye positioning device, the spatial position of the user's eyes can be determined with high precision, thereby being able to provide a 3D display image of the display object that matches the spatial position of the user's eyes, improving the 3D display quality and enhancing the viewing experience. Based on the actual spatial position of the user's eyes, the viewpoint position of the user's eyes can be determined, so as to provide the user with a more accurate 3D display with a higher degree of freedom.

In some embodiments, the eye positioning image processor is further configured to recognize the presence of eyes based on the black and white image.

In some embodiments, the eye positioning device includes an eye positioning data interface configured to transmit eye spatial position information including the spatial position of the eye.

In some embodiments, the depth-of-field acquisition device is a structured light camera or a TOF camera.

In some embodiments, the eye positioning device further includes a viewing angle determining device configured to calculate the user's viewing angle with respect to the 3D display device.

According to the viewing angle, 3D display images of the display object viewed from different angles can be generated in a follow-up manner, so that the user can watch the 3D display images consistent with the viewing angle, and enhance the realism and immersion of the 3D display.

In some embodiments, the black and white camera is configured to capture a sequence of black and white images.

In some embodiments, the eye positioning image processor includes: a buffer configured to buffer multiple black-and-white images in a black-and-white image sequence; a comparator configured to compare multiple black-and-white images before and after the black-and-white image sequence; Configured so that when the comparator does not recognize the presence of eyes in the current black-and-white image in the black-and-white image sequence and recognizes the presence of eyes in the previous or subsequent black-and-white images, it will be based on the previous or subsequent black-and-white images The eye space position information determined by the acquired depth information is used as the current eye space position information.

Based on this, for example, when a black-and-white camera is stuck or frame skipping, it can provide users with a more consistent display screen to ensure a viewing experience.

In one solution, a 3D display device is provided, including: a multi-viewpoint 3D display screen, including a plurality of sub-pixels corresponding to a plurality of viewpoints; the eye positioning device as described above, is configured to determine the spatial position of a user's eyes And a 3D processing device configured to determine the viewpoint according to the spatial position of the user's eyes, and render sub-pixels corresponding to the viewpoint based on the 3D signal.

In some embodiments, the multi-view 3D display screen includes a plurality of composite pixels, each of the plurality of composite pixels includes a plurality of composite sub-pixels, and each composite sub-pixel of the plurality of composite sub-pixels is corresponding to a plurality of composite sub-pixels. It is composed of multiple sub-pixels for each viewpoint.

In some embodiments, the 3D processing device and the eye positioning device are communicatively connected through an eye positioning data interface.

In some embodiments, the 3D display device further includes a 3D photographing device configured to capture 3D images, and the 3D photographing device includes a depth-of-field camera and at least two color cameras.

In some embodiments, the eye positioning device is integrated with the 3D camera.

In some embodiments, the 3D camera is placed in front of the 3D display device.

In one solution, an eye positioning method is provided, including: taking a black and white image of a user's face; acquiring depth information of the face; and determining the spatial position of the eye based on the black and white image and the depth information.

In some embodiments, the eye positioning method further includes: recognizing the presence of the eye based on the black and white image.

In some embodiments, the eye positioning method further includes: transmitting eye spatial position information including the spatial position of the eye.

In some embodiments, the eye positioning method further includes: photographing a black-and-white image sequence including black-and-white images.

In some embodiments, the eye positioning method further includes: buffering multiple black-and-white images in a black-and-white image sequence; comparing multiple black-and-white images before and after the black-and-white image sequence; when the current black-and-white image in the black-and-white image sequence is not recognized by comparing When the eye exists and the existence of the eye is recognized in the previous or subsequent black and white images, the eye spatial position information determined based on the previous or subsequent black and white image and the acquired depth information is used as the current eye spatial position information.

In one solution, a 3D display method is provided, including: determining the spatial position of a user's eyes; determining a viewpoint according to the spatial position of the user's eyes, and rendering sub-pixels corresponding to the viewpoint based on a 3D signal; wherein the 3D display device includes The multi-viewpoint 3D display screen includes multiple sub-pixels corresponding to multiple viewpoints.

In one solution, a 3D display terminal is provided, including a processor, a memory storing program instructions, and a multi-view 3D display screen. The processor is configured to execute the 3D display method described above when executing the program instructions.

The computer-readable storage medium provided by the embodiment of the present disclosure stores computer-executable instructions, and the above-mentioned computer-executable instructions are configured to execute the above-mentioned eye positioning method and 3D display method.

The computer program product provided by the embodiments of the present disclosure includes a computer program stored on a computer-readable storage medium. The above-mentioned computer program includes program instructions. When the program instructions are executed by a computer, the above-mentioned computer executes the above-mentioned eye positioning method, 3D display method.

The above general description and the following description are only exemplary and explanatory, and are not used to limit the application.

Description of the drawings

One or more embodiments are exemplified by the accompanying drawings. These exemplified descriptions and drawings do not constitute a limitation on the embodiments, and the drawings do not constitute a scale limitation, and among them:

Fig. 1 is a schematic diagram of an eye positioning device according to an embodiment of the present disclosure;

2A and 2B are schematic diagrams of a 3D display device according to an embodiment of the present disclosure;

FIG. 3 is a schematic diagram of using an eye positioning device according to an embodiment of the present disclosure to determine the spatial position of an eye;

4 is a schematic diagram of the steps of an eye positioning method according to an embodiment of the present disclosure;

Fig. 5 is a schematic diagram of steps of an eye positioning method according to an embodiment of the present disclosure;

Fig. 6 is a schematic diagram of steps of an eye positioning method according to an embodiment of the present disclosure;

FIG. 7 is a schematic diagram of the steps of a 3D display method according to an embodiment of the present disclosure;

Fig. 8 is a schematic structural diagram of a 3D display terminal according to an embodiment of the present disclosure.

Reference signs:

100: 3D display device; 101: processor; 122: register; 110: multi-view 3D display screen; 120: 3D camera; 121: camera unit; 121a: first color camera; 121b: second color camera; 121c: Depth-of-field camera; 125: 3D image output interface; 126: 3D image processor; 130: 3D processing device; 131: buffer; 140: signal interface; 150: eye positioning device; 151: eye positioning device; 151a: black and white Camera; 151b: Depth of Field Acquisition Device; 152: Eye Positioning Image Processor; 155: Viewing Angle Determining Device; 156: Buffer; 157: Comparator; 153: Eye Positioning Data Interface; FP: Focal Plane; O: Lens Center ; F: focal length; MCP: black and white camera plane; R: user’s right eye; L: user’s left eye; P: interpupillary distance; XR: X coordinate of the user’s right eye imaging in the focal plane; XL: user’s The X-axis coordinates of the left eye imaging in the focal plane; βR: tilt angle; βL: tilt angle; α: included angle; DR: the distance of the user's right eye R relative to the black and white camera plane MCP; DL: the user's left eye L The distance relative to the MCP of the black and white camera plane; DLP: the display plane; DLC: the center of the display; HFP: the plane where the face is located; 800: 3D display terminal; 810: multi-view 3D display; 811: memory; 812: communication interface ; 813: bus; 814: processor.

Detailed ways

In order to have a more detailed understanding of the features and technical content of the embodiments of the present disclosure, the implementation of the embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. The attached drawings are for reference only and are not used to limit the embodiments of the present disclosure.

In an embodiment of the present disclosure, there is provided an eye positioning device configured to be used in a 3D display device. The eye positioning device includes: an eye locator, including a black-and-white camera configured to take a black-and-white image, and a black-and-white camera configured to take a black-and-white image. A depth-of-field acquisition device for acquiring depth-of-field information; an eye positioning image processor configured to recognize the presence of the eye based on the black and white image and determine the spatial position of the eye based on the black-and-white image and the acquired depth information. Such an eye positioning device is exemplarily shown in FIG. 1.

In an embodiment of the present disclosure, a 3D display device is provided, including: a multi-viewpoint 3D display screen (for example, a multi-viewpoint naked eye 3D display screen), including a plurality of sub-pixels corresponding to a plurality of viewpoints; a 3D processing device configured to Rendering the sub-pixels corresponding to the viewpoint based on the 3D signal; wherein the viewpoint is determined by the spatial position of the user's eyes; and according to the eye positioning device described above.

By way of explanation and not limitation, the determination of the viewpoint by the spatial position of the eye may be implemented by a 3D processing device, or may be implemented by an eye positioning image processor of an eye positioning device.

In some embodiments, the 3D processing device is communicatively connected with the multi-view 3D display screen.

In some embodiments, the 3D processing device is communicatively connected with the driving device of the multi-view 3D display screen.

In an embodiment of the present disclosure, an eye positioning method is provided, including: taking a black and white image; acquiring depth information; recognizing the presence of the eye based on the black and white image; and determining the spatial position of the eye based on the black and white image and the acquired depth information.

In an embodiment of the present disclosure, a 3D display method is provided, which is suitable for a 3D display device. The 3D display device includes a multi-viewpoint 3D display screen, including multiple sub-pixels corresponding to multiple viewpoints; the 3D display method includes: transmitting a 3D signal; Use the eye positioning method described above to determine the spatial position of the user's eyes; determine the viewpoint where the eye is based on the spatial position of the eye; and render the sub-pixels corresponding to the viewpoint based on the 3D signal.

FIG. 2A shows a schematic diagram of a 3D display device 100 according to an embodiment of the present disclosure. 2A, in an embodiment of the present disclosure, a 3D display device 100 is provided, including a multi-view 3D display screen 110, a signal interface 140 configured to receive a video frame of a 3D signal, and a 3D processing device communicatively connected to the signal interface 140 The device 130 and the eye positioning device 150. The eye positioning device 150 is communicatively connected to the 3D processing device 130, so that the 3D processing device 130 can directly receive eye positioning data.

In some embodiments, the 3D processing device is configured to determine the viewpoint of the user's eyes from the spatial position of the eyes. In other embodiments, the determination of the viewpoint of the user's eyes from the spatial position of the eyes can also be achieved by an eye positioning device, and the 3D processing device receives the eye positioning data including the viewpoint.

By way of explanation and not limitation, the eye positioning data may include the spatial position of the eyes, such as the distance between the user's eyes and the multi-viewpoint 3D display screen, the viewpoint where the user's eyes are located, and the user's perspective.

The multi-view 3D display screen 110 may include a display panel and a grating (not labeled) covering the display panel. In the embodiment shown in FIG. 2A, the multi-view 3D display screen 110 may include m columns and n rows, that is, m×n composite pixels, thereby defining a display resolution of m×n.

In some embodiments, the resolution of m×n may be a resolution above Full High Definition (FHD), including but not limited to 1920×1080, 1920×1200, 2048×1280, 2560×1440, 3840×2160, etc.

By way of explanation and not limitation, each composite pixel includes a plurality of composite sub-pixels, and each composite sub-pixel is composed of i sub-pixels of the same color corresponding to i viewpoints, i≥3. In the embodiment shown in FIG. 2A, i=6, but it is conceivable that i is another value. In the illustrated embodiment, the multi-viewpoint 3D display screen may correspondingly have i (i=6) viewpoints (V1-V6), but it is conceivable that there may be more or fewer viewpoints correspondingly.

By way of explanation and not limitation, in the embodiment shown in FIG. 2A, each composite pixel includes three composite sub-pixels, and each composite sub-pixel is composed of 6 same-color sub-pixels corresponding to 6 viewpoints (i=6). Pixel composition. The three composite sub-pixels respectively correspond to three colors, namely red (R), green (G) and blue (B). In the embodiment shown in FIG. 2A, the three composite sub-pixels in each composite pixel are arranged in a single column, and the six sub-pixels of each composite sub-pixel are arranged in a single row. However, it is conceivable that multiple composite sub-pixels in each composite pixel are arranged in different forms; it is also conceivable that multiple sub-pixels in each composite sub-pixel are arranged in different forms.

By way of explanation and not limitation, for example, as shown in FIG. 2A, the 3D display apparatus 100 may be provided with a single 3D processing device 130. The single 3D processing device 130 simultaneously processes the rendering of the sub-pixels of each composite pixel of each composite pixel of the multi-view 3D display screen 110. In other embodiments, the 3D display device 100 may also be provided with more than one 3D processing device 130, which process the composite sub-pixels of the composite pixels of the multi-view 3D display screen 110 in parallel, serial, or a combination of series and parallel. Rendering of sub-pixels. Those skilled in the art will understand that more than one 3D processing device can be allocated in other ways and process multiple rows and multiple columns of composite pixels or composite sub-pixels of the multi-view 3D display screen 110 in parallel, which falls within the scope of the embodiments of the present disclosure.

In some embodiments, the 3D processing device 130 may also optionally include a buffer 131 to buffer the received video frames.

In some embodiments, the 3D processing device is an FPGA or ASIC chip or FPGA or ASIC chipset.

Continuing to refer to FIG. 2A, the 3D display device 100 may further include a processor 101 communicatively connected to the 3D processing device 130 through the signal interface 140. In some embodiments shown herein, the processor 101 is included in a computer or a smart terminal, such as a mobile terminal, or as a processor unit thereof. However, it is conceivable that, in some embodiments, the processor 101 may be arranged outside the 3D display device. For example, the 3D display device may be a multi-view 3D display with a 3D processing device, such as a non-intelligent 3D TV.

For the sake of simplicity, the following exemplary embodiments of the 3D display device include a processor inside. Based on this, the signal interface 140 is an internal interface connecting the processor 101 and the 3D processing device 130. In some embodiments shown herein, the signal interface as the internal interface of the 3D display device may be MIPI, mini-MIPI interface, LVDS interface, min-LVDS interface or Display Port interface. In some embodiments, as shown in FIG. 2A, the processor 101 of the 3D display device 100 may include a register 122. The register 122 can be configured to temporarily store instructions, data, and addresses. In some embodiments, the register 122 may be configured to receive information about the display requirements of the multi-view 3D display screen 110

In some embodiments, the 3D display device 100 may further include a codec configured to decompress and encode and decode the compressed 3D signal and send the decompressed 3D signal to the 3D processing device 130 via the signal interface 140.

Referring to FIG. 2B, the 3D display device 100 further includes a 3D photographing device 120 configured to capture 3D images. The eye positioning device 150 is integrated in the 3D photographing device 120, or it is conceivable to be integrated into a conventional photographing device of a processing terminal or a display device. . In the illustrated embodiment, the 3D camera 120 is a front camera. The 3D photographing device 120 includes a camera unit 121, a 3D image processor 126, and a 3D image output interface 125.

As shown in FIG. 2B, the camera unit 121 includes a first color camera 121a, a second color camera 121b, and a depth camera 121c. In other embodiments, the 3D image processor 126 may be integrated in the camera unit 121. In some embodiments, the first color camera 121a is configured to obtain a first color image of the subject, and the second color camera 121b is configured to obtain a second color image of the subject, and the intermediate point is obtained by combining the two color images. The composite color image; the depth-of-field camera 121c is configured to obtain depth information of the subject. The synthesized color image and depth information obtained by synthesis form a 3D image. In the embodiment of the present disclosure, the first color camera and the second color camera are the same color camera. In other embodiments, the first color camera and the second color camera may also be different color cameras. In this case, in order to obtain a composite color image, the first color image and the second color image can be calibrated or corrected. The depth-of-field camera 121c may be a TOF (time of flight) camera or a structured light camera. The depth camera 121c may be arranged between the first color camera and the second color camera.

In some embodiments, the 3D image processor 126 is configured to synthesize the first color image and the second color image into a synthetic color image, and synthesize the obtained synthetic color image and depth information into a 3D image. The formed 3D image is transmitted to the processor 101 of the 3D display device 100 through the 3D image output interface 125.

Optionally, the first color image, the second color image, and the depth information are directly transmitted to the processor 101 of the 3D display device 100 via the 3D image output interface 125, and the processor 101 performs the aforementioned synthesis of the two color images and forms the 3D image Wait for processing.

Optionally, the 3D image output interface 125 may also be communicatively connected to the 3D processing device 130 of the 3D display device 100, so that the 3D processing device 130 can perform processing such as synthesizing color images and forming 3D images.

In some embodiments, at least one of the first color camera and the second color camera is a wide-angle color camera.

Continuing to refer to FIG. 2B, the eye positioning device 150 is integrated in the 3D photographing device 120 and includes an eye locator 151, an eye positioning image processor 152 and an eye positioning data interface 153.

The eye locator 151 includes a black-and-white camera 151a and a depth-of-field acquisition device 151b. The black-and-white camera 151a is configured to capture black-and-white images, and the depth-of-field acquisition device 151b is configured to acquire depth-of-field information. In the case where the 3D camera 120 is front-facing and the eye positioning device 150 is integrated in the 3D camera 120, the eye positioning device 150 is also front-facing. Then, the subject of the black-and-white camera 151a is the user's face, and the face or eye is recognized based on the black-and-white image captured, and the depth-of-field acquiring device acquires at least the depth information of the eye, and may also acquire the depth information of the face.

In some embodiments, the eye positioning data interface 153 of the eye positioning device 150 is communicatively connected to the 3D processing device 130 of the 3D display device 100, so that the 3D processing device 130 can directly receive the eye positioning data. In other embodiments, the eye positioning image processor 152 may be communicatively connected to the processor 101 of the 3D display device 100, so that the eye positioning data may be transmitted from the processor 101 to the 3D processing through the eye positioning data interface 153装置130。 Device 130.

In some embodiments, the eye positioning device 150 is communicatively connected with the camera unit 121, so that the eye positioning data can be used when shooting 3D images.

Optionally, the eye locator 151 is also provided with an infrared emitting device 154. When the black-and-white camera 151a is working, the infrared emitting device 154 is configured to selectively emit infrared light to supplement the light when the ambient light is insufficient, for example, when shooting at night, so that shooting can also be performed under weak ambient light conditions. Can recognize black and white images of faces and eyes.

In some embodiments, the eye positioning device 150 or the processing terminal or display device integrated with the eye positioning device may be configured to, when the black-and-white camera is working, based on the received light sensing signal, for example, it is detected that the light sensing signal is lower than When the threshold is given, control the opening of the infrared emitting device or adjust its size. In some embodiments, the light sensing signal is received from an ambient light sensor integrated in the processing terminal or the display device.

Optionally, the infrared emitting device 154 is configured to emit infrared light with a wavelength greater than or equal to 1.5 microns, that is, long-wave infrared light. Compared with short-wave infrared light, long-wave infrared light has a weaker ability to penetrate the skin, so it is less harmful to the eyes.

The captured black and white image is transmitted to the eye positioning image processor 152. Exemplarily, the eye positioning image processor is configured to have a visual recognition function, such as a face recognition function, and is configured to recognize a face and eyes based on a black and white image. Based on the identified eyes, the viewing angle of the user relative to the display screen of the display device can be obtained, which will be described below.

The depth information of the eyes or the face acquired by the depth acquisition device 151b is also transmitted to the eye positioning image processor 152. The eye positioning image processor 152 is configured to determine the spatial position of the eye based on the black and white image and the acquired depth information, which will be described below.

In some embodiments, the depth-of-field acquisition device 151b is a structured light camera or a TOF camera.

As an explanation and not limitation, the TOF camera includes a projector and a receiver. The projector transmits light pulses to the observed object, and then receives the light pulses reflected back from the observed object through the receiver, passing the round trip time of the light pulse To calculate the distance between the observed object and the camera.

By way of explanation and not limitation, the structured light camera includes a projector and a collector. The surface structured light, such as coded structured light, is projected onto the observed object through the projector to form a distorted image of the surface structured light on the surface of the observed object. Then, the distorted image is collected and analyzed by the collector, so as to restore the three-dimensional outline and spatial information of the observed object.

In some embodiments, the black and white camera 151a is a wide-angle black and white camera.

In some embodiments, the depth-of-field acquisition device 151b and the depth-of-field camera 121c of the 3D photographing device 120 may be the same. In this case, the depth acquisition device 151b and the depth camera 121c may be the same TOF camera or the same structured light camera. In other embodiments, the depth-of-field acquisition device 151b and the depth-of-field camera 121c may be different.

In some embodiments, the eye positioning device 150 includes a viewing angle determining device 155, which is configured to calculate the user's viewing angle with respect to the 3D display device or its display screen or black and white camera.

Based on the black-and-white image captured by the black-and-white camera 151a, the angle of view includes, but is not limited to, the inclination angle of the user's monocular and black-and-white camera lens center O/display center DLC relative to the black and white camera plane MCP/display plane DLP, and the binocular connection The inclination angle of the connection between the midpoint (center of both eyes) and the black-and-white camera lens center O/display center DLC relative to the black-and-white camera plane MCP/display plane DLP.

On this basis, combined with the depth image acquired by the depth-of-field acquisition device 151b, in addition to the above-mentioned inclination angle, the angle of view can also include the inclination angle of the binocular line with respect to the black and white camera plane MCP/display plane DLP, and the plane of the face HFP with respect to The tilt angle of the black and white camera plane MCP/display plane DLP, etc. Among them, the plane HFP of the face can be determined by extracting several facial features, such as the eyes and ears, the corners of the eyes and the mouth, the eyes and the chin, and so on. In the embodiment of the present disclosure, since the eye positioning device 150 and its black-and-white camera 151a are front-facing with respect to the 3D display device or its display screen, the black-and-white camera plane MCP can be regarded as the display screen plane DLP.

By way of explanation and not limitation, the inclination angle of the line with respect to the plane described above includes but is not limited to the angle between the line and the projection of the line in the plane, the angle between the projection of the line in the plane and the horizontal direction of the plane, The angle between the projection of the line in the plane and the vertical direction of the plane. Wherein, the angle between the line and the projection of the line in the plane may have a horizontal component and a vertical component.

In some embodiments, as shown in FIG. 2B, the viewing angle determining device 155 may be integrated in the eye positioning image processor 152. As described above, the eye positioning image processor 152 is configured to determine the spatial position of the eye based on the black and white image and the depth information. In the embodiments of the present disclosure, the spatial position of the eye includes but is not limited to the above-described angle of view, the distance of the eye relative to the black and white camera plane MCP/display plane DLP, the eye relative to the eye positioning device or its black and white camera /3D display equipment or the spatial coordinates of its display screen, etc. In some embodiments, the eye positioning device 150 may further include a viewing angle data output interface configured to output the viewing angle calculated by the viewing angle determining device.

In other embodiments, the viewing angle determination device may be integrated in the 3D processing device.

By way of explanation and not limitation, the black-and-white image captured by the black-and-white camera 151a including the left and right eyes of the user can be known as the X-axis (horizontal direction ) Coordinates and Y-axis (vertical direction) coordinates. As shown in FIG. 3, taking the lens center O of the black and white camera 151a as the origin, the X axis and the Y axis (not shown) perpendicular to the X axis form a black and white camera plane MCP, which is parallel to the focal plane FP; The axis direction is the Z axis, and the Z axis is also the depth direction. That is to say, in the XZ plane shown in FIG. 3, the X-axis coordinates XR and XL of the left and right eyes imaging in the focal plane FP are known; moreover, the focal length f of the black and white camera 151a is known; In this case, the inclination angle β relative to the X axis of the projection of the line connecting the left eye and the right eye and the black and white camera lens center O in the XZ plane can be calculated, which will be further described below. Similarly, in the YZ plane (not shown), the Y-axis coordinates of the left eye and right eye imaging in the focal plane FP are known, and combined with the known focal length f, the left eye and right eye can be calculated The inclination angle of the projection of the line connecting the lens center O of the black and white camera lens in the YZ plane with respect to the Y axis of the black and white camera plane MCP.

By way of explanation and not limitation, the black-and-white images including the left and right eyes of the user captured by the black-and-white camera 151a and the depth information of the left and right eyes acquired by the depth-of-field acquisition device 151b can be known as the left and right eyes. The spatial coordinates (X, Y, Z) in the coordinate system of the black-and-white camera 151a, where the Z-axis coordinates are the depth information. Based on this, as shown in Figure 3, the angle α between the projection of the line connecting the left eye and the right eye in the XZ plane and the X axis can be calculated. Similarly, in the YZ plane (not shown), the angle between the projection of the line connecting the left eye and the right eye in the YZ plane and the Y axis can be calculated.

FIG. 3 schematically shows a top view of a geometric relationship model that uses a black and white camera 151a and a depth acquisition device 151b (not shown) to determine the spatial position of the eye. Wherein, R and L represent the user's right eye and left eye, respectively, and XR and XL are respectively the X-axis coordinates of the user's right eye R and left eye L in the focal plane FP of the black and white camera 151a. Given the known focal length f of the black and white camera 151a and the X-axis coordinates XR and XL of the eyes in the focal plane FP, it can be concluded that the line connecting the user's right eye R and left eye L to the lens center O is in the XZ plane The inclination angles βR and βL of the projection relative to the X axis are:

On this basis, through the depth information of the right eye R and the left eye L obtained by the depth acquisition device 151b (not shown), it can be known that the user's right eye R and left eye L are relative to the black and white camera plane MCP/screen plane. DLP distance DR and DL. Based on this, it can be obtained that the angle α between the projection of the user's eyes in the XZ plane and the X axis and the interpupillary distance P are respectively:

The above calculation methods and mathematical expressions are only illustrative, and those skilled in the art can think of other calculation methods and mathematical expressions to obtain the desired spatial position of the eye. Those skilled in the art can also think of transforming the coordinate system of the black and white camera with the coordinate system of the display device or its display screen when necessary.

In some embodiments, when the distances DR and DL are not equal and the included angle α is not zero, it can be considered that the user is obliquely looking at the display screen plane DLP; when the distance DR and DL are equal and the viewing angle α is zero, it can be considered that the user is looking up at the display screen. Flat DLP. In other embodiments, a threshold may be set for the included angle α, and when the included angle α does not exceed the threshold, it can be considered that the user is looking up at the plane of the display screen DLP.

In some embodiments, the eye positioning device 150 includes an eye positioning data interface 153 configured to transmit eye spatial position information, including but not limited to the tilt angle, included angle, and spatial coordinates as described above. The use of eye spatial position information can provide users with targeted or customized 3D display screens.

By way of explanation and not limitation, the angle of view, such as the angle between the center of the user’s eyes and the center of the display DLC relative to the horizontal direction (X axis) or the vertical direction (Y axis), is transmitted to the 3D processing device 130. Based on the received viewing angle, the 3D processing device 130 generates a 3D display screen corresponding to the viewing angle, so as to be able to present display objects viewed from different angles to the user.

Exemplarily, based on the angle between the center of the user's eyes and the center of the display screen DLC relative to the horizontal direction (X-axis), the following effect can be presented in the horizontal direction; based on the relationship between the center of the user's eyes and the center of the display screen DLC The angle in the vertical direction (Y-axis) can present a follow-up effect in the vertical direction.

By way of explanation and not limitation, the spatial coordinates of the user’s left and right eyes are transmitted to the 3D processing device 130 through the eye positioning data interface 153. The 3D processing device 130 determines based on the received spatial coordinates that the user’s eyes are located and determined by The multi-viewpoint 3D display screen 110 provides viewpoints, and renders corresponding sub-pixels based on the video frame of the 3D signal.

Exemplarily, when it is determined based on the spatial position information of the eyes that the eyes of the user each correspond to one viewpoint, the video frame based on the 3D signal renders the sub-pixels corresponding to the two viewpoints among the multiple composite sub-pixels of each composite pixel. The sub-pixels corresponding to the viewpoints adjacent to the two viewpoints among the plurality of complex sub-pixels of each complex pixel may be additionally rendered.

Exemplarily, when it is determined based on the spatial position information of the eyes that the eyes of the user are respectively located between two viewpoints, the video frame based on the 3D signal renders the multiple composite sub-pixels of each composite pixel corresponding to the four viewpoints. Sub-pixels.

Exemplarily, when it is determined based on the spatial position information of the eyes that at least one of the eyes of the user has moved, the next video frame of the 3D signal may be rendered based on a new predetermined viewpoint among multiple composite sub-pixels of each composite pixel. The corresponding sub-pixel.

Exemplarily, when it is determined that there is more than one user based on the eye spatial position information, multiple composite sub-pixels of each composite pixel and sub-pixels corresponding to the viewpoints of each user's eyes may be rendered based on the video frame of the 3D signal.

In some embodiments, the user's viewing angle and viewpoint position are determined separately, and a 3D display screen that changes with the viewing angle and viewpoint position is provided accordingly to improve the viewing experience.

In other embodiments, the eye spatial position information can also be directly transmitted to the processor 101 of the 3D display device 100, and the 3D processing device 130 receives/reads the eye spatial position from the processor 101 through the eye positioning data interface 153 information.

In some embodiments, the black-and-white camera 151a is configured to capture a sequence of black-and-white images, which includes a plurality of black-and-white images arranged in time.

In some embodiments, the eye positioning image processor 152 includes a buffer 156 and a comparator 157. The buffer 156 is configured to buffer a plurality of black-and-white images arranged sequentially in time in the black-and-white image sequence. The comparator 157 is configured to compare a plurality of black and white images taken before and after time in the black and white image sequence. By comparison, for example, it can be judged whether the spatial position of the eye has changed or whether the eye is still in the viewing range, and so on.

In some embodiments, the eye localization image processor 152 further includes a judge (not shown) configured to, based on the comparison result of the comparator, the presence of the eye is not recognized in the current black-and-white image in the black-and-white image sequence And when the presence of eyes is recognized in the previous or subsequent black and white images, the eye spatial position information determined based on the previous or subsequent black and white images is used as the current eye spatial position information. In this case, for example, the user briefly turns his head. In this case, the user's face and eyes may not be recognized for a short time.

Exemplarily, there are several black and white images in the black and white image sequence in the buffer segment of the buffer 156. In some cases, the face and eyes cannot be recognized from the current black and white image that is cached, but the face and eyes can be recognized from the black and white images before or after the cache. In this case, the eye spatial position information determined based on the black-and-white image that is later than the current black-and-white image can be used as the current eye spatial position information; it can also be based on the eye-space position information before the current black-and-white image. , That is, the eye space position information determined by taking the black and white image earlier is used as the current eye space position information. In addition, it is also possible to average the spatial position information of the eyes determined based on the black and white images that can recognize the face and eyes before and after the above, perform data fitting, interpolation or other methods, and the obtained The result is the current spatial position information of the eye.

In some embodiments, the black and white camera 151a is configured to capture a sequence of black and white images at a frequency of 24 frames per second or more. Illustratively, the shooting is performed at a frequency of 30 frames per second. Illustratively, shooting is performed at a frequency of 60 frames per second.

In some embodiments, the black and white camera 151a is configured to shoot at the same frequency as the refresh frequency of the display screen of the 3D display device.

The embodiments of the present disclosure may also provide an eye positioning method, which is implemented by using the eye positioning device in the above-mentioned embodiment.

Referring to FIG. 4, in some embodiments, the eye positioning method includes:

S401: Take a black and white image of the user's face;

S402: Acquire depth information of the face;

S403: Determine the spatial position of the eye based on the captured black and white image and the depth of field information.

Referring to FIG. 5, in some embodiments, the eye positioning method includes:

S501: Take a black and white image of the user's face;

S502: Acquire depth information of the face;

S503: Recognizing the presence of eyes based on the captured black and white image;

S504: Determine the spatial position of the eye based on the captured black and white image and depth of field information;

S505: Transmit the spatial position information of the eye including the spatial position of the eye.

Referring to FIG. 6, in some embodiments, the eye positioning method includes:

S601: Take a black and white image sequence including black and white images of the user's face;

S602: Cache multiple black and white images in the black and white image sequence;

S603: compare multiple black and white images before and after in the black and white image sequence;

S604: Acquire depth information of the face;

S605: When the presence of eyes is not recognized by comparing the current black and white images in the black and white image sequence and the presence of eyes is recognized in the black and white images before or after, based on the black and white images before or after and the acquired depth information The determined eye spatial position information is used as the current eye spatial position information.

The embodiments of the present disclosure may also provide a 3D display method, which is applicable to the 3D display device in the above embodiment. The 3D display device includes a multi-viewpoint 3D display screen, and the multi-viewpoint 3D display screen includes a plurality of sub-pixels corresponding to a plurality of viewpoints.

Referring to FIG. 7, in some embodiments, the 3D display method includes:

S701: Determine the spatial position of the user's eyes;

S702: Determine the viewpoint according to the spatial position of the user's eyes, and render sub-pixels corresponding to the viewpoint based on the 3D signal.

The embodiment of the present disclosure provides a 3D display terminal 800. Referring to FIG. 8, the 3D display terminal 800 includes a processor 814, a memory 811, a multi-view 3D display screen 810, and may also include a communication interface 812 and a bus 813. Among them, the multi-view 3D display screen 810, the processor 814, the communication interface 812, and the memory 811 communicate with each other through the bus 813. The communication interface 812 can be used for information transmission. The processor 814 may call logic instructions in the memory 811 to execute the 3D display method of the foregoing embodiment.

In addition, the logic instructions in the memory 811 can be implemented in the form of a software functional unit and when sold or used as an independent product, they can be stored in a computer readable storage medium.

As a computer-readable storage medium, the memory 811 can be configured to store software programs and computer-executable programs, such as program instructions/modules corresponding to the methods in the embodiments of the present disclosure. The processor 814 executes functional applications and data processing by running the program instructions/modules stored in the memory 811, that is, realizes the eye positioning method and/or the 3D display method in the foregoing method embodiment.

The memory 811 may include a program storage area and a data storage area. The program storage area may store an operating system and an application program required by at least one function; the data storage area may store data created according to the use of the terminal device, and the like. In addition, the memory 811 may include a high-speed random access memory, and may also include a non-volatile memory.

The technical solutions of the embodiments of the present disclosure can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes one or more instructions to enable a computer device (which can be a personal computer, a server, or a network). Equipment, etc.) execute all or part of the steps of the method of the embodiment of the present disclosure. The aforementioned storage media can be non-transitory storage media, including: U disk, mobile hard disk, read-only memory, random access memory, magnetic disk or optical disk, and other media that can store program codes, or it can be a transient storage medium. .

Those skilled in the art may realize that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be implemented by electronic hardware or a combination of computer software and electronic hardware. Whether these functions are performed by hardware or software may depend on the specific application and design constraint conditions of the technical solution. Those skilled in the art may use different methods for each specific application to realize the described functions, but such realization should not be considered as going beyond the scope of the embodiments of the present disclosure.

In the embodiments disclosed herein, the disclosed methods and products (including but not limited to devices, equipment, etc.) can be implemented in other ways. For example, the device or device embodiments described above are merely illustrative. For example, the division of units may only be a logical function division, and there may be other divisions in actual implementation, for example, multiple units or components may be combined. Or it can be integrated into another system, or some features can be ignored or not implemented. In addition, the displayed or discussed mutual coupling or direct coupling or communication connection may be indirect coupling or communication connection through some interfaces, devices or units, and may be in electrical, mechanical or other forms. The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to implement this embodiment. In addition, the functional units in the embodiments of the present disclosure may be integrated into one processing unit, or each unit may exist alone physically, or two or more units may be integrated into one unit.

In addition, in the description corresponding to the flowchart in the accompanying drawings, the operations or steps corresponding to different blocks can also occur in a different order than disclosed in the description, and sometimes there is no specific operation or step between different operations or steps. order.

Claims

An eye positioning device, including:

The eye locator includes a black and white camera configured to take a black and white image of the user's face and a depth acquisition device configured to acquire depth information of the face;

The eye positioning image processor is configured to determine the spatial position of the eye based on the black and white image and the depth information.
The eye positioning device according to claim 1, wherein the eye positioning image processor is further configured to recognize the presence of the eye based on the black and white image.
The eye positioning device according to claim 1, further comprising an eye positioning data interface configured to transmit eye spatial position information including the spatial position of the eye.
The eye positioning device according to claim 1, wherein the depth-of-field acquisition device is a structured light camera or a TOF camera.
The eye positioning device according to claim 1, further comprising a viewing angle determining device configured to calculate the viewing angle of the user with respect to the 3D display device.
The eye positioning device according to any one of claims 1 to 5, wherein the black and white camera is configured to capture a sequence of black and white images.
The eye positioning device according to claim 6, wherein the eye positioning image processor comprises:

A buffer, configured to buffer a plurality of black and white images in the black and white image sequence;

A comparator configured to compare multiple black-and-white images before and after the black-and-white image sequence;

The arbiter is configured to, when the comparator does not recognize the presence of eyes in the current black-and-white image in the sequence of black-and-white images and recognizes the presence of eyes in previous or subsequent black-and-white images, The eye spatial position information determined based on the black and white images before or after and the acquired depth information is used as the current eye spatial position information.
A 3D display device, including:

Multi-viewpoint 3D display screen, including multiple sub-pixels corresponding to multiple viewpoints;

The eye positioning device according to any one of claims 1 to 7, configured to determine the spatial position of the user's eyes; and

The 3D processing device is configured to determine a viewpoint according to the spatial position of the user's eyes, and render sub-pixels corresponding to the viewpoint based on a 3D signal.
8. The 3D display device according to claim 8, wherein the multi-viewpoint 3D display screen includes a plurality of composite pixels, each of the plurality of composite pixels includes a plurality of composite sub-pixels, and the plurality of composite pixels Each composite sub-pixel in the sub-pixel is composed of a plurality of sub-pixels corresponding to a plurality of viewpoints.
The 3D display device according to claim 8 or 9, wherein the 3D processing device and the eye positioning device are communicatively connected through an eye positioning data interface.
The 3D display device according to claim 8 or 9, further comprising:

3D camera, configured to capture 3D images,

The 3D photographing device includes a depth-of-field camera and at least two color cameras.
11. The 3D display device according to claim 11, wherein the eye positioning device is integrated with the 3D photographing device.
11. The 3D display device according to claim 11, wherein the 3D photographing device is placed in front of the 3D display device.
An eye positioning method, including:

Take a black and white image of the user's face;

Acquiring depth information of the face;

The spatial position of the eye is determined based on the black and white image and the depth information.
The eye positioning method according to claim 14, further comprising: recognizing the presence of the eye based on the black and white image.
The eye positioning method according to claim 14, further comprising: transmitting eye spatial position information including the spatial position of the eye.
The eye positioning method according to claim 14, further comprising: photographing a black and white image sequence including the black and white image.
The eye positioning method according to claim 17, further comprising:

Buffering a plurality of black and white images in the black and white image sequence;

Comparing multiple black and white images before and after in the black and white image sequence;

When the presence of eyes is not recognized by comparing the current black and white images in the black and white image sequence and the presence of eyes is recognized in the black and white images before or after, based on the black and white images before or after and the acquired The eye space position information determined by the depth of field information is used as the current eye space position information.
A 3D display method, including:

Determine the spatial position of the user’s eyes;

Determine the viewpoint according to the spatial position of the user's eyes, and render sub-pixels corresponding to the viewpoint based on the 3D signal;

Wherein, the 3D display device includes a multi-viewpoint 3D display screen, and the multi-viewpoint 3D display screen includes a plurality of sub-pixels corresponding to a plurality of viewpoints.
A 3D display terminal includes a processor, a memory storing program instructions, and a multi-view 3D display screen. The processor is configured to execute the 3D display method according to claim 19 when the program instructions are executed.
A computer-readable storage medium storing computer-executable instructions configured to execute the method according to any one of claims 14 to 19.
A computer program product, comprising a computer program stored on a computer-readable storage medium, the computer program comprising program instructions, when the program instructions are executed by a computer, the computer executes any one of claims 14 to 19 The method described.