JP2022113478A - Integral stereoscopic display system and method thereof - Google Patents

Abstract

To provide an integral stereoscopic display system that can suppress a communication data amount, and an arithmetic processing amount of a mobile terminal.SOLUTION: An integral stereoscopic display system 100 displays an element image generated by a server 2 on a mobile terminal 1 so as to follow the point of view of a viewer who views with the mobile terminal 1. The mobile terminal 1 includes communication means 10, photographing means 11, face area detection means 12, face part detection means 13, and display means 14. The server 2 includes communication means 20, three-dimensionalizing means 21, and element image generation means 22.SELECTED DRAWING: Figure 3

Description

The present invention relates to an integral stereoscopic display system and method.

There is a demand for video media that can provide a highly realistic experience regardless of location. If such video media is realized, it can be applied to various fields such as entertainment, education, and medical care. Three-dimensional video is known as one of the video media that allows users to experience a high sense of presence.

Among several stereoscopic methods for displaying 3D images, a ray reproduction stereoscopic method that reproduces the same light rays as those that actually exist is attracting attention. An integral method using a lens array is known as one of the three-dimensional methods for reproducing light rays. This integral method can be realized simply by superimposing a lens array on a general display such as a liquid crystal display, and can reproduce binocular parallax and vertical and horizontal motion parallax.

In recent years, mobile terminals such as smartphones and tablets have become popular, and it is becoming common to view video media on mobile terminals. Therefore, viewing of high-quality 3D video using a mobile terminal is examined. In the integral method, there is a trade-off relationship between image quality such as the resolution of the three-dimensional image, the viewing area, and the depth reproduction range. In order to improve all these elements, it is necessary to improve the performance of display devices, but there is a limit to the performance improvement of display devices such as mobile terminals.

Therefore, application of viewpoint tracking technology capable of improving video quality by having a viewer follow a narrow viewing zone is studied (Patent Document 1, Non-Patent Document 1). Hereinafter, the integral method in which the viewpoint-tracking technology is applied to the mobile terminal will be referred to as "portable-terminal-type viewpoint-tracking IP method." For example, in the mobile terminal type viewpoint tracking IP method, the viewer's viewpoint position is measured by the in-camera of the mobile terminal, the server generates a 3D image focusing on the viewpoint position, and the 3D image is played back on the mobile terminal. do.

JP 2019-213127 A

Okaichi et al., “Viewpoint Tracking Integral 3D Image Display Using 8K Display”, Institute of Image Information and Television Engineers Winter Conference, 23D-3, 2018

In the mobile terminal type viewpoint tracking IP system, it is necessary to communicate various data such as viewpoint positions and three-dimensional images between the server and the mobile terminal. Furthermore, in the mobile terminal type viewpoint tracking IP method, since the elemental images are displayed according to the viewpoint position of the viewer, it is necessary to measure the viewpoint position and generate the elemental images from the three-dimensional model according to the viewpoint position. . In the mobile terminal type viewpoint tracking IP method, if these processes cannot be executed in real time, it becomes impossible to reproduce correct 3D video due to delay. However, in the mobile terminal type viewpoint tracking IP method, the processing capability of the mobile terminal is limited, and it is difficult to perform large-capacity data communication and real-time processing of complicated calculations.

SUMMARY OF THE INVENTION An object of the present invention is to provide an integral stereoscopic display system and method that can reduce the communication data amount and the arithmetic processing amount of a mobile terminal.

In order to solve the above problems, an integral 3D display system according to the present invention provides an integral 3D display system that displays element images generated by an element image generation device on a mobile terminal so as to follow the viewpoint of a person viewing the image on the mobile terminal. In the display system, the portable terminal includes photographing means, face area detection means, face part detection means, and display means, and the element image generation device includes three-dimensional position calculation means and element image generation means. And, it was configured to include.

According to such a configuration, the mobile terminal captures the face image of the person by the imaging means. Also, the mobile terminal detects the face area of the person from the face image by the face area detection means. Then, the mobile terminal detects the image coordinates of the face parts including at least both eyes of the person from the face area by the face part detection means. Further, the mobile terminal displays the element images generated by the element image generation device by means of display means.

Further, the elemental image generation device calculates the three-dimensional position of the face part from the image coordinates of the face part by the three-dimensional position calculation means. Then, the element image generating device generates an element image with the three-dimensional position of both eyes included in the face part as the viewpoint position by the element image generating means.

In order to solve the above-mentioned problems, an integral stereoscopic display system according to the present invention provides an integral stereoscopic display system that displays element images generated by an element image generation device on a mobile terminal so as to follow the viewpoint of a person viewing on the mobile terminal. In the three-dimensional display system, the mobile terminal includes photographing means, face area detection means, face part detection means, three-dimensional position calculation means, and display means, and the element image generation device includes element images. and generating means.

According to such a configuration, the mobile terminal captures the face image of the person by the imaging means. Also, the mobile terminal detects the face area of the person from the face image by the face area detection means. Then, the mobile terminal detects the image coordinates of the face parts including at least both eyes of the person from the face area by the face part detection means. Furthermore, the mobile terminal calculates the three-dimensional position of the face part from the image coordinates of the face part by means of the three-dimensional position calculation means. Further, the mobile terminal displays the element images generated by the element image generation device by means of display means.

Further, the element image generating device generates an element image with the three-dimensional positions of both eyes included in the face part as the viewpoint position by the element image generating means.

In order to solve the above-described problems, an integral stereoscopic display method according to the present invention provides an integral stereoscopic display method for displaying elemental images generated by an elemental image generation device on a mobile terminal so as to follow the viewpoint of a person viewing the image on the mobile terminal. A three-dimensional display method comprising a face image photographing step, a face area detection step, a face part detection step, a three-dimensional position calculation step, an element image generation step, and an element image display step. .

According to this procedure, in the face image capturing step, the mobile terminal captures the face image of the person. Also, in the face area detection step, the mobile terminal detects the face area of the person from the face image. Then, in the face part detection step, the mobile terminal detects image coordinates of face parts including at least both eyes of the person from the face area. Furthermore, in the three-dimensional position calculation step, the mobile terminal or the elemental image generation device calculates the three-dimensional positions of the facial parts from the image coordinates of the facial parts.

Further, in the elemental image generation step, the elemental image generation device generates an elemental image with the three-dimensional positions of both eyes included in the face part as the viewpoint position. Then, in the elemental image display step, the mobile terminal displays the elemental image generated by the elemental image generation device.

As described above, in the present invention, the mobile terminal performs viewpoint position measurement processing, and the elemental image generation device performs elemental image generation processing. Therefore, in the present invention, since there is no need to transmit and receive face images, the amount of communication data can be reduced, and since there is no need for the mobile terminal to perform elemental image generation processing, it is possible to reduce the amount of arithmetic processing of the mobile terminal.

According to the present invention, it is possible to suppress the communication data amount and the arithmetic processing amount of the mobile terminal.

1 is a schematic configuration diagram of an integral stereoscopic display system according to each embodiment; FIG. 4 is a flowchart of viewpoint position measurement processing in each embodiment. 1 is a block diagram showing the configuration of an integral stereoscopic display system according to a first embodiment; FIG. FIG. 4 is an explanatory diagram for explaining viewpoint position measurement processing in the first embodiment; FIG. 4 is an explanatory diagram for explaining searching for a face area in the first embodiment; (a) and (b) are explanatory diagrams for explaining the movement of the virtual camera array in the first embodiment. FIG. 4 is an explanatory diagram for explaining generation of element images in the first embodiment; 4 is a sequence diagram showing the operation of the integral 3D display system according to the first embodiment; FIG. FIG. 11 is a block diagram showing the configuration of an integral stereoscopic display system according to a second embodiment; FIG. FIG. 11 is a sequence diagram showing the operation of the integral stereoscopic display system according to the second embodiment; (a) and (b) are explanatory diagrams for explaining the shift of elemental images in the third embodiment. FIG. 11 is an explanatory diagram for explaining rotation of a three-dimensional model in the third embodiment;

Hereinafter, each embodiment of the present invention will be described with reference to the drawings. However, each embodiment described below is for embodying the technical idea of the present invention, and unless there is a specific description, the present invention is not limited to the following. In addition, the same reference numerals may be given to the same means, and the description thereof may be omitted.

[Study of system configuration]
Before describing each embodiment, a system configuration for realizing the mobile terminal type viewpoint-tracking integral stereoscopic display system 100 of FIG. 1 will be examined. Hereinafter, the ``portable terminal type viewpoint tracking integral 3D display system'' will be simply abbreviated as ``integral 3D display system''.
As shown in FIG. 1, an integral stereoscopic display system 100 is composed of a mobile terminal 1 used by a viewer (person) α, and a server (element image generation device) 2 that provides various data to the mobile terminal 1. ing.

As shown in FIG. 2, in the integral stereoscopic display system 100, a viewpoint position measurement process S1 for measuring the viewpoint position of the viewer α, an element image generation process (element image generation step) S2 for generating element images, and an element Element image display processing (element image display step) S3 for displaying an image is performed. In this integral stereoscopic display system 100, it is necessary to perform the viewpoint position measurement process S1 and the element image generation process S2 for each frame.

The specific processing contents of the viewpoint position measurement processing S1, the elemental image generation processing S2, and the elemental image display processing S3 will be described later. Also, since the processing load of the elemental image display processing S3 is so low that it can be ignored, it will not be considered. Further, it is preferable that the cycle of the viewpoint position measurement processing S1 and the element image generation processing S2 is the same, but it is not essential.

In the integral stereoscopic display system 100, two methods are conceivable for distributing the viewpoint position measurement processing S1 and the elemental image generation processing S2 between the mobile terminal 1 and the server 2. FIG. In the first method, the server 2 transmits a three-dimensional model necessary for generating elemental images to the portable terminal 1, and the portable terminal 1 performs viewpoint position measurement processing S1 and elemental image generation processing S2. However, in the first method, it is necessary to transmit the three-dimensional model to the mobile terminal 1, which increases the amount of communication data and the processing load of the mobile terminal 1 extremely.

As described above, in order to realize the integral stereoscopic display system 100, it is necessary to suppress the communication data amount and the arithmetic processing amount of the mobile terminal 1. FIG. In the first method, the processing load of the mobile terminal 1 is too high, so the processing time per frame becomes long, and discomfort such as stuttering occurs when viewing. Thus, the first method cannot keep up with viewpoint tracking and is not realistic.

In the second method, the processing load on the mobile terminal 1 can be reduced by distributing the processing between the server 2 and the mobile terminal 1 so that the server 2 executes the element image generation processing S2, which has a large processing load. . Therefore, the second method is further considered.

As shown in FIG. 2, the viewpoint position measurement process S1 is subdivided into a face image photographing step S10, a face area detection step S11, a face part detection step S12, and a three-dimensional conversion step (three-dimensional position calculation step) S13. can. Here, consider a case where the mobile terminal 1 performs the face image photographing step S10 and the server 2 performs the processes after the face area detection step S11. In this case, the mobile terminal 1 needs to perform encoding processing (for example, H.264) in order to transmit the face image, which increases the processing load of the mobile terminal 1 . Furthermore, there is a problem that the amount of communication data is large even if the face image is encoded. Therefore, it is considered that the integral stereoscopic display system 100 can be realized by performing the viewpoint position measurement processing S1 up to the facial part detection step S12 or the three-dimensional conversion step S13 in the portable terminal 1, and performing the subsequent processing in the server 2. .

(First embodiment)
[Configuration of integral stereoscopic display system]
The configuration of an integral 3D display system 100 according to the first embodiment will be described with reference to FIG.
The integral 3D display system 100 displays elemental images generated by the server 2 on the mobile terminal 1 so as to follow the viewpoint of the viewer α viewing on the mobile terminal 1 . As shown in FIG. 1 , the integral stereoscopic display system 100 includes a mobile terminal 1 and a server 2 .

The mobile terminal 1 performs the viewpoint position measurement processing S1 up to the facial parts detection step S12, and also performs the element image display step S3. Here, the mobile terminal 1 has a lens array for displaying elemental images on the display surface of a general smart phone or tablet. In addition, the mobile terminal 1 has an in-camera for photographing the face image of the viewer α and a communication function for communicating various data with the server 2 . Note that the mobile terminal 1 is not limited to a smart phone or a tablet as long as it has the functions described above.

The server 2 performs the three-dimensional conversion step S<b>13 and the element image generation process S<b>2 to generate element images, and transmits the generated element images to the portable terminal 1 . In this embodiment, the server 2 is a general server connected to a network NW such as the Internet and capable of communicating various data with the mobile terminal 1 .

[Configuration of mobile terminal]
The configuration of the mobile terminal 1 will be described with reference to FIGS. 3 and 4. FIG.
As shown in FIG. 3, the mobile terminal 1 includes communication means 10, photographing means 11, face area detection means 12, face part detection means 13, and display means .

The communication means 10 communicates various data with the server 2 via the network NW. Specifically, the communication means 10 transmits the image coordinates of the facial parts of the viewer α to the server 2 and receives the element images from the server 2 . For example, the communication means 10 transmits and receives various data using arbitrary wireless communication such as WiFi and packet communication.

The photographing means 11, as shown in FIG. 4, performs a facial image photographing step S10 for photographing a facial image 90 of the viewer α. For example, the photographing means 11 may be an in-camera of a smartphone or a tablet. In this embodiment, in order to follow the viewpoint of the viewer α, the photographing means 11 photographs the face image 90 of the viewer α in real time, and outputs the photographed face image 90 to the face area detection means 12 frame by frame. .

The face area detection means 12 performs a face area detection step S11 for detecting the face area 91 of the viewer α from the face image 90 input from the photographing means 11 . In the present embodiment, the photographing means 11 detects the face area 91 from each frame of the face image 90 using a known method in order to follow the viewpoint of the viewer α (eg References 1 and 2). The face area detection means 12 then outputs the detected face area 91 to the face part detection means 13 .

Reference 1: “OpenCV”, [online], [searched on December 10, 2020], Internet <URL: https://opencv.org/>
Reference 2: “Dlib”, [online], [searched on December 10, 2020], Internet <URL: http://dlib.net/>

<Face area detection>
The detection of the face area 91 by the face area detection means 12 will be described in detail below.
Since detection of the face area 91 is executed for each frame of the face image 90, speeding up is possible. As shown in FIG. 5, it is assumed that _a face area 91-1 is detected in the center of _a face image 90-1 of a certain frame. In this case, there is _a high possibility that the face area _91-2 can be detected in the vicinity of the face area _91-1 of the previous frame in the face image 90-2 of the next frame. Therefore, the face area detection means 12 detects the face area _91-2 from the face image 90-2 of the _next frame, using the periphery of the face area 91-1 of the previous frame as the detection region 92. _FIG . That is, the face area detection means ₁₂ detects the face area _91-2 within the range of the detection area 92 set in the face image 90-2 of the next frame. For example, the detection area 92 is _a rectangular area obtained by enlarging the face area 911 detected in the previous frame by a factor of two or less. As described above, in the face area detection means 12, since the detection area 92 is narrower than the entire face image 90, speeding up can be achieved. It should be noted that the face area detection means 12 detects the face area 91 from the entire face image 90 in the case of the top frame or when the detection of the face area 91 fails.

The face parts detection means 13 performs a face parts detection step S12 of detecting image coordinates 93 of face parts including at least both eyes of the viewer α from the face area 91 . Here, the face parts are parts such as the eyes, nose, and mouth that constitute the face of the viewer α. In this embodiment, the face part detection means 13 detects the image coordinates of both eyes, nose and mouth of the viewer α as face parts from the face area 91 input from the face area detection means 12 . For example, the facial parts detection means 13 can detect facial parts by a known technique (eg, References 1 and 2 above). As shown in FIG. 4, when the face part detection means 13 detects a face part, the two-dimensional coordinates of both eyes in the face image 90 are known.
After that, the facial parts detection means 13 transmits the image coordinates 93 of the detected facial parts to the server 2 via the communication means 10 . Since the image coordinates 93 of this face part are data describing two-dimensional coordinate values of both eyes, the amount of data is smaller than that of the face image.

In addition, in order to solve the PnP problem described later in the server 2, the internal parameters and lens distortion coefficients of the photographing means (in-camera) 11 are required. Therefore, it is preferable that the mobile terminal 1 perform camera calibration in advance to acquire parameters and transmit the parameters to the server 2 . For example, a technique for camera calibration is described in Reference 3.

Reference 3: Z. Zhang, "A flexible new technique for camera calibration", IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 22, No. 11, pp. 1330-1334 (2000)

The display means 14 performs element image display processing S3 for displaying the element images generated by the server 2 . Specifically, the display means 14 receives element images from the server 2 via the communication means 10 and displays the received element images. For example, the display means 14 is composed of a liquid crystal element or organic EL element display and a lens array arranged on the display surface (not shown).

[Server configuration]
The configuration of the server 2 will be described with reference to FIGS. 3 and 4. FIG.
As shown in FIG. 3 , the server 2 includes communication means 20 , three-dimensionalization means (three-dimensional position calculation means) 21 , and element image generation means 22 .

The communication means 20 communicates various data with the mobile terminal 1 via the network NW. Specifically, the communication unit 20 receives the image coordinates 93 of the facial parts from the mobile terminal 1 and transmits the element images to the mobile terminal 1 .

As shown in FIG. 4, the three-dimensionalization means 21 performs a three-dimensionalization step S13 of calculating the three-dimensional positions 95 of both eyes from the image coordinates 93 of the face parts of the viewer α. First, the three-dimensionalization means 21 receives image coordinates 93 of facial parts from the mobile terminal 1 via the communication means 20 . Next, the three-dimensionalization means 21 calculates the three-dimensional position 94 of the facial part from the received image coordinates 93 of the facial part (see Reference 4, for example). Specifically, the 3D rendering means 21 solves a PnP (Perspective n-Point) problem from preset 3D positions of facial parts and the image coordinates 93 of the facial parts, so that both eyes are included in the camera coordinate system. A three-dimensional position 94 of the face part is calculated. As shown in FIG. 4, the 3D position 95 of both eyes is included in the 3D position 94 of the face part and can be described by the 3D coordinates of the right and left eyes.
After that, the three-dimensionalization means 21 outputs the calculated three-dimensional positions 95 of both eyes to the elemental image generation means 22 . Note that the three-dimensionalization means 21 does not need to output the three-dimensional coordinates of the eyes other than those of both eyes to the element image generation means 22 because the three-dimensionalization means 21 is unnecessary for generating the elemental images.

Reference 4: Digital Image Processing (Revised New Edition), CG-ARTS Society, 2015

The elemental image generating means 22 performs an elemental image generating process S2 for generating (rendering) an elemental image with the 3D position 95 of both eyes input from the 3D rendering means 21 as the viewpoint position. Then, the element image generation means 22 transmits the generated element images to the mobile terminal 1 via the communication means 20 .

<Generation of Element Image>
Generation of the element image E by the element image generation means 22 will be described in detail with reference to FIGS. 6 and 7. FIG. In this embodiment, the elemental image generating means 22 renders the elemental images E from the three-dimensional model M that is the basis of the stereoscopic image. Specifically, the element image generating means 22 arranges a three-dimensional model M as a subject in a computer graphics (CG) space, and photographs the three-dimensional model M with a virtual camera array C. FIG.

In FIG. 6, the virtual camera array C is composed of five virtual cameras C ₁ to C ₅ arranged in the horizontal direction. It is assumed that all the virtual cameras C ₁ to C ₅ are arranged facing the virtual lens array L, and the internal parameters are set in advance so that the area of the virtual lens array L can be photographed.

At this time, the elemental image generating means 22 sets the midpoint P of the three-dimensional position 95 (FIG. 4) of both eyes obtained in the viewpoint position measurement process S1 as the center of the virtual camera array C as the viewpoint position. As shown in FIG. 6A, the viewer α faces the virtual lens array L, and in the virtual lens array coordinate system having the center of the virtual lens array L as the origin, x− A state in which both y-coordinates are zero is taken as a reference. Note that the horizontal direction is the x-axis, the vertical direction is the y-axis, and the depth direction is the z-axis.

The element image generating means 22 similarly moves the virtual camera array C up, down, left, and right as the viewer α moves up, down, left, and right from this state. As shown in FIG. 6(b), when the viewer α moves to the left, the elemental image generating means 22 moves the virtual camera array C to the left. Furthermore, when the viewing distance changes, the elemental image generating means 22 moves the virtual camera array C back and forth according to the change in the viewing distance D. FIG. At this time, the elemental image generating means 22 also changes the internal parameters so that the area of the virtual lens array L can be photographed according to the movement of the virtual camera array C. FIG. Then, the elemental image generating means 22 generates an elemental image E by rearranging the pixel positions of the multi-viewpoint image V captured by the virtual camera array C, as shown in FIG.

As described above, it is preferable to match the middle point P of both eyes in the virtual lens array coordinate system with the center of the virtual camera array C. FIG. On the other hand, the three-dimensional position obtained by the viewpoint position measurement processing S1 is expressed in the camera coordinate system of the in-camera. The positional relationship between the virtual lens array coordinate system and the camera coordinate system is parallel and the distance is short, and the difference in the positional relationship between the two coordinate systems is not a big problem and can be ignored. Furthermore, the positional relationship between the two coordinate systems may be adjusted by applying preset offset values for the two coordinate systems.

[Operation of Integral Stereoscopic Display System]
The operation of the integral stereoscopic display system 100 will be described with reference to FIG.
As shown in FIG. 8 , the portable terminal 1 performs a face image photographing step S10 by the photographing means 11 .
The mobile terminal 1 uses the face area detection means 12 to perform a face area detection step S11.
The mobile terminal 1 performs a face part detection step S<b>12 by the face part detection means 13 and transmits the image coordinates 93 of the detected face parts to the server 2 .

The server 2 receives the image coordinates 93 of the facial parts from the mobile terminal 1 by the three-dimensionalization means 21, and performs the three-dimensionalization step S13.
The server 2 performs element image generation processing S<b>2 by the element image generation means 22 and transmits the generated element image E to the portable terminal 1 .
The portable terminal 1 receives the elemental image from the server 2 by the display means 14, and performs elemental image generation display processing S3.

[Action/effect]
As described above, in the integral 3D display system 100, the mobile terminal 1 performs the viewpoint position measurement processing S1 up to the face part detection step S12, and the server 2 performs the element image generation processing S2. Therefore, in the integral stereoscopic display system 100, since there is no need to transmit and receive face images, the amount of communication data is suppressed, and since it is not necessary for the mobile terminal 1 to perform the elemental image generation processing S2, the amount of arithmetic processing of the mobile terminal 1 is also suppressed. can. Furthermore, the integral 3D display system 100 enables the viewer α to easily enjoy 3D images on the go using the portable terminal 1, and is useful for quick and efficient information acquisition in the event of a disaster.

(Second embodiment)
[Configuration of integral stereoscopic display system]
With reference to FIG. 9, the configuration of an integral 3D display system 100A according to the second embodiment will be described with respect to the differences from the first embodiment.
The integral stereoscopic display system 100A differs from the first embodiment in that the portable terminal 1A performs the three-dimensionalization step S13.

As shown in FIG. 9, the integral stereoscopic display system 100A includes a mobile terminal 1A and a server 2A.
The portable terminal 1A performs all viewpoint position measurement processing S1 including the three-dimensionalization step S13.
In the server 2A, the element image generation device performs the element image generation processing S2. That is, unlike the first embodiment, the server 2A does not perform the three-dimensionalization step S13.

[Configuration of mobile terminal]
Next, the configuration of the mobile terminal 1 will be described.
As shown in FIG. 9, mobile terminal 1A includes communication means 10A, photographing means 11, face area detection means 12, face part detection means 13, display means 14, and three-dimensionalization means (three-dimensional position calculation). means) 15; Since the configuration other than the communication means 10A and the three-dimensionalization means 15 is the same as that of the first embodiment, the description thereof is omitted.

The communication means 10A communicates various data with the server 2A via the network NW. Specifically, the communication means 10A transmits the three-dimensional position 95 of both eyes of the viewer α to the server 2A and receives the element image E from the server 2A. Otherwise, the communication means 10A is the same as in the first embodiment, so further explanation is omitted.

The three-dimensionalization means 15 performs the three-dimensionalization step S13 as in the first embodiment. First, the three-dimensionalization means 15 calculates the three-dimensional positions 95 of both eyes from the image coordinates 93 of the face parts of the viewer α input from the face part detection means 13 . The three-dimensional rendering means 15 then transmits the three-dimensional positions 95 of both eyes of the viewer α to the server 2A via the communication means 10 . The three-dimensional position 95 of both eyes has a smaller amount of data than the face image.

[Server configuration]
Next, the configuration of the server 2A will be explained.
As shown in FIG. 9, the server 2 includes communication means 20A and element image generation means 22A.

The communication means 20A communicates various data with the mobile terminal 1A via the network NW. Specifically, the communication means 20A receives the three-dimensional position 95 of both eyes of the viewer α from the mobile terminal 1A and transmits the element images to the mobile terminal 1A. Otherwise, the communication means 20A is the same as in the first embodiment, so further explanation is omitted.

The elemental image generating means 22A receives the three-dimensional positions 95 of both eyes of the viewer α from the portable terminal 1A via the communication means 20A. Then, the elemental image generating means 22A performs elemental image generation processing S2 as in the first embodiment.

[Operation of Integral Stereoscopic Display System]
The operation of the integral stereoscopic display system 100A will be described with reference to FIG.
As shown in FIG. 10, the mobile terminal 1A performs a three-dimensionalization step S13 by the three-dimensionalization means 15. FIG.
Other than that, the operation of the integral stereoscopic display system 100A is the same as that of the first embodiment, so further explanation is omitted.

[Action/effect]
As described above, in the integral stereoscopic display system 100A, the mobile terminal 1A performs all the viewpoint position measurement processing S1, and the server 2A performs the element image generation processing S2. Therefore, in the integral stereoscopic display system 100A, since there is no need to transmit and receive face images, the amount of communication data is suppressed, and since it is not necessary to perform the elemental image generation processing S2 in the mobile terminal 1A, the amount of arithmetic processing of the mobile terminal 1A is suppressed. can. Furthermore, the integral 3D display system 100A enables the viewer α to easily enjoy 3D images on the go using the portable terminal 1A, and is useful for quick and efficient information acquisition in the event of a disaster.

Advantages of each of the first and second embodiments will now be described. The integral 3D display system 100 of FIG. 3 has the advantage that the three-dimensionalization step S13 is performed by the server 2, so that the computational processing amount of the mobile terminal 1 can be greatly reduced. On the other hand, the integral stereoscopic display system 100A of FIG. 9 transmits and receives coordinate data of two points, ie, the three-dimensional position 95 of both eyes, and therefore has the advantage of being able to greatly reduce the amount of communication data.

(Third Embodiment)
The integral stereoscopic display system 100B according to the third embodiment will be described below with respect to the differences from the first and second embodiments.

With viewpoint tracking, it is possible to widen the viewing area of 3D images, but it can also be used in other ways. Specifically, it is to always display the 3D image of the front viewpoint by following the viewpoint. In many cases, the 3D video of the front viewpoint is considered to be the video of the viewpoint position that the creator of the content most wants to show to the viewer α. In the integral method, the appearance of the stereoscopic image changes according to the viewing direction of the viewer α, so unless the viewer α is correctly facing the front, the viewer cannot see the 3D video from the front viewpoint. Depending on the details of the content and the viewing environment, there may be a case where the user wants to always view the 3D video from the front viewpoint. In order to fix the viewpoint position to the front viewpoint, the following processing may be added when the elemental image generating means 22B in FIG. 2 renders the elemental images. Since there are two types of processing methods, they will be described in order.

<Shifting Element Images>
The elemental image generating means 22B shifts the generated elemental image E to the front of the viewpoint position. The position of the virtual camera array C should be fixed to the front of the three-dimensional model M if only the element image E of the front viewpoint is to be rendered. In this case, as shown in FIG. 11A, the middle point P of both eyes of the viewer α does not match the front face Q of the three-dimensional model M, so a stereoscopic image cannot be displayed. Here, the front face Q of the three-dimensional model M becomes the center of the virtual camera array C. FIG. Therefore, as shown in FIG. 11(b), the elemental image generating means 22 shifts the elemental image E after rendering to generate an elemental image E' of the front viewpoint. In the example of FIG. 11, the midpoint P of both eyes, which is the viewpoint position, is on the left side of the front Q of the three-dimensional model M, so the element image E is shifted to the opposite right side. Note that the shift of the elemental image E may cause distortion in the stereoscopic image.

A method of calculating the shift value s' of the element image E will be described below. The shift value s' is an integer that takes a positive or negative value, and the unit is a pixel. As shown in FIG. 11A, the distance F between the virtual lens array L and the virtual display K, the midpoint P=[P _x, P _y, P _z ] ^T of both eyes in the virtual lens array coordinate system, the viewing distance Let D=|P _z |. Note that the distance F is equal to the focal length of the virtual lens array L. FIG. At this time, the shift value s is obtained by the following formula (1).

Since the unit of this shift value s is distance, the pixel pitch PP of the virtual display _K is used to convert the unit into pixels. Therefore, the shift value s' is represented by the following equation (2). Note that round( ) is a rounding function.

That is, after generating the elemental image E, the elemental image generating means 22B calculates the shift value s' using the equations (1) and (2), and shifts the elemental image E by the shift value s'. '. Note that the elemental image generating means 22B may shift the elemental image E in the y-axis direction as well as in the x-axis direction. Also, the element image generating means 22B preferably generates a large-sized element image E in advance so as not to generate a place where the element image E cannot be displayed after the shift.

<Rotation of 3D model>
The element image generating means 22B rotates the three-dimensional model M toward the viewpoint position and generates an element image E from the rotated three-dimensional model M. FIG. As shown in FIG. 12, the element image generating means 22B rotates the three-dimensional model M so that the front face Q of the three-dimensional model M before rotation faces the middle point P of both eyes. In the example of FIG. 12, the midpoint P of both eyes, which is the viewpoint position, is on the left side of the front face Q of the three-dimensional model M before rotation, so the three-dimensional model M is rotated clockwise so that it faces the opposite right side. When the rotation angle θ of the three-dimensional model M is large, part of the three-dimensional model may be separated from the lens surface of the virtual lens array L, resulting in blurring.

Here, the rotation angle θ of the three-dimensional model M is obtained by the following formula (3). In other words, the element image generating means 22 should calculate the rotation angle θ using the equation (3) and generate the element image E from the three-dimensional model M rotated by the calculated rotation angle θ. Note that the element image generating means 22B may rotate the three-dimensional model M in the x-axis direction as well as in the y-axis direction.

Note that, in the integral stereoscopic display system 100B, normal viewing by viewpoint tracking and viewing with a fixed front viewpoint can be switched by an arbitrary method. For example, in the integral stereoscopic display system 100B, the viewer α may operate the mobile terminal 1 to switch between normal viewing and viewing with a fixed front viewpoint.

[Action/effect]
As described above, the integral 3D display system 100B can display the 3D image from the front viewpoint that the creator of the content most wants to show to the viewer α. can be improved.

(Modification)
Although the respective embodiments of the present invention have been described in detail above, the present invention is not limited to the above-described respective embodiments, and includes design changes and the like within the scope of the present invention.

In viewpoint position measurement processing, wide-angle shooting is often performed in order to widen the shooting range, and the influence of lens distortion becomes very large. Therefore, in the integral stereoscopic display system, by removing this lens distortion, it is possible to improve the accuracy of the three-dimensional positions of both eyes. Specifically, the timing of removing this lens distortion is considered to be the face image photographing step and the face part detection step. If the lens distortion is strong and the face area cannot be detected correctly, the lens distortion may be detected in the face image photographing step. For example, the "undistort( ) function" described in Reference 1 above can be used to remove lens distortion across face images. Further, in the face part detection step, by using the "undistortPoints( ) function" described in Reference 1, lens distortion can be removed for each point in the face image, and the amount of calculation processing can be suppressed.

In the integral 3D display system, by transmitting the touch panel operation of the mobile terminal to the server, the 3D model can be viewed from various angles without changing the viewpoint position. For example, the integral stereoscopic display system may generate the element images after enlarging or reducing the three-dimensional model according to pinch-in/out. In addition, in an integral stereoscopic display system, element images may be generated after rotating a three-dimensional model according to swiping up, down, left, or right.

1, 1A Mobile terminal 2, 2A Server (element image generation device)
10, 10A communication means 11 imaging means 12 face area detection means 13 face part detection means 14 display means 15 three-dimensionalization means (three-dimensional position calculation means)
20, 20A communication means 21 three-dimensionalization means (three-dimensional position calculation means)
22, 22A, 22B elemental image generating means 90 face image 91 face area 92 detection area 93 face part image coordinates 94 face part three-dimensional position 95 two-eye three-dimensional position 100, 100A, 100B integral stereoscopic display system E element image NW Network P Midpoint V Multi-viewpoint image

Claims (6)

An integral stereoscopic display system that displays element images generated by an element image generation device on a mobile terminal so as to follow the viewpoint of a person viewing the mobile terminal,
The mobile terminal is
a photographing means for photographing the face image of the person;
face area detection means for detecting the face area of the person from the face image;
face part detection means for detecting image coordinates of a face part including at least both eyes of the person from the face area;
a display means for displaying the element image generated by the element image generation device,
The element image generation device is
3D position calculation means for calculating the 3D position of the face part from the image coordinates of the face part;
and an elemental image generating means for generating the elemental images with the three-dimensional positions of both eyes included in the face parts as viewpoint positions. An integral stereoscopic display system that displays element images generated by an element image generation device on a mobile terminal so as to follow the viewpoint of a person viewing the mobile terminal,
The mobile terminal is
a photographing means for photographing the face image of the person;
face area detection means for detecting the face area of the person from the face image;
face part detection means for detecting image coordinates of a face part including at least both eyes of the person from the face area;
3D position calculation means for calculating the 3D position of the face part from the image coordinates of the face part;
a display means for displaying the element image generated by the element image generation device,
The element image generation device is
An integral stereoscopic display system, comprising: element image generation means for generating the element image with a three-dimensional position included in the face part as a viewpoint position. 3. The face area detection means detects the face area from the face image of the next frame, using a periphery of the face area detected from the face image of the previous frame as a detection region. Item 3. The integral stereoscopic display system according to item 2. 4. The integral stereoscopic display system according to any one of claims 1 to 3, wherein the elemental image generating means shifts the generated elemental images to the front of the viewpoint position. 4. The element image generating means rotates the three-dimensional model toward the viewpoint position, and generates the element image from the rotated three-dimensional model. The integral stereoscopic display system according to the paragraph. An integral stereoscopic display method for displaying an element image generated by an element image generation device on a mobile terminal so as to follow the viewpoint of a person viewing on the mobile terminal,
a face image capturing step in which the mobile terminal captures a face image of the person;
a face area detection step in which the mobile terminal detects the face area of the person from the face image;
a face part detection step in which the mobile terminal detects image coordinates of a face part including at least both eyes of the person from the face area;
a three-dimensional position calculation step in which the mobile terminal or the element image generation device calculates the three-dimensional position of the face part from the image coordinates of the face part;
an element image generation step in which the element image generation device generates the element image with the three-dimensional positions of both eyes included in the face part as viewpoint positions;
and an element image display step in which the mobile terminal displays the element images generated by the element image generation device.

Images