WO2023042266A1

WO2023042266A1 - Video processing device, video processing method, and video processing program

Info

Publication number: WO2023042266A1
Application number: PCT/JP2021/033764
Authority: WO
Inventors: 誉宗巻口; 大樹吹上; 卓佐野; 仁志瀬下
Original assignee: 日本電信電話株式会社
Priority date: 2021-09-14
Filing date: 2021-09-14
Publication date: 2023-03-23

Abstract

According to an embodiment, this video processing device comprises an image acquiring unit, an optimization processing unit, a pattern generating unit, and an image generating unit. The image acquiring unit acquires a left-eye viewpoint image obtained by imaging a display area from a left-eye position, a right-eye viewpoint image obtained by imaging the display area from a right-eye position, and a center viewpoint image obtained by imaging the display area from a center position between the left-eye position and the right-eye position. The optimization processing unit optimizes, on the basis of the left-eye viewpoint image and the right-eye viewpoint image, a phase shift amount and a weight both calculated on the basis of the center viewpoint image. The pattern generating unit generates, on the basis of the optimized phase shift amount and the optimized weight, a parallax induction pattern corresponding to the parallax between the left-eye position and the right-eye position. The image generating unit generates stereo-paired images on the basis of the center viewpoint image and the parallax induction pattern.

Description

VIDEO PROCESSING DEVICE, VIDEO PROCESSING METHOD, AND VIDEO PROCESSING PROGRAM

The present invention relates to a video processing device, a video processing method, and a video processing program.

The "HiddenStereo method" is known as a stereo image generation technology that applies the visual mechanism that works when humans perceive depth. In the HiddenStereo method, a parallax induced pattern having a phase difference of 90 degrees with respect to a reference image is generated, and a stereo pair of images created by adding or subtracting them to the reference image is displayed on a 3D display. By using this method, a user wearing 3D glasses can perceive a 3D image with binocular stereoscopic vision, and a user without 3D glasses can see a 2D image (reference image described above) without ghosts or double images. perceptible.

However, in this method, since the phase difference between the reference image and the parallax induction pattern is fixed at 90 degrees, the left and right parallax induction amounts with respect to the reference image are always equal. As a result, if the parallax is left-right symmetrical, for example, an object in front of the user can be accurately reproduced in depth, but if the parallax is left-right asymmetric, for example, a position away from the front of the user in the horizontal direction can be reproduced. It has been difficult to reproduce accurate depth in the case of objects such as This phenomenon is particularly conspicuous in objects at the edges of the screen on a large 3D display. Therefore, there is a demand for a stereo image generation technique that can appropriately reproduce depth even when parallax is left-right asymmetric.

An object of the present invention is to provide a video processing device, a video processing method, and a video processing program capable of appropriately reproducing depth even when parallax is left-right asymmetric.

According to the embodiment, the video processing device includes an image acquisition unit, an optimization processing unit, a pattern generation unit, and an image generation unit. The image acquisition unit obtains a left eye viewpoint image obtained by photographing the display area from the left eye position, a right eye viewpoint image obtained by photographing the display area from the right eye position, and an intermediate viewpoint image obtained by photographing the display area from an intermediate position between the left eye position and the right eye position. to get The optimization processing unit optimizes the phase shift amount and weight calculated based on the intermediate viewpoint image based on the left-eye viewpoint image and the right-eye viewpoint image. A pattern generator generates a parallax induction pattern corresponding to the parallax between the left eye position and the right eye position based on the optimized phase shift amount and the optimized weights. The image generator generates stereo pair images based on the intermediate viewpoint images and the parallax induction pattern.

According to the embodiments, it is possible to provide a video processing device, a video processing method, and a video processing program capable of appropriately reproducing depth even when parallax is left-right asymmetric.

FIG. 1 is an explanatory diagram illustrating parallax that occurs at a user's viewpoint when reproducing a 3D object using the video processing device according to the embodiment. FIG. 2 is a diagram illustrating an example of the configuration of the video processing device according to the embodiment; FIG. 3 is a diagram illustrating an example of the functional configuration of the video processing device according to the embodiment; FIG. 4 is an explanatory diagram illustrating an example of processing for generating stereo pair images by the video processing device according to the embodiment. FIG. 5 is a flowchart illustrating an example of processing executed by the video processing device according to the embodiment; FIG. 6 is an explanatory diagram illustrating an example of screen division when the video processing device according to the modification of the embodiment generates stereo pair images. FIG. 7 is an explanatory diagram illustrating an example of processing in which the video processing device according to the modification of the embodiment generates stereo pair images. FIG. 8 is a flowchart illustrating an example of processing executed by a video processing device according to a modification of the embodiment;

An embodiment of the present invention will be described in detail with reference to the drawings as appropriate.

FIG. 1 is an explanatory diagram for explaining parallax that occurs at the user's viewpoint when reproducing a 3D object using the video processing device according to the embodiment. In FIG. 1, a depth direction (direction indicated by arrows Y1 and Y2) and a horizontal direction (direction indicated by arrows X1 and X2) are defined. The depth direction intersects (perpendicularly or substantially perpendicularly to) the vertical direction. The horizontal direction intersects (perpendicular or nearly perpendicular) both the depth direction and the vertical direction. In FIG. 1, the user is positioned on the near side (arrow Y2 side) with respect to the actual display area RS in the depth direction. Therefore, the user perceives the 3D object by viewing the display area RS from the near side. At this time, let distance D be the depth width reproduced by the 3D object. The distance D is the distance in the depth direction between the real display area RS and the virtual display surface VS. Also, in FIG. 1, a left eye position PL, a right eye position PR, and an intermediate position PC are defined. The left eye position PL is the position of the assumed viewpoint corresponding to the left eye among the assumed viewpoints of both eyes of the user who perceives the 3D object. The right eye position PR is the position of the assumed viewpoint corresponding to the right eye among the assumed viewpoints of both eyes of the user who perceives the 3D object. The intermediate position PC is a horizontal position intermediate between the left eye position PL and the right eye position PR.

For example, when the user perceives a 3D object by viewing the visual recognition area DL, the user perceives a virtual point VL on a virtual display plane VS at a distance D away. At this time, as shown in the enlarged view of the visual recognition area DL, when the image of the display area RS is perceived at the left eye position PL and when the image of the display area RS is perceived at the intermediate position PC, the user sees the horizontal direction. A parallel parallax W1 is generated. Similarly, when the image of the display region RS is perceived at the right eye position PR and when the image of the display region RS is perceived at the intermediate position PC, the user has a parallax W2 along the horizontal direction. In this example, as shown in the enlarged view of the visible area DL, when the user views the visible area DL, the parallax W1 and the parallax W2 are almost equal.

On the other hand, when the user views the visible region DR, the visible region DR is further away from the user than the visible region DL in the horizontal direction. In this case, parallax W2 is larger than parallax W1, as shown in the enlarged view of visual recognition region DR. Thus, the parallax W1 and the parallax W2 change as the user's viewing area changes. In the video processing device 20 of the present embodiment, by generating a parallax induction pattern corresponding to such a change in parallax, even a 3D object displayed at a position horizontally distant from the user, for example, can be displayed by the user. can be perceived with correct depth representation.

FIG. 2 is a diagram showing an example of the configuration of the video processing device 20. As shown in FIG. The video processing device 20 is, for example, a computer. The video processing device 20 comprises a processor 201, a storage medium 202, a user interface 203, and a communication module 204, for example. Processor 201 , storage medium 202 , user interface 203 and communication module 204 are connected to each other via bus 205 .

The processor 201 includes any of a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an ASIC (Application Specific Integrated Circuit), a microcomputer, an FPGA (Field Programmable Gate Array), and a DSP (Digital Signal processor). . The storage medium 202 may include a secondary storage device in addition to a main storage device such as memory.

The main memory is a non-temporary storage medium. The main storage device is, for example, a non-volatile memory such as a HDD (Hard Disk Drive) or an SSD (Solid State Drive) that can be written and read at any time, a non-volatile memory such as a ROM (Read Only Memory). Also, a combination of these nonvolatile memories may be used. Secondary storage is a tangible storage medium. The auxiliary storage device is a combination of non-volatile memory and volatile memory such as RAM (Random Access Memory). In the video processing device 20, only one processor 201 and storage medium 202 may be provided, or a plurality of them may be provided.

In the video processing device 20, the processor 201 performs processing by executing programs and the like stored in the storage medium 202. In the video processing device 20, the program executed by the processor 201 may be stored in a computer (server) connected via a network such as the Internet, a server in a cloud environment, or the like. In this case, processor 201 downloads the program via the network.

In the user interface 203, the user of the video processing device 20 inputs various operations and the like, and information and the like to be notified to the user are notified by display or the like. The user interface 203 may be a display unit such as a display, or an input unit such as a touch panel or keyboard. A device connected to the video processing device 20 may be used as the input unit, or an input unit of another processing device capable of communicating via a network may be used.

FIG. 3 is a diagram showing an example of the functional configuration of the video processing device 20. As shown in FIG. As shown in FIG. 3, the video processing device 20 includes, for example, an image acquisition section 31, an optimization processing section 32, a pattern generation section 33, an image generation section 34, and a communication section . The processes of the image acquisition unit 31, the optimization processing unit 32, the pattern generation unit 33, the image generation unit 34, and the communication unit 35 are realized by the processor 201 and the communication module 204, for example.

The image acquisition unit 31 acquires viewpoint images used by the video processing device 20 . The image acquisition unit 31 is, for example, a camera. The optimization processing unit 32 executes predetermined processing based on the viewpoint image acquired by the image acquisition unit 31 . The pattern generation unit 33 generates a parallax induction pattern based on the viewpoint image acquired by the image acquisition unit 31 and the processing result of the optimization processing unit 32 . The parallax induction pattern realizes the parallax W1 and the parallax W2 described with reference to FIG. 1, for example, by performing predetermined processing together with a predetermined image that serves as a reference. The image generation unit 34 generates stereo pair images based on the parallax induction pattern generated by the pattern generation unit 33 and a predetermined reference image. The communication unit 35 transmits the stereo pair images generated by the image generation unit 34 by a predetermined method. For example, the communication unit 35 outputs the stereo pair images from the image output device by transmitting the stereo pair images to the image output device connected to the video processing device 20 .

Next, the method by which the video processing device 20 generates stereo pair images will be described in detail. FIG. 4 is an explanatory diagram for explaining an example of a process in which the video processing device 20 generates stereo pair images. The image acquiring unit 31 acquires the left-eye viewpoint image PLP, the intermediate viewpoint image PCP, and the right-eye viewpoint image PRP, as described above. The optimization processing unit 32 generates a parallax induction pattern ID based on these three images. In this embodiment, the intermediate viewpoint image PCP is used as a predetermined reference image (reference image). The optimization processing unit 32 transforms each of the left-eye viewpoint image PLP, intermediate-viewpoint image PCP, and right-eye viewpoint image PRP into frequency-phase components. Conversion to frequency-phase components is performed by the same process as the method described in Non-Patent Document 1. The phase component at frequency i and position j after conversion of the intermediate viewpoint image PCP into frequency-phase components is denoted by X(i, j). The phase component of frequency i and position j after conversion of the left eye viewpoint image PLP into frequency-phase components is denoted by L(i, j). The phase component of frequency i and position j after conversion of the right-eye viewpoint image PRP into frequency-phase components is denoted by R(i, j).

The optimization processing unit 32 phase-shifts the intermediate viewpoint image PCP by y degrees. The optimization processing unit 32 generates a phase-shifted intermediate viewpoint image PCP ^shift by adding a phase shift amount y(i, j) to the phase component X(i, j) of the intermediate viewpoint image PCP. The optimization processing unit 32 generates an estimated left-eye viewpoint image PLP ^asm by estimating the left-eye viewpoint image PLP by adding the intermediate viewpoint image PCP and the phase-shifted intermediate viewpoint image PCP ^shift . At this time, the optimization processing unit 32 multiplies the phase-shifted intermediate viewpoint image PCP ^shift by the weight A, and performs addition with the intermediate viewpoint image PCP. As for the value of the weight A, for example, a predetermined initial value is set in advance. Based on the estimated left-eye viewpoint image PLP ^asm , the optimization processing unit 32 calculates an estimated phase shift amount zL(i, j) from the intermediate viewpoint image PCP in the estimated left-eye viewpoint image PLP ^asm .

Similarly, the optimization processing unit 32 subtracts the phase-shifted intermediate viewpoint image PCP ^shift from the intermediate viewpoint image PCP to generate an estimated right-eye viewpoint image PRP ^asm by estimating the right-eye viewpoint image PRP. At this time, the optimization processing unit 32 multiplies the phase-shifted intermediate viewpoint image PCP ^shift by the weight A, and performs subtraction from the intermediate viewpoint image PCP. Based on the estimated right-eye viewpoint image PRP ^asm , the optimization processing unit 32 calculates an estimated phase shift amount zR(i, j) from the intermediate viewpoint image PCP in the estimated right-eye viewpoint image PRP ^asm .

The optimization processing unit 32 optimizes the set (A, y) of the weight A and the phase shift amount y under the condition of minimizing the error N represented by Equation (1). Since the estimated phase shifts zL(i,j) and zR(i,j) vary depending on both the weight A and the phase shift y, minimizing the error N yields the set (A,y) is optimized. A set of weight A and phase shift amount y is determined by exhaustive search, for example. By minimizing the error N, the optimization processing unit 32 calculates a set (A ^opt , y ^opt ) of the optimum weight A ^opt and the optimum phase shift amount y ^opt .

Based on the optimum weight A ^opt and the optimum phase shift amount y ^opt calculated by the optimization processing unit 32, the pattern generation unit 33 generates a parallax induction pattern ID. The parallax induction pattern ID is generated by the same process as the method described in Non-Patent Document 1. The image generation unit 34 generates one of the stereo pair images at the left eye position PL by adding the parallax induction pattern ID generated by the pattern generation unit 33 and the intermediate viewpoint image PCP. The image generation unit 34 generates the other of the stereo pair images at the right eye position PR by subtracting the parallax induction pattern ID generated by the pattern generation unit from the intermediate viewpoint image PCP. In this manner, the image generator 34 generates stereo pair images.

The optimization process described above will be described in detail in the case where the level at the viewpoint position phase θ is represented by the intensity of the sine wave. In this case, the intensity at the phase X of the intermediate position PC is represented as sin(X). At this time, if the phase X is phase-shifted by y degrees, the intensity after the phase shift is expressed as sin(X+y). The optimization processing unit 32 calculates the result of adding the intensity sin(X) at the phase X and Asin(X+y) obtained by weighting the intensity sin(X+y) after the phase shift with the weight A at the phase L at the left eye position PL. Estimated intensity. When the estimated intensity at this time is expressed using the phase X, the estimated phase shift amount zL, and the weight BL, it is expressed as Equation (2).

Therefore, the estimated phase shift amount zL is represented by Equation (3).

Similarly, the optimization processing unit 32 subtracts the result of subtracting Asin(X+y), which is obtained by weighting the phase-shifted intensity sin(X+y) with the weight A, from the intensity sin(X) at the phase X, Let be the estimated intensity at phase R. When the estimated intensity at this time is expressed using the phase X, the estimated phase shift amount zR, and the weight BR, it is expressed as in Equation (4).

Therefore, the estimated phase shift amount zR is represented by Equation (5).

Using these, the optimization processing unit 32 optimizes the set (A, y) of the weight A and the phase shift amount y under the condition of minimizing the error N represented by Equation (1).

FIG. 5 is a flowchart illustrating an example of processing executed by the video processing device 20 according to the embodiment. The processing in FIG. 5 is repeatedly executed at the timing when the video processing device 20 generates stereo pair images. Therefore, the process of FIG. 5 is an example of a flowchart in one process of image generation processing for generating stereo pair images.

At the timing of generating the stereo pair images, the video processing device 20 acquires the viewpoint images by the image acquiring unit 31 (S501). At this time, the image acquisition unit 31 acquires the left-eye viewpoint image PLP, the intermediate viewpoint image PCP, and the right-eye viewpoint image PRP. The video processing device 20 converts the acquired left-eye viewpoint image PLP, intermediate-viewpoint image PCP, and right-eye viewpoint image PRP into frequency-phase components as described above (S502). The video processing device 20 generates the phase-shifted intermediate viewpoint image PCP ^shift as described above (S503). As described above, the video processing device 20 generates the estimated left-eye viewpoint image PLP ^asm and the estimated right-eye viewpoint image PRP ^asm based on the intermediate viewpoint image PCP and the phase-shifted intermediate viewpoint image PCP ^shift (S504). . The video processing device 20 estimates an estimated phase shift amount zL(i, j) based on the estimated left-eye viewpoint image PLP ^asm , and estimates an estimated phase shift amount zR(i, j) based on the estimated right-eye viewpoint image PRP ^asm . (S505). The image processing device 20 optimizes the set of the phase shift component y(i,j) and the weight A(i,j) under the condition of minimizing the error N as described above (S506). The video processing device 20 generates the parallax induction pattern ID based on the optimum weight A ^opt and the optimum phase shift amount y ^opt as described above (S507). As described above, the video processing device 20 generates stereo pair images based on the intermediate viewpoint image PCP and the parallax induction pattern ID (S508). As described above, the video processing device 20 completes the stereo pair image generation processing.

In this embodiment, the video processing device 20 includes an image acquisition section 31 , an optimization processing section 32 , a pattern generation section 33 and an image generation section 34 . The image acquisition unit 31 obtains a left-eye viewpoint image PLP obtained by photographing the display region RS from the left-eye position PL, a right-eye viewpoint image PRP obtained by photographing the display region RS from the right-eye position PR, and a display region from the middle between the left-eye position PL and the right-eye position PR. and an intermediate viewpoint image PCP obtained by photographing the RS. The optimization processing unit 32 optimizes the phase shift amount y and the weight A calculated based on the intermediate viewpoint image PCP based on the left eye viewpoint image PLP and the right eye viewpoint image PRP. The pattern generator 33 generates a parallax induction pattern ID corresponding to the parallax between the left eye position PL and the right eye position PR based on the optimized optimum phase shift amount y ^opt and the optimized optimum weight A ^opt . . The image generator 34 generates stereo pair images based on the intermediate viewpoint image PCP and the parallax induction pattern ID. In this way, since the video processing device 20 optimizes the phase shift amount and the weight A based on the left-eye viewpoint image PLP and the right-eye viewpoint image PRP, it is possible to provide the user with an appropriate parallax even when the parallax is left-right asymmetric. can be done. Therefore, the video processing device 20 can provide appropriate depth representation to the user.

(Modification)
FIG. 6 is an explanatory diagram illustrating an example of a method of dividing the display area RS when the video processing device 20 according to the modification generates stereo pair images. In this modification, the actual display area RS is divided into a predetermined number with reference to the intermediate position PC, and the phase shift amount y and the weight A are optimized for each divided area. That is, the phase shift amount y is the same and the weight A is the same within the divided regions. The actual number of divisions of the display area RS is not particularly limited. In the example of FIG. 6, the actual display area RS is divided into three areas, left area AL, center area AC, and right area AR. In this case, the image processing device 20 optimizes the phase shift amount y and the weight A in the left area AL, optimizes the phase shift amount y and the weight A in the central area AC, and optimizes the phase shift amount y and the weight A in the right area AR. to optimize.

FIG. 7 is an explanatory diagram illustrating an example of processing for generating stereo pair images by the video processing device 20 according to the modification. In this modified example, when the optimization processing unit 32 phase-shifts the intermediate viewpoint image PCP by y degrees, a common phase shift amount y ^part is used in each of the divided regions into which the display region RS is divided. The optimization processing unit 32 adds a phase shift amount y ^part to the phase component X(i, j) of the intermediate viewpoint image PCP in each of the divided regions, thereby obtaining a phase-shifted intermediate viewpoint image (PCP ^shift ) ^part . , generated for each segmented region.

The optimization processing unit 32 adds the intermediate viewpoint image PCP and the phase-shifted intermediate viewpoint image (PCP ^shift ) ^part for each divided region, thereby estimating the corresponding portion of the left eye viewpoint image PLP to obtain an estimated left eye viewpoint image. Generate (PLP ^asm ) ^part . At this time, the optimization processing unit 32 multiplies the phase-shifted intermediate viewpoint image (PCP ^shift ) ^part by the weight A ^part , and performs addition with the intermediate viewpoint image PCP. The optimization processing unit 32 estimates the estimated phase shift amount zL ^part from the intermediate viewpoint image PCP based on the estimated left-eye viewpoint image (PLP ^asm ) ^part for each divided area.

The optimization processing unit 32 calculates a set (A part , y part ⁾ of the weight A ^part and the phase shift amount y ^part for each divided region under the condition of minimizing the error N represented by Equation (1) ^. Optimize. Based on the optimum weight (A ^part ) ^opt and the optimum phase shift amount (y ^part ) ^opt calculated for each divided area by the optimization processing unit 32, the pattern generation unit 33 generates a parallax induction pattern ID ^part for each divided area. do. The image generating unit 34 adds the parallax induction pattern ID ^part generated by the pattern generating unit 33 and the part of the intermediate viewpoint image PCP corresponding to the divided area to obtain a stereo pair at the left eye position PL corresponding to the divided area. Generate one of the images. The image generation unit 34 subtracts the parallax induction pattern ID ^part generated by the pattern generation unit 33 from the intermediate viewpoint image PCP corresponding to the division area, thereby generating a stereo pair image at the right eye position PR corresponding to the division area. generate the other. After completing the generation of the stereo pair images corresponding to the divided regions in all the divided regions, the image generator 34 synthesizes the stereo pair images corresponding to the respective divided regions to obtain the stereo pair images corresponding to the display region RS. to generate

FIG. 8 is a flowchart illustrating an example of processing executed by the video processing device 20 of this modification. At the timing of generating the stereo pair images, the video processing device 20 acquires the viewpoint images by the image acquiring unit 31 (S801). At this time, the image acquisition unit 31 acquires the left-eye viewpoint image PLP, the intermediate viewpoint image PCP, and the right-eye viewpoint image PRP. Video processing device 20 converts acquired left eye viewpoint image PLP, middle viewpoint image PCP, and right eye viewpoint image PRP into frequency-phase components as described above (S802). The video processing device 20 generates a phase-shifted intermediate viewpoint image (PCP ^shift ) ^part for each divided area as described above (S803). As described above, the video processing device 20 generates an estimated left-eye viewpoint image (PLP ^asm ) ^part ^and an estimated right-eye viewpoint image ( ^PRP ^asm ) ^part is generated for each divided area (S804).

The video processing device 20 estimates an estimated phase shift amount zL ^part for each divided area based on the estimated left-eye viewpoint image (PLP ^asm ) ^part , and estimates an estimated phase shift amount zR ^part based on the estimated right-eye viewpoint image (PRP ^asm ) ^part . is estimated for each divided area (S805). As described above, the image processing device 20 optimizes the set of the phase shift amount y ^part and the weight A ^part for each divided area under the condition that the error N is minimized (S806). As described above, the image processing device 20 generates the parallax induction pattern ID ^part for each divided area based on the optimum weight (A ^part ) ^opt and the optimum phase shift amount (y ^part ) ^opt (S807). As described above, the video processing device 20 generates stereo pair images for each divided area based on the intermediate viewpoint image PCP and the parallax induction pattern ID ^part (S808). The video processing device 20 generates a stereo pair image corresponding to the display area RS by synthesizing the stereo pair images for each divided area (S809). As described above, the video processing device 20 completes the generation of stereo pair images.

Also in this modified example, as in the above-described embodiment, the video processing device 20 optimizes the phase shift amount y and the weight A based on the left-eye viewpoint image PLP and the right-eye viewpoint image PRP, so the parallax is left-right asymmetric. Appropriate parallax can be given to the user even in this case. Therefore, the video processing device 20 can give the user an appropriate representation of depth.

In another modification, the video processing device 20 may acquire the left eye position PL, right eye position PR, and intermediate position PC in real time. Acquisition of the left eye position PL, right eye position PR, and intermediate position PC in real time is performed, for example, by head tracking of the user. In this case, the video processing device 20 generates stereo pair images in real time based on the left eye position PL, right eye position PR, and intermediate position PC acquired in real time. Therefore, the video processing device 20 executes the processing shown in FIG. 5 or 8 each time the left eye position PL, right eye position PR, and intermediate position PC are updated. By acquiring the left eye position PL, the right eye position PR, and the intermediate position PC in real time in this way, stereo pair images reflecting the user's viewpoint in real time are generated. Therefore, the video processing device 20 can provide the user with more appropriate depth representation.

The methods described in the above-described embodiments and the like can be stored and distributed as programs (software) that can be executed by computers, for example, in storage media such as magnetic disks, optical disks, and semiconductor memories. Storage media are not limited to those for distribution, and include storage media such as magnetic disks and semiconductor memories provided inside computers or devices connected via a network. Also, the techniques described in the embodiments may be transmitted and distributed over a communication medium. The programs stored on the medium side also include a setting program that configures in the computer software to be executed by the computer. Software includes not only execution programs but also tables and data structures. A computer that realizes this system reads a program recorded in a storage medium and executes the above-described processing by controlling the operation by software. The software may be constructed by a computer using a configuration program.

It should be noted that the present invention is not limited to the above-described embodiments, and can be variously modified in the implementation stage without departing from the gist of the present invention. Further, each embodiment may be implemented in combination as appropriate, in which case the combined effect can be obtained. Furthermore, various inventions are included in the above embodiments, and various inventions can be extracted by combinations selected from a plurality of disclosed constituent elements. For example, even if some constituent elements are deleted from all the constituent elements shown in the embodiments, if the problem can be solved and effects can be obtained, the configuration with the constituent elements deleted can be extracted as an invention.

DESCRIPTION OF SYMBOLS 20... Video processing apparatus 201... Processor 202... Storage medium 203... User interface 204... Communication module 31... Image acquisition part 32... Optimization process part 33... Pattern generation part 34... Image generation part 35... Communication part ID... Parallax induction pattern RS...Display area PL...Left eye position PR...Right eye position PC...Intermediate position PLP...Left eye viewpoint image PRP...Right eye viewpoint image PCP...Intermediate viewpoint image

Claims

A left eye viewpoint image obtained by photographing the display area from the left eye position, a right eye viewpoint image obtained by photographing the display area from the right eye position, and an intermediate viewpoint image obtained by photographing the display area from the intermediate position between the left eye position and the right eye position. an image acquisition unit to acquire;
an optimization processing unit that optimizes the phase shift amount and weight calculated based on the intermediate viewpoint image based on the left-eye viewpoint image and the right-eye viewpoint image;
a pattern generator that generates a parallax induction pattern corresponding to the parallax between the left eye position and the right eye position based on the optimized phase shift amount and the optimized weight;
an image generation unit that generates a stereo pair image based on the intermediate viewpoint image and the parallax induction pattern;
A video processing device comprising:
The optimization processing unit
shifting the phase of the intermediate viewpoint image based on the phase shift amount;
generating an estimated left-eye viewpoint image by estimating the left-eye viewpoint image and an estimated right-eye viewpoint image by estimating the right-eye viewpoint image based on the weight, the intermediate viewpoint image, and the phase-shifted intermediate viewpoint image;
calculating an estimated phase shift amount from the intermediate viewpoint image in the estimated left eye viewpoint image and an estimated phase shift amount from the intermediate viewpoint image in the estimated right eye viewpoint image;
optimizing the weight and the phase shift amount based on the estimated phase shift amount of the estimated left-eye viewpoint image and the estimated phase shift amount of the right-eye viewpoint image;
The video processing device according to claim 1.
The image generator is
generating an image corresponding to the left eye position among the stereo pair images by adding the intermediate viewpoint image and the parallax induction pattern;
generating an image corresponding to the right eye position among the stereo pair images by subtracting the parallax induction pattern from the intermediate viewpoint image;
The video processing device according to claim 1.
The display area includes a plurality of the display areas,
The optimization processing unit optimizes the phase shift amount and the weight in each of the plurality of display areas,
The pattern generation unit generates the parallax induction pattern in each of the plurality of display areas,
The image generation unit generates the stereo pair images in each of the plurality of display areas, and converts the stereo pair images corresponding to the plurality of display areas to the stereo pair images corresponding to the entire display area. to generate
The video processing device according to claim 1.
the image acquisition unit acquires the left eye position, the right eye position, and the intermediate position in real time;
The video processing device according to claim 1.
A left eye viewpoint image obtained by photographing the display area from the left eye position, a right eye viewpoint image obtained by photographing the display area from the right eye position, and an intermediate viewpoint image obtained by photographing the display area from the intermediate position between the left eye position and the right eye position. to obtain;
optimizing the phase shift amount and weight calculated based on the intermediate viewpoint image based on the left-eye viewpoint image and the right-eye viewpoint image;
generating a parallax induced pattern corresponding to the parallax between the left eye position and the right eye position based on the optimized phase shift amount and the optimized weights;
generating a stereo pair image based on the intermediate viewpoint image and the parallax induction pattern;
A video processing method comprising:
to the computer,
A left eye viewpoint image obtained by photographing the display area from the left eye position, a right eye viewpoint image obtained by photographing the display area from the right eye position, and an intermediate viewpoint image obtained by photographing the display area from the intermediate position between the left eye position and the right eye position. let me get
optimizing the phase shift amount and weight calculated based on the intermediate viewpoint image based on the left eye viewpoint image and the right eye viewpoint image;
generating a parallax induced pattern corresponding to the parallax between the left eye position and the right eye position based on the optimized phase shift amount and the optimized weight;
generating a stereo pair image based on the intermediate viewpoint image and the parallax induction pattern;
video processing program.