JP4617965B2 - Image processing method, apparatus and program - Google Patents

Description

  The present invention relates to an image processing method, apparatus and program for performing stereo image matching processing.

In order to generate a virtual camera viewpoint image at a viewpoint position (virtual viewpoint) that does not actually exist from a plurality of captured images, an inter-image matching technique is indispensable. Such a correlation technique between images is one of the most important and difficult problems in “stereo vision”.
For example, let us consider the correspondence of stereo vision by two cameras, which is the most basic configuration. For a point on one image, the corresponding point is on a straight line in the other image observed from another viewpoint. In general, this straight line L1 ′ is referred to as an epipolar line. Conventionally, “corresponding point matching” methods that are often used include pixel-based matching, area-based matching, and feature-based matching.
In pixel-based matching, the correspondence between points in one image is searched for in the other image as it is (Non-patent Document 1). In area-based matching, when searching for the correspondence of a point in one image in the other image, a search is performed using a local image pattern around the point (Non-Patent Documents 2 and 3). In feature-based matching, features such as shading edges are extracted from images, and association is performed using only features between images (Non-Patent Documents 4 and 5). In these methods, a certain pixel Pa (x, y) on the reference camera image is basically searched on the epipolar line corresponding to the other camera image, and the pixel having the highest similarity is used as the corresponding point.
Then, the virtual viewpoint image is generated by mixing the pixel values of the pixels of the two corresponding points at a ratio according to the position of the virtual viewpoint.
C. Lawrence Zitnick and Jon A. Webb: Multi-baseline Stereo Using Surface Extraction, Technical Report, CMU-CS-96-196, (1996) Okutomi.M and Kanade.T: A locally adaptive window for signal matching, Int. Journal of Computer Vision, 7 (2), pp.143-162, (1992) Okutomi, Kinde: Stereo matching using multiple baseline lengths, IEICE Transactions D-II, Vol. J75-D-II, No. 8, pp. 1317-1327, (1992) H. Baker and T. Binford: Depth from edge and intensity based stereo, In Proc.IJCAI'81, (1981) WELGrimson: Computational experiments with a feature based stereo algorithm, IEEE Trans.PAMI, Vol.7, No.1, pp.17-34, (1985) Ohta.Y and Kanade.T .: Stereo by intra- and inter-scanline search using dynamic programming, IEEE PAMI-7 (2), 139-154, (1985) Cox IJ et al .: A Maximum likelihood stereo algorithm, Computer Vision and Image Understanding, 63 (3), 542-567, (1996) Y.Liu et al .: Eye-Contact Visual Communication with Virtual View Synthesis, Proc.CCNC2005, (2005)

  Conventionally, as shown in FIG. 20, there is a problem that the virtual viewpoint image is deteriorated when the corresponding point associated by the matching process is in the occlusion area.

  The present invention has been made in view of the above-described problems of the prior art, and an object thereof is to provide an image processing method, an apparatus, and a program for generating a high-quality virtual viewpoint image even when there is an occlusion area.

The image processing method of the light first emitting, for each of the plurality of images, based on a first step of generating the parallax data without matching method of sequentially corresponding constraint, the parallax data generated in the first step In addition, for each of the plurality of images, a second step of separating the image into a non-order-corresponding image whose parallax is greater than or equal to a reference and other order-corresponding images, and the second step of the plurality of images more in a separate said order corresponding images and non-sequentially corresponding image, a third step of performing individual matching process by the matching technique has the sequence corresponding restraint, associated with the matching processing of the third step A fourth step of determining whether or not a pixel of the pixel exists in the occlusion region, and based on the determination result of the fourth step, the pixel is present in the occlusion region of the plurality of associated pixels Do not pixel Using pixel values, possess a fifth step of generating a virtual pixel values corresponding to the plurality of pixels in the virtual viewpoint image, the said fifth step, the determination result of the fourth step Based on the determination result of the fourth step, the virtual pixel value is generated using the pixel values of the two pixels when both of the two associated pixels do not exist in the occlusion region. The order-corresponding image and the non-order-corresponding image separated in the second step when one of the two pixels is present in the occlusion region and the other is not present in the occlusion region. For at least one of the two, the virtual pixel value corresponding to the two pixels in the virtual viewpoint image is generated using the pixel value of the pixel that does not exist in the occlusion area among the two associated pixels.

Second inventions of the image processing apparatus, for each of the plurality of images to generate parallax data without matching method of sequentially corresponding constraint, based on the parallax data, for each of the plurality of images, the image Separating means for separating the non-order-corresponding image whose parallax is equal to or higher than the reference and other order-corresponding images, and the order-corresponding image and the non-order-corresponding image separated by the separating means of the plurality of images. A matching means for performing matching processing individually by a matching method having a correspondence constraint; a determination means for determining whether or not a plurality of pixels associated by the matching processing of the matching means are present in an occlusion region; and Based on the determination result of the determination means, the pixel value of the pixel that does not exist in the occlusion area among the plurality of associated pixels is used to Possess a generating means for generating a virtual pixel value corresponding to the number of pixels, wherein the generating means, based on the determination result of said determining means, both occlusion regions of the two pixels the associated The virtual pixel value is generated using the pixel values of the two pixels, and one of the two pixels is present in the occlusion region based on the determination result of the determination unit, When the other does not exist in the occlusion area, at least one of the order-corresponding image and the non-order-corresponding image separated by the separating unit exists in the occlusion area among the two associated pixels. A virtual pixel value corresponding to the two pixels in the virtual viewpoint image is generated using the pixel value of the non-performed pixel.

Third inventions of the program is a program executed by the computer image processing apparatus has, for each of the plurality of images, a first procedure for generating the parallax data without matching method of sequentially corresponding constraint, the A second procedure for separating each of the plurality of images into an unordered image having a parallax of a reference or higher and an other image corresponding to the order based on the parallax data generated in the first procedure. A third procedure for individually matching the order-corresponding image and the non-order-corresponding image separated in the second procedure of the plurality of images by a matching method having the order correspondence constraint , a fourth step in which a plurality of pixels associated with the matching process of step 3 it is determined whether or not present in the occlusion region, based on the determination result of the fourth step, the urging corresponding It was using the pixel values of the pixels that do not exist in the occlusion region of the plurality of pixels, and a fifth step of generating a virtual pixel values corresponding to the plurality of pixels in the virtual viewpoint image, the said computer In the fifth procedure, based on the determination result of the fourth procedure, when both of the two associated pixels do not exist in the occlusion area, the pixel values of the two pixels are calculated. The virtual pixel value is used to generate one of the two pixels in the occlusion area and the other is not in the occlusion area based on the determination result of the fourth procedure. The pixel value of a pixel that does not exist in the occlusion area among the two associated pixels with respect to at least one of the order-corresponding image and the non-order-corresponding image separated in the second procedure. Used to generate a virtual pixel values corresponding to the two pixels in the virtual viewpoint image.

  According to the present invention, it is possible to provide an image processing method, apparatus and program for generating a high-quality virtual viewpoint image even when there is an occlusion area.

Hereinafter, the best mode for carrying out the present invention will be described in detail with reference to the drawings.
First, an example of the correspondence relationship between the configuration of the present embodiment and the configuration of the present invention will be described.
The parallax map data Dm1 and Dm2 are an example of the parallax data of the present invention.
The unordered image data Pe1b and Pe2b are an example of the unordered image data of the present invention.
Order-corresponding image data Pe1a, Pe2a, Pe1c, and Pe2c are examples of the order-corresponding image data of the present invention.

Steps ST6 and ST8 shown in FIG. 18 are an example of the first step of the present invention.
Steps ST7 and ST9 shown in FIG. 18 correspond to the second and third steps of the present invention.
Here, steps ST21 and ST23 shown in FIG. 15 are examples of the second step of the present invention, and steps ST22, ST24 and ST25 are examples of the third step of the present invention.
Step ST3 shown in FIG. 18 is an example of the fourth process of the present invention, and step ST4 is an example of the fifth process.
Step ST10 shown in FIG. 18 is an example of step ST6 of the present invention.

The image parallelization processing unit 20 shown in FIG. 2 is an example of the disparity generation unit of the present invention, the non-order corresponding region separation unit 28 is an example of the separation unit of the present invention, and the DP matching processing unit 29 is the matching of the present invention. It is an example of a processing means.
A program PRG shown in FIG. 18 is an example of the program of the present invention.

In the communication system 1 to which the present invention is applied, for example, as shown in FIG. 1, a user a at a point A and a user b at a point B are remotely interacting with each other while viewing a display image of the other party from a distant place. System.
At the point A, a camera 11a and a camera 12a that capture the user a as a subject to be photographed from different viewpoints, and a display 5a for displaying an image of the user b captured at the point B side to the user a, An image processing apparatus 2a is provided that generates a virtual viewpoint image Ima based on the images Pa1 and Pa2 captured by the cameras 11a and 12a and transmits the virtual viewpoint image Ima to the point B via the network 7.
At the point B, a camera 11b and a camera 12b that capture images of the user b as a photographing target from different viewpoints, and a display 5b for displaying an image of the user a captured at the point A side to the user b, An image processing device 2b that generates a virtual viewpoint image Imb based on the images Pb1 and Pb2 captured by the cameras 11b and 12b and transmits the image to the point A via the network 7 is disposed.
The virtual viewpoint images Ima and Imb generated by the image processing apparatuses 2a and 2b are images picked up by a virtual camera virtually installed near the center of the display 5a or 5b on which the display image of the other party is projected. It corresponds to.
The cameras 11a and 11b are respectively installed on the left side of the displays 5a and 5b when viewed from the users a and b, and the cameras 12a and 12b are respectively installed on the right side of the display when viewed from the users a and b. It becomes. The cameras 11 and 12 are installed with the shooting direction and the shooting angle of view being fixed, but these may be freely changed based on information input from the users a and b. By the way, in this communication system 1, description will be given by taking as an example a case where an imaging target is imaged by two cameras installed in accordance with the user's line of sight.
The displays 5a and 5b display images based on the virtual viewpoint images Imb and Ima supplied from the counterpart point via the network 7 via a liquid crystal display surface, for example. The liquid crystal display surfaces of the displays 5a and 5b are composed of a large number of liquid crystal display elements and the like, and the liquid crystal display elements are optically modulated in accordance with output signals based on the virtual viewpoint images Imb and Ima to create an image to be displayed to the user.

The image processing apparatuses 2a and 2b are usually constituted by electronic devices such as a personal computer (PC). These image processing apparatuses 2a and 2b have a function of communicating with each other via the network 7, and transmit images and sounds in response to requests from the other party. The configuration of the image processing apparatuses 2a and 2b will be described in detail later.
The network 7 includes information such as an Internet network connected to the image processing apparatus 2 via a telephone line, ISDN (Integrated Services Digital Network) / B (broadband) -ISDN connected to a TA / modem, and the like. It is a public communication network that enables two-way transmission / reception. Incidentally, when the communication system 1 is operated in a certain narrow area, the network 7 may be configured by a LAN (Local Area Network). Further, when transmitting moving images, the network 7 is continuously transmitted from one channel having moving images including, for example, MPEG (Moving Picture Experts Group) data, based on the Internet protocol (IP). . In addition, when transmitting a still image, the image is transmitted at regular intervals from a channel different from the channel for transmitting a moving image. Note that a network server (not shown) may be connected to the network 7. This network server (not shown) manages, for example, Internet information, receives a request from the image processing apparatus 2, and transmits predetermined information stored in itself.

Hereinafter, the image processing apparatus 2a will be described.
FIG. 2 is a block diagram of the image processing apparatus 2a shown in FIG.
The image processing device 2b has the same configuration as the image processing device 2a.
Hereinafter, the configuration of the image processing apparatus 2a will be described.
As shown in FIG. 2, the image processing apparatus 2a includes, for example, an image parallelization processing unit 20, a calibration unit 26, an unordered corresponding region detection unit 27, an unordered corresponding region separation unit 28, a DP matching processing unit 29, Each region has a virtual viewpoint image generation unit 30 and a virtual viewpoint image generation unit 31.

The image processing apparatus 2a detects and separates regions that cannot satisfy the order correspondence constraint as described above based on distance information, and obtains corresponding points by DP matching processing for each separated region. Then, based on the virtual camera viewpoint position information, virtual viewpoint videos of each region are generated, and finally, the virtual viewpoint videos are synthesized as a single virtual viewpoint video so that the one with a short distance is located in front.
In the image processing apparatus 2a, each region virtual viewpoint image generation unit 30 generates a virtual viewpoint pixel value (virtual pixel value) based on the pixel values of two pixels associated in the DP matching process.

[Image parallel processing unit 20]
The image parallel processing unit 20 performs geometric image correction processing and normalization processing on the image data Pa1 and Pa2 input from the cameras 11a and 12a.
The image parallelization processing unit 20 corrects the image data Pa1 and Pa2 based on the control information including the geometric positional relationship between the cameras 11a and 12a. The geometric positional relationship between the cameras 11a and 12a may be parameterized based on the control information generated by the camera calibration unit 26 described above.
Further, when imaging is performed while changing the shooting direction and the shooting angle of view of each camera 11a, 12a, these are parameterized in the camera calibration unit, and these parameters are used as control information when correcting the image. It may be included. Thereby, the image parallelization processing unit 20 can perform image correction in real time according to the shooting direction and / or the shooting angle of view of each camera 11a, 12a.
The camera calibration unit 26 similarly shows chromatic aberration, distortion, and optical axis shift in each lens of the cameras 11a and 12a, for example, in Japanese Patent Laid-Open No. 2000-350239 and Japanese Patent Laid-Open No. 11-53549. The parameters may be parameterized based on the camera calibration method, and these may be output to the image parallelization processing unit 20. In this case, the image parallelization processing unit 20 adjusts the images from the cameras 11a and 12a to the reference image by projective transformation based on the acquired parameters. Similarly, the luminance component and the like are adjusted with each other by using a projective transformation matrix that minimizes the luminance error between the images from the cameras 11a and 12a.

A user a as a subject to be photographed is photographed from different angles by the cameras 11a and 12a. As a result, the line-of-sight direction, the face direction, etc. of the user a on the image data Pa1, Pa2 generated by the cameras 11a, 12a are different from each other. Such images Pa1 and Pa2 are respectively supplied to the image parallelization processing unit 20, and are based on parameters such as lens distortion and image center of each camera 11a and 12a obtained in advance by the camera calibration unit 26. And correct so that the center of the image does not shift.
The image parallelization processing unit 20 generates normalized image data Pm1 and Pm2 by normalizing the corrected image data Pa1 and Pa2 as shown below, and outputs this to the non-order corresponding region detection unit 27. .
FIG. 3 shows a case where the image data Pa1 and Pa2 captured by the cameras 11a and 12a are normalized. As shown in FIG. 3, when imaging is performed by aligning the optical axis from different viewpoints to M points to be photographed by the optical centers C1 and C2 of the cameras 11a and 12a, on the imaging surfaces of the cameras 11a and 12a. The normal directions k1 and k2 of the obtained images Pa1 and Pa2 indicate different directions. That is, the direction of the straight line connecting each camera 11a, 12a and the point M coincides with the normal direction k1, k2 of each image data Pa1, Pa2 imaged by each camera, but these indicate different directions. . By performing geometric normalization on each of the images Pa1 and Pa2, normalized image data Pm1 and Pm2 whose image planes having the normal directions k1 ′ and k2 ′ in the same direction are parallel to each other are generated.
This geometric normalization is performed by estimating camera internal parameters A1, A2, rotation matrices R1, R2, and transfer matrices T1, T2 using projection matrices P1, P2 obtained in advance by the camera calibration unit 26. Realize. As a result, it is possible to generate normalized image data Pm1 and Pm2 in which the normal directions k1 ′ and k2 ′ of the corrected image pickup surface are parallelized.
Incidentally, when performing this geometric normalization, a virtual plane π including the optical centers C1 and C2 is set, and the normal lines of the normalized images Pm1 and Pm2 with respect to the normal direction of the virtual plane π. You may make it normalize using projection matrix P1, P2 so that a direction may become the same direction.

[Non-order corresponding area detection unit 27]
The non-order corresponding region detection unit 27 receives the normalized image data Pm1 and Pm2 from the image parallelization processing unit 20.
As shown in FIG. 4, the non-order corresponding region detection unit 27 removes the background image data Pk1 and Pk2 from the normalized image data Pm1 and Pm2, and generates foreground image data Pf1 and Pf2.
In this manner, by removing the background image, it is possible to perform an efficient matching process that reduces the influence of the background image.

The non-order corresponding region detection unit 27 generates (estimates) the parallax map data Dm1 and Dm2 based on the foreground image data Pf1 and Pf2 by a stereo matching method without order correspondence constraint.
In the present embodiment, an area-based matching method or the like is used as a stereo matching method with no order correspondence constraint.
For each of the parallax map data Dm1 and Dm2, the non-order corresponding region detection unit 27 sets pixel data constituting the data to pixel data (previous pixel data) that is equal to or greater than a predetermined threshold value and less than the threshold value. And pixel data.
Then, the non-order-corresponding region detection unit 27 generates non-order-corresponding region parallax data composed of pixel data equal to or greater than the threshold value.
For example, as shown in FIG. 5, the non-order corresponding area detection unit 27 generates non-order corresponding area parallax data CL1 and CL2 from the parallax map data Dm1 and Dm2, respectively.
The non-order corresponding region detection unit 27 outputs the foreground image data Pf1 and Pf2 and the non-order corresponding region parallax data CL1 and CL2 to the non-order corresponding region separation unit 28.

[Non-order corresponding region separation unit 28]
As shown in FIG. 5, the non-order corresponding region separation unit 28 extracts non-order corresponding image data Pe1b and Pe2b from the foreground image data Pf1 and Pf2 based on the non-order corresponding region parallax data CL1 and CL2. The data is output to the DP matching processing unit 29.
Further, the non-order corresponding region separation unit 28 extracts the order corresponding image data from the foreground image data Pf1 and Pf2 based on the non-order corresponding region parallax data CL1 and CL2.
Then, as shown in FIG. 6, the non-order-corresponding region separation unit 28 converts the portion of the non-order-corresponding image data in the extracted order-corresponding image data into a label image (fixed luminance and color difference with predetermined pixel values). Order-corresponding image data Pe1a and Pe2a replaced with (labeled) images are generated and output to the DP matching processing unit 29.

[DP matching processing unit 29]
The DP matching processing unit 29 performs DP matching processing between the order corresponding image data Pe1a and Pe2a.
The DP matching processing unit 29 outputs the order correspondence image data Pe1c and Pe2c obtained by correcting the labels of the order correspondence image data Pe1a and Pe2a and the DP matching processing result DPa to each region virtual viewpoint image generation unit 30.
Further, the DP matching processing unit 29 performs DP matching processing between the non-order corresponding image data Pe1b and Pe2b.
The DP matching processing unit 29 outputs the unordered correspondence image data Pe1b and Pe2b and the DP matching processing result DPb to each region virtual viewpoint image generation unit 30.

First, DP matching processing of the order corresponding image data Pe1a and Pe2a will be described.
As described above, the DP matching processing unit 29 individually performs the DP matching process between the order-corresponding image data Pe1a and Pe2a and the DP matching process between the non-order-corresponding image data Pe1b and Pe2b.
As a result, as shown in FIGS. 7A and 7B, the order correspondence image data (body region) and the non-order correspondence image data (hand region) can be correctly associated.
Therefore, the virtual viewpoint image generation unit 31 in the subsequent stage can generate the virtual viewpoint image shown in FIG.

Conventionally, as shown in FIG. 8, the foreground images pic1 and pic2 are associated with each imaging position while being associated with the imaging target. For the foreground images pic1 and pic2, when the pixels on the scan line L4 in the foreground image pic1 in which the user a as the subject is imaged are associated with the pixels on the scan line L4 ′ in the foreground image pic2, the scan line The point sequence R1 of the feature points on L4 is {c1, c2, c3, c4} in order from the left, and the point sequence R2 of the feature points on the scan line L4 ′ is {d1, d2, d3, d4} in order from the left. To do.
Here, when the feature points R1 and R2 on the scan lines L4 and L4 ′ are associated with each other in the relationship with the subject, first, the feature point on L4 ′ corresponds to c1 and corresponds 1: 1. However, the feature point on L4 ′ corresponds to the feature point c2 constituting the neck of the user “a”. Similarly, d2 and d4 correspond to the feature points on L4 ′ with respect to the feature points c3 and c4 constituting the hand of the user a.
In this way, in the foreground images pic1 and pic2 obtained by imaging from different viewpoints, the feature point associations intersect between image regions having a large parallax based on the distance from the subject to the camera. As a result, the disturbance of the image as shown in FIG. 9 occurs for the virtual viewpoint image created in the virtual viewpoint image generation unit at the subsequent stage.

  Incidentally, as shown in FIG. 7, the DP matching processing unit 29 can correctly associate the order correspondence image data (body region) and the non-order correspondence image data (hand region). As shown in FIG. 7A, when the DP matching process is performed as it is for the order-corresponding image data Pe1a and Pe2a in which the part of the non-order-corresponding image data in the extracted order-corresponding image data is replaced (labeled) with the label image. ) Originally, the left image L1 should correspond to the right image R1 ′, but the left image L1 corresponds to the right image R1. As a result, in the virtual viewpoint video shown in FIG. 7C, the position of the contour of the face in the virtual viewpoint video using the coordinate values of L1 of the left image and R1 of the right image is shifted, and the virtual image having a sense of incongruity It becomes a viewpoint video.

In order to reduce the above-mentioned uncomfortable feeling, especially the DP matching processing unit 29 improves the image quality around the contour of the face or body region, the order-corresponding image data Pe1a and Pe2a labeled using the symmetry of the person region. Perform the label correction process.
Specifically, as shown in FIG. 6, the DP matching processing unit 29 extracts contours in the opposite direction from the label regions of the order-corresponding image data Pe1a and Pe2a, and uses them as a mask to label them. The region is corrected to generate correction order corresponding image data Pe1c and Pe2c.
FIG. 10A shows an example of the processing. A solid line indicates a contour detection result of an area on the opposite side of the mask area, and a dotted line indicates a mask area corrected using the outline on the opposite side.
FIG. 10B shows correction order correspondence image data Pe1c and Pe2c after label area correction, and FIG. 10C shows an association result using the image after the label area correction, and a virtual viewpoint using the result. This is image data Ipna. In this way, by correcting the label area, the virtual viewpoint image data Ipna can be brought closer to the original subject image as compared to the case shown in FIG.

Hereinafter, DP matching by the DP matching processing unit 29 will be described.
The DP matching processing unit 29 obtains a correspondence relationship for each pixel position constituting the two image regions to be DP matched.
This association is performed, for example, by extracting pixel positions and luminance components at the same location constituting the face of the user a between the correction order correspondence image data Pe1c and Pe2c.
For example, as illustrated in FIG. 11, the corresponding point of the pixel P11 on the scan line L1 of the correction order corresponding image data Pe1c exists on the scan line L1 ′ of the correction order corresponding image data Pe2c, and the L1 By searching “up”, the most similar pixel position P11 ′ can be detected as a corresponding point. Incidentally, the matching unit 29 may be executed only for the part where the feature is extracted for this association, or may be executed for all the pixels constituting the correction order correspondence image data Pe1c and Pe2c.

In the present embodiment, the image parallelization processing unit 20 in the previous stage of the DP matching processing unit 29 normalizes the image in advance and parallelizes the epipolar lines, so that the robustness of the pixel search can be improved.
In the correction order corresponding image data Pe1c and Pe2c shown in FIG. 11, the corresponding point of the pixel P11 on the scan line L1 is present on the scan line L1 ′, and the correspondence is obtained by searching on the L1 ′. A pixel P11 ′ as a point can be detected.

For example, as shown in FIG. 12, the pixels on the scan line L1 in the correction order corresponding image data Pe1c in which the user a as the subject is respectively copied and the pixels on the scan line L1 ′ in the correction order corresponding image data Pe2c. In the case of association, the point sequence R1 of the feature points on the scan line L1 is {a1, a2, a3, a4, a5} in order from the left, and the point sequence R2 of the feature points on the scan line L1 ′ is sequentially from the left Let {b1, b2, b3, b4, b5}.
Here, when the feature points R1 and R2 on the scan lines L1 and L1 ′ are associated with each other in the relationship with the subject, first, the feature point on L1 ′ corresponds to a1 and corresponds 1: 1. However, b2 corresponds to the feature points on L1 ′ corresponding to the feature points a2 and a3 constituting the right ear of the user “a”, and corresponds to 2: 1. Similarly, b3 and b4 correspond to the feature points on L1 ′ with respect to the feature point a4 constituting the left ear of the user a, and correspond to 1: 2. Note that b5 corresponds to the feature point on L1 ′ with respect to a5, and corresponds to 1: 1.

As described above, in the correction order corresponding image data Pe1c and Pe2c obtained by imaging from different viewpoints, the content displayed in the ear portion of the user a is different depending on the parallax based on the distance from the subject to the camera. Come. Hereinafter, such a region is referred to as an occlusion region. In such an occlusion area, the corresponding point of the subject displayed in one correction order corresponding image data is hidden by the other correction order corresponding image data due to the parallax. , b1), (a2, b2), (a3, b3), (a4, b4), (a5, b5)}, an error occurs.
For this reason, the DP matching processing unit 29 identifies the parallax and thereby obtains the point sequences R1 and R2 of the feature points of the correction order corresponding image data shown in FIG. 12A as a result as shown in FIG. As shown in (2), control is performed so as to be associated with {(a1, b1), (a2, b2), (a3, b2), (a4, b3), (a4, b4), (a5, b5)} .

Specifically, the DP matching processing unit 29 performs dynamic programming (DP: shortest path search) as shown in FIG. 12C for all the pixels on the scan line in the correction order corresponding image data Pe1c and Pe2c. Performs dynamic association using.
In FIG. 12C, a point sequence R1 {a1, a2, a3, a4, a5} of feature points on the scan line L1 is arranged on the x axis, and the feature points on the scan line L1 ′ are arranged on the y axis. When the column R2 {b1, b2, b3, b4, b5} is applied, when the correspondence shown in FIG. 12B is applied to this graph, the path indicated by the bold line shown in FIG. 12C is taken. become. Hereinafter, a straight line connecting corresponding points indicated by bold lines is referred to as an optimum route.

When linearly increasing to the upper right in this optimum route, when the lines on the scan lines L1 and L1 ′ are shifted from left to right for association, the feature points are sequentially shifted by 1: 1 to correspond to each other. Is shown. As an example of the optimal path linearly increasing to the upper right, the DP matching processing unit 29 shifts the feature points (a2, b1) on the scan lines L1, L1 ′ one by one from the left to the right one by one. , b2) can be accurately associated.
In addition, when shifting in the horizontal direction in this optimal path, as a result of the occurrence of parallax between the correction order corresponding image data Pe1c and Pe2c, the feature points indicated in the correction order corresponding image data Pe1c correspond to the correction order. This suggests that the image data Pe2c is hidden.
In such a case, the DP matching processing unit 29 associates a plurality of feature points on the correction order corresponding image data Pe2c with one feature point on the correction order corresponding image data Pe1c. As an example of the optimum path that shifts in the horizontal direction, b2 indicating the right ear of the user a at the feature points (a2, b2) on the scan lines L1, L1 ′ further corresponds to a3 due to the above-described parallax. Is maintained as it is, and a3 is associated with it.

  Further, when shifting in the vertical direction on this optimum path, as a result of the occurrence of parallax between the correction order corresponding image data Pe1c and Pe2c, the feature points indicated in the correction order corresponding image data Pe2c are corrected order corresponding image data. This suggests that it has been hidden in Pe1c. In such a case, the DP matching processing unit 29 associates a plurality of feature points on the correction order corresponding image data Pe2c with one feature point on the correction order corresponding image data Pe1c. As an example of the optimum path that shifts in the vertical direction, a4 indicating the left ear of the user a at the feature points (a4, b3) on the scan lines L1, L1 ′ further corresponds to b4 due to the above-described parallax. B4 is associated with this while maintaining the above.

  The DP matching processing unit 29 executes these associations between the scan lines L1 and L1 'constituting all or part of the mutual order correspondence image data Pe1a and Pe2a. Then, by obtaining the above-described optimum route for each of the scan lines L1, L1 ', the feature points are associated with each other between the point sequences R1, R2.

FIG. 13 shows a case in which the correspondence (x1, x2) between arbitrary feature points (x1, y) and (x2, y) on the scan lines L1, L1 ′ is obtained.
The optimum path to the point (x1, x2) increases linearly in the upper right in the graph shown in FIG. 13 by shifting the point (x1-1, x2-1) one by one from left to right, or By shifting one point in the horizontal direction while maintaining x2 at the point (x1-1, x2), the graph moves from the point (x1-1, x2) in the horizontal direction in the graph shown in FIG. Further, the optimum path to the point (x1, x2) is shifted by one shift in the vertical direction while maintaining x1 at the point (x1, x2-1), so that the point (x1, x2) in the graph shown in FIG. -1) to move in the vertical direction.

That is, the optimum route passing through the point (x1, x2) is the points (x1-1, x2), (x1-1, x2-1), ( either x1, x2-1).
The DP matching processing unit 29 determines which point (x1-1, x2), (x1-1, x2-1), (x1, x2-1) and the point (x1, x2) are reached. It is determined by sequentially finding the functions to be explained.
The DP matching processing unit 29 obtains a matching cost function d (x1, x2) and dynamic occlusion cost functions dx1 (x1, x2), dx2 (x1, x2) shown below, and determines the obtained functions. Accordingly, the optimum route described above is obtained.
The matching cost function d (x1, x2) is a function indicating the similarity between the luminance component and the color component between the pixel positions for which the correspondence is obtained.
The occlusion cost function dx1 (x1, x2) is a function indicating the degree of hiding of the subject image with respect to the image Pe2a of the order corresponding image data Pe1a.
The occlusion cost function dx2 (x1, x2) is a function indicating the degree of hiding of the subject image with respect to the order corresponding image data Pe1a of the image Pe2a. These occlusion cost functions dx1 (x1, x2) and dx2 (x1, x2) have a shape reflecting the parallax between the images of the subject.

First, a method for obtaining the matching cost function d (x1, x2) will be described.
The DP matching processing unit 29 determines which of the luminance component or the color component to be compared is weighted for d (x1, x2).
This weighting is performed based on the following formula (1) using a weighting coefficient α.

[Equation 1]
dk (s, t) = α × dYk (s, t) + (1−α) dCk (s, t)
... (1)

Here, (s, t) represents a pixel position in the correction order corresponding image data Pe1c and Pe2c corresponding to the point (x1, x2).
In addition, k indicates which line of the correction order correspondence image data Pe1c and Pe2c corresponds (that is, k = y).
In the above equation (1), dYk (s, t) represents the absolute difference value of the luminance component between the coordinates (s, t) between the correction order corresponding image data Pe1c and Pe2c, and is expressed by the following equation (2). Defined.

[Equation 2]
dYk (s, t) = | Y1k (s, t) −Y2k (s, t) |
... (2)

  In the above equation (1), dCk (s, t) represents the absolute value of the difference between the color components between the correction order corresponding image data Pe1c and Pe2c, and is defined by the following equation (3).

[Equation 3]
dCk (s, t) = | C1k (s, t) −C2k (s, t) |
... (3)

  That is, by setting α higher in the above equation (1), the component of the luminance component difference absolute value dYk (s, t) can be more reflected in the obtained dk (s, t). In addition, by setting α to be smaller in the above equation (1), the component of the color component difference absolute value dCk (s, t) can be more reflected in the obtained dk (s, t). The DP matching processing unit 29 may assign an average value of the color component matching cost and the luminance component matching cost for α.

  d (x1, x2) is further obtained by the following equation (4) based on dk (s, t) obtained by the above equation (1).

[Equation 4]
d (x1, x2) = (Σdk (s, t)) / 2K k = −K,..., K−1
(4)

The above equation (4) means that d (x, y) can be obtained by taking an average with each pixel located above and below the scan line.
By the above equation (4), it is possible to reflect the correlation between each pixel located above and below the scan line for the obtained d (x1, x2). As a result, it is possible to greatly improve the association accuracy.
That is, the matching cost d (x1, x2) obtained by the above method increases as the absolute value of the difference between the luminance component or the color component at the pixel position (s, t) of the correction order corresponding image data Pe1c and Pe2c increases. To do. In other words, it increases as the difference in luminance component or color component at the pixel position (s, t) between the correction order corresponding image data Pe1c and Pe2c increases, and decreases as they are similar. That is, the matching cost d (x1, x2) makes it possible to identify the luminance component or color component similarity at the pixel positions (s, t) of the images Pe1a and Pe2a.

Next, a method for obtaining the occlusion cost functions dx1 (x1, x2) and dx2 (x1, x2) will be described.
Each of these occlusion cost functions dx1 (x1, x2), dx2 (x1, x2) is generated based on disparity information. As the distance from the cameras 11a and 12a to the user a as the subject decreases (as the parallax increases), the probability of occurrence of an occlusion area increases. In such a case, the matching unit 29 responds by lowering the occlusion cost functions dx1 (x1, x2) and dx2 (x1, x2). On the other hand, as the distance from the cameras 11a and 12a to the user a as the subject becomes longer (as the parallax becomes smaller), the probability of occurrence of an occlusion area decreases.
In such a case, the DP matching processing unit 29 responds by increasing the occlusion cost functions dx1 (x1, x2) and dx2 (x1, x2).
Each occlusion cost function dx1 (x1, x2), dx2 (x1, x2) can be obtained based on the following equations (5) and (6).

[Equation 5]
dx1 (x1, x2) = β × d (x1, x2) + T0
... (5)

[Equation 6]
dx2 (x1, x2) = γ × d (x1, x2) + T1
(6)

Here, d (x1, x2) is a matching cost, and dynamically adjusts the occlusion cost in order to eliminate variations in luminance values and hues of the left and right images. β and γ represent the rate of change of dp (x1, x2) and can be experimentally obtained in advance. T0 and T1 are initial occlusion cost constants, which can also be obtained experimentally in advance.
The DP matching processing unit 29 obtains these functions dx1 (x1, x2), d (x1, x2), dx2 (x1, x2), and then, based on the following formulas (7) to (9), respectively. Accumulated matching costs C (x1-1, x2), C (x1-1, x2-1), and C (x1, x2-1) are added to calculate total costs k1, k2, and k3.

[Equation 7]
k1 = C (x1-1, x2) + dx1 (x1, x2)
... (7)

[Equation 8]
k2 = C (x1-1, x2-1) + d (x1, x2)
(8)

[Equation 9]
k3 = C (x1, x2-1) + dx2 (x1, x2)
... (9)

  Here, C (x1-1, x2), C (x1-1, x2-1), and C (x1, x2-1) are points (x1-1, x2) and (x1-1, x2-1), respectively. ), (X1, x2-1) shows the accumulated matching cost obtained. Incidentally, the cumulative matching cost C (x1, x2) at the point (x1, x2) is assigned the smallest one among the obtained k1, k2, and k3 as shown in the following equation (10).

[Equation 10]
C (x1, x2) = min {k1, k2, k3}
(10)

  The DP matching processing unit 29 obtains the optimum route by selecting the smallest one of the obtained total costs k1, k2, and k3, and uses the result as the DP matching processing result DPa for each region virtual viewpoint image generation unit 30. Output to.

Here, when k1 is the minimum, it means that the feature point indicated in the correction order corresponding image data Pe1c is shielded in the correction order corresponding image data Pe2c by increasing the parallax. In such a case, as shown by the arrow J1 in FIG. 13, the optimum route is obtained so as to reach the point (x1, x2) by shifting in the horizontal direction from the point (x1-1, x2).
Further, when k3 is the minimum, it means that the feature point indicated in the correction order corresponding image data Pe2c is shielded in the correction order corresponding image data Pe1c by increasing the parallax. In such a case, as shown by the arrow J3 in FIG. 13, the optimum path is obtained so as to reach the point (x1, x2) by shifting from the point (x1, x2-1) in the vertical direction.
Further, when k2 is the minimum, it means that the similarity of the luminance component or the color component at the pixel position (s, t) of the correction order corresponding image data Pe1c, Pe2c is high. In such a case, an optimum path is obtained so as to reach the point (x1, x2) by shifting one by one in the horizontal and vertical directions from the point (x1-1, x2-1) as shown by the arrow J2 in FIG. Will be.

 The DP matching processing unit 29 sets the condition 1 “Y1k (s, t) and Y2k (s, t) for the corrected label regions L1c and L2c in the correction order corresponding image data Pe1c and Pe2c shown in FIG. When “both pixels are not in the occlusion region”, the matching cost function of the following equation (11) is used.

[Equation 11]
dk (s, t) = αdYk (s, t) + (1−α) · dCk (s, t)
... (11)

  Further, the DP matching processing unit 29 sets condition 2 “Y1k (s, t) and Y2k (s, t) for the corrected label areas L1c and L2c in the correction order corresponding image data Pe1c and Pe2c shown in FIG. If at least one of the pixels in the occlusion region is satisfied, a matching cost function of the following equation (12) is used.

[Equation 12]
dk (s, t) = Const (constant value)
(12)

  As described above, the DP matching processing unit 29 uses different functions for the corrected label areas L1c and L2c and other areas as the matching cost function of the correction order corresponding image data Pe1c and Pe2c. Thereby, compared with the case where the same function as the other areas is used for the label areas L1c and L2, the correction order corresponding image data Pe1c and Pe2c can be matched with high accuracy.

Next, the DP matching process for the unordered correspondence image data Pe1b and Pe2b shown in FIG. 5 will be described.
Unlike the order-corresponding image data Pe1a and Pe2a, the non-order-corresponding image data Pe1b and Pe2b do not have a label area.
Therefore, the DP matching processing unit 29 calculates the cost based on the above formulas (1) to (10) other than the above formulas (11) and (12) for the label region, performs the DP matching processing, and the result The DP matching processing result DPb is output to each region virtual viewpoint image generation unit 30.

[Each Area Virtual Viewpoint Image Generation Unit 30]
Each region virtual viewpoint image generation unit 30 generates virtual viewpoint order correspondence image data Ipna based on the order correspondence image data Pe1c and Pe2c input from the DP matching processing unit 29 and the DP matching processing result DPa. Is output to the virtual viewpoint image generation unit 31.
Each region virtual viewpoint image generation unit 30 generates the virtual viewpoint non-order corresponding image data Ipnb based on the non-order corresponding image data Pe1b and Pe2b input from the DP matching processing unit 29 and the DP matching processing result DPb. This is output to the virtual viewpoint image generation unit 31.

For example, as illustrated in FIG. 14, each region virtual viewpoint image generation unit 30 performs DP matching processing on the pixel position P11 in the order corresponding image data Pe1c as the corresponding point with respect to the pixel position P11 in the order correspondence image data Pe1c. When it is specified by the result DPa, the coordinates of the pixel position P11 are (x1, y1) as shown in FIG. 14, and the coordinates of the pixel position P11 ′ are (x2, y2).
The virtual viewpoint image region generation unit 30 uses the coordinates (xt, yt) of the pixel position on the virtual viewpoint image data Ipna corresponding to the pixel positions P11 and P11 ′ based on m (≦ 1) as relative position information. It can be determined by the following equation (13).

[Equation 13]
(Xt, yt) = m × (x1, y1) + (1−m) × (x2, y2)
... (13)

  Further, when the luminance components at the pixel positions P11 and P11 ′ are J11 and J11 ′, the respective region virtual viewpoint image generation unit 30 calculates the luminance component (virtual pixel value) at the pixel position Ph on the virtual viewpoint image data Ipna. Pt can be determined by image interpolation as shown in FIG.

  For example, as shown in FIG. 15, each area virtual viewpoint image generation unit 30 is based on whether or not the pixel positions P11 and P11 ′ of the order-corresponding image data Pe1c and Pe2c shown in FIG. 14 are in the occlusion area. The formula for calculating the luminance component (virtual pixel value) Pt at the pixel position Ph of the virtual viewpoint image data Ipna is switched.

  That is, as shown in FIG. 15, each area virtual viewpoint image generation unit 30 determines whether or not both of the pixel positions P11 and P11 ′ are in the occlusion area (step ST21), and enters the occlusion area. If not, the luminance component Pt is calculated by the following formula (14) (step ST22).

[Formula 14]
Pt = m × J11 + (1−m) × J11 ′
... (14)

  Each region virtual viewpoint image generation unit 30 determines whether or not the pixel positions P11 and P11 ′ are in the occlusion region based on the result of the DP matching process in step ST6 (ST8) in step ST17 shown in FIG. Judging.

If each region virtual viewpoint image generation unit 30 determines in step ST22 that one of the pixel positions P11 and P11 ′ is in the occlusion region, the process proceeds to step ST23.
Then, when each region virtual viewpoint image generation unit 30 determines that the pixel position P11 is in the occlusion region, the luminance component at the pixel position P11 ′ is used as the luminance component Pt (step ST25).
On the other hand, if each area virtual viewpoint image generation unit 30 determines that the pixel position P11 is not in the occlusion area, the luminance component at the pixel position P11 is used as the luminance component Pt (step ST24).

According to the present embodiment, as shown in FIG. 15, when any one of the pixel positions P11 and P11 ′ is in the occlusion region, the occupancy is calculated for calculating the luminance component Pt of the virtual viewpoint image data Ipna. Only the luminance component at the pixel position not in the occlusion area is used without using the luminance component at the pixel position in the resolution area.
Thereby, high-quality virtual viewpoint image data Ipna can be generated.
In the method shown in FIG. 15, in addition to the generation of the virtual viewpoint image data Ipna, each region virtual viewpoint image generation unit 30 generates the virtual viewpoint image data Ipnb based on the non-order corresponding image data Pe1b and Pe2b. The same applies to the above.

The virtual viewpoint image region generation unit 30 can determine the coordinates of each pixel constituting the virtual viewpoint image data Ipna and its luminance component according to m as the relative position information, as shown in the above equation (14). it can.
Here, m decreases as the virtual viewpoint in the virtual camera approaches the camera 11a, and increases as the virtual viewpoint approaches the camera 12a.
For this reason, as shown in FIG. 17, the coordinates (xt, yt) determined based on the above equation (13) approach the coordinates (x1, y1) of the pixel position P11 as the virtual viewpoint approaches the camera 11a. Further, as the virtual viewpoint approaches the camera 12a, it approaches the coordinates (x2, y2) of the pixel position P12. That is, since the coordinates (xt, yt) can be freely determined according to the position of the virtual camera, the position of the user a displayed on the virtual viewpoint image data Ipna can be freely changed.

The luminance component Pt determined based on the above formula (14) approaches the luminance component J11 at the pixel position P11 as the virtual viewpoint approaches the camera 11a, and the luminance at the pixel position P11 ′ as the virtual viewpoint approaches the camera 12a. The component J11 ′ is approached. That is, the pixels constituting the user a on the virtual viewpoint image Ipna can be brought close to the luminance component J11 or the luminance component J11 ′ according to the position of the virtual camera.
In particular, since the shooting directions of the camera 11a and the camera 12a are different from each other, the luminance between the pixel position P11 on the order-corresponding image data Pe1c and the corresponding pixel position P11 ′ on the order-corresponding image data Pe2c. The ingredients are different from each other. Depending on the position of the virtual camera by linearly increasing or decreasing the luminance component Pt according to m as the relative position information so that one of the different luminance components is the minimum value and the other is the maximum value. Thus, it is possible to determine the luminance component of the pixels constituting the user a displayed on the virtual viewpoint image data Ipna. Further, since the generated virtual viewpoint image data Ipna is generated based on the relationship associated with the matching unit 29 described above, it is possible to further reduce image quality deterioration of the obtained image.

By sequentially determining the coordinates (xt, yt) and the luminance component Pt at the pixel position Ph as described above, the generated virtual viewpoint image data Ipna has the gaze direction, the face direction, etc. of the user a to be displayed. The order-corresponding image data Pe1a and Pe2a which are different from each other are always facing the front.
The virtual viewpoint image data Ipna generated in this way is sent to the virtual viewpoint image generation unit 31.

  Each region virtual viewpoint image generation unit 30 generates the above-described virtual viewpoint image data Ipna based on the unordered correspondence image data Pe1b and Pe2b input from the DP matching processing unit 29 and the DP matching processing result DPb. Similarly, the virtual viewpoint non-order corresponding image data Ipnb is generated and output to the virtual viewpoint image generating unit 31.

[Virtual viewpoint image generation unit 31]
As shown in FIG. 16, the virtual viewpoint image generation unit 31 combines the virtual viewpoint image data Ipna and Ipnb to generate virtual viewpoint image data Ima.
The virtual viewpoint image generation unit 31 outputs the virtual viewpoint image data Ima to the other image processing device 2b via the network 7.
Here, the virtual viewpoint image data Ima is data for displaying an image as if captured by a virtual camera installed near the center of the screen. As a result, the users a and b shown in FIG. 1 can interact with each other through the displays 5a and 5b with their lines of sight matched.

[Operation example]
Hereinafter, an operation example of the image processing apparatus 2a illustrated in FIG. 1 will be described.
FIG. 18 is a flowchart for explaining an operation example of the image processing apparatus 2a shown in FIG.
Step ST1:
The image parallel processing unit 20 performs image correction processing and normalization processing on the image data Pa1 and Pa2 input from the cameras 11a and 12a to generate normalized image data Pm1 and Pm2, and the non-order corresponding region detection unit 27 Output to.
At this time, the image parallelization processing unit 20 performs the image correction processing based on parameters such as lens distortion and image center of the cameras 11 a and 12 a input from the calibration unit 26.

Step ST2:
As shown in FIG. 4, the non-order corresponding region detection unit 27 removes the background image data Pk1 and Pk2 from the normalized image data Pm1 and Pm2, and generates foreground image data Pf1 and Pf2.
In this manner, by removing the background image, it is possible to perform an efficient matching process that reduces the influence of the background image.

Step ST3:
The non-order corresponding region detection unit 27 generates (estimates) the parallax map data Dm1 and Dm2 based on the foreground image data Pf1 and Pf2 by a stereo matching method without order correspondence constraint.
In the present embodiment, an area-based matching method or the like is used as a stereo matching method with no order correspondence constraint.

Step ST4:
For each of the parallax map data Dm1 and Dm2, the non-order corresponding region detection unit 27 sets pixel data constituting the data to pixel data (previous pixel data) that is equal to or greater than a predetermined threshold value and less than the threshold value. And pixel data.
Then, the non-order-corresponding region detection unit 27 generates non-order-corresponding region parallax data composed of pixel data equal to or greater than the threshold value.
For example, as shown in FIG. 5, the non-order corresponding area detection unit 27 generates non-order corresponding area parallax data CL1 and CL2 from the parallax map data Dm1 and Dm2, respectively.
The non-order corresponding region detection unit 27 outputs the foreground image data Pf1 and Pf2 and the non-order corresponding region parallax data CL1 and CL2 to the non-order corresponding region separation unit 28.
As shown in FIG. 5, the non-order corresponding region separation unit 28 extracts non-order corresponding image data Pe1b and Pe2b from the foreground image data Pf1 and Pf2 based on the non-order corresponding region parallax data CL1 and CL2. The data is output to the DP matching processing unit 29.
Further, the non-order corresponding region separation unit 28 extracts the order corresponding image data from the foreground image data Pf1 and Pf2 based on the non-order corresponding region parallax data CL1 and CL2.
Then, as shown in FIG. 6, the non-order-corresponding region separation unit 28 replaces (labels) the order-corresponding image data Pe1a in the extracted order-corresponding image data with a label image. , Pe2a is generated and output to the DP matching processing unit 29.

Step ST5:
As shown in FIGS. 6 and 10, the DP matching processing unit 29 performs label correction processing on the order-corresponding image data Pe1a and Pe2a labeled using the symmetry of the person region, and the correction order-corresponding image data Pe1c, Generate Pe2c.

Step ST6:
The DP matching processing unit 29 performs DP matching processing between the correction order corresponding image data Pe1c and Pe2c by the cost calculation shown in the above formulas (1) to (12), and generates a DP matching processing result DPa. Are output to each region virtual viewpoint image generation unit 30.
Here, the DP matching processing unit 29 performs a cost calculation for the label area different from the other areas by the above formulas (11) and (12). That is, for the label area, the DP matching processing unit 29 uses the pixel values of the corresponding points on the condition that both of the corresponding points of the image data Pe1c and Pe2c do not exist in the occlusion area. The matching cost is calculated in the same manner as the region, but if the above condition is not satisfied, the matching cost is set to a predetermined (fixed) value.
In addition, the DP matching processing unit 29 outputs the correction order correspondence image data Pe1c and Pe2c to each region virtual viewpoint image generation unit 30.

Step ST7:
Each region virtual viewpoint image generation unit 30 performs the virtual viewpoint order by, for example, the procedure shown in FIG. 15 based on the order correspondence image data Pe1a and Pe2a input from the DP matching processing unit 29 and the DP matching processing result DPa. Corresponding image data Ipna is generated and output to the virtual viewpoint image generating unit 31.

Step ST8:
The DP matching processing unit 29 performs DP matching processing between the correction order corresponding image data Pe1b and Pe2b by the cost calculation shown in the above formulas (1) to (10), and generates a DP matching processing result DPb. Are output to each region virtual viewpoint image generation unit 30.
In addition, the DP matching processing unit 29 outputs the correction order corresponding image data Pe1b and Pe2b to each region virtual viewpoint image generation unit 30.

Step ST9:
Each region virtual viewpoint image generation unit 30 performs the virtual viewpoint order, for example, in the procedure shown in FIG. 15 based on the order correspondence image data Pe1b and Pe2b input from the DP matching processing unit 29 and the DP matching processing result DPb. Corresponding image data Ipnb is generated and output to the virtual viewpoint image generation unit 31.

  As illustrated in FIG. 16, the virtual viewpoint image generation unit 31 combines the virtual viewpoint image data Ipna and Ipnb input from each region virtual viewpoint image generation unit 30 to generate virtual viewpoint image data Ima.

[effect]
As described above, in the image processing apparatus 2a, as shown in FIG. 15, when one of the pixel positions P11 and P11 ′ is in the occlusion area, the luminance component Pt of the virtual viewpoint image data Ipna is For the calculation, only the luminance component at the pixel position not in the occlusion area is used without using the luminance component at the pixel position in the occlusion area.
Thereby, high-quality virtual viewpoint image data Ipna can be generated.
In the method shown in FIG. 15, in addition to the generation of the virtual viewpoint image data Ipna, each region virtual viewpoint image generation unit 30 generates the virtual viewpoint image data Ipnb based on the non-order corresponding image data Pe1b and Pe2b. The same applies to the above.

In the image processing device 2a, the captured image is separated into the order-corresponding image data Pe1a and Pe2a that satisfy the order-corresponding constraint and the non-order-corresponding image data Pe1b and Pe2b that do not satisfy the DP matching. The matching process can be performed.
Further, the image processing apparatus 2a combines the virtual viewpoint image data Ipna and Ipnb generated for the separated order-corresponding image data Pe1a and Pe2a and the non-order-corresponding image data Pe1b and Pe2b, respectively, thereby obtaining an accurate virtual viewpoint image. Data Ima can be generated.

  Also. The image processing apparatus 2a generates order-corresponding image data Pe1a and Pe2a in which the part of the non-order-corresponding image data is replaced (labeled) with the label image, and using this, the above-described formula (11), DP matching processing is performed using the function of (12). Thereby, accurate matching processing can be performed for the order-corresponding image data.

In particular, in this communication system 1, it is sufficient to arrange at least two cameras 11 and 12 on both sides of the display 5, and it is not necessary to extract 3D information of the subject each time, so that the entire system becomes complicated. There is also an advantage of eliminating.
Further, in the communication system 1, it is not necessary to use a special device such as a half mirror, a hologram screen, or a projector, and a simple and inexpensive system can be configured.

The present invention is not limited to the embodiment described above.
That is, those skilled in the art may make various modifications, combinations, subcombinations, and alternatives regarding the components of the above-described embodiments within the technical scope of the present invention or an equivalent scope thereof.
For example, all or a part of the configuration or steps of the image processing apparatus 2a shown in FIG. 2 may be realized as dedicated hardware for performing it.
Further, for example, as illustrated in FIG. 19, the image processing device 2 a may be realized as an image processing device 102 in which an interface 121, a memory 122, and a processing circuit 123 are connected via a bus 120.
In this case, the processing circuit 123 reads the program PRG from the memory 122 and executes the steps shown in FIGS. 15 and 18.
At this time, the processing circuit 123 appropriately stores the intermediate data obtained by the processing of each step shown in FIGS. 15 and 18 in the memory 122 or other buffer memory, and reads it out as necessary for use in the processing.
The memory 122 is a recording medium such as a semiconductor memory, an optical disk, a magnetic disk, or a magneto-optical disk.

FIG. 1 is a diagram showing an outline of a communication system according to a first embodiment of the present invention. FIG. 2 is a block diagram of the image processing apparatus shown in FIG. FIG. 3 is a diagram for explaining normalization for matching the normal directions of the images Pa1 and Pa2 shown in FIG. FIG. 4 is a diagram for explaining background image deletion performed by the non-order corresponding region detection unit illustrated in FIG. 1. FIG. 5 is a diagram for explaining the generation processing of the parallax map data Dm1 and Dm2 in the image processing apparatus shown in FIG. FIG. 6 is a diagram for explaining the label processing performed by the non-order corresponding region separating unit shown in FIG. FIG. 7 is a diagram for explaining the effect of the label processing shown in FIG. FIG. 8 is a diagram for explaining problems of the related technology of the present embodiment. FIG. 9 is a diagram for explaining a problem of the related technology of the present embodiment. FIG. 10 is a diagram for explaining a process for correcting a label in order-corresponding image data. FIG. 11 is a diagram for explaining the DP matching processing. FIG. 12 is a diagram for explaining the DP matching processing. FIG. 13 is a diagram for explaining the DP matching processing. FIG. 14 is a diagram for explaining virtual viewpoint image data generation processing performed by each region virtual viewpoint image generation unit illustrated in FIG. 2. FIG. 15 is a flowchart of virtual viewpoint image data generation processing performed by each region virtual viewpoint image generation unit illustrated in FIG. 2. FIG. 16 is a diagram for explaining processing of the virtual viewpoint image generation unit illustrated in FIG. FIG. 17 is a diagram for explaining virtual viewpoint image data corresponding to a virtual viewpoint. FIG. 18 is a flowchart for explaining an operation example of the image processing apparatus shown in FIG. FIG. 19 is a diagram for explaining a modification of the image processing apparatus shown in FIG. FIG. 20 is a diagram for explaining a problem of stereo image processing.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 ... Communication system, 2a, 2b ... Image processing apparatus, 11a, 11b ... Camera, 20 ... Image parallelization process part, 26 ... Calibration part, 27 ... Unorder corresponding | compatible area | region detection part, 28 ... Unorder corresponding | compatible area | region separation part , 29 ... DP matching processing unit, 30 ... each region virtual viewpoint image generation unit, 31 ... virtual viewpoint image generation unit

Claims (5)

For each of the plurality of images, a first step of generating parallax data by a matching method without order correspondence constraint;
Based on the parallax data generated in the first step, for each of the plurality of images, a second image that separates the image into a non-order-corresponding image whose parallax is greater than or equal to a reference and other order-corresponding images Process,
A third step of individually performing a matching process on the order-corresponding image and the non-order-corresponding image separated in the second step of the plurality of images by a matching method having the order correspondence constraint ;
A fourth step of determining whether or not a plurality of pixels associated in the matching process of the third step exist in the occlusion region;
Based on the determination result of the fourth step, a virtual value corresponding to the plurality of pixels in the virtual viewpoint image is obtained using a pixel value of a pixel that does not exist in the occlusion area among the plurality of associated pixels. A fifth step of generating pixel values;
I have a,
The fifth step includes
Based on the determination result of the fourth step, the virtual pixel value is generated using the pixel values of the two pixels when both of the two associated pixels do not exist in the occlusion region. ,
Based on the determination result of the fourth step, when one of the two pixels is present in the occlusion region and the other is not present in the occlusion region, the separation is performed in the second step. For at least one of the order-corresponding image and the non-order-corresponding image, corresponding to the two pixels in the virtual viewpoint image using the pixel value of the pixel that does not exist in the occlusion area among the two associated pixels. An image processing method for generating a virtual pixel value . The image processing method according to claim 1 , further comprising: a sixth step of synthesizing the virtual viewpoint image of the order-corresponding image and the virtual viewpoint image of the non-order-corresponding image. A seventh step of performing a label process of rewriting a portion corresponding to the non-order-corresponding image in the order-corresponding image separated in the second step into a label image indicating a certain pixel value;
The third step includes
In the order of the pixels along the epipolar lines of the plurality of images, the cost corresponding to the difference in pixel values between the pixels of the order-corresponding images that have been subjected to the label processing in the seventh step of the two images. Searching for a correspondence between the pixels that minimizes the cumulative value;
The cost is set to a constant value when at least one of the two pixels to be matched is located in an occlusion area, and otherwise, the cost is determined according to the difference between the pixel values of the two pixels. The image processing method according to claim 2 , wherein the cost is calculated. For each of the plurality of images, disparity data is generated by a matching method without order correspondence constraint, and based on the disparity data, for each of the plurality of images, the image is a non-order correspondence image with a parallax equal to or higher than a reference, Separation means for separating into other order-corresponding images,
A matching unit that individually performs a matching process on the order-corresponding image and the non-order-corresponding image separated by the separating unit of the plurality of images by a matching method having the order-corresponding constraint ;
Determining means for determining whether or not a plurality of pixels associated in the matching process of the matching means exist in an occlusion region;
Based on the determination result of the determination unit, a virtual pixel value corresponding to the plurality of pixels in the virtual viewpoint image using a pixel value of a pixel that does not exist in the occlusion area among the plurality of associated pixels. Generating means for generating
I have a,
The generating means includes
Based on the determination result of the determination means, when both of the two associated pixels do not exist in the occlusion area, the virtual pixel value is generated using the pixel value of the two pixels,
Based on the determination result of the determination unit, when one of the two pixels exists in the occlusion region and the other does not exist in the occlusion region, the order correspondence image separated by the separation unit and For at least one of the unordered images, a virtual pixel value corresponding to the two pixels in the virtual viewpoint image using a pixel value of a pixel that does not exist in the occlusion area among the two associated pixels An image processing apparatus for generating A program executed by a computer included in an image processing apparatus ,
For each of the plurality of images, a first procedure for generating parallax data by a matching method without order correspondence constraint;
Based on the parallax data generated in the first procedure, for each of the plurality of images, a second image that separates the image into a non-order-corresponding image whose parallax is equal to or higher than a reference and other order-corresponding images Procedure and
A third step of performing a matching process individually by the plurality of the said order corresponding images and non-order corresponding image separated in the second step of the image, matching technique has the sequence corresponding restraint,
A fourth procedure for determining whether or not a plurality of pixels associated in the matching process of the third procedure exist in the occlusion region;
Based on the determination result of the fourth procedure, a virtual value corresponding to the plurality of pixels in the virtual viewpoint image is obtained using a pixel value of a pixel that does not exist in the occlusion area among the plurality of associated pixels. A fifth procedure for generating pixel values;
To the computer ,
In the fifth procedure,
Based on the determination result of the fourth procedure, the virtual pixel value is generated using the pixel values of the two pixels when both of the two associated pixels do not exist in the occlusion region. ,
Based on the determination result of the fourth procedure, when one of the two pixels is present in the occlusion region and the other is not present in the occlusion region, the separation is performed in the second procedure. For at least one of the order-corresponding image and the non-order-corresponding image, corresponding to the two pixels in the virtual viewpoint image using the pixel value of the pixel that does not exist in the occlusion area among the two associated pixels. A program that generates virtual pixel values .

Images