JP2006039770A - Image processing apparatus, method, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To reduce irregularities in an image that may occur while a virtual image is generated, when generating an image (hereinafter referred to as the virtual image) at a virtual visual point as a virtual visual point, between respective images acquired by photographing a subject from mutually different visual points by at least two or more cameras. <P>SOLUTION: In an image acquired by at least two cameras 11a, 12a, each correspondence result of each pixel is obtained by two matching means 28, 29 for obtaining corresponding relation of each mutually different pixel, the presence of irregularities in the image is verified from the comparison of each correspondence by an information generation part 33, and an appropriate composite image is generated by a virtual visual point image generation part 30 based on the verification result, so that the irregularities in a virtual visual point image are reduced. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention is applied to a video conference system or a video phone system, for example, and relates to an image processing apparatus, method, and program for capturing an image to be transmitted and received and reconstructing it into a virtual viewpoint image obtained as a result of capturing the image with a virtual camera.

  As represented by a video phone system, a video conference system, and the like, a system has been proposed in which a plurality of users visually recognize a display image of the other party at a location apart from each other and perform remote dialogue. In such a system, a display image of the other party is displayed on the display, and a user who visually recognizes the display is imaged as a subject to be photographed, and the obtained image signal is transmitted through a network such as a public line or a dedicated line. By transmitting to the image processing apparatus, it is possible to give both users a sense of reality.

  In a conventional video conference system, a user who visually recognizes the display image of the other party displayed near the center of the display is captured by the camera at the top of the display, so that the image of the user facing down is displayed on the other party's display. Will be displayed. For this reason, there is a problem in that the users who actually view the display are interacted with each other in a state where their lines of sight are inconsistent, giving a sense of discomfort to each other.

  Ideally, if the camera is installed at a location where the display image of the other party is projected, the user naturally looks at the camera, so that the conversation can be realized with the eyes of both users matched. However, if a camera is installed on the display image of the other party, there is a problem that becomes very obtrusive.

  In order to solve the problem related to the line-of-sight mismatch, three-dimensional information of the subject is extracted based on input images taken by a plurality of cameras arranged on both sides of the display, and the extracted three-dimensional information and the receiver's There has been proposed an image processing apparatus that reconstructs an output image of a subject in accordance with information on a viewpoint position and displays the image on a display on the other side (see, for example, Patent Document 1). In this image processing apparatus, by synthesizing a virtual viewpoint camera image centered on the screen using an epipolar plane image generated from a plurality of camera images arranged on a straight line, the user's line of sight is made coincident and a sense of presence is realized. We realize high communication.

  In addition to the above devices, in order to match each other's line of sight in a video conference, an image communication device that captures images with two cameras installed on the left and right of the screen and generates three-dimensional position information based on the obtained images ( For example, see Patent Document 2).

  By the way, between the images Ps1 and Ps2 obtained by imaging the user from different viewpoints as shown in FIG. 17, as shown in FIG. 17, a repetitive pattern (for example, a check pattern) or a so-called non-characteristic point that hardly changes in luminance ( For example, it is difficult to make a correspondence in a single color wall).

  In addition, in the images Ps1 and Ps2 obtained by imaging from different viewpoints, for example, the contents displayed on the cheeks and ears shown in FIG. 17 differ depending on the parallax from the subject to the camera. . Hereinafter, such a region is referred to as an occlusion region. In the occlusion area, due to the parallax, the corresponding point of the object displayed in one image Ps1 is hidden in the other image Ps2, and thus there is a problem that inconvenience occurs when performing the association.

  In addition, images Ps1 and Ps2 obtained by imaging from different viewpoints are, for example, in regions where brightness varies depending on the viewing direction, such as a window portion, or in regions where specular reflection occurs, such as a user's nose portion. However, there is a problem that a difference occurs between the luminance component and the color component and an error occurs in the association, so that the virtual viewpoint image generated based on the association is disturbed.

  As a method different from the above improvement measures, a user interface of an image communication terminal device for guiding a subject to an appropriate position / posture and acquiring an appropriate video has been proposed (for example, Patent Documents). 3). In this user interface, if the user as the subject is too close to the camera and deviates from the camera frame using the result of detecting the position and orientation of the subject from the camera system, the user interface Improve live-action video by giving appropriate instructions. However, although this conventional image communication terminal device is effective in terms of ensuring that the subject fits within the camera frame under an environment that assumes a live-action video, the disturbance factor is not the subject when generating the virtual viewpoint image. It is difficult to say that it is effective because there are many features of the object (background, etc.). Furthermore, it is very difficult to guide the appropriate position and orientation of the subject while taking into account the depth and occlusion between the complex background and the foreground.

JP 2001-52177 A JP 2002-300602 A JP-A-8-251561

  Accordingly, the present invention has been devised in view of the above-described problems, and the object of the present invention is to automatically and efficiently prevent disturbance of a virtual viewpoint image created from an image obtained from each camera. An object of the present invention is to provide an image processing apparatus, method, and program that can be accurately removed.

  In order to solve the above-described problem, an image processing apparatus to which the present invention is applied has a correspondence relationship for each pixel between each image obtained by imaging a subject from different viewpoints by at least two cameras. A first matching means to be obtained; a second matching means for obtaining a correspondence relationship for each pixel based on a technique different from the first matching means; a correspondence relationship obtained by the first matching means; And a relative positional relationship for each camera with a virtual viewpoint set to obtain an image from a virtual viewpoint that is a virtual viewpoint. Information generating means for generating the relative position information shown based on the comparison result, and the first matching means and / or the second matching means Based on the relative position information of the virtual viewpoint generated by the information generation means from the positions and luminance components of the pixels associated with each other, the positions of the pixels constituting the virtual viewpoint image at the virtual viewpoint and the luminance components thereof Image generating means for obtaining

  In order to solve the above-described problem, an image processing apparatus to which the present invention is applied is related to the subject among images obtained by capturing subjects from different viewpoints by at least two cameras. A first matching unit that obtains a correspondence relationship for each pixel on the same horizontal line, and a block in which a plurality of pixels are two-dimensionally arranged are cut out for each image, and luminance components between the cut out blocks are calculated. A second matching means for sequentially calculating a sum of absolute differences and obtaining a correspondence for each pixel based on a set of blocks having the smallest calculated sum of absolute differences; and a correspondence obtained by the first matching means And the correspondence obtained by the second matching means for each pixel, and set to obtain an image from a certain viewpoint Information generating means for generating relative position information indicating the relative positional relationship between the virtual viewpoint and each of the cameras based on the comparison result, and the first matching means and / or the second matching means. Image generation for determining the position and luminance component of a pixel constituting the virtual viewpoint image to be generated by the virtual viewpoint based on the relative position information generated by the information generation means from the position and luminance component of the associated pixel The first matching means identifies the similarity by comparing the luminance component and the color component for each pixel position for which the correspondence is determined, and identifies the parallax between the images of the subject, The correspondence is determined according to the identified similarity and parallax.

  In order to solve the above-described problems, an image processing method to which the present invention is applied is related to the subject between the images obtained by capturing the subject from different viewpoints by at least two cameras. A first matching step for obtaining a correspondence relationship for each pixel on the same horizontal line, and a block in which a plurality of pixels are two-dimensionally arranged are cut out for each image, and luminance components between the cut out blocks are calculated. A second matching step for sequentially calculating a sum of absolute differences, and obtaining a correspondence for each pixel based on a set of blocks having the smallest calculated sum of absolute differences; and a correspondence obtained in the first matching step In addition, the correspondence obtained in the second matching step is compared for each pixel, and an image from a certain viewpoint is obtained. An information generating step for generating relative position information indicating a relative positional relationship with respect to each of the cameras with a virtual viewpoint set for the purpose based on the comparison result, and the first matching step and / or the second matching An image for obtaining the position of the pixel constituting the virtual viewpoint image to be generated by the virtual viewpoint and the luminance component thereof based on the relative position information generated by the information generation step from the position and luminance component of the pixels associated with each other in the step The first matching step identifies the similarity by comparing the luminance component and the color component for each pixel position for which the correspondence relationship is obtained, and the parallax between the images of the subject. And the correspondence is determined according to the identified similarity and parallax.

  In order to solve the above-described problem, the program to which the present invention is applied is identical to each other while relating to the subject among the images obtained by capturing the subject from different viewpoints by at least two cameras. A first matching step for obtaining a correspondence relationship for each pixel on the horizontal line, and a block in which a plurality of pixels are two-dimensionally arranged are cut out for each image, and the luminance component difference between the cut out blocks is absolute A second matching step for sequentially calculating a sum of values, and obtaining a correspondence relationship for each pixel based on a set of blocks in which the calculated sum of absolute differences is minimized; a correspondence relationship obtained in the first matching step; The correspondence obtained in the second matching step is compared for each pixel and an image from a certain viewpoint is obtained. An information generation step for generating relative position information indicating a relative positional relationship with respect to each of the cameras with a virtual viewpoint set for the purpose based on the comparison result, and the first matching step and / or the second matching An image for obtaining the position of the pixel constituting the virtual viewpoint image to be generated by the virtual viewpoint and the luminance component thereof based on the relative position information generated by the information generation step from the position and luminance component of the pixels associated with each other in the step The first matching step identifies the similarity by comparing the luminance component and the color component for each pixel position for which the correspondence relationship is obtained, and the parallax between the images of the subject. And calculating the correspondence according to the identified similarity and parallax. Make it run.

  In the present invention, when it is confirmed from the comparison result that the virtual viewpoint image to be generated is disturbed, the image is disturbed due to a parallax error by bringing the virtual viewpoint position closer to each camera. A virtual viewpoint image that is not present can be displayed to the user. As a result, it is possible to prevent the other user from generating a distorted image and not to give an uncomfortable feeling. In particular, in the present invention, since the disturbance at the time of creating the virtual viewpoint image can be automatically removed, consideration for moving each user to an appropriate position in relation to the camera is reduced. Further, by removing the disturbance of the virtual viewpoint image, the image quality itself in the virtual viewpoint image can be improved.

  Hereinafter, the best mode for carrying out the present invention will be described in detail with reference to the drawings.

  In the communication system 1 to which the present invention is applied, for example, as shown in FIG. 1, a user a at a point A and a user b at a point B are remotely interacting with each other while viewing a display image of the other party from a distant place. System.

  At the point A, a camera 11a and a camera 12a that capture the user a as a subject to be photographed from different viewpoints, and a display 5a for displaying an image of the user b captured at the point B side to the user a, An image processing apparatus 2a is provided that generates a virtual viewpoint image Ima based on the images Pa1 and Pa2 captured by the cameras 11a and 12a and transmits the virtual viewpoint image Ima to the point B via the network 7.

  At the point B, a camera 11b and a camera 12b that capture images of the user b as a photographing target from different viewpoints, and a display 5b for displaying an image of the user a captured at the point A side to the user b, An image processing device 2b that generates a virtual viewpoint image Imb based on the images Pb1 and Pb2 captured by the cameras 11b and 12b and transmits the image to the point A via the network 7 is disposed.

  The virtual viewpoint images Ima and Imb generated by the image processing apparatuses 2a and 2b are images picked up by a virtual camera virtually installed near the center of the display 5a or 5b on which the display image of the other party is projected. It corresponds to.

  The cameras 11a and 11b are respectively installed on the left side of the displays 5a and 5b when viewed from the users a and b, and the cameras 12a and 12b are respectively installed on the right side of the display when viewed from the users a and b. It becomes. The cameras 11 and 12 are installed with the shooting direction and the shooting angle of view being fixed, but these may be freely changed based on information input from the users a and b. By the way, in this communication system 1, description will be given by taking as an example a case where an imaging target is imaged by two cameras installed in accordance with the user's line of sight.

  The displays 5a and 5b display images based on the virtual viewpoint images Imb and Ima supplied from the counterpart point via the network 7 via a liquid crystal display surface, for example. The liquid crystal display surfaces of the displays 5a and 5b are composed of a large number of liquid crystal display elements and the like, and the liquid crystal display elements are optically modulated in accordance with output signals based on the virtual viewpoint images Imb and Ima to create an image to be displayed to the user.

  The image processing apparatuses 2a and 2b are usually constituted by electronic devices such as a personal computer (PC). These image processing apparatuses 2a and 2b have a function of communicating with each other via the network 7, and transmit images and sounds in response to requests from the other party. The configuration of the image processing apparatuses 2a and 2b will be described in detail later.

  The network 7 includes information such as an Internet network connected to the image processing apparatus 2 via a telephone line, ISDN (Integrated Services Digital Network) / B (broadband) -ISDN connected to a TA / modem, and the like. It is a public communication network that enables two-way transmission / reception. Incidentally, when the communication system 1 is operated in a certain narrow area, the network 7 may be configured by a LAN (Local Area Network). Further, when transmitting moving images, the network 7 is continuously transmitted from one channel having moving images including, for example, MPEG (Moving Picture Experts Group) data, based on the Internet protocol (IP). . In addition, when transmitting a still image, the image is transmitted at regular intervals from a channel different from the channel for transmitting a moving image. Note that a network server (not shown) may be connected to the network 7. This network server (not shown) manages, for example, Internet information, receives a request from the image processing apparatus 2, and transmits predetermined information stored in itself.

  Next, the configuration of the image processing apparatus 2 will be described using the image processing apparatus 2a as an example. As shown in FIG. 2, the image processing apparatus 2a includes a block matching unit 28 and a DP matching unit 29 to which images Pa1 and Pa2 are supplied from the connected cameras 11a and 12a, and these block matching unit 28 and DP matching unit 29. The virtual viewpoint image Ima generated by the virtual viewpoint image generation unit 30 connected to the information generation unit 33, the virtual viewpoint image generation unit 30 connected to the information generation unit 33, and the counterpart terminal device 2b. And an output control unit 31 for transmitting to.

  The block matching unit 28 is supplied with images Pa1 and Pa2 captured by the cameras 11a and 12a, respectively. The block matching unit 28 cuts out a block in which a plurality of pixels are two-dimensionally arranged for each image Pa1, Pa2 based on a so-called block matching method, and sequentially calculates the sum of absolute differences of luminance components between the cut out blocks. A calculation is performed, and a correspondence relationship is obtained for each pixel between blocks in which the calculated sum of absolute differences is minimum.

  The DP matching unit 29 is supplied with images Pa1 and Pa2 captured by the cameras 11a and 12a, respectively. The DP matching unit 29 obtains a correspondence relationship for each pixel position constituting the images Pa1 and Pa2.

  Incidentally, the association in the DP matching unit 29 is performed by extracting the pixel position and the luminance component at the same location constituting the face of the user a between the images Pa1 and Pa2. For example, as shown in FIG. 3, the corresponding point of the pixel P11 on the epipolar line L1 of the image Pa1 is present on the epipolar line L1 ′ of the image Pa2, and by searching on that L1 ′, A similar pixel position P11 ′ can be detected as a corresponding point. Incidentally, the matching unit 29 may be executed only for the part where the feature is extracted for this association, or may be executed for all the pixels constituting the images Pa1 and Pa2.

  The information generation unit 33 compares the correspondence obtained by the block matching unit 28 with the correspondence obtained by the DP matching unit 29 for each pixel. The information generation unit 33 generates relative position information indicating a relative positional relationship between the optical centers of the virtually installed virtual cameras with respect to the cameras 11a and 12a based on the comparison result.

  The virtual viewpoint image generation unit 30 is notified of the relative position information notified from the information generation unit 33. In addition, the virtual viewpoint image generation unit 30 is notified of the pixel positions and the luminance components associated with each other by the block matching unit 28 and / or the DP matching unit 29 from the information generation unit 33. Based on the relative position information generated by the information generation unit 33, the virtual viewpoint image generation unit 30 obtains a pixel position and a luminance component thereof constituting the virtual viewpoint image Ima to be generated by the virtual camera. The virtual viewpoint image generation unit 30 transmits a virtual viewpoint image Ima configured by the obtained pixel position and its luminance component to the output control unit 31.

  The output control unit 31 performs control so that the transmitted virtual viewpoint image Ima is transmitted to the image processing device 2b via the network 7. In such a case, the output control unit 31 may perform control so that the images Pa1 and Pa2 generated by the cameras 11a and 12a are independently transmitted to the image processing apparatus 2b.

  Next, specific operations in the image processing apparatus 2a will be described.

  A user a as a subject to be photographed is photographed from different angles by the cameras 11a and 12a. As a result, the line of sight of the user a on the images Pa1 and Pa2 generated by the cameras 11a and 12a, the face orientation, and the like are different from each other. Such images Pa1 and Pa2 are associated with each pixel position while being associated with the imaging target in the DP matching unit 29.

  When the DP matching unit 29 associates the images Pa1 and Pa2, for example, as shown in FIG. 4A, pixels on the epipolar line L1 in the image Pa1 in which the user a as the subject is respectively projected. And the pixel on the epipolar line L1 ′ in the image Pa2 are associated with the point R1 of the feature point on the epipolar line L1 in order from the left {a1, a2, a3, a4, a5}, and the epipolar line The point R2 of the feature point of L1 ′ is {b1, b2, b3, b4, b5} in order from the left. Here, when the feature points R1 and R2 on the epipolar lines L1 and L1 ′ are associated with each other in relation to the subject, first, the feature point on L1 ′ corresponds to a1 and corresponds 1: 1. However, b2 corresponds to the feature points on L1 ′ corresponding to the feature points a2 and a3 constituting the right ear of the user “a”, and corresponds to 2: 1. Similarly, b3 and b4 correspond to the feature points on L1 'with respect to the feature point a4 constituting the left ear of the user a, and correspond to 1: 2. Note that b5 corresponds to the feature point on L1 'with respect to a5, and corresponds to 1: 1.

  As described above, in the images Pa1 and Pa2 obtained by imaging from different viewpoints, the content displayed in the ear portion of the user a and the like varies depending on the parallax based on the distance from the subject to the camera. Hereinafter, such a region is referred to as an occlusion region. In such an occlusion area, the corresponding point of the subject displayed in one image is hidden in the other image due to the parallax, and therefore, {(a1, b1), (a2, b2) as in the conventional case. ), (A3, b3), (a4, b4), (a5, b5)}, an error occurs.

  For this reason, the DP matching unit 29 in the image processing apparatus 2a to which the present invention is applied recognizes such parallax, and as a result, the point sequences R1 and R2 of the feature points of the image shown in FIG. Corresponding to {(a1, b1), (a2, b2), (a3, b2), (a4, b3), (a4, b4), (a5, b5)} as shown in FIG. To be controlled.

  Specifically, dynamic association using dynamic programming (DP: shortest path search) as shown in FIG. 4C is performed for all pixels on the epipolar lines in the images Pa1 and Pa2.

  In FIG. 4C, the point sequence R1 {a1, a2, a3, a4, a5} of the feature points on the epipolar line L1 is arranged on the x axis, and the feature points on the epipolar line L1 ′ are arranged on the y axis. When the column R2 {b1, b2, b3, b4, b5} is applied and the correspondence shown in FIG. 4B is applied to this graph, the path indicated by the bold line shown in FIG. 4C is taken. become. Hereinafter, a straight line connecting corresponding points indicated by bold lines is referred to as an optimum route.

  When linearly increasing to the upper right in this optimal path, when the epipolar lines L1 and L1 ′ are correlated by shifting from left to right, the feature points are sequentially shifted by 1: 1 to correspond. Is shown. As an example of the optimal path linearly increasing in the upper right, the feature points (a2, b2) are accurately handled by shifting each one from the left to the right from the feature points (a1, b1) on the epipolar lines L1, L1 ′. Can be attached.

  Further, when shifting in the horizontal direction in this optimum path, it is suggested that the feature point shown in the image Pa1 is hidden in the image Pa2 as a result of the occurrence of parallax between the images Pa1 and Pa2. . In such a case, a plurality of feature points on the image Pa1 are associated with one feature point on the image Pa2. As an example of the optimal path shifted in the horizontal direction, b2 indicating the right ear of the user a at the feature points (a2, b2) on the epipolar lines L1, L1 ′ further corresponds to a3 due to the above-described parallax. Is maintained as it is, and a3 is associated with it.

  Further, when shifting in the vertical direction in this optimum path, it is suggested that the feature point shown in the image Pa2 is hidden in the image Pa1 as a result of the parallax between the images Pa1 and Pa2. . In such a case, a plurality of feature points on the image Pa2 are associated with one feature point on the image Pa1. As an example of the optimum path shifted in the vertical direction, a4 indicating the left ear of the user a at the feature points (a4, b3) on the epipolar lines L1, L1 ′ further corresponds to b4 due to the above-described parallax. B4 is associated with this while maintaining the above.

  The DP matching unit 29 executes these associations between the epipolar lines L1 and L1 'that constitute all or part of the images Pa1 and Pa2. Then, by obtaining the above-described optimum path for each of the epipolar lines L1, L1 ', the feature points are associated with each other between the point sequences R1, R2.

  FIG. 5 shows a case where an optimum path to an arbitrary feature point (x, y) on the epipolar lines L1 and L1 'is obtained.

  The optimum path to the feature point (x, y) is linearly increased to the upper right in the graph shown in FIG. 5 by shifting the feature point (x-1, y-1) one by one from the left to the right. Alternatively, the feature point (x-1, y) is shifted from the feature point (x-1, y) in the horizontal direction in the graph shown in FIG. Further, the optimum path to the feature point (x, y) is shifted by one shift in the vertical direction while maintaining x as it is at the feature point (x, y-1). It moves from x, y-1) in the vertical direction.

  That is, the optimum path passing through the feature point (x, y) is the feature point (x-1, y), (x-1, y-1) located at the left, lower left, and lower in the graph shown in FIG. , (X, y-1). The DP matching unit 29 determines which feature point (x-1, y), (x-1, y-1), (x, y-1) to reach the feature point (x, y) as follows. This is determined by sequentially obtaining the functions described in (1).

  The DP matching unit 29 obtains a matching cost function d (x, y) and a dynamic occlusion cost function dx (x, y), dy (x, y) shown below, and responds to the obtained functions. To obtain the above-mentioned optimum route. The matching cost function d (x, y) is a function indicating the similarity between the luminance component and the color component between the pixel positions for which the correspondence relationship is obtained, and the occlusion cost function dx (x, y) is the image Pa1. The occlusion cost function dy (x, y) is a function indicating the degree of hiding of the subject image relative to the image Pa1 of the image Pa2. These occlusion cost functions dx (x, y) and dy (x, y) are in a form that reflects the parallax between the images of the subject.

  For example, as shown in FIG. 6, when the point w that can be imaged only from the viewpoint C1 of one camera 11a cannot be imaged from the viewpoint C2 of the other camera 12a, the degree of hiding of the subject image increases. In such a case, the occlusion cost function dx (x, y), dy (x, y) becomes large, and the region including the point w is an occlusion region.

  Next, a method for obtaining the matching cost function d (x, y) will be described.

For d (x, y), it is determined which of the luminance component or color component to be compared is to be weighted. This weighting is performed based on the following formula (1) using a weighting coefficient α.
d _k (s, t) = α × dY _k (s, t) + (1−α) dC _k (s, t) (1)
Here, (s, t) represents the pixel position in the image Pa1 and the image Pa2 corresponding to the feature point (x, y). In addition, k indicates which line in the image Pa1 and the image Pa2 corresponds (that is, k = y). In this equation (1), dY _k (s, t) represents the absolute value of the difference in luminance component between the coordinates (s, t) between the image Pa1 and the image Pa2, and is defined by the following equation (2).
dY _k (s, t) = | Y1 _k (s, t) −Y2 _k (s, t) | (2)
Further, in this equation (1), dC _k (s, t) represents the absolute difference value of the color component between the image Pa1 and the image Pa2, and is defined by the following equation (3).
dC _k (s, t) = | C1 _k (s, t) −C2 _k (s, t) | (3)
That is, by setting α higher in the above equation (1), the component of the luminance component difference absolute value dY _k (s, t) can be more reflected in the obtained d _k (s, t). Further, by setting α to be smaller in the above formula (1), the component of the color component difference absolute value dC _k (s, t) can be more reflected in the obtained d _k (s, t). Incidentally, for α, an average value of the matching cost of the color component and the matching cost of the luminance component may be assigned.

d (x, y) is further obtained by the following equation (4) based on d _k (s, t) obtained by equation (1).
d (x, y) = (Σd _k (s, t)) / 2K k = −K,..., K−1
.... (4)
This equation (4) means that d (x, y) can be obtained by taking an average with each pixel located above and below the epipolar line. By this equation (4), it is possible to reflect the correlation with each pixel located above and below the epipolar line for the obtained d (x, y). As a result, it is possible to greatly improve the association accuracy.

  That is, the matching cost d (x, y) obtained by the above method increases as the absolute value of the difference between the luminance component or the color component at the pixel position (s, t) between the image Pa1 and the image Pa2 increases. In other words, it increases as the difference in luminance component or color component at the pixel position (s, t) between the image Pa2 and the image Pa1 increases, and decreases as they are similar. That is, it is possible to identify the luminance component or the similarity of the color component at the pixel position (s, t) of the image Pa1 and the image Pa2 by the matching cost d (x, y).

  Next, a method for obtaining the occlusion cost function dx (x, y), dy (x, y) will be described.

  Each of these occlusion cost functions dx (x, y), dy (x, y) is generated based on the disparity information generated by the information generating unit 33. As the distance from the cameras 11a and 12a to the user a as the subject decreases (as the parallax increases), the probability of occurrence of an occlusion area increases. In such a case, the DP matching unit 29 responds by lowering the occlusion cost function dx (x, y), dy (x, y). On the other hand, as the distance from the cameras 11a and 12a to the user a as the subject becomes longer (as the parallax becomes smaller), the probability of occurrence of an occlusion area decreases. In such a case, the DP matching unit 29 responds by increasing the occlusion cost function dx (x, y), dy (x, y).

Each occlusion cost function dx (x, y), dy (x, y) can be obtained based on the following equations (5) and (6).
dx (x, y) = β × dp (x, y) + T0 (5)
dy (x, y) = γ × dp (x, y) + T1 (6)
Here, dp (x, y) is dominated by the acquired parallax information, and decreases as the parallax becomes larger, and increases as the parallax becomes smaller. β and γ represent the rate of change of dp (x, y), and can be obtained experimentally in advance. T0 and T1 are initial occlusion cost constants, which can also be obtained experimentally in advance.

The DP matching unit 29 calculates these functions dx (x, y), d (x, y), and dy (x, y), and then accumulates them based on the following formulas (7) to (9). Matching costs C (x−1, y), C (x−1, y−1), and C (x, y−1) are added to calculate total costs k1, k2, and k3.
k1 = C (x-1, y) + dx (x, y) (7)
k2 = C (x-1, y-1) + d (x, y) (8)
k3 = C (x, y-1) + dy (x, y) (9)
Here, C (x-1, y), C (x-1, y-1), and C (x, y-1) are characteristic points (x-1, y) and (x-1, y-, respectively). 1) The accumulated matching cost obtained in (x, y-1) is shown. Incidentally, as the accumulated matching cost C (x, y) at the feature point (x, y), the smallest one among the obtained k1, k2, and k3 is assigned as shown in the following equation (10).
C (x, y) = min {k1, k2, k3} (10)
The DP matching unit 29 obtains the optimum route by selecting the smallest one of the obtained total costs k1, k2, and k3.

  Here, when k1 is the minimum, it means that the feature point shown in the image Pa1 is shielded in the image Pa2 by increasing the parallax. In such a case, the optimum path is obtained so as to reach the feature point (x, y) by shifting in the horizontal direction from the feature point (x-1, y) as shown by the arrow J1 in FIG.

  Further, when k3 is the minimum, it means that the feature point shown in the image Pa2 is shielded in the image Pa1 by increasing the parallax. In such a case, as shown by an arrow J3 in FIG. 5, an optimum route is obtained so as to reach the feature point (x, y) by shifting the feature point (x, y-1) in the vertical direction. .

  Furthermore, when k2 is minimized, it means that the similarity of the luminance component or the color component at the pixel position (s, t) between the image Pa1 and the image Pa2 is high. In such a case, as shown by the arrow J2 in FIG. 5, the optimum path is reached from the feature point (x-1, y-1) to the feature point (x, y) by shifting one by one in the horizontal and vertical directions. Will be required.

  FIG. 7B shows a correspondence relationship between pixel components (luminance or color components) of each feature point on the epipolar lines L1 and L1 ′ in the image Pa1 and the image Pa2 shown in FIG.

  In the image Pa1 shown in FIG. 7A, the wall recesses L2 and L3 shown on both sides of the user a and the wall recesses L2 ′ and L3 ′ shown on both sides of the user a in the image Pa2 are shown in FIG. In (b), it appears clearly as a line difference as a difference between pixel components. Similarly, pixel components constituting the user a clearly appear as pixel component differences near the center of FIG.

  In FIG. 7B, when the optimum path passes through the intersection of the same pixel component at each feature point on the epipolar lines L1 and L1 ′, k2 is minimized, and as a result, the pixel position This means that priority is given to a high similarity between the luminance component or the color component in (s, t). On the other hand, when the optimum path passes through the feature points on the epipolar lines L1 and L1 ′ other than the intersection of the same pixel component, k1 or k3 is minimized, and as a result, the disparity is This means that the images Pa1 and Pa2 are shielded.

  In this way, an optimum path from any one of the feature points (x-1, y), (x-1, y-1), (x, y-1) to the feature point (x, y) is obtained. Will be. The determination of the optimum path means that an optimum correspondence is determined according to the similarity between the luminance component and the color component and the parallax between the images Pa1 and Pa2. C (x, y) obtained from the equation is also defined as the optimum cumulative matching cost for obtaining the correspondence for each pixel position. These accumulated matching costs C (x, y) can be cumulatively taken into account when obtaining the optimum route to the feature points located on the upper, upper right, and right in the above graph. It can be improved further.

  Further, since the DP matching unit 29 calculates each function dx (x, y), d (x, y), dy (x, y) based on a completely different concept, the total cost k1, k2, When k3 is simply compared based on Expression (10), an error may occur depending on the shooting environment and the subject. In such a case, β, γ, T0, and T1 in equations (5) and (6) are optimized in advance according to the shooting environment and the subject, so that each function dx (x, y), d (x, The disparity between y) and dy (x, y) can be eliminated.

  As described above, the DP matching unit 29 in the image processing apparatus 2a to which the present invention is applied determines which of the identified similarity and parallax should be prioritized when obtaining the correspondence between the images Pa1 and Pa2. When giving priority to similarity, the same pixel positions on the horizontal line are associated with each other. When giving priority to parallax, a plurality of pixel positions on the same horizontal line are associated with one pixel position. As a result, even when the occlusion area exists, more accurate association can be performed. Further, by performing the association based on the above-described method, for example, the accuracy of the association can be improved even in a repetitive pattern such as a part of both eyes or a so-called non-feature point (for example, a wall portion) that hardly changes in luminance. Can do. Furthermore, even in areas where brightness varies depending on the viewing direction, such as the window part, and in areas where specular reflection occurs, such as the user's nose part, it may be dominated by the difference in luminance and color components. There are few, and it can be matched comparatively easily.

  In the DP matching unit 29 in the image processing apparatus 2a to which the present invention is applied, the similarity and the parallax are identified to the last, and if the correspondence is obtained according to the identified similarity and parallax, Of course, the effect can be obtained.

  The step of obtaining the correspondence in the DP matching unit 29 is referred to as step S11 in FIG. Incidentally, in parallel with this step S11, the block matching unit 28 obtains a correspondence relationship based on the block matching method (step S12). Based on this correspondence relationship, a parallax histogram in the cameras 11a and 12a is created, and the matching result is obtained. Verify the presence or absence of image disturbance for.

  In this step S12, as shown in FIG. 9A, when the sum of absolute differences in the matching result becomes small, and there are a plurality of peaks whose matching frequency increases, for example, FIG. 9B This means that the face and hand of the user a are at different distances from the cameras 11a and 12a.

  In general, when the user's face U and hand V are at substantially the same distance from the cameras 11a and 12a, as shown in FIG. For each position of the face image U ′ and the hand image V ′ obtained as a result of imaging the face U and the hand V, the face image U ″ and the hand image V obtained as a result of imaging from the viewpoint C2 of the camera 12a. Each position of "" will be arranged in order. That is, since no parallax occurs between the face U and the hand V, it is possible to ensure matching accuracy and to suppress disturbance of the generated virtual viewpoint image Ima.

  On the other hand, for example, as shown in FIG. 10 (b), when the user's face A and hand V are at different distances from the cameras 11a and 12a, for example, the user a For each position of the face image U ′ and the hand image V ′ obtained as a result of imaging the face U and the hand V, the face image U ″ and the hand image V obtained as a result of imaging from the viewpoint C2 of the camera 12a. Each position of "" will be reversed. That is, as a result of the reversal of parallax between the face U and the hand V, matching accuracy cannot be ensured, and as a result, there is a risk that the generated virtual viewpoint image Ima is disturbed.

  For this reason, in step S13, for example, based on the procedure shown in FIG. 11, it is more efficiently identified whether or not these parallax inversions occur via the peak of the matching frequency.

  Details of step S13 for verifying the matching result in step S12 are shown in FIG. In FIG. 11, it is first determined in step S21 whether or not there are two or more such peaks. As a result, when there are two or more peaks of matching frequency, the process proceeds to step S22, and when only one peak exists, the process proceeds to step S25.

  When the process proceeds to step S22, it is identified whether or not the interval between the two or more peaks exceeds a predetermined threshold value. As a result, when the peak interval exceeds a predetermined threshold, it is identified that the peak interval is sufficiently wide, and the process proceeds to step S23. Otherwise, the process proceeds to step S25. In general, the situation in which occlusion is likely to occur is that a plurality of different peaks often appear at distant positions in the parallax histogram as shown in FIG. 9 (a). It becomes possible to identify how easily the image is disturbed.

  When the process proceeds to step S23, it is detected whether or not the number of pixels near the two peaks is considerable. The determination of whether or not there is a substantial amount may be performed based on whether or not the number of pixels exceeds a threshold value, as in step S22. If it is determined that the number of pixels is considerable, the process proceeds to step S24, and otherwise, the process proceeds to step S25.

  In step S24, since the parallax is reversed for each subject, it is determined that the virtual viewpoint image Ima to be generated may be disturbed. If such a determination is made, the process proceeds from step S13 to step S14.

  On the other hand, when the process proceeds to step S25, it is determined that there is no possibility that the virtual viewpoint image Ima to be generated is disturbed because there is no parallax for the subject. If such a determination is made, the process proceeds from step S13 to step S17.

  In step S14, the correspondence obtained by the DP matching unit 29 in step S11 is compared with the correspondence obtained by the block matching unit 28 in step S12.

  In step S14, these correspondences are compared based on the procedure shown in FIG. First, in step S31 in FIG. 12, a counter Counter for measuring the number of pixels having different parallaxes is initialized. Next, the process proceeds to step S32, and the processing described below is executed for all the pixels constituting the image Pa1 and the image Pa2.

  In step S33, the difference between the parallax D1 obtained based on the block matching method in step S12 and the parallax D2 obtained based on the DP matching in step S11 is taken. Incidentally, the parallax D1 is based on the correspondence obtained by the block matching unit 28, and the parallax D2 is based on the correspondence obtained by the DP matching unit 29. In this step S33, it is identified whether or not the difference absolute value (= | D1-D2 |) between the parallax D1 and the parallax D2 exceeds the threshold value Thr. As a result, when the difference absolute value exceeds the threshold Thr, the process proceeds to step S34. On the other hand, when the difference absolute value is equal to or smaller than the threshold Thr, the process proceeds to step S35.

  When the process proceeds to step S34, 1 is added to the value of the counter Counter, and the process proceeds to step S35.

  When the process proceeds to step S <b> 35, it is confirmed whether or not the above-described processing has been executed for all the pixels constituting the image Pa <b> 1 and the image Pa <b> 2. As a result, when the above-described processing is executed for all pixels, the process proceeds to step S36. On the other hand, if the execution of the above-described processing has not been completed for all the pixels, the process proceeds to step S32 again, and the processing is repeated for the remaining pixels. As a result, the counter Counter counts the number of pixels whose difference absolute value exceeds the threshold Thr.

  That is, it can be performed for all the pixels constituting the image Pa1 and the image Pa2 as to whether or not the absolute difference value exceeds the threshold Thr. Thereby, the number of pixels whose difference absolute value exceeds the threshold value Thr can be identified. Incidentally, the number of pixels is equivalent to the total number of pixels having a difference between the correspondence obtained by the block matching unit 28 and the correspondence obtained by the DP matching unit 29. In addition, since such pixels may induce image disturbance due to parallax for the virtual viewpoint image Ima to be generated, the following step S36 indicates how much of these pixels occupy the entire image. Determine in.

  In step S36, it is determined whether or not the counter Counter exceeds a threshold value R. The threshold value R is optimized in advance according to the minimum number of pixels necessary for inducing image disturbance. Thereby, it becomes possible to determine whether or not the number of pixels exceeding the actually identified threshold value Thr induces image distortion. In step S36, if the counter Counter exceeds the threshold value R, the process proceeds to step S37, and it is determined that image disturbance is detected. On the other hand, if the counter Counter is equal to or smaller than the threshold value R in step S36, the process proceeds to step S38 and it is determined that there is no image disturbance.

  Returning to the description of FIG. As a result of comparing the correspondences in step S14, when the disturbance of the image is detected and the above-described step S37 is reached, the process proceeds from step S15 to step S16. On the other hand, if the image distortion is not detected and the process proceeds to step S38, the process proceeds to step S17 via step S15.

  When the process proceeds to step S17, the distances from the camera 11a and the camera 12a in the user a are identified. As a result, when the user a is closer to the camera 11a and the camera 12a, the process proceeds to step S18 on the assumption that the image may be disturbed due to parallax. Incidentally, the distance from the cameras 11a and 12a for the user a is determined according to whether or not the distance is larger than a predetermined threshold by using a parallax histogram based on the correspondence obtained in step S12. You may make it do. On the other hand, when the user a is away from the camera 11a and the camera 12a, it is assumed that there is no possibility of image disturbance due to parallax, and the process proceeds to step S19.

  When the process proceeds to steps S16, S18, and S19, the virtual viewpoint image generation unit 30 performs a correction process as described later when the virtual viewpoint image Ima is generated.

  First, a method of creating the virtual viewpoint image Ima by the virtual viewpoint image generation unit 30 will be described.

The virtual viewpoint image generation unit 30 generates the virtual viewpoint image Ima based on the correspondence obtained by the DP matching unit 29 and the correspondence obtained by the block matching unit 28 as described above. For example, in the DP matching unit 29, when the pixel position P11 ′ in the image Pa2 is specified as the corresponding point with respect to the pixel position P11 in the image Pa1, the coordinates of the pixel position P11 are as shown in FIG. x1, y1) and the coordinates of the pixel position P11 ′ are (x2, y2). The virtual viewpoint image generation unit 31 uses the coordinates (xt, yt) of the pixel position on the virtual viewpoint image Ima corresponding to the pixel positions P11 and P11 ′ as follows based on m (≦ 1) as the relative position information. It can be determined by equation (11).
(Xt, yt) = (1−m) × (x1, y1) + m × (x2, y2) (11)
When the luminance components at the pixel positions P11 and P11 ′ are J11 and J11 ′, respectively, the luminance component Pt at the pixel position Ph on the virtual viewpoint image Ima can be determined by the following equation (12).

Pt = (1−m) × J11 + m × J11 ′ (12)
As described above, the virtual viewpoint image generation unit 31 can determine the coordinates of each pixel constituting the virtual viewpoint image Ima and its luminance component according to m as the relative position information. As shown in FIG. 14, m decreases as the virtual viewpoint in the virtual camera approaches the camera 11a, and increases as the virtual viewpoint approaches the camera 12a.

  Therefore, the coordinates (xt, yt) determined based on Expression (11) approach the coordinates (x1, y1) of the pixel position P11 as the virtual viewpoint approaches the camera 11a, and the virtual viewpoint approaches the camera 12a. As a result, it approaches the coordinates (x2, y2) of the pixel position P12. That is, since the coordinates (xt, yt) can be freely determined according to the position of the virtual camera, the position of the user a displayed on the virtual viewpoint image Ima can be freely changed.

  In addition, the luminance component Pt determined based on Expression (12) approaches the luminance component J11 at the pixel position P11 as the virtual viewpoint approaches the camera 11a, and the luminance component at the pixel position P11 ′ as the virtual viewpoint approaches the camera 12a. It will approach J11 '. That is, the pixels constituting the user a on the virtual viewpoint image Ima can be brought close to the luminance component J11 or the luminance component J11 'according to the position of the virtual camera.

  In particular, since the shooting directions of the camera 11a and the camera 12a are different from each other, the luminance component is different between the pixel position P11 on the image Pa1 and the corresponding pixel position P11 'on the image Pa2. Depending on the position of the virtual camera by linearly increasing or decreasing the luminance component Pt according to m as the relative position information so that one of the different luminance components is the minimum value and the other is the maximum value. Thus, it is possible to determine the luminance component of the pixels constituting the user a displayed on the virtual viewpoint image Ima. In addition, since the generated virtual fulcrum image Ima is generated based on the relationship associated in the DP matching unit 29 described above, it is possible to further reduce image quality degradation of the obtained image.

  By sequentially determining the coordinates (xt, yt) and the luminance component Pt at the pixel position Ph as described above, the generated virtual viewpoint image Ima has the line-of-sight direction of the displayed user a, the face direction, etc. The different images Pa1, Pa2 are always facing the front.

  The generated virtual viewpoint image Ima is sent to the network 7 under the control of the output control unit 31. Then, the virtual viewpoint image Ima transmitted to the image processing device 2b on the other side is displayed on the display 5b under the control of the image processing device 2b. The user b interacts while visually recognizing the user a on the virtual viewpoint image Ima displayed on the display 5b. However, since the user a's face and line-of-sight direction are facing the front, It is possible to enjoy a feeling of visually recognizing an image taken by a virtual camera installed near the center. Similarly, the user a interacts while viewing the user b on the virtual viewpoint image Imb displayed on the display 5a. However, the user b facing the front can be visually recognized. That is, in this communication system 1, visual communication in which the line of sight is always matched between users who are interacting can be realized, and a more realistic and realistic remote dialog can be realized.

In particular, in the communication system 1, at least two cameras 11 and 12 are connected to the display 5.
It is sufficient to dispose them on both sides, and it is not necessary to extract the three-dimensional information of the subject each time, so that there is an advantage that the entire system is not complicated.

  Further, in the communication system 1, it is not necessary to use a special device such as a half mirror, a hologram screen, or a projector, and a simple and inexpensive system can be configured.

  In the communication system 1 to which the present invention is applied, when it is determined that the image is disturbed and the process proceeds to step S16, the following correction process is performed when the virtual viewpoint image Ima is generated. Apply.

First, it is assumed that the initial position of the virtual camera is at a position where the relative position information m in FIG. 14 is represented by 0.5. At this time, if the position αp of the virtual camera in the previous frame of the virtual viewpoint image Ima is larger than 0, the position αc of the virtual camera in the current frame of the virtual viewpoint image Ima is calculated based on the following equation (13).
αc = Max {(αp−Vc_Step), 0} (13)
Here, Vc_Step means a step width when shifting the viewpoint of the virtual camera from the initial position to the viewpoint of the camera 11a or the camera 12a. The position (αp−Vc_Step) obtained by subtracting the step width from the virtual camera position αp in the previous frame becomes the viewpoint position approaching the camera 11a by the step width Vc_Step. It is determined which of (αp−Vc_Step) and 0 is larger. As a result, when (αp−Vc_Step) is large, the viewpoint position αc in the current frame is set to (αp−Vc_Step) based on the equation (13).

  Even when moving to the next frame, if the next frame is the current frame, the previous frame is set to the viewpoint position that is closer to the camera 11a by the step width Vc_Step, and therefore the step width Vc_Step is further increased. It is determined which of the position subtracted by the minute and 0 is greater. That is, as the frame moves backward, the viewpoint position of the virtual camera gradually approaches the camera 11a by the step width Vc_Step. Finally, the viewpoint position of the virtual camera becomes equivalent to the viewpoint position of the camera 11a.

  That is, when it is confirmed that the virtual viewpoint image Ima to be generated is disturbed by moving to step S16, the viewpoint position of the virtual camera is brought closer to the camera 11a for each step width Vc_Step. It is possible to display a viewpoint image from the camera 11a that does not cause image disturbance due to parallax to the user. As a result, it is possible to prevent the other party user b from displaying a distorted image and not to give a sense of incongruity. In particular, in the communication system 1, since the disturbance at the time of creating the virtual viewpoint image Ima can be automatically removed, it is not necessary for each user to move to an appropriate position in relation to the cameras 11a and 12a. Also, by removing the disturbance of the virtual viewpoint image Ima, it is possible to improve the image quality itself of the image.

Note that the step width Vc_Step may be set freely on the user side in order to change the viewpoint position as smoothly as possible according to the shooting environment. Also, the viewpoint position of the virtual camera is not limited to approaching the camera 11a, but may be approached to the viewpoint of the camera 12a for each step width Vc_Step. At this time, as to whether the viewpoint position of the virtual camera is closer to the camera 11a or the camera 12a, for example, when the position of the user's hand V causing the occlusion is on the camera 11a side, go to the camera 11a. When the virtual viewpoint is brought closer and the position of the hand V is on the camera 12a side, the virtual viewpoint may be brought closer to the camera 12a.
When the viewpoint of the virtual camera is set to the viewpoint of the camera 11a and the image transition is not detected and the process proceeds to step S19 described above, when αp <0.5, the following is performed. You may make it return the viewpoint position of a virtual camera based on (14) Formula.
αc = Min {(αp + Vc_Step), α0} (14)
As shown in equation (14), as the frame moves backward, the viewpoint position of the virtual camera approaches α0 for each step width Vc_Step. Finally, the original virtual viewpoint position α0 is returned.

  The image processing apparatus 2 to which the present invention is applied is not limited to the above-described embodiment. For example, as shown in FIG. 15, the image processing apparatus 110 may be applied as a notification unit 101 that notifies the user of information corresponding to the comparison result of the correspondence relationship in step S <b> 14. In FIG. 15, the same components and members as those of the image processing apparatus 2 shown in FIG. 2 described above are denoted by the same reference numerals and description thereof is omitted.

The notification unit 101 is connected to the information generation unit 33, and when the comparison result of the correspondence described above is sent, notifies the user of appropriate information according to the comparison result.
For example, when it is identified from the occlusion detection result by the block matching unit 28 that the position of the user a is too close to the cameras 11a and 12a, the fact that the user a Notify Receiving such notification, the user a recognizes that his / her position is too close to the cameras 11a and 12a, and moves backward. As a result, the reversal of parallax can be prevented, and it is possible to prevent the virtual viewpoint image Ima having an uncomfortable feeling from being transmitted to the user b on the other side.
In addition, the present invention is not limited to the case where the user a is in front of the camera. In order to guide the position of the user a with respect to the cameras 11a and 12a to an appropriate position without parallax from the viewpoint of creating a virtual viewpoint image Ima that is not disturbed. May be output.
Further, the present invention is not limited to guiding the user a to an appropriate position, and any information may be notified as long as the information is based on the association result in step S11 or step S12.

  FIG. 16 shows an operation procedure when such a notification unit 101 is provided. When each of the images Pa1 and Pa2 captured by the cameras 11a and 12a includes a subject (such as a user a's hand) that causes a shift in parallax, the virtual viewpoint position α0 without any correction processing being performed. If the virtual viewpoint image Ima is created in step S31, the image is disturbed as shown in step S31.

  In such a case, the process proceeds to step S32, an error notification is sent to the user via the notification unit 101, and a message such as “Too far out” is output, for example, to a position where no parallax occurs. A virtual viewpoint image Ima in which image disturbance is suppressed is created by urging the movement and moving the virtual viewpoint to the viewpoint of the camera 11a or the camera 12a, and this is transmitted to the image processing apparatus 2 on the other side. Thereby, until the user a moves backward, the virtual viewpoint image Ima in which the viewpoint position is moved is continuously transmitted, and after the user a moves backward, the virtual viewpoint image Ima at the virtual viewpoint position α0 is created again. As a result, it is possible to continue to transmit the virtual viewpoint image Ima without any disturbance to the other user, and more natural communication can be realized.

  In particular, when it is desired to continue outputting the virtual viewpoint image, the communication system 1 can present appropriate means to the user, so that the possibility that the virtual viewpoint image is output is higher than in the conventional method.

  The present invention is not limited to the above-described embodiment, and the association method in the DP matching unit 29 and the association method in the block matching unit 28 may be any methods as long as they are different from each other. Good.

It is a figure which shows the outline of the communication system to which this invention is applied. It is a figure for demonstrating per structure of an image processing apparatus. It is a figure for demonstrating about the matching in a matching part. It is a figure for demonstrating about the correlation using DP. It is a figure shown about the case where the optimal path | route to the arbitrary feature points (x, y) on epipolar line L1, L1 'is calculated | required. It is a figure for demonstrating the occlusion which arises between the image Pa1 and the image Pa2. It is a figure which shows the correspondence of the pixel component (luminance or color component) of each feature point on the epipolar lines L1 and L1 'in the image Pa1 and the image Pa2. It is a flowchart which shows the process sequence in an image processing apparatus. It is a figure which shows the frequency of matching with respect to the distance from a camera. It is a figure for demonstrating the inversion phenomenon of the parallax according to the position of the user's face and hand with respect to a camera. It is a flowchart which discriminate | determines whether these parallax inversion has occurred via the peak of matching frequency. It is a flowchart which shows the procedure which compares the correspondence calculated | required by DP matching and block matching. It is a figure for demonstrating about the method of producing a virtual viewpoint image. It is a figure shown about the method of correcting a virtual viewpoint position. It is a figure for demonstrating per structure of the image processing apparatus provided with the notification part. It is a figure which shows the operation | movement procedure in the case of providing a notification part. It is a figure for demonstrating the problem of a prior art.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Communication system, 2 Image processing apparatus, 5 Display, 7 Network, 11,12 Camera, 28 Block matching, 29 DP matching part, 30 Virtual viewpoint image generation part, 31 Output control part, 33 Information generation part

Claims (14)

First matching means for obtaining a correspondence relationship for each pixel between each image obtained by imaging a subject from different viewpoints by at least two cameras;
Second matching means for obtaining a correspondence relationship for each pixel based on a method different from the first matching means;
To compare the correspondence obtained by the first matching means with the correspondence obtained by the second matching means for each pixel and to obtain an image from a virtual viewpoint that is a virtual viewpoint. Information generating means for generating relative position information indicating a relative positional relationship with respect to each of the cameras with the virtual viewpoint to be set based on the comparison result;
The virtual viewpoint based on the relative position information of the virtual viewpoint generated by the information generating means from the positions and luminance components of the pixels associated with each other by the first matching means and / or the second matching means. An image processing apparatus comprising: an image generation means for obtaining a position of a pixel constituting a virtual viewpoint image and a luminance component thereof. First matching that obtains a correspondence relationship for each pixel on the same horizontal line while being related to the subject between the images obtained by imaging the subject from different viewpoints by at least two cameras. Means,
A set of blocks in which a block in which a plurality of pixels are two-dimensionally arranged is cut out for each of the above images, the sum of absolute differences of luminance components between the cut out blocks is sequentially calculated, and the calculated sum of absolute differences is minimized. A second matching means for obtaining a correspondence relationship for each pixel based on:
A virtual viewpoint that is set to obtain an image from a certain viewpoint while comparing the correspondence obtained by the first matching means and the correspondence obtained by the second matching means for each pixel. Information generating means for generating relative position information indicating a relative positional relationship with respect to each of the cameras based on the comparison result;
The virtual to be generated from the virtual viewpoint based on the relative position information generated by the information generating unit from the position and luminance component of the pixels associated with each other by the first matching unit and / or the second matching unit. Image generation means for obtaining the position of the pixel constituting the viewpoint image and its luminance component;
The first matching means identifies the similarity by comparing the luminance component and the color component for each pixel position for which the correspondence relationship is obtained, identifies the parallax between the images of the subject, and identifies the identified similarity An image processing apparatus characterized by obtaining the correspondence relationship according to the degree and parallax. The image processing apparatus according to claim 2, wherein the information generation unit generates the relative position information so that a relative positional relationship of the virtual viewpoint with respect to the cameras continuously changes in units of frames. . The information generation means obtains an absolute difference value between the disparity based on the correspondence obtained by the first matching means and the disparity based on the correspondence obtained by the second matching means for each pixel. The relative position information is determined so that the virtual viewpoint approaches the one of the cameras stepwise when the difference absolute value exceeds the first threshold and the determined number exceeds the second threshold. The image processing apparatus according to claim 3, wherein: The information generation means generates the relative position information so that the virtual viewpoint approaches stepwise to an intermediate position between the cameras when the determined number is less than a second threshold. Item 5. The image processing apparatus according to Item 4. The second matching means identifies a parallax based on a geometric positional relationship of the subject with respect to each of the cameras;
The image processing apparatus according to claim 2, wherein the information generation unit further generates the relative position information based on the parallax identified by the second matching unit. The image processing apparatus according to claim 2, further comprising notification means for notifying a user of predetermined information according to the comparison result by the information generation means. The matching unit determines which of the identified similarity and parallax should be prioritized when obtaining the correspondence between the images, and when prioritizing the similarity, the same pixel on the horizontal line The image processing apparatus according to claim 2, wherein the association is performed between positions, and when priority is given to parallax, a plurality of pixel positions on the same horizontal line are associated with the position of the corresponding pixel. The image processing apparatus according to claim 2, wherein the matching unit weights either a luminance component or a color component to be compared when identifying the similarity. The image processing apparatus according to claim 2, wherein the matching unit identifies the similarity while reflecting a correlation between a luminance component and a color component at each pixel position located above and below the horizontal line. . 3. The image processing according to claim 2, wherein the image correction unit matches a normal direction of each image captured by each camera with a normal direction of a virtual plane on which the image of the virtual viewpoint is projected. apparatus. The image correction means obtains a projection transformation matrix for projecting each image from each camera onto the virtual plane, and sets the normal direction of each image based on the obtained projection transformation matrix. The image processing apparatus according to claim 11, wherein the image processing apparatus is adapted to a direction. First matching that obtains a correspondence relationship for each pixel on the same horizontal line while being related to the subject between the images obtained by imaging the subject from different viewpoints by at least two cameras. Steps,
A set of blocks in which a block in which a plurality of pixels are two-dimensionally arranged is cut out for each of the above images, the sum of absolute differences of luminance components between the cut out blocks is sequentially calculated, and the calculated sum of absolute differences is minimized. The second matching step for obtaining the correspondence relationship for each pixel based on the above, the correspondence relationship obtained in the first matching step, and the correspondence relationship obtained in the second matching step are compared for each pixel, An information generation step of generating relative position information indicating a relative positional relationship with respect to each camera with a virtual viewpoint set to obtain an image from a certain viewpoint based on the comparison result;
A virtual viewpoint image to be generated based on the virtual viewpoint based on the relative position information generated by the information generation step from the position and luminance component of the pixels associated with each other in the first matching step and / or the second matching step. An image generation step for obtaining the position of the pixel constituting the pixel and its luminance component,
In the first matching step, the similarity is identified by comparing the luminance component and the color component for each pixel position for which the correspondence is obtained, and the parallax between the images of the subject is identified, and the identified An image processing method characterized in that the correspondence is obtained according to similarity and parallax. First matching that obtains a correspondence relationship for each pixel on the same horizontal line while being related to the subject between the images obtained by imaging the subject from different viewpoints by at least two cameras. Steps,
A set of blocks in which a block in which a plurality of pixels are two-dimensionally arranged is cut out for each of the above images, the sum of absolute differences of luminance components between the cut out blocks is sequentially calculated, and the calculated sum of absolute differences is minimized. The second matching step for obtaining the correspondence relationship for each pixel based on the above, the correspondence relationship obtained in the first matching step, and the correspondence relationship obtained in the second matching step are compared for each pixel, An information generation step of generating relative position information indicating a relative positional relationship with respect to each camera with a virtual viewpoint set to obtain an image from a certain viewpoint based on the comparison result;
A virtual viewpoint image to be generated based on the virtual viewpoint based on the relative position information generated by the information generation step from the position and luminance component of the pixels associated with each other in the first matching step and / or the second matching step. An image generation step for obtaining the position of the pixel constituting the pixel and its luminance component,
In the first matching step, the similarity is identified by comparing the luminance component and the color component for each pixel position for which the correspondence is obtained, and the parallax between the images of the subject is identified, and the identified A program for causing a computer to obtain the correspondence relationship according to similarity and parallax.

Images