JP7078564B2 - Image processing equipment and programs - Google Patents

Description

The present invention relates to an image processing device and a program capable of obtaining an appropriate three-dimensional model even when the subject is shielded.

Conventionally, a technique for generating an image from a free viewpoint (hereinafter referred to as a free viewpoint image) that has not been shot by a camera has been proposed for sports scenes and the like. This technology synthesizes images from virtual viewpoints that are not arranged based on images taken by multiple cameras, and displays the results on the screen to enable video viewing from various viewpoints. Is to be.

Here, among the free-viewpoint image synthesis techniques, there is an existing technique for synthesizing high-quality free-viewpoint images by generating a 3D computer graphics (3DCG) model of the subject using a principle called the visual volume crossing method. Exists (Non-Patent Document 1). In this method, the outline information of the subject obtained from a plurality of cameras is back-projected into a three-dimensional space, and they are described in a huge number of point cloud data to accurately reproduce the outline of the subject. By inputting a 3DCG model of the subject generated in advance, deciding the position of the virtual viewpoint and rendering it on the display, a free viewpoint image is generated. In addition to this, a technique for realizing a visual volume crossing method using a virtual plane group without using point cloud data has been proposed (Patent Documents 1 and 2).

As described above, a plurality of inventions have been proposed in which a subject is 3DCG modeled from a plurality of camera images and a virtual image of an arbitrary viewpoint is synthesized, and most of them are based on the principle of the visual volume crossing method.

Japanese Patent Application No. 2017-167472 Japanese Patent Application No. 2018-161868

Laurentini, A. "The Visual Hull Concept for Silhouette Based Image Understanding." IEEE PAMI, 16,2 (1994), 150-162

Although it is a visual volume crossing method that enables 3DCG modeling of a subject in various scenes, there are preconditions such as "in principle, all cameras capture the subject". That is, when the subject is shielded by any of the cameras and a part of the recognition result of the subject on the camera image is missing, the subject model is also lost and the composite image is significantly deteriorated in quality. In particular, many of the commonly used person extraction techniques assume that the person is moving, and this deterioration becomes remarkable when the subject is shielded by a stationary object. In addition, when the subject is shielded by the front-back relationship between the subjects, the shape of the subject on the background side can be supplemented by the shape of the foreground. In person extraction using machine learning, the same problem occurs when the subject person is shielded by a stationary object other than the person.

This problem becomes remarkable especially in sports images that emphasize the interaction between a subject and a stationary object. For example, considering sports climbing in which a schematic example of the state of the venue is shown in FIG. 1, the subject PL is a structure that is fixed to the wall W and protrudes from the wall W in order to climb the wall W. When grasping the foothold and handle for climbing the player PL), it is not modeled because the hand is shielded by the hold HL, and the composite image that the hand part of the player PL is missing It often happens. Also, if a part of the selected PL is shielded by the hold HL when shooting from any camera, not limited to the hand, the composite image will be similarly missing. (In the schematic example of FIG. 1, the player PL is shown to be on the ground GR before climbing the wall W.)

As described above, in the conventional visual volume crossing method, when the subject to be modeled in 3DCG (for example, athlete PL) is shielded by something other than the target of the modeling (for example, hold HL), There was a problem that an appropriate model could not be obtained. Then, even if a virtual image of an arbitrary viewpoint is synthesized from such an inappropriate model, the composite image will be inappropriate as if the subject has a defect.

In view of the above problems of the prior art, it is an object of the present invention to provide an image processing apparatus and a program capable of obtaining an appropriate three-dimensional model even when the subject is shielded.

In order to achieve the above object, the present invention is an image processing apparatus, from a first extraction unit that extracts a subject area as a first mask from each image of a multi-viewpoint image, and from each image of the multi-viewpoint image. A second extraction unit that extracts a region of a foreground object as an object different from the subject as a second mask, an identification unit that identifies an individual region in the second mask, and the first mask. The integrated first mask is obtained by adding the identified individual regions of the second mask that are determined to cause a defect due to shielding in the first mask based on the positional relationship. It is characterized by including an integrated unit for obtaining, and a generating unit for generating a three-dimensional model of the subject by using the integrated first mask corresponding to each image of the multi-viewpoint image. Further, the program is characterized in that the computer functions as the image processing device.

According to the present invention, among the identified individual regions of the second mask in the integrated portion, those determined to have caused a defect due to shielding in the first mask based on the positional relationship are added. This is a case where the subject is shielded by generating a three-dimensional model of the subject by using the integrated first mask in the synthesis unit after obtaining the integrated first mask. However, an appropriate 3D model can be obtained.

It is a figure which shows the schematic example of the state of a sport climbing venue as an example of shooting an image in which a problematic situation occurs. It is a functional block diagram of the image processing apparatus which concerns on one Embodiment. It is a figure which shows the schematic example of the processing by an identification part. It is a figure which shows the schematic example of the 2nd Embodiment regarding the labeling of the additional processing in the identification part. It is a figure which shows the schematic example of the integration process by the integration part. It is a figure which shows the example of the hardware composition of the general computer apparatus which can realize the image processing apparatus.

FIG. 2 is a functional block diagram of the image processing apparatus 10 according to the embodiment. As shown in the figure, the image processing apparatus 10 includes a first extraction unit 1, a second extraction unit 2, an identification unit 3, an integration unit 4, a generation unit 5, a synthesis unit 6, and a calibration unit 7. As its overall operation, the image processing device 10 receives the multi-viewpoint image as each time frame of the multi-viewpoint video as input in the first extraction unit 1, the second extraction unit 2, and the calibration unit 7, and each unit shown in the figure. After processing in, a 3DCG model excluding the effect of defects due to shielding is created in the generation unit 5, and by using this model, the composite image at the virtual viewpoint position specified by user input etc. is output from the synthesis unit 6. do. By outputting this composite image for each time of the input multi-viewpoint video, the image processing device 10 can output the free-viewpoint video.

In the following description, an arbitrary one-time frame in a multi-viewpoint video will be described as a multi-viewpoint image. (That is, when the image processing apparatus 10 handles video, the processing at each time can be basically the same. Therefore, in the following description, the video and time are particularly explicitly referred to. Except for, the explanation is about the common processing at each time.)

As the input / output data contents of each functional unit are shown in FIG. 2, the outline of the processing of each functional unit of the image processing apparatus 10 is as follows. As is known in the technical fields of the visual volume crossing method and free viewpoint image generation, the multi-viewpoint image as an input to the image processing device 10 surrounds the same scene (for example, the venue for sports climbing in FIG. 1). It is obtained by being photographed by a plurality of (at least two) cameras arranged.

The first extraction unit 1 extracts the area of the subject to which the 3DCG model is generated by the generation unit 5 on the rear stage side as the first mask from the image of each viewpoint of the multi-viewpoint image as an input, and the integration unit 4 Output to. The second extraction unit 2 extracts from the image of each viewpoint of the multi-viewpoint image as an input a region other than the target (stationary object) for which the 3DCG model is generated by the generation unit 5 on the subsequent stage side as the second mask. , Output to the identification unit 3. The calibration unit 7 obtains calibration data (camera calibration data) of the camera taking the multi-viewpoint image by analyzing the multi-viewpoint image as an input, and outputs the calibration data (camera calibration data) to the identification unit 3 and the generation unit 5.

The identification unit 3 obtains a second mask identified by identifying individual regions with respect to the second mask (generally configured as one or more closed regions) obtained from the second extraction unit 2. , Output to the integration unit 4.

In the identification unit 3, in one embodiment, as a coping process for dealing with a situation in which two or more different stationary objects are originally photographed, but the image area may be connected and configured as one closed area. , Each of the closed regions in the second mask is given an identification as to whether it is a closed region caused by one stationary object or a closed region caused by two or more stationary objects. This identified second mask can be output to the integration unit 4. In order to perform this identification process, the identification unit 3 can use the calibration data obtained from the calibration unit 7.

Here, when one closed region is caused by two or more stationary objects, it is identified after being given a distinction as to which part of this one closed region is caused by which stationary object. The second mask is output.

In any of the embodiments, the second mask identified by the identification unit 3 is a second mask in which an individual region for each individual stationary object is identified. (This identification is not given to the second mask obtained from the second extraction unit 2.)

In the integration unit 4, shielding occurs in the first mask by using the second mask in which the individual region obtained from the identification unit 3 is identified with respect to the first mask obtained from the first extraction unit 1. The processing (integration processing) for filling the shielding occurrence portion is performed by adding the corresponding individual areas of the second mask to the portion determined to be, and the integrated first mask obtained by this integration processing is generated. Output to part 5. (In addition, depending on the determination, the addition of the individual areas is not performed, and the first mask obtained from the first extraction unit 1 and the first mask integrated by the integration unit 4 may be the same as the data. It is possible.)

The first mask output by the first extraction unit 1, the second mask output by the second extraction unit 2, the identified second mask output by the identification unit 3, and the integration unit 4 output. The integrated first mask is obtained for each camera viewpoint image of the multi-viewpoint image as an input, and the processing for obtaining the mask is common to each of the functional units 1, 2, 3, and 4. be.

In the generation unit 5, the multi-viewpoint image is captured by applying the visual volume crossing method to the integrated first mask for each image of each camera viewpoint of the multi-viewpoint image obtained from the integration unit 4. Generates a 3DCG model of the subject and outputs it to the synthesizer 6. When the generation unit 5 generates the data, the calibration data obtained from the calibration unit 7 can be referred to and used.

The compositing unit 6 synthesizes and outputs a composite image by rendering the 3DCG model obtained from the generating unit 5 at a virtual viewpoint position designated by user input or the like. As described above, the composite image obtained by outputting this composite image in each time frame of the input multi-viewpoint image becomes the free viewpoint image of the subject at the designated virtual viewpoint position.

In the following, the details of each functional unit whose outline has been described above will be described.

<Calibration unit 7>
It is assumed that a predetermined in-space plane (for example, wall WL or ground GR in the example of FIG. 1) is captured in the multi-viewpoint image. In the calibration unit 7, the characteristic points of a predetermined in-space plane in the image of each viewpoint of the multi-viewpoint image (for example, the intersection of the white lines of the sports court) and the in-space plane in the real space to be photographed. Correspondence with points is performed, and calibration data is calculated as camera parameters (external parameters and internal parameters). For example, when the multi-viewpoint image is for a general sports image, the size of the court is standardized, so by using this as prior knowledge (field model of the court), the corner detection of the existing method is performed. It is possible to easily calculate which coordinates in the real space (world coordinate system) given by this prior knowledge correspond to the points on the image plane detected by the feature point detection or the like.

This camera calibration in the calibration unit 7 can be performed manually or by using the automatic calibration of any existing method. For example, as a manual method, the camera parameters can be estimated by selecting the intersection of the white lines on the screen by user operation and associating with the field model measured in advance. If the screen is distorted, the internal parameters may be estimated first.

If the input multi-viewpoint image is premised on shooting with a fixed camera, the camera calibration process in the calibration unit 7 may be performed only once at the beginning of the image. That is, the calibration data obtained at a certain time may not change with time, and may be referred to and used at a later time. Further, when a mobile camera is assumed, the camera parameters may be calculated in each time frame by the automatic calibration process by the above-mentioned arbitrary existing method.

<1st extraction unit 1>
The first extraction unit 1 extracts a first mask that expresses the shape of the subject (animal body) in each image of the multi-viewpoint image as a 0,1 binary mask image. That is, at each pixel position of each image of the multi-viewpoint image, the position where the value is 0 means that the background is the background and the subject does not exist, and the position where the value is 1 is the foreground and the subject exists. The first mask is extracted as a binary mask image meaning. (The rule of which of the binary values is the foreground or the background may use the reverse rule.) The first mask as the extracted binary mask image was integrated in the integration unit 4 on the rear stage side. After being processed into the first mask (given in the data format of the binary mask image as before the integration), it is input to the generation unit 5 and used to generate the 3DCG model shape of the subject.

In the first extraction unit 1, for example, the background subtraction method, which is an existing technique, can be used as a method for obtaining the first mask as a binary mask. In this technique, statistical information such as an image without a subject or its average value is registered as background statistical information in advance, and the difference between the background statistical information and the camera image at the target time (that is, the input multi-viewpoint image) is taken. On the other hand, the subject area is extracted by performing threshold processing. In addition, the first mask can be extracted by widely using existing techniques such as a technique for extracting a person or the like using machine learning (object recognition technique).

<Second extraction unit 2>
In the second extraction unit 2, apart from the subject mask obtained as the first mask by the first extraction unit 1, a still object in the foreground that shields the subject in each image of the multi-viewpoint image (for example, the sport of FIG. 1). Climbing Hold HL) is masked as a foreground object to obtain a second mask. An existing method can be used for the process of obtaining the second mask in the second extraction unit 2, and for example, an area division technique for dividing an image into small areas using color information or the like can be used.

Specifically, the average value of the color information contained in each small area obtained by dividing the area of the image as an existing method and the predetermined color information of the foreground object given as prior knowledge. A second mask can be obtained by comparing statistical information such as the median and the median and extracting a similar image region.

Here, as the threshold value for the similarity determination, a predetermined value set by the user may be used. After that, 1 (foreground) is assigned to the pixels in the area determined to be the foreground object, 0 (background) is assigned to the pixels in the other areas, and the second mask is assigned as a binary mask image in the same data format as the first mask. Obtainable. At this time, the second mask as the foreground object mask data does not appear in front of the subject in the actual scene (that is, does not shield the subject), and the foreground (foreground as the object to which the visual volume crossing method is applied). There may be regions that do not become, but in the second extraction unit 2, it is not necessary to distinguish between them, and a second mask can be obtained. It should be noted that the processing for distinguishing these is realized in the integrated unit 4 on the subsequent stage side.

In addition, for the extraction of the foreground object in the second extraction unit 2, existing technologies such as object recognition by machine learning, semantic segmentation, and extraction by chroma key can be widely used in addition to the above-mentioned area division. Prior knowledge about foreground objects such as color information in the case of division (information on the type of object to be recognized in the case of object recognition, etc.) shall be given in the same way. Further, in this specification, an explanatory example mainly for an artificial object that is completely stationary is used, but even if it is a natural object with minute movements such as a tree, segmentation using machine learning is similarly used. By using it and extracting its shape mask, it can be treated as a stationary object. That is, the stationary object extracted by the second mask may be an object that can be determined to be stationary.

In terms of terms, a movable object that forms the foreground in the first mask is a subject, and an object that forms the foreground in the second mask as a separate object from this subject (a stationary object (including a case with minute movement as described above)). Is called a foreground object.

<Identification unit 3>
The identification unit 3 identifies the second mask as the foreground object mask data output from the second extraction unit 2 by assigning an ID (labeling) for each closed region on these image coordinates. The second mask data is obtained. For this labeling process, existing techniques such as the area expansion method may be used.

In one embodiment of the present invention, the following additional processing may be performed after labeling each closed region on the image coordinates by such an existing technique. That is, depending on the foreground object, a plurality of objects may be connected and share one closed area on the image. In order to deal with this case, additional processing is performed to separate labels for different foreground objects in the same closed area (the same label is given by labeling of existing technology), and the labeling result in the additional processing is finally obtained. It may be output to the integration unit 4 as the identified second mask. (Alternatively, the labeling result as a closed area on the image coordinates by the existing technique may be output to the integration unit 4 as the identified second mask without performing additional processing.)

FIG. 3 is a diagram showing a schematic example of processing by the identification unit 3, in which the second mask F _{[before identification] before} identification output from the second extraction unit 2 is on the upper side, and before this identification is on the lower side. The second mask F _{[before identification]} identified by the processing by the identification unit 3 is shown. In the second mask F before _{identification [before identification]} , the area shown in black is only obtained as the foreground object mask, and the individual areas are not identified, whereas in the identified second mask F, the total is obtained. The identification results of 10 individual regions F ₁₀ to F ₁₉ are obtained.

The identified second mask F in FIG. 3 is an example when the identification unit 3 performs additional processing for identification. That is, the two individual regions F ₁₃ and F ₁₄ (of which the individual regions F ₁₃ are shown in light color for distinction) are on the image coordinates as can be seen in the state of the second mask F _{before identification [before identification]} . They are connected to each other in a single closed area, are labeled with the same label by existing technology labeling, and are further labeled with two different labels by additional processing to identify them. Similarly, the three distinct regions F ₁₇ , F ₁₈ and F ₁₉ (of which the distinct regions F ₁₇ and F ₁₈ are shown in light color for distinction) can be seen in the pre-discrimination second mask F _{[pre-discrimination]} state. As described above, they are connected to each other forming a single closed area on the image coordinates, and are labeled with the same label by the labeling of the existing technique, and further, three different labels are given by the additional processing to be identified. There is.

For the other individual areas F ₁₀ , F ₁₁ , F ₁₂ , F ₁₅ and F ₁₆ , additional processing was performed after the same label was given as a closed area on the image coordinates by labeling the existing technology. Remains the same label and does not change.

Hereinafter, the first embodiment and the second embodiment regarding the labeling of the additional processing in the identification unit 3 will be described.

In the first embodiment, the additional processing can be labeled by the following procedures (1) to (3).
(1) Visual volume intersection using the calibration data as the camera parameter obtained by the camera calibration in the calibration unit 7 and the second mask as the foreground object mask data extracted by the second extraction unit 2. Perform the method to obtain the three-dimensional shape of the foreground object. At this time, the visual volume crossing method may be applied using all the images of each camera viewpoint of the multi-viewpoint image. As for the visual volume crossing method, any existing method may be used as in the visual volume crossing method in the generation unit 5 described later. In addition, when it is judged that the image quality is poor due to reasons such as the foreground object not being captured at a short distance such as a hold, or the image from the camera viewpoint where the accuracy of the camera parameters is poor, the visual volume crossing method is applied. May be excluded from.
(2) Then, the acquired three-dimensional shape is labeled for each connection region by the existing technique, so that different objects can be distinguished from each other in the three-dimensional space.
(3) After that, the three-dimensional shape is reprojected onto each camera image plane (second mask as foreground object mask data), and the mask regions belonging to different objects distinguished by labeling in the above three-dimensional space. Give different labels.

According to the above steps (1) to (3), even when the masks of a plurality of different objects are connected and share one closed area in a certain foreground object mask data (second mask), this two-dimensional mask is used. It is possible to obtain additional labeling results identified as different objects (different objects by considering separation in three-dimensional space) in the foreground object mask data as an image.

In the first embodiment, it may be omitted to perform labeling for each closed region by the existing technique as a pretreatment for the second mask. (That is, in the first embodiment, since a unique label is obtained in the procedure (3), the labeling result may be obtained only by the above steps (1) to (3), not as an additional process. )

The second embodiment is different from the first embodiment in which the foreground object mask on almost all camera images of the multi-viewpoint image is used, and labeling is performed using only the viewpoints of a small number of cameras among the multi-viewpoint images. It is also a possible embodiment.

In the second embodiment, the result of the area division and the recognition by machine learning used in the second extraction unit 2 can be used. Specifically, for each small area (or recognized small area) that has already been divided into areas, a representative point (or area) such as the center of gravity is projected onto the three-dimensional space. The same operation is performed over a few cameras, and the representative point of the other foreground object mask in the projected ray of the above representative point (the ray projected through the optical center and the representative point of the camera). When the distance from the light beam of the above is less than or equal to a certain threshold value, the identification information that the area corresponds to the same object between different camera images is obtained. By this operation, it is possible to assign different labels to foreground object masks that are concatenated in one camera image but are clearly distant in other cameras, and output the labeled foreground mask data.

FIG. 4 is a diagram showing a schematic example of a second embodiment relating to labeling of additional processing in the identification unit 3. In the three-dimensional space to be photographed, it is assumed that there are two holds as the first hold HL1 and the second hold HL2 as described in FIG. 1 as the objects to be extracted as the foreground object on the second map. By taking these hold HL1 and HL2 from the front of the camera C1, they are obtained separately from the closed regions R11 and R12 (two single closed regions R11 and R12) on the image P1 (second map), respectively. The representative points are points p11 and p12, respectively.

On the other hand, these holds HL1 and HL2 are obtained by taking pictures from a very tilted direction with the camera C2 and connecting them to a single closed region R20 on the image P2 (second map), which is a representative point. Is p20. In the example of FIG. 4, the labeling results of the additional processing can be obtained by the following (processing 1) to (processing 3).

(Process 1)
By obtaining the following two threshold determination results (a) and (b), identification information that a single closed region R20 of the camera C2 contains two different foreground objects is obtained.
(A) Since the distance between the light rays C1-p11 in the camera C1 and the light rays C2-p20 in the camera C2 is equal to or less than the preset threshold value th, the regions R11 and the regions R20 (or their representative points p11 and p20). Is determined to correspond to.
(B) Since the distance between the light rays C1-p12 in the camera C1 and the light rays C2-p20 in the camera C2 is also equal to or less than the same threshold value, the regions R12 and the regions R20 (or their representative points p12 and p20) correspond to each other. It is determined that it is.
(Here, for example, in the case of "ray C1-p11", C1 represents the optical center of the camera C1, and the ray as a straight line passing through the optical center C1 and the point p11 is "ray C1-p11". .)

Therefore, since the above (a) and (b) are satisfied with respect to the single closed region R20 of the camera C2, the single closed region R20 of the camera C2 is appropriately used in another camera C1. It is possible to obtain identification information that two separated closed regions R11 and R12 are concatenated, that is, hold HL1 and HL2 as two different foreground objects are concatenated. ..

(Process 2)
After obtaining this identification information, the representative points p21 and p22 (indicated by white circles in the image P2 in FIG. 3) corresponding to the holds HL1 and HL2 as two different foreground objects are the coordinates in the image P2 of the camera C2. Ask as. Specifically, after obtaining the spatial coordinates x1 and x2 (not shown in Fig. 3) as representative points of the hold HL1 and HL2 in the three-dimensional space, the ray C2-x1 and the image P2 (in the epipolar geometric model). As is known, the representative point p21 corresponding to the first hold HL1 is obtained as the intersection with the image P2) as the projection plane, and similarly, it corresponds to the second hold HL2 as the intersection between the ray C2-x2 and the image P2. The representative point p22 can be obtained. If the representative points p21 and p22 obtained as intersections are not included in the region R20, the points included in the region R20 in the vicinity of the obtained points may be set as the representative points p21 and p22 again.

Here, the spatial coordinates x1 and x2 as representative points of the holds HL1 and HL2 in the three-dimensional space may be obtained as follows. That is, although not shown in FIG. 4, hold HL1 is used as a closed region R31 and a closed region R32 (representative points are points p31 and p32, respectively) in the image P3 of another camera C3 in addition to the image P1 of the camera C1. And HL2 are obtained separately, and there is a correspondence between the image P1 and the image P1 by the threshold value determination of the distance between the rays (the same determination as in the above determinations (a) and (b)), that is, the closed region R11. It is assumed that the correspondence between R31 and the correspondence between the closed regions R12 and R32 are obtained. This makes it possible to obtain the spatial coordinates x1 as the intersections of the rays C1-p11 and C3-p31, as is known in the epipolar geometric model, and similarly, the spatial coordinates as the intersections of the rays C1-p12 and the rays C3-p32. You can find x2. If an intersection where these rays actually completely intersect cannot be obtained due to an error or the like, the midpoint at the position where the rays are closest to each other may be set as the intersection.

(Process 3)
Then, for each pixel position in the single closed region R20 of the image P2, for example, the distances from the representative points p21 and p22 are calculated, and if the distance from the representative points p21 is smaller, the pixel position is the first. Labeling can be performed as corresponding to the 1-hold HL1, and when the distance from the representative point p22 is small, the pixel position can be labeled as corresponding to the 2nd hold HL2. Here, as the distance, only the Euclidean distance on the image coordinates may be used as the evaluation target, but by adding the distance on the color space to the evaluation target, the representative points p21, p22 and the texture etc. can be obtained. The labeling result may be obtained assuming that they are similar to each other and are located close to the representative points p21 and p22 to some extent. When performing evaluation on the color space, evaluation may be performed using a histogram or the like including a small area in the vicinity of the pixel to be evaluated. In the example of the single closed region R20 in the image P2 of FIG. 4, the case where two different foreground objects are connected has been described, but when three or more objects are connected, labeling is performed in exactly the same manner. It can be carried out.

As described above, the second embodiment described in the schematic example in FIG. 4 can be more generally realized by the following (procedure 1) to (procedure 3). As is known, the calibration data obtained from the calibration unit 7 can be used when calculating the epipolar geometric model (calculation regarding light rays and calculation regarding reprojection).

(Procedure 1)
By using an epipolar geometric model between camera images (second map) from different viewpoints, it is possible to individually determine the distance between corresponding rays (rays passing through the optical center of the camera and the representative point of the region, the same applies hereinafter). It is presumed that there is one foreground object corresponding to each closed region of each camera image in the three-dimensional space by comprehensively obtaining the correspondence between the closed regions of the above and whether or not the correspondence is duplicated. The identification information of whether it is presumed to be two or more is obtained. It should be noted that the closed region in which the correspondence with the closed regions of the camera images of different viewpoints cannot be obtained may be presumed to have one foreground object.

It should be noted that the correspondence relationship may be obtained only between the camera images in which the difference in the camera viewpoint satisfies a certain condition in the procedure 1. For example, it may be limited to only between camera images in which the difference in camera viewpoint is within a certain range.

(Procedure 2)
With respect to the first closed region of the first camera image presumed to correspond to two or more foreground objects, a plurality of second camera images and third camera images different from the first closed region corresponding to the first closed region. The closed region (a plurality of closed regions equal to each other in the second camera image and the third camera image) is presumed to be one foreground object, and is obtained by referring to the second camera image and the second camera image. The ray intersections of the epipolar geometric model are obtained for the representative points of the corresponding regions of the three camera images, and the ray intersections are representative of each of the plurality of foreground objects connected inside the first closed region of the first camera image. Let it be a point (spatial coordinate).

Regarding the first closed region of the first camera image, N closed regions correspond to each other in the second camera image, and M different closed regions correspond to each other in the third camera image (N ≠ M, N> M). However, if the generality is not lost), the M of each of the M rays of the third camera image and the ray having the smallest error when finding the intersection among the N rays of the second camera image. The intersections may be obtained as representative points (spatial coordinates) of a plurality of foreground objects.

(Procedure 3)
A representative point obtained by reprojecting the representative points (spatial coordinates) of the plurality of foreground objects obtained above onto the image coordinates of the first camera image is obtained, and each pixel in the first closed region of the first camera image is combined with the pixel. An ID corresponding to the representative point with the closest distance (not only the Euclidean distance on the image coordinates but also the distance on the color space may be used) is assigned, and the labeling result is obtained. If the reprojected image coordinates are not included in the first closed region, a point included in the first closed region near the reprojected image coordinates may be used as a representative point.

It should be noted that (Process 1) to (Process 3) in the description of FIG. 4 correspond to the specific examples of the above (Procedure 1) to (Procedure 3), respectively. As a premise that (Procedure 2) is possible, the multi-viewpoint image needs to be composed of three or more viewpoints.

<Integration Department 4>
The integration unit 4 utilizes the first mask as the subject mask obtained from the first extraction unit 1 and the identified second mask as the labeled foreground object mask obtained from the identification unit 3. , The generation unit 5 on the rear side of the integration unit 4 generates an integrated first mask as input data for executing the visual volume crossing method.

The processing in the integration unit 4 is based on the following considerations. That is, it is desirable that the data input to the visual volume crossing method does not include noise or the like even on the image plane and well represents the shape of the subject. On the other hand, as described above as a problem, the region shielded by the stationary object is lost in the first mask obtained by the first extraction unit 1, so that a process for filling the region is required. Therefore, by adding a part of the identified second mask obtained by the identification unit 3 (only a part considered to be the minimum necessary to fill the defect) to the first mask, the above-mentioned visual observation can be performed as much as possible. Keep the properties of the volume crossing method as input data.

Specifically, in the integration unit 4, the first mask is M, a certain mask closed area included in the first mask M is Mi ( _i is an index assigned to each closed area), and the identified second mask is F. Assuming that a certain closed area included in the second mask F is F _l (l is a label value as an identification result), the second mask closed area M _i is as shown in the following equations (1) and (2). It is determined whether or not to assign each closed region F _l of the mask, and if there is a grant determination, the region M _i is updated to the region M _i ', and these are updated as in Eq. (3). A new mask image M', that is, an integrated first mask, is output as the sum of the regions M _i '.

That is, a closed region F _l that satisfies the judgment condition “TH 1> | M _i ∩ F _l |> TH 2” shown in the equation (2) with respect to the label value l is a closed region M _i as shown in the equation (1). By adding to (the union is obtained for the image region as a set), the closed region M _i is updated to the closed region M _i ', and these updated closed region M _i'are as shown in Eq. (3). As a whole, a new mask image M'that integrates the original mask image M can be obtained. In the equation (2) in which the judgment condition is included as the condition of the element l belonging to the set L, | ・ | (absolute value symbol) is an operator that returns the number of pixels included in the image area "・" which is its argument. (The same applies to the equation (4) described later), where TH1 and TH2 are predetermined upper and lower thresholds for the number of pixels, respectively.

That is, as shown in Eq. (2), the number of pixels of the overlapping point M _i ∩ F _l between the closed area M _i and the closed area F _l | M _i ∩ F _l | is larger than the lower threshold value TH 2 and is higher. When it is smaller than the threshold value TH1, the closed region F _l may be added to the closed region M _i as shown in Eq. (1). For the threshold values TH1 and TH2, fixed values may be given by user setting, or the number of pixels in the closed region F _l | F _l | as shown in the following equations (4-1) and (4-2). Alternatively, it may be given as the smaller of the numbers obtained by multiplying the number of pixels | M _i | in the closed region M _i by a fixed ratio r1 and r2 (0 <r2 <r1 <1), respectively.
TH1 = min {r1 * | F _l |, r1 * | M _i |}… (4-1)
TH2 = min {r2 * | F _l |, r2 * | M _i |}… (4-2)

In Eq. (2), regarding the closed region F _l that determines whether to add by using the lower threshold value TH 2, it is considered that the overlapping location M _i ∩ F l is accidentally overlapped due to noise or the like because the overlapping point M i ∩ F _l is too small. Can be excluded from the addition target. In addition, with respect to the closed domain F _l that determines whether to add by using the upper threshold value TH 1, the closed domain M _i is completely covered inside (the relationship of "M _i ⊂ F _l " as the inclusion relationship of the set). Can be excluded from the addition target when the situation is close to or close to the above, for example, when the foreground area completely or almost covers the background of the subject.

Instead of using both the upper threshold TH1 and the lower threshold TH2 instead of the equation (2), use only one of them as in the following equation (2-1) or (2-2). You may try to.
L = {l ｜ TH1> | M _i ∩ F _l |}… (2-1)
L = {l ｜| M _i ∩ F _l |> TH2}… (2-2)

The process of identifying each closed region M _i in the first mask M may be performed at the same time when the first mask is extracted by the first extraction unit 1. For this process, a method similar to the labeling of the existing technique described in the second extraction unit 2 (such as a region expansion method on image coordinates) may be used.

FIG. 5 is a diagram showing a schematic example of the integration process by the integration unit 4 separately from the data D1 to D4. Data D1 shows an example in which the first mask M is composed of two closed regions M ₁ and M ₂ as an example of the first mask M as an input. This first mask M is an example in which the arm portion of the selected PL as shown in FIG. 1 is shielded by the hold HL and a defect occurs. In the data D2, as an example of the identified second mask F as an input, the identified second mask F has two identified closed regions F ₁ and F ₂ (hold HL as shown in FIG. 1). An example is shown which consists of a closed region corresponding to each.

In FIG. 5, the data D3 shows a state in which the first mask M of the data D1 and the identified second mask F of the data D2 are overlapped as a schematic example of the process for determining the condition of the equation (2). D4 shows a schematic example of the integrated first mask M'obtained by adding the equation (1) based on this condition determination and obtaining all of them as in the equation (3).

In the example of data D3 and D4, the condition judgment in Eq. (2) holds as "TH 1> | M ₁ ∩ F ₁ |> TH 2" only for the combination of the closed region M ₁ and the closed region F ₁ , and other combinations. Is an example of the case where it was not established. That is, the equation (2) is "L = {1}" for the closed domain M ₁ , and the equation (2) is "L = empty set (not applicable)" for the closed domain M ₂ . ing. Therefore, these two closed regions M ₁ and M ₂ in the first mask M are updated as the following equations (1-1) and (1-2) by applying the equation (1), respectively. By applying the equation (3) based on these, the first mask M'integrated as shown in the following equation (3-1) can be obtained in the example of FIG.
M ₁ '= M ₁ + F ₁ … (1-1)
M ₂ '= M ₂ … (1-2)
M'= M ₁ '+ M ₂ '
= M + F ₁ … (3-1)

In the above schematic example of FIG. 5, the integration process is given by the above equation (3-1), and when the integrated first mask M'is obtained, the cause of the defect in the first mask M is obtained. It can be seen that only the closed region F ₁ is added, and the closed region F ₂ that is not the cause of the defect is not added.

As already described, in the processing of the integration unit 4, the same processing is independently performed on the first mask corresponding to the image of each camera viewpoint of the multi-viewpoint image and the identified second mask. At this time, since the first mask and the second mask differ depending on the camera viewpoint, the determination result of the equation (2) may differ depending on the camera viewpoint. Therefore, for example, in one camera viewpoint, one or more individual areas in the second mask are added by the integration process, but in another camera viewpoint, no individual area in the second mask is added by the integration process. It is possible that.

<Generator 5>
The generation unit 5 estimates the three-dimensional shape of the subject and generates a 3DCG model. For the specific generation process, for example, the above-mentioned Patent Documents 1 and 2 and Non-Patent Document 1 and any other existing visual volume crossing method may be used. In general, the visual volume crossing method is a 3DCG model by taking the intersection of visual cones based on the subject shape information of all cameras (in the present embodiment, the integrated first mask obtained by the integrated unit 4). Get the shape of. Therefore, if even one of the plurality of camera binary masks contains a missing area, the 3DCG model shape is also lost. However, in the present embodiment, as described above, the integration unit 4 performs the process of filling the defect due to the shielding of the stationary object, so that the generation unit 5 generates an appropriate 3DCG model in which the influence of the defect is eliminated. Is possible.

<Synthesis unit 6>
In the compositing unit 6, the image from the final virtual viewpoint is synthesized according to the shape of the 3DCG model obtained in the generating unit 5. Any existing rendering method may be used for the compositing process, and the camera texture in the vicinity (the camera viewpoint is close to the virtual viewpoint among the multi-view images) according to the position coordinates of the virtual viewpoint given by the user input or the like. The color information of the subject 3D model can be determined using the texture of the object. For backgrounds other than subjects that are not 3DCG models, a background model created in advance (for example, if you are shooting sports, hold the sports venue or equipment as the background (in the case of sports climbing, hold it). The final composite image can be obtained by superimposing it on the above 3DCG model using the CG model).

As described above, according to one embodiment of the present invention, it is possible to synthesize a free-viewpoint image without damaging the shape of the subject in an image in which an interaction between a subject and a stationary object such as sports climbing occurs. Hereinafter, supplementary explanations will be given regarding individual matters.

(1) In the integration unit 4, the integration process is performed by the above equations (1) to (3), and the equation (2) that specifies the condition for determining the target to be integrated at this time, that is, the first mask is shielded. More generally, the following equation (5) may be used to determine the closed region of the second mask, which is considered to cause a defect.
L = {l | The positional relationship between the closed area M _i and the closed area F _l satisfies a predetermined condition. } …(Five)

The above-mentioned equation (2) specifically determines that the positional relationship in the above equation (5) satisfies a predetermined condition based on the overlap between the closed region M _i and the closed region F _l . be. As another embodiment, the condition determination of the positional relationship in the equation (5) may be determined (affirmative determination) by, for example, the distance between the closed region M _i and the closed region F _l being smaller than a predetermined threshold value. good. This distance may be obtained as the distance between the representative points (center of gravity, etc.) of the closed region M _i and the closed region F _l , or the point on the boundary of the closed region M _i and the point on the boundary of the closed region F _l . It may be obtained as the minimum distance between the two. In addition to using a fixed value for the distance threshold, the size of the closed region M _i and / or the closed region F _l (one of the rectangles surrounding the region) is set similar to the above equation (4) regarding the threshold value for duplication. It may be set as a fixed ratio value based on the size of the side, etc.).

Further, the determination in the equation (5) may be a determination in which the determination based on the overlap between the closed regions and the determination based on the distance as described above are combined.

(2) FIG. 6 is a diagram showing an example of a hardware configuration of a general computer device 30 capable of realizing the image processing device 10. As shown in FIG. 6, the computer device 30 is a CPU (central processing unit) 101 that executes a predetermined instruction, and a dedicated processor 102 that executes a part or all of the execution instructions of the CPU 101 on behalf of the CPU 101 or in cooperation with the CPU 101. (GPU (graphic calculation device), deep learning dedicated processor, etc.), RAM 103 as the main storage device that provides a work area for the CPU 101 and the dedicated processor 102, ROM 104 as the auxiliary storage device, communication IF (interface) 201, and display 202. , A bus B for sending and receiving data between them.

Each functional unit of the image processing apparatus 10 shown in FIG. 2 can be realized by a CPU 101 and / or a dedicated processor 102 that reads and executes a predetermined program corresponding to the processing content of each functional unit from the ROM 104. Here, when communication-related processing related to data transmission / reception via the network is performed, the communication IF 201 further operates in conjunction with each other, and when display-related processing is performed, the display 202 further operates in conjunction with the display-related processing. .. For example, the multi-viewpoint image as an input may be received via the communication IF201 and the composite image as an output may be transmitted via the communication IF201. The composite image obtained by the composite unit 6 may be displayed on the display 202.

Further, the image processing device 10 may be realized by only one computer device 30, or two or more computer devices 30 that communicate with each other via a network each have a function of the image processing device 10 shown in FIG. It may be realized as a system in which one or more of the parts are shared and shared.

10 ... Image processing device, 1 ... 1st extraction unit, 2 ... 2nd extraction unit, 3 ... Identification unit, 4 ... Integration unit, 5 ... Generation unit, 6 ... Synthesis unit

Claims (9)

A first extraction unit that extracts the area of the subject as the first mask from each image of the multi-viewpoint image,
A second extraction unit that extracts a region of a foreground object as an object different from the subject as a second mask from each image of the multi-viewpoint image.
An identification unit that identifies an individual area in the second mask,
To the first mask, among the identified individual regions of the second mask, those determined to have caused a defect due to shielding in the first mask based on the positional relationship are added. To obtain the integrated first mask by
A generation unit that generates a three-dimensional model of the subject by using the integrated first mask corresponding to each image of the multi-viewpoint image is provided .
With respect to the first mask, the integrated unit determines that the identified individual regions of the second mask, in which the size of the overlap satisfies a predetermined condition, cause the defect. An image processing device characterized by adding as . The image processing apparatus according to claim 1 , wherein the predetermined condition includes that the size of the overlap is larger than the lower threshold value. The image processing apparatus according to claim 2 , wherein the predetermined condition includes that the size of the overlap is smaller than the upper threshold value. In the identification unit, a three-dimensional model of the foreground object is generated and labeled by applying the visual volume crossing method using the second mask corresponding to each image of the multi-viewpoint image, and the label is assigned. The image processing apparatus according to any one of claims 1 to 3 , wherein the three-dimensional model is reprojected onto the second mask to identify individual regions in the second mask. A first extraction unit that extracts the area of the subject as the first mask from each image of the multi-viewpoint image,
A second extraction unit that extracts a region of a foreground object as an object different from the subject as a second mask from each image of the multi-viewpoint image.
An identification unit that identifies an individual area in the second mask,
To the first mask, among the identified individual regions of the second mask, those determined to have caused a defect due to shielding in the first mask based on the positional relationship are added. To obtain the integrated first mask by
A generation unit that generates a three-dimensional model of the subject by using the integrated first mask corresponding to each image of the multi-viewpoint image is provided.
In the identification unit, after identifying individual closed regions on the image coordinates with respect to the second mask corresponding to each image of the multi-viewpoint image, the identification unit is used.
By using the epipolar geometry model
The correspondence between the individual closed regions between the different second masks corresponding to the images of different camera viewpoints in the multi-viewpoint image was obtained.
In the obtained correspondence relationship, when a plurality of closed regions correspond to one closed region of the first camera viewpoint in each of the second camera viewpoint and the third camera viewpoint, the plurality of closed regions are used. Find the corresponding spatial coordinates for each
An image processing apparatus comprising identifying the one closed region as a plurality of different individual regions on image coordinates based on the obtained plurality of spatial coordinates. The image processing apparatus according to any one of claims 1 to 5 , wherein the first extraction unit extracts the first mask by applying a background subtraction method or object recognition. Any of claims 1 to 6 , wherein the second extraction unit extracts the second mask by applying region division or object recognition using the prior knowledge given about the foreground object. The image processing device described in Crab. The invention according to any one of claims 1 to 7 , further comprising a compositing unit that synthesizes a free-viewpoint image corresponding to the multi-viewpoint image by rendering the three-dimensional model of the subject from a designated virtual viewpoint. Image processing equipment. A program characterized in that a computer functions as the image processing device according to any one of claims 1 to 8 .

Images