JP7191514B2 - Image processing device, image processing method, and program - Google Patents

Description

The present invention relates to an image processing technique for synthesizing images.

2. Description of the Related Art In a 3D compatible digital camera or the like, there is an image synthesizing technique for adjusting the depth relationship between a subject and a menu screen and synthesizing them. For example, Patent Literature 1 discloses a technique of adjusting the positional relationship between the depth of the photographed subject image and the menu screen displayed on the screen so that the menu screen is positioned in front of the subject when displayed on the screen. .

Some recent cameras are capable of not only capturing an image, but also measuring a distance in the depth direction (hereinafter referred to as depth distance) and recording the depth distance information together with the captured image. Using a camera that can record depth and distance information together with such captured images, multiple images can be captured and, for example, multiple people can be synthesized against a single background. It becomes possible. For example, when only a person is synthesized for a plurality of people, the depth distance information is used to separate the person object and the background from the captured image, and the front-back relationship of the separated person object is calculated based on the depth distance information. Synthesis is performed after determination.

JP-A-2010-86228

By the way, when extracting and synthesizing an object of a subject captured in a plurality of shot images, if the depth distance values of the subject appearing in each shot image are very close, the synthesis result will be an image that looks strange. Sometimes it becomes For example, when objects of subjects with close depth distances are mixed and synthesized, a synthesized image with a sense of incongruity may be generated.

SUMMARY OF THE INVENTION Accordingly, it is an object of the present invention to obtain a synthesis result that does not give a sense of discomfort when synthesizing images using depth distance information.

The present invention includes extraction means for extracting an object from a plurality of images, and determining a representative distance value representing a depth distance for each object extracted from the plurality of images based on depth distance information included in the images. and synthesizing means for synthesizing the objects to generate a synthesized image based on the representative distance value for each of the extracted objects, wherein the determining means determines when the objects are synthesized When the objects overlap each other, the representative distance value is determined for the overlapping objects, and the initial value is determined as the representative distance value for the objects that do not overlap each other .

According to the present invention, when synthesizing an image using depth distance information, even if a plurality of subjects are synthesized, a synthetic result with little sense of incongruity can be obtained.

1 is a diagram showing a schematic configuration of an image synthesizing device according to an embodiment; FIG. FIG. 10 is a diagram used for explaining the relationship between an image having depth distance information and 3D rendering; 9 is a flow chart of synthesis processing; FIG. 10 is a diagram showing an example of synthesis when there are non-overlapping objects; 10 is a flowchart of composition processing when there is overlap determination; 8 is a flow chart of synthesis processing when face recognition is performed; FIG. 10 is a diagram used for explaining screen transitions when selecting a background and a composition target; FIG. 10 is a diagram used for explaining the presence/absence of a face and the composition order; FIG. 10 is a diagram used for explaining an example of comparing partial parts of a face; FIG. 11 is a diagram used for explaining an example of collectively changing depth distances of objects;

Preferred embodiments of the present invention will be described below with reference to the drawings.
<First embodiment>
FIG. 1 is a diagram showing a schematic configuration of an image synthesizing device, which is an example of an image processing device according to this embodiment.
A control unit 105 performs overall control of the image synthesizing apparatus. Further, each component of the image synthesizing apparatus, which will be described below, uses the memory 106 to perform each process.
The image data reading unit 100 acquires image data from an external medium or the like (not shown). The image data described in the present embodiment is data that includes an image and information on the distance in the depth direction for each pixel of the image (hereinafter referred to as depth distance). Details of the depth distance and its information (referred to as depth distance information) will be described later.

The decoding unit 101 decodes the image data acquired by the image data reading unit 100 and makes it recognizable as an image. Further, the decoding unit 101 has a function of acquiring depth distance information included in the image, and associates each pixel of the decoded image with the depth distance.

The object extraction unit 103 extracts an object such as a subject from an image to which depth distance information is assigned to each pixel as described above. This object extraction is a process of extracting an object composed of pixels whose depth distance is shorter (closer) than a predetermined distance from the image. The remaining image area where the object is separated from the image becomes the background. In this way, the object extraction unit 103 uses a predetermined distance as a threshold, and separates an object whose depth distance is shorter (closer) than the threshold from the input image to which depth distance information is added from the background other than that. . Object extraction may be performed using other existing object extraction techniques that do not use depth distance information.

The distance calculation unit 107 generates a depth map, which will be described later, from the depth distance information for each pixel, and also calculates a representative distance value, which will be described later, based on the depth distance information of each pixel of the object extracted by the object extraction unit 103. do. The details of how to generate the depth map and how to calculate the representative distance value will be described later.

The synthesizing unit 102 synthesizes a plurality of objects extracted from the plurality of input images by the object extracting unit 103 as described above and an image used as a background (hereinafter referred to as a background image). Although the details will be described later, the synthesizing unit 102 uses the representative distance value calculated by the distance calculating unit 107 to synthesize a plurality of objects and a background image.
The display unit 104 displays images. The display unit 104 also displays a synthesized image generated by the synthesizing process in the synthesizing unit 102 . Note that the display unit 104 may have a touch panel function capable of receiving instructions from a user or the like.

Note that the control unit 105, the image data reading unit 100, the decoding unit 101, the distance calculation unit 107, the synthesizing unit 102, the object extraction unit 103, etc. in FIG. May be. When the processing of each part is executed by the software configuration, it is realized by executing the program (software) according to the present embodiment by the CPU or the like. The program according to this embodiment may be read from the memory 106 (including ROM and removable semiconductor memory), or may be downloaded from a network such as the Internet (not shown).

Next, image data having depth distance information will be described with reference to FIGS. 2(a) to 2(c).
An image 200 shown in FIG. 2A is an image having depth distance information for each pixel. Assume that the image 200 illustrated in FIG. 2A is an image in which an object 204 corresponding to the face of a human subject is shown in the center.

In the image 200, depth distance information is stored for each pixel forming the image. The depth distance information stored in each pixel will be described below.
A pixel 201 in FIG. 2( a ) is one of the pixels included in the image 200 . The data 202 forming this pixel 201 has a configuration (ZRGB configuration) in which depth distance information Z and color information RGB are combined. The larger the value of the depth/distance information Z, the closer it was to the camera when the image was taken (the closer it is to the front). The color information RGB is information on each of the three primary colors of red (R), green (G), and blue (B). Each of the depth distance information Z and the color information RGB is 8-bit information, and a total of 32-bit data per pixel. Here, as an example, the 8-bit depth distance information Z of one pixel is inserted into the 32-bit ZRGB configuration, but the present invention is not limited to this example. Any format may be used.

A planar image 210 shown in FIG. 2(a) is created only from color information RGB. A depth map 211 is generated from the depth distance information Z for each pixel. The depth map 211 is a map in which depth distance information for each pixel is two-dimensionally arranged corresponding to the arrangement of each pixel. Note that in the example of FIG. 2A, the depth map 211 is represented by a coarse mesh with a reduced number of pixels for the sake of simplification. In the depth map 211 exemplified in FIG. 2( a ), a hatched mesh portion 205 is a region corresponding to the face object 204 in the image 200 . In the depth map 211 of FIG. 2A, the numerical value of the depth distance information Z of the mesh portion 205 corresponding to the object 204 is larger than the numerical value of the other portions. This indicates that it existed at a position close to

Then, based on the plane image 210 and the depth map 211, it is possible to arrange the image in the virtual 3D space. FIG. 2B is a diagram showing a schematic image of a 3D wire frame 207 of a result of arranging the image 200 in the virtual 3D space based on the plane image 210 and the depth map 211 . In FIG. 2B, the x direction indicates the horizontal direction of the image 200 (the direction in which the x value increases from the right side of the image), the y direction indicates the vertical direction of the image 200 (the direction in which the y value increases from the upper side of the image), and z The direction represents the depth direction. In the 3D wire frame 207 of FIG. 2B, the larger the value in the depth direction (depth distance), the higher the height in the z direction, indicating that the object is closer to the camera when the image is captured. In the case of the arrangement result example shown in FIG. 2(b), the region 206 with increased height in the z direction corresponds to the face object 204 in FIG. 2(a).

Next, a separation method for separating an object and a background (extracting an object) from an input image will be described.
As described above, since the input image has depth distance information for each pixel, a predetermined distance is set as a threshold for the depth distance information Z, and the object is separated from the background depending on whether the depth distance is within the threshold. can be done.

Next, a method of calculating a representative distance value for each object extracted from an image having depth distance information will be described.
The face object 204 separated as described above is higher in the z direction (depth distance) than the background (closer to the camera) as shown in region 206 in FIG. is represented as a hemisphere.
Here, a plurality of methods are conceivable as a method of calculating the representative distance value, and in this embodiment, two examples of the following first calculation method and second calculation method will be given.
The first calculation method calculates the average distance value per pixel by adding the values of the depth distances of the pixels that make up the object and dividing the addition result by the total number of pixels in the object. This is a method of using a representative distance value.
A second calculation method is to obtain the maximum depth distance value 221 shown in FIG.

In the present embodiment, the representative distance value is calculated using either of the first and second calculation methods, and the representative distance value is applied to the depth distance of all the pixels forming the previously separated object. , depth distance information represented by a cylindrical shape 230 as shown in FIG. 2(c) is set. In addition, FIG.2(c) represents the schematic image of a 3D wire-frame similarly to FIG.2(b).

Next, each process performed by the image synthesizing apparatus of this embodiment when synthesizing a plurality of images will be described with reference to FIGS. 3 and 7. FIG.
FIG. 3 is a flow chart showing the flow of processing performed in the image synthesizing apparatus of this embodiment. In the following description of the flowchart, processing steps S300 to S306 are abbreviated as S300 to S306. The same applies to other flowcharts to be described later. As an example, the processing of the flowchart of FIG. 3 is started when the user inputs a start instruction from an operation unit such as a touch panel (not shown) of the image synthesizing apparatus of the present embodiment. Note that the processing of the flowchart of FIG. 3 may be started when a start instruction is input via an external device, network, or the like, or at a timing such as a preset time.

3, the control unit 105 first initializes the memory 106, and then advances the process to S301.
In step S301, the control unit 105 instructs the image data reading unit 100 to acquire image data. Accordingly, the image data reading unit 100 acquires image data by reading it from an external medium (not shown) or the like. Then, the control unit 105 stores the image data acquired by the image data reading unit 100 in the memory 106 .
Furthermore, the control unit 105 causes the display unit 104 to display a UI (user interface) screen for the user to select a background image. At this time, the control unit 105 reads out the image data stored in the memory 106 and causes the decoding unit 101 to decode it, converts the captured image into a thumbnail image, and displays it on the screen of the display unit 104 in a list format.

FIG. 7A shows an example of a UI screen 710 displayed on the screen of the display unit 104 when selecting a background image. The UI screen 710 displays a list of thumbnail images of captured images and a message prompting the user to select a background image. FIG. 7A shows, as thumbnail images, an image A with a person (A), an image B with a person (B), an image C with a person (C), and an image D with no person. 4 shows an example in which four images of are displayed in a list. When the user inputs a selection instruction for a desired image among these images, a check mark 713 is displayed on the selected image. A check mark 713 is an icon that allows the user to recognize that the image is selected. FIG. 7A shows an example in which a check mark 713 is displayed when the user selects an image D that does not include a person. A completion button 712 is also displayed on the UI screen 710 in FIG. 7A. When the user selects one of the images and instructs the completion button 712 to make a decision, the control unit 105 determines that image as a background image to be used for synthesis. In the example of FIG. 7(a), it is assumed that image D with no person is confirmed as the background image by pressing the completion button 712 while a check mark 713 is displayed on image D. .

Then, when the background image is confirmed, the control unit 105 advances the processing to S302.
In step S302, the control unit 105 causes the display unit 104 to display a UI screen for the user to select an image to be synthesized (hereinafter referred to as an image to be synthesized).

FIG. 7B shows an example of a UI screen 720 displayed on the screen of the display unit 104 when selecting an image to be synthesized. As with the UI screen 710, the UI screen 720 shows an example in which four thumbnail images of image A, image B, image C, and image D are displayed as a list of thumbnail images of captured images. At this time, the UI screen 720 also displays a message prompting the user to select an image to be synthesized. Then, when the user inputs a selection instruction for a desired image on the UI screen 720, the check mark 713 similar to that described above is displayed on the selected image. In FIG. 7B, an image A including a person (A), an image B including a person (B), and an image C including a person (C) are checked by the user. An example in which a mark 713 is displayed is shown. A completion button 722 is also displayed on the UI screen 720 of FIG. 7B. When the user selects a desired image from among the images and issues a determination instruction to the completion button 722, the control unit 105 determines that image as an image to be synthesized. In the example of FIG. 7B, when the check mark 713 is displayed on the image A, the image B, and the image C, the image A, the image B, and the image C are displayed when the completion button 722 is pressed. are determined as images to be synthesized. Then, when the composition target image is determined, the control unit 105 advances the processing to S303.

Proceeding to S303, the control unit 105 determines whether or not the next representative distance value calculation processing of S304 has been completed for all the synthesis target images confirmed in S302. Then, if there is an unprocessed synthesis target image for which the representative distance value calculation process has not been performed, the control unit 105 causes the unprocessed synthesis target image to be subjected to the representative distance value calculation process of S304.

In S304, the control unit 105 controls the object extraction unit 103 and the distance calculation unit 107 to perform representative distance value calculation processing on the synthesis target image confirmed in S302. As the process of S304, the object extraction unit 103 first extracts a subject (for example, a human object) from one synthesis target image and separates it from the background. Next, the distance calculation unit 107 calculates the representative distance value based on the depth distance information of each pixel forming the object extracted by the object extraction unit 103, as described above. Then, the control unit 105 causes the memory 106 to store the representative distance value calculated by the distance calculation unit 107 in association with the extracted object. After that, the control unit 105 returns the processing to S303, and determines again whether or not the representative distance value calculation processing for all the images to be combined has ended. In this manner, the control unit 105 repeats the determination process of S303 and the representative distance value calculation process of S304 until the representative distance value calculation process of all synthesis target images is completed. Then, if the control unit 105 determines in S303 that the representative distance value calculation processing for all the synthesis target images confirmed in S302 has ended, the processing advances to S305.

Proceeding to S305, the control unit 105 performs a process of determining the composition order of each object based on the representative distance value stored in the memory 106 in association with each object (human object) processing to sort the order). The synthesis order is, for example, in ascending order of the representative distance value (that is, in ascending order of distance from the camera at the time of photographing). After that, the control unit 105 advances the processing to S306.

Proceeding to S306, the control unit 105 causes the synthesis unit 102 to sequentially read objects from the memory 106 in the synthesis order determined in S305, that is, in ascending order of the representative distance value (in order of distance from the camera). are combined with the background image (D) in this order. Then, the control unit 105 causes the memory 106 to store the synthesized image data obtained by the synthesizing process performed by the synthesizing unit 102 .

Here, among the three images A to C in which a person is photographed, the representative distance value of the object of image A is "100", the representative distance value of the object of image B is "10", and the object of image C is "10". is "50", and the larger the value, the closer it is to the camera. In this case, the composition order is the order of the image B object, the image C object, and the image A object. Therefore, in this case, the synthesizing unit 102 performs synthesizing processing such that the images of the respective objects are superimposed on the background image (D) in the order of the object of image B, then the object of image C, and finally the object of image A. will be As a result, an object with a large representative distance value (close to the camera) is image synthesized last.

In this embodiment, the representative distance value is calculated for each object, and the depth distance is uniformed within each object. I never get tired of it. Therefore, according to the present embodiment, it is possible to obtain a synthesized image that gives less discomfort.

<Second embodiment>
Next, as a second embodiment, an example will be described in which overlap determination between objects is performed, and representative distance value calculation processing is skipped for objects that do not overlap when combined. Note that the configuration of the image synthesizing apparatus of the second embodiment is the same as that of FIG. 1, so illustration and description thereof will be omitted.
An example in which the objects included in the synthesis target image do not overlap will be described with reference to FIG.

In the example of FIG. 4, assume that images 400, 401, and 402 are images to be synthesized. Also assume that objects 404, 405 and 406 are objects separated from the background in the corresponding images 400, 401 and 402, respectively. It is also assumed that the object 405 of the image 401 and the object 406 of the image 402 are objects that overlap each other when synthesized, while the object 404 of the image 400 is an object that does not overlap with objects of other images.

An image 403 represents a composite image after each of the objects 404, 405, and 406 has been composited with the background image. That is, in the example of FIG. 4, non-overlapping object 404 is placed in image 403 as object 407, while object 405 is placed in image 403 as object 408 and object 406 as object 409. FIG.

In the case of the second embodiment, the control unit 105 performs overlap determination processing for determining whether or not the plurality of objects extracted from the plurality of synthesis target images by the object extraction unit 103 overlap each other. That is, before the representative distance value is calculated, the control unit 105 recognizes the positional relationship of each of the objects 404, 405, and 406 extracted by the object extraction unit 103, and determines whether or not the objects overlap each other. judge. Then, the control unit 105 detects an object such as the object 404 that does not overlap with other objects, and skips the representative distance value calculation processing for that object 404 .

FIG. 5 is a flow chart of processing in the second embodiment.
In the flowchart of FIG. 5, the same reference numerals are given to the same processing steps as in the flowchart of FIG.

In the case of the flowchart of FIG. 5, after S302, the control unit 105 advances the process to S503.
Proceeding to S503, the control unit 105 determines whether or not all objects extracted by the object extraction unit 103 from all the compositing target images determined in S302 will overlap each other at the time of compositing. For example, in the case of FIG. 4, the control unit 105 determines that objects 405 and 406 overlap, and determines that object 404 does not overlap any other object. After S503, the control unit 105 advances to S303. If it is determined that all synthesis target images have been processed, the control unit 105 advances the processing to S305. Proceed with processing.

In step S501, the control unit 105 causes the representative distance value to be calculated in step S304 for objects that are determined to overlap when combined based on the result of the overlap determination performed in step S503, and then returns the processing to step S303. . On the other hand, the control unit 105 performs the processing of S502 for the objects determined in S501 not to overlap at the time of composition.

In step S502, the control unit 105 sends a predetermined initial value to the distance calculation unit 107 because the object does not overlap other objects during synthesis and the synthesis processing based on the representative distance value becomes unnecessary. set. Then, the control unit 105 stores the initial representative distance value in the memory 106 in association with the object. After S502, the control unit 105 advances the process to S305. Since the processing after S305 is the same as described above, the description is omitted.

According to the second embodiment, by skipping the representative distance value calculation processing for objects that do not overlap each other at the time of combining, it is possible to reduce the processing load due to the representative distance value calculation. In addition, as in the case of the first embodiment, objects that overlap each other at the time of synthesis are synthesized based on the representative distance value. image can be obtained.

<Third Embodiment>
Next, in the third embodiment, an image synthesizing method in which the object to be synthesized is a specific object, for example, a human face object will be described.
Here, in the first embodiment, a person object was taken as an example of an object to be extracted and synthesized, but in the third embodiment, a case in which a person's face object is included in an image to be synthesized will be described. In the case of the third embodiment, the object extraction unit 103 also has a face determination function for determining whether or not a person's face object is included in the composition target image. In the case of the third embodiment, the object extraction unit 103 measures the depth distance of the object as described above and also determines whether or not there is a face. Then, the control unit 105 associates the face presence/absence information and the representative distance value determined by the object extraction unit 103 with the object, stores them in the memory 106, and uses them in the synthesis process.

FIG. 6 is a flow chart of processing in the third embodiment.
In the flowchart of FIG. 6, the same processing steps as those of the flowchart of FIG. 3 are denoted by the same reference numerals, and descriptions thereof are omitted, and only different processing steps (S601, S602, S603, S604) are described here. .

In the case of the flowchart of FIG. 6, after S302, the control unit 105 advances the process to S601.
Upon proceeding to S601, the control unit 105 determines whether or not the representative distance value calculation processing and face recognition processing for all the compositing target images confirmed in S302 have ended. Then, if there is an unprocessed composition target image for which the representative distance value calculation processing and the face recognition processing have not been performed, the control unit 105 performs the representative distance value calculation processing and face recognition processing of S602 on the unprocessed composition target image. Let the recognition process run.

After proceeding to S602, the control unit 105 controls the object extraction unit 103 and the distance calculation unit 107 to perform representative distance value calculation processing and face recognition processing on the synthesis target image determined in S302. As the process of S304, the object extraction unit 103 first extracts a human object from one image to be synthesized. Furthermore, the object extraction unit 103 determines whether or not there is a face object of the person, and if it determines that there is a face object, stores the face presence/absence information in the memory 106 in association with the image. Keep Further, the distance calculation unit 107 calculates a representative distance value based on the depth distance information of each pixel forming the object extracted by the object extraction unit 103, in the same manner as described above. Then, the control unit 105 causes the memory 106 to store the representative distance value calculated by the distance calculation unit 107 in association with the extracted object. After that, the control unit 105 returns the processing to S601, and determines again whether or not the representative distance value calculation processing and the face recognition processing for all synthesis target images have been completed. In this manner, the control unit 105 repeats the determination processing in S601 and the representative distance value calculation processing and face recognition processing in S602 until the representative distance value calculation processing and face recognition processing for all synthesis target images are completed. Then, if the control unit 105 determines in S601 that the representative distance value calculation processing and the face recognition processing for all composition target images have ended, the processing advances to S603.

Proceeding to S603, the control unit 105 sorts the compositing order of each object based on the representative distance value and face presence/absence information stored in the memory 106 in association with each object. In this case, the control unit 105 acquires from the memory 106 the representative distance values of the plurality of objects extracted from the plurality of synthesis target images, and determines whether the difference between the representative distance values is a close value within a certain range. determine what Further, the control unit 105 determines whether or not any of the objects includes the specific object when the representative distance values of the objects are close. In this example, the control unit 105 uses face presence/absence information to determine whether an object including a face object exists among objects having close representative distance values. Then, when an object including a face object exists among objects having close representative distance values, the control unit 105 determines the synthesis order so that the object is synthesized on the near side (synthesed later) when synthesizing. In this case, the control unit 105 arranges the face objects so that, among the objects whose representative distance value difference is within a certain range, the object including the face object is synthesized in the front side during synthesis. Change the representative distance value for objects containing . After S603, the control unit 105 advances the process to S604.

Note that the control unit 105 changes the representative distance value of the object that does not include the face object so that the object that does not include the face object is synthesized on the rear side during synthesis. can be
Further, for example, when the difference in the representative distance value between objects that both include faces is within a certain range, the synthesis order is determined according to the representative distance value for each object that includes the face. In this way, when the representative distance values of objects including faces are close to each other, synthesis may be performed using the representative distance values of facial parts, etc., as in the fourth embodiment described later.

In step S604, the control unit 105 causes the combining unit 102 to combine objects with the background image in the order of composition according to the representative distance value determined in step S603, that is, in the order of objects with smaller representative distance values (in order of distance from the camera). . That is, in this case, in the image synthesis, among the objects whose representative distance value difference is within a certain range, the object not including the face is synthesized with the background image, and then the object including the face is synthesized. . Then, the control unit 105 causes the memory 106 to store the synthesized image data obtained by the synthesizing process performed by the synthesizing unit 102 .

Image composition in the third embodiment will be specifically described with reference to FIG.
FIG. 8 shows an example in which three images, image E, image F, and image G, are images to be synthesized. It is assumed that the object of image E has a representative distance value of 1.0 m, and that it has been determined that there is no face object as a result of face recognition. Image F and image G have the same representative distance value of 0.8 m for each object, but image F is judged to have a face object, while image G is judged to have no face object. It is assumed that

In this case, the synthesis order is determined as follows.
Objects in image E whose representative distance value is far from the camera are determined as the composition order in which they are first composited with the background image. Since the objects in image F and image G have the same representative distance value, presence/absence of a face is determined. Since image F contains a face object and image G does not contain a face object, the composition order of image G objects is determined first, and the composition order of image F objects is determined after image G objects. It is determined. By doing so, a composite image can be obtained in which the object of image E is arranged on the farthest side, the object of image G is arranged next, and the face object of image F is arranged in the foreground. Become.

In the third embodiment, synthesis is performed while using representative distance values. Objects having close representative distance values are judged whether or not they contain a face object. Determines the order of compositing on the near side. As a result, a natural synthesized image can be obtained even in the third embodiment.

<Fourth Embodiment>
In the fourth embodiment, a different calculation method for calculating the representative distance value described in the first embodiment will be described.
In the method of calculating the representative distance value in the fourth embodiment, the depth distance information of the same part is calculated for each of two or more objects whose overlaps are to be compared, and is used as the representative distance value.

The representative distance value calculation method of the fourth embodiment will be described with reference to FIGS. 9(a) and 9(b). In FIG. 9A, an image 900 is an image in which three persons are photographed as subjects. An image 900 is an example in which face images 901, 902, and 903 of three persons, person (A), person (B), and person (C) are shown. A face image 901 is a face image of a person (A) wearing a hat 904 . It is assumed that the face image 902 is the image of the face of the person (B), and there is nothing that hides the face. A face image 903 is a face image of a person (C), and a hand 905 is in front of the face.

FIG. 9(b) is a diagram showing the distance map 910 in the example of FIG. 9(a) in the same manner as described above. As shown in FIG. 9(b), the distance map 910 is a distance map including the depth distance between each subject (person) and the hat or hand in FIG. 9(a). In the distance map 910 of FIG. 9(b), a hatched mesh portion 911 corresponds to the face image 901 and the hat 904, a mesh portion 912 corresponds to the face image 902, and a mesh portion 913 corresponds to the face image 903 and the hat 904. It corresponds to hand 905 . In the mesh portion 912, the depth distance value of the portion 917 corresponding to the center of the face image 902 is large. On the other hand, in the case of the mesh portion 911 , the depth distance value of the portion 914 corresponding to the hat 904 is larger than the depth distance value of the portion 916 corresponding to the center of the face image 901 . In the case of the mesh portion 913 , the value of the depth distance of the portion 915 corresponding to the hand 905 is larger than that of the portion 919 corresponding to the center of the face image 903 .

The distance calculation unit 107 calculates a representative distance value for each object including the face image 901 and hat 904 of person (A), the face image 902 of person (B), the face image 903 and hand 905 of person (C). Then, the distance calculation unit 107 sums and averages the depth distances of the pixels in the mesh portions 911, 912, and 913 corresponding to the respective objects, and calculates a representative distance value. However, in the case of the image 900 in FIG. 9A, the calculation result of the representative distance value is affected by the depth distance of the hat 904, the hand 905, and the like.

Therefore, in the fourth embodiment, the hat 904, the hand 905, and other factors are excluded, and only the positions of the faces are compared. Here, in order to simply compare the depth distances of only the faces, the control unit 105 instructs the object extraction unit 103 to extract, for example, only nose parts from each of the face images 901, 902, and 903. instruct to do so. Since the technology for recognizing a part of the face such as the nose is an existing technology, the description thereof will be omitted. Then, the distance calculation unit 107 averages the values of the depth distance of each pixel corresponding to the position of the nose extracted by the object extraction unit 103 to calculate a representative distance value. In the case of the example of FIG. 9B, the depth distance value corresponding to the nose portion is the distance value "9" for the portion 916 corresponding to the center of the face described above, the distance value "10" for the portion 917, and the distance value "10" for the portion 919 distance value "8".

Thus, in the fourth embodiment, by calculating the representative distance value from a part of the face (parts), in addition to reducing the calculation area, elements such as hats and hands are omitted to represent the face object. A distance value can be calculated. Here, the nose is used as a part of the face, but the present invention is not limited to this, and any part such as the eyes, mouth, etc. may be used as long as it has a feature, is easy to recognize, and is easy to compare depth distances. Moreover, even if it is not a human body, it may be anything that can be extracted by the object extraction unit 103, such as tire positions of automobiles, when comparing automobiles.

<Fifth Embodiment>
In the fifth embodiment, a method of editing the depth distance information of the object described in the first embodiment and synthesizing it with another object will be described.
A synthesizing method according to the fifth embodiment will be described with reference to FIG.
In FIG. 10, an image 1000 is a person (A) and an image 1010 is a composition target image in which a person (B) is shown, and objects 1001 and 1011 are shown in each of them. A map 1002 is a map of depth distance information Z of the image 1000 . A similar map 1012 is a map of depth distance information Z of the image 1010 . Map 1002 includes mesh portion 1003 corresponding to object 1001 , and map 1012 includes mesh portion 1013 corresponding to object 1011 .

In the example of FIG. 10, comparing the mesh portion 1003 of the object 1001 and the mesh portion 1013 of the object 1011 in the depth distance information maps 1002 and 1012, it can be seen that the object 1011 is closer. Here, when the synthesizing unit 102 synthesizes these objects, a synthesized image 1020 is obtained. This synthesized image 1020 is an image synthesized so that the object 1021 corresponding to the object 1011 of the person (B) is in front.

Next, it is assumed that the user places the object 1001 representing the person (A) in front of the object 1011 representing the person (B) and issues an image composition instruction. In this case, the control unit 105 collectively changes the depth distance information Z of the object 1001 in response to the user's instruction. That is, the control unit 105 uniformly sets the distance value X for the portion of the depth distance information map 1002 of the object 1001 in which the value Z(a, b) of each depth distance is not 0, that is, the mesh portion 1003 corresponding to the object 1001. to add. The distance value X may be, for example, a value designated by the user or a predetermined value, or an appropriate value determined by the control unit 105 based on depth distance information between the object designated by the user and other objects. A value determined as Thereby, new depth distance information NewZ(a, b) is calculated.

At this time, the control unit 105 calculates new depth distance information NewZ(a, b) using the following formula.

NewZ(a,b)=0 //Z(a,b)==0
=Z(a,b)+X //Z(a,b)! == 0

In the example of the fifth embodiment, the map 1004 shows an example in which 5 is added to a location where the depth distance information Z is not 0 by the control unit 105 when X=5. is the area of the depth distance information NewZ.

By performing such processing, the synthesizing unit 102 generates a synthesized image as shown in the image 1022 . In this image 1022, the object of the person (A) does not change in appearance as an image, but is rearranged in the front position by a distance corresponding to a value of 5 in the depth direction. This results in a value of 14 (9+5) for mesh portion 1005 and a value of 10 for mesh portion 1013 . The synthesizing unit 102 synthesizes the object 1001 and the object 1011 with the background image based on the depth distance information of the mesh portion 1003 of the map 1004 to generate an image 1022 . As a result, the image 1022 is a composite image in which the person (A) object is placed in front of the person (B) object.

According to the fifth embodiment, by editing the depth distance information of the objects and synthesizing the respective objects, it is possible to obtain a synthetic image in which the positions of the respective objects are interchanged without causing a sense of incongruity.

The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in the computer of the system or apparatus reads and executes the program. It can also be realized by processing to It can also be implemented by a circuit (for example, ASIC) that implements one or more functions.

All of the above-described embodiments merely show specific examples for carrying out the present invention, and the technical scope of the present invention should not be construed to be limited by these. That is, the present invention can be embodied in various forms without departing from its technical spirit or main features.

100: image data reading unit, 101: decoding unit, 102: synthesis unit, 103: object extraction unit, 104: display unit, 105: control unit, 106: memory, 107: distance calculation unit

Claims (11)

an extraction means for extracting an object from a plurality of images;
Determination means for determining a representative distance value representing the depth distance for each object extracted from the plurality of images, based on the depth distance information included in the images;
synthesizing means for synthesizing the objects to generate a synthetic image based on the representative distance value for each of the extracted objects;
The determining means determines the representative distance value for the overlapping objects when the objects overlap when the objects are combined , and determines an initial value as the representative distance value for the objects that do not overlap . An image processing apparatus characterized by: The determining means determines that an object including a specific object exists among the extracted objects, and a difference in the representative distance value between the object including the specific object and the object not including the specific object is within a certain range. 2, the representative distance value is changed so that an object including the specific object is in front of an object not including the specific object during the synthesis. The image processing device according to . 3. The image according to claim 1, wherein for each of the extracted objects, the determining means determines an average of values representing depth distances of all pixels included in the object as the representative distance value. processing equipment. 3. The method according to claim 1, wherein, for each of the extracted objects, the determining means determines, as the representative distance value, a value representing a maximum depth distance among all pixels included in the object. image processing device. 3. The image processing according to claim 1, wherein said determining means determines said representative distance value based on depth distance information of all pixels included in a specific portion of said extracted object. Device. 6. The image processing apparatus according to claim 5 , wherein said determining means determines an average of values representing depth distances of all pixels included in said specific region as said representative distance value. 7. The image processing apparatus according to any one of claims 1 to 6 , wherein said determining means has changing means for changing a value representing the depth distance of said extracted object. 8. The image processing apparatus according to claim 7 , wherein said changing means changes the value of said depth distance of an object designated by a user. 9. The image processing apparatus according to claim 7 , wherein the changing unit uniformly adds a distance value to the value of the depth distance of the object to change the depth distance. An image processing method executed by an image processing device,
an extraction step of extracting an object from a plurality of images;
a determination step of determining a representative distance value that represents the depth distance for each object extracted from the plurality of images based on the depth distance information included in the images;
a synthesizing step of synthesizing the objects to generate a synthetic image based on the representative distance value for each of the extracted objects;
In the determining step, if the objects overlap when the objects are merged, the representative distance value is determined for the overlapping objects, and an initial value is determined as the representative distance value for the objects that do not overlap . An image processing method characterized by: A program for causing a computer to function as each means of the image processing apparatus according to any one of claims 1 to 9 .

Images