JP2019045993A - Image processing device, image processing method and program - Google Patents

Abstract

To extract a subject area from a photographic image with high accuracy.SOLUTION: An image processing device according to the present invention acquires location information and azimuth information as a photographic condition when photographic means photographs, and generates a model image on the basis of the photographic condition, and then extracts a subject area from a photographic image on the basis of the generated model image.SELECTED DRAWING: Figure 4

Description

  The present invention relates to a technique for extracting a subject area from an image.

  As one of the image recognition methods, there is a method of dividing an image obtained by photographing into a plurality of areas and identifying a class related to the classification of an object for each of the divided areas. This method identifies the class of each area based on the feature value extracted from each area. Properly dividing the image into areas facilitates many image processing such as object recognition and scene recognition, and correction of image quality according to the object.

  However, if identification is performed based on only the feature amount of each region, erroneous detection may occur despite the high reliability (the identification score and the identification likelihood are high). For example, although a blue sky and a blue wall can be easily distinguished by humans, it is difficult for a classifier to distinguish an area cut out of a part of the sky from an area cut out of a part of the blue wall.

  Therefore, Patent Document 1 discloses a technique for estimating a category or a part of a subject based on GPS information or shooting information obtained when shooting with a camera.

Patent No. 5565521 gazette

R. Socher, “Parsing Natural Scenes and Natural Language with Recursive Neural Networks”, International Conference on Machine Learning 2011. P. Krahenbuhl, "Efficient Inference in Fully Connected CRFs with Gaussian Edge Potentials", Neural Information Processing Systems 2011. J. Tighe, "SuperParsin: Scalable Nonparametric Image Parsing with Superpixels", European Conference on Computer Vision 2010. Y. Taigman, "DeepFace: Closing the Gap to Human-Level Performance in Face Verification", Conference on Computer Vision and Pattern Recognition 2014.

  However, as in the technique of Patent Document 1, there are cases where it is not possible to accurately extract the subject region in the photographed image only by the information obtained from the GPS information and the photographing information. Therefore, an object of the present invention is to enable accurate extraction of a subject region from a captured image.

  The present invention acquires, as a photographing condition, a first acquisition unit that acquires a photographed image in which a subject is photographed by a photographing unit, and position information and orientation information of the photographing unit when the acquired photographed image is photographed. Extraction means for extracting the area of the subject from the photographed image based on the second acquisition means, generation means for generating a model image of the subject based on the acquired photographing conditions, and the generated model image And.

  According to the present invention, it is possible to accurately extract a subject area from a captured image.

FIG. 1 is a view showing an example of the arrangement of an image processing system according to the first embodiment; FIG. 3 is a view showing an example of an image in which a subject region is extracted in the first embodiment. FIG. 1 is a schematic block diagram showing a hardware configuration of an image processing apparatus according to a first embodiment. FIG. 1 is a schematic block diagram showing a functional configuration of an image processing apparatus according to each embodiment. FIG. 10 is a schematic block diagram showing the functional configuration of an image processing apparatus according to a third embodiment. FIG. 1 is a schematic block diagram showing a functional configuration of an image processing apparatus according to each embodiment. 5 is a flowchart showing processing executed by the image processing apparatus according to each embodiment. FIG. 2 is a view showing an example of a subject model in the first embodiment. FIG. 3 is a view showing an example of a subject model image in the first embodiment. FIG. 6 is a view showing another example of the subject model image in the first embodiment. FIG. 8 is a view for explaining the details of processing of the subject region extraction step in the present embodiment; FIG. 2 is a view showing an example of a subject model coordinate system in the first embodiment. FIG. 6 is a view showing an example of edge-based three-dimensional model fitting in the first embodiment. FIG. 7 is a view for explaining a method of calculating the position and orientation of a three-dimensional model using line segment information in the first embodiment. The schematic block diagram which shows the function structure of the registration object image setting part which concerns on 3rd Embodiment. The flowchart which shows the processing by the registered object picture setting part concerning a 3rd embodiment. The figure which shows the example of the setting of a registration object area | region in 3rd Embodiment.

First Embodiment
Hereinafter, the details of the first embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a view showing a schematic configuration example of a system provided with an image processing apparatus according to the present embodiment. In the system of FIG. 1, a camera 10 and an image processing apparatus 20 are connected via a network 15. The image processing apparatus 20 and the camera 10 may be integrally configured.

  The camera 10 captures an image to be subjected to image processing by the image processing apparatus 20. In FIG. 1, a tree 30a, a car 30b, a building 30c, a sky 30d, a road 30e, a human body 30f, etc. exist within the angle of view (shooting range) An example in which the camera 10 captures a scene 30 to be shot is shown. The image processing apparatus 20 extracts each subject area in the scene 30 captured by the camera 10.

  FIG. 2 is a view showing an example of an image in which a subject region is extracted. FIG. 2A shows a photographed identification target image 100, in which images of subjects 101 to 105 appear. A subject area extracted from the identification target image 100 is indicated by 200 in FIG. Although FIG. 2B shows an example in which only the subject area 200 of the building 101 is extracted, a plurality of subject areas may be extracted. FIG. 2C shows an example in which a plurality of subject areas 201 and 202 are extracted. The method of extracting the subject areas 200 to 202, and the processing method for the extracted subject areas 200 to 202 and the other areas 102 to 106 will be described later.

  FIG. 3 is a schematic block diagram showing an example of the hardware configuration of the image processing apparatus 20 according to the present embodiment. The CPU 401 controls the entire image processing apparatus 20. When the CPU 401 executes a program stored in the ROM 403, the HD 404, or the like, the functional configuration of the image processing apparatus 20 described later and the processing of the flowchart according to the image processing apparatus 20 are realized. The RAM 402 is a storage area that functions as a work area in which the CPU 401 develops and executes a program. The ROM 403 is a storage area for storing programs executed by the CPU 401 and the like. The HD 404 is a storage area for storing various data including various programs required when the CPU 401 executes a process, data on a threshold, and the like. The operation unit 405 receives an input operation by the user. The display unit 406 displays information of the image processing apparatus 20. A network I / F 407 connects the image processing apparatus 20 to an external device.

  FIG. 4 is a schematic block diagram showing a functional configuration of the image processing apparatus 20 in each embodiment, and FIG. 4A is a block diagram in the present embodiment. In addition to the functional blocks in the image processing apparatus 20, FIG. 4A also illustrates an imaging unit 500 corresponding to the camera 10. The imaging unit 500 corresponds to the camera 10 and acquires an identification target image. The image processing apparatus 20 according to the present embodiment includes a photographed image acquisition unit 501, a photographing condition acquisition unit 502, an object information acquisition unit 503, an object model image generation unit 504, an object area extraction unit 505, an image processing unit 506, an object information holding unit It has 507. The subject information holding unit 507 may be connected to the computer 20 as a non-volatile storage device. Details of each of these functions of the image processing apparatus 20 will be described later.

  FIG. 7 is a flowchart showing an outline of processing executed by the image processing apparatus 20 in each embodiment, and FIG. 7A is a flowchart of this embodiment. In the flowchart of FIG. 7A, in the captured image acquisition step S110, the captured image acquisition unit 501 receives an identification target image captured by the imaging unit 500 as input data. The acquired identification target image is transmitted to the subject model image generation unit 504 and the subject region extraction unit 505.

  Next, in the photographing condition acquisition step S120, the photographing condition acquisition unit 502 acquires the photographing condition corresponding to the identification target image photographed by the photographing unit 500. The specific content of the imaging information will be described later. The acquired shooting conditions are transmitted to the subject information acquisition unit 503 and the subject model image generation unit 504. Next, in the subject information acquisition step S130, a three-dimensional model of the subject estimated from the position information and orientation information (direction and direction) of the imaging unit acquired in the imaging condition acquisition step S120 and surface reflection characteristic information of each surface (texture Information) is acquired from the subject information holding unit 507. Details will be described later. The three-dimensional model of the subject and the surface reflection characteristic information (texture information) of each surface, which are the subject information acquired, are transmitted to the subject model image generation unit 504.

  Next, in the subject model image generation step S140, a subject model image is generated based on the photographing conditions acquired by the photographing condition acquisition unit 502 and the subject information acquired by the object information acquisition unit 503. The generation method will be described later. The generated subject model image has the same size as the captured image acquired by the captured image acquisition unit 501, and is an image related to the same object. The generated subject model image is transmitted to the subject region extraction unit 505. Next, in the subject region extraction step S150, the subject region in the photographed image is extracted by fitting the subject model image generated by the subject model image generation unit 504 and the photographed image acquired by the photographed image acquisition unit 501. The specific method will be described later. Information of the extracted subject area is transmitted to the image processing unit 506.

  Finally, in the image processing step S160, image processing is performed based on the subject area extracted by the subject area extraction unit 505. Specific processing content will be described later. By the above processing, the subject region in the photographed image is extracted, and the image is image-processed based on the subject region and the contour of the region.

  Next, the specific flow of each process will be described with reference to the flowchart shown in FIG. In the photographed image acquiring step S110, an image obtained by the photographing unit 500 photographing the scene 30 as shown in FIG. 1 is acquired as the identification target image 100. The identification target image may be an image stored in an external device (not shown). In that case, the input unit 501 acquires an image read from an external device as an identification target image. The image stored in the external device may be, for example, an image captured in advance by the imaging unit 500 or the like, or an image stored after being acquired by another method such as passing through a network or the like. It is also good. In that case, it is necessary to save in advance information (shooting information) such as shooting parameters at the time of shooting in addition to the shot image. The specific content of the imaging information will be described in the next imaging condition acquisition step S120.

  In the photographing condition acquisition step S120, the photographing condition when the photographing unit 500 photographs the scene 30 as shown in FIG. 1 is acquired. The shooting conditions include the position information of the shooting unit 500 obtained based on GPS information, etc., the azimuth (direction / direction) of the shooting unit 500 when shooting the identification target image 100, and shooting parameters such as the angle of view and shutter speed. It shows. Also, light source information and environmental information at the time of shooting may be acquired. For example, it may be a photometric value acquired by a sensor of a camera when photographing with a camera or the like, a value such as color temperature or AF. In addition, environmental information such as weather and temperature may be acquired via a network using GPS information or the like. Those pieces of information are used when generating a subject model image, and a higher definition subject model image can be generated.

  In the subject information acquisition step S130, subject information is acquired based on the imaging conditions acquired in the imaging condition acquisition step S120. Among the shooting conditions, subject information is acquired based on position information acquired particularly by GPS information or the like. Furthermore, the subject itself may hold information and may be held in an RFID tag or the like, and the image processing apparatus may receive the information to obtain subject information. Thereby, for example, subject information on a moving subject can be acquired. In any case, in the present embodiment, the subject information is information used in the next subject model image generating step S140. FIG. 8 shows an example of a subject model in the present embodiment, and subject information includes, for example, a three-dimensional model of a subject, color / texture information of each surface, and the like as shown in 300 to 302 shown in FIG. It represents the reflectance, transparency, and refractive index of each surface. Based on this information, a subject model image under the current shooting conditions is generated.

  In the present embodiment, although a three-dimensional model of a subject is acquired, it is sufficient if a subject model image can be generated, so a model image of a subject obtained from each viewpoint may be held. In the case of an articulated object such as a person or a character as shown in FIG. 8B, the model of each part may be held separately.

  In the subject model image generation step S140, a subject model image is generated using the photographing conditions and the subject information obtained in the previous photographing condition acquisition step S120 and the subject information acquisition step S130. In order to generate, rendering technology such as ray tracing may be used. A subject model image close to a photographed image can be generated by using position information of the photographing apparatus, an azimuth, an angle of view, photographing parameters, and information of an imaging apparatus and an environment in which the subject is present. FIGS. 9 and 10 are views showing an example of a subject model image according to this embodiment, and in the case of a subject model as shown in FIG. 8A, as shown in FIGS. 9A and 9B. An object model image is generated. Further, in the case of an object model of a multi-joint object as shown in FIG. 8B, as shown in FIGS. Since there are also parameters of joint angles between parts, multiple subject model images are generated. The generation method may be randomly generated, or the posture (frequency of each joint angle) taken by the subject often as subject model information may be stored in advance and may be generated based thereon. In FIGS. 9 and 10, the subject model image is generated from a single subject model, but a plurality of models may be used to generate a subject model image.

  In the subject region extraction step S150, the subject region is extracted by fitting the photographed image acquired in the photographed image acquisition step S110 with the object model image acquired in the object model image generation step S140. FIG. 11 is a diagram for explaining the details of the processing of the subject region extraction step in the present embodiment. As shown in FIG. 11, the subject region is extracted by fitting the contour 600 of the subject model acquired from the subject model image to the captured image 100.

  In the present embodiment, the subject region is extracted by performing edge-based three-dimensional model fitting. According to this method, it is possible to calculate the three-dimensional position and orientation in the coordinate system which has been previously defined at the same time as the subject area. FIG. 12 shows an example of a subject model coordinate system in the present embodiment. Here, as shown in FIG. 12, an object model coordinate system 304 having a position at an object model 300 as an origin is defined, and the three-dimensional position and orientation of the camera 10 in the object model coordinate system 304 is calculated. In the present embodiment, only the subject 101 is described, but it may be performed on a plurality of subjects, or fitting is performed on one subject, and the relative position and orientation relationship with the subject is used. Other subject areas may be extracted.

FIG. 13 is a diagram showing an example of edge-based three-dimensional model fitting in the present embodiment. As shown in FIG. 13, in the edge-based three-dimensional model fitting, processing is performed at each division point obtained by dividing each edge of the model at equal intervals. The processing is performed on the image plane by projecting a three-dimensional model on the image plane. At each dividing point 52, the edge of the object to be measured is searched on a line segment (hereinafter, search line 53) which passes through the dividing point perpendicular to each projection line 51 of the three-dimensional model. After processing for all division points, the position and orientation of the three-dimensional model are calculated. Here, let N _C be the total number of corresponding points that could be dealt with among the division points. The position and orientation of the three-dimensional model are calculated by repeatedly correcting.

Next, a method of calculating the position and orientation of a three-dimensional model using information on line segments (edges) will be described. FIG. 14 is a diagram for explaining a method of calculating the position and orientation of a three-dimensional model using line segment information. In FIG. 14, the horizontal direction of the image is taken as the u axis, and the vertical direction as the v axis. The coordinates of a certain division point 52 are represented by (u _k , v _k ), and the inclination of the line segment 51 to which the division point belongs on the image is represented by the inclination θ with respect to the u-axis. Also, let the normal vector of the line segment 51 be (sin θ, −cos θ). Further, the coordinates of the corresponding point 60 on the edge of the object to be measured with respect to the division point 52 are set to (u ′ _k , v ′ _k ). Here, the distance from the dividing point 52 to the corresponding point 60 is calculated. Here, the point (u, v) on the line segment 51 satisfies the formula 1.

However,

It is. A point (u, v) on a straight line parallel to the line segment 51 passing through the corresponding point 60 satisfies Equation 3.

However,

It is. Therefore, the distance between the dividing point 52 and the corresponding point 60 is calculated by dr. The coordinate values of the corresponding point 60 in the image coordinate system are a function of the position and orientation of the three-dimensional model. There are six degrees of freedom in the position and orientation of the three-dimensional model. Here, a parameter representing the position and orientation of the three-dimensional model is represented by p. p is a six-dimensional vector, and includes three elements representing the position of the three-dimensional model and three elements representing the posture. The three elements representing the posture are represented, for example, by Euler angles or a three-dimensional vector in which the direction represents the rotation axis and the size represents the rotation angle.

If (u, v) is approximated by a first-order Taylor expansion in the vicinity of (u _k , v _k ) as coordinates of the division point 52 in the image coordinate system, it can be expressed as Expression 5.

Here, partial differential coefficients ∂u / ∂p _i and ∂v / ∂p _i are image Jacobians.

  The correction value Δp of the position / posture parameter p of the three-dimensional model is calculated such that (u, v) represented by Formula 5 exists on the straight line represented by Formula 3. Substituting Equation 5 into Equation 3, Equation 6 is obtained.

If Formula 6 is arranged, it can be expressed like Formula 7.

Since Equation 7 holds for N _C division points, linear simultaneous equations for Δp like Equation 8 hold.

Here, Equation 8 is briefly expressed as Equation 9.

From Equation 9, Δp is determined using the generalized inverse matrix (J ^T · J) ⁻¹ of the matrix J. Finally, calculation is repeated until Δp becomes equal to or less than the threshold. After repeated calculation, the subject area is extracted. When a plurality of subject model images are generated in the subject model image generating step S140, the subject area at the time when Δp becomes the smallest among the results of fitting each subject model image and the photographed image is the optimum subject area. do it. In the case of an articulated object, three-dimensional model fitting may be performed for each part. Alternatively, three-dimensional model fitting may be performed on a plurality of subject model images as shown in FIGS. 10A to 10D, and the subject area when Δp becomes the smallest may be set as the optimal subject area. .

  In the present embodiment, only the area related to the subject holding the subject model is extracted, but other areas may also be extracted. In that case, region division and category identification may be performed using a region division method as disclosed in Non-Patent Documents 1 and 2. Alternatively, the class of each small region of the identification target image may be determined based on the similar image selected by the similar image search based on the global feature of the identification target image as disclosed in Non-Patent Document 3. When searching for the similar image, the search may be performed based on the position information of the camera acquired in the photographing information acquisition step S120 or the subject information acquired in the subject information acquisition step S130.

  In the present embodiment, edge-based three-dimensional model fitting is performed to extract the subject area, but a method of extracting the subject area by performing a graph cut with the subject area as the foreground and the other areas as the background is used. May be Also, fitting may be performed using image feature quantities other than edges.

  In the image processing step S160, the image quality enhancement processing is performed on the captured image using the subject region extracted in the subject region extracting step S150. A specific process will be described in the case where the subject region 200 is extracted for the subject 101 as shown in FIG. 2B. Color tone correction, white balance adjustment, noise adjustment, and the like are performed on the subject 101 for which the subject area 200 has been extracted, using color information of each surface of the subject model. Further, unnecessary objects are removed, blurring processing, and the like are performed on unnecessary objects such as the object 106 other than the object 101. When removing unnecessary objects, the background is interpolated using a subject model image. When a plurality of subject areas 201 and 202 are extracted as shown in FIG. 2C, separate processes can be performed on each subject area.

  Further, as described above, when an area other than the subject holding the subject model is also extracted, the high image quality processing may be performed according to the category or the area. For example, in the case of FIG. 2C, when the person 103 and the tree 106 can be extracted, the high image quality processing may be performed according to the person and the tree.

  As described above, according to the present embodiment, the image processing apparatus 20 acquires a three-dimensional model of an object from the imaging conditions at the time of imaging, and generates an object model image. By fitting the generated object model image and the photographed image, the object region in the photographed image can be extracted, and the image quality improvement processing can be performed for each region based on the object region. By generating the subject model image from the three-dimensional model of the subject and the shooting conditions, it is possible to accurately extract the subject region in the identification target image.

Second Embodiment
Next, a second embodiment of the present invention will be described. In the present embodiment, after the subject region in the captured image is extracted by fitting the captured image and the subject model image generated once, the position / orientation of the imaging unit or the light source information and environment information are extracted from the extraction result of the subject region. Re-estimate. Then, the subject model image is generated again from the re-estimated information, and the subject region in the photographed image is accurately estimated by fitting it to the photographed image. Hereinafter, a second embodiment of the present invention will be described with reference to the drawings. In addition, about the structure already demonstrated in 1st Embodiment, the description is abbreviate | omitted and the same code | symbol is attached | subjected.

  FIG. 4B shows a functional configuration of the image processing apparatus 20 in the present embodiment. The difference from the first embodiment is that a photographing condition re-estimation unit 508 and a subject region re-extraction unit 509 are added.

  In the first embodiment, one or more subject model images are created using the shooting conditions and the subject model, and the subject region is extracted by fitting the subject model image and the photographed image. On the other hand, in the present embodiment, the fitting result is fed back to re-estimate the shooting conditions, and the subject region is extracted again. This method accurately estimates imaging information such as position information even when the environment information obtained from the position information obtained by GPS etc. or sensor information from the camera is not accurate or rough, and the subject area is also highly accurate. It can be extracted.

  FIG. 7B is a flowchart showing processing by the image processing apparatus in the present embodiment. The processing contents in the photographed image acquisition step S210 to the subject area extraction step S250 and the image processing step S280 of the present embodiment are the same as the photographed image acquisition step S110 to the subject area extraction step S150 and the image processing step S160 in the first embodiment. is there. In the present embodiment, a photographing condition re-estimation step S260 and a subject region re-extraction step S270 are additionally added.

  In the photographing condition re-estimating step S260, the photographing condition re-estimating unit 508 re-estimates the photographing conditions from the subject region estimation result extracted in the subject region extracting step S250. The re-estimated shooting conditions are sent to the subject area re-extraction unit 509. Next, in the subject region re-extracting step S260, the subject region extraction unit 509 regenerates the subject model image from the photographing conditions re-estimated in the photographing condition re-estimating step S260, and uses the subject model image and the photographed image. Extract The extracted subject area is transmitted to the image processing unit 506.

  Next, specific processing contents of the photographing condition re-estimating step S260 and the subject region re-extracting step S270 will be described. In the photographing condition re-estimating step S260, photographing conditions are re-estimated from the result of extracting the subject region extracted in the subject region extracting step S250. Specifically, as shown in FIG. 12, the position / posture of the camera at the object model coordinates with the three-dimensional position of the object model as the origin is used. Position / posture information of the subject in the world coordinate system is obtained in advance to convert the position / posture of the camera in the estimated subject model coordinates into position information obtained from GPS information or the like. The position and orientation of the camera in the obtained world coordinate system are higher in precision than the position and orientation of the camera acquired in the photographing condition acquisition step S220. In addition, if texture information of each surface of the subject is also stored, it is more accurate than the environment information acquired in the photographing condition acquisition step S220 from the texture of each surface of the photographed image and the position and orientation of the camera in the previous world coordinate system. Environmental information and light source information can be acquired.

  In the subject region re-extracting step S270, the subject model image is generated again using the photographing conditions re-estimated in the photographing condition re-estimating step S260, and three-dimensional model fitting is performed again using the subject model image and the photographed image. Thus, it is possible to extract a subject area with higher accuracy than the subject area extracted in the subject area extraction step S250.

  As described above, according to the present embodiment, the image processing apparatus 20 acquires a three-dimensional model of an object from the imaging conditions at the time of imaging, and generates an object model image. A subject area in the photographed image is extracted by fitting the generated subject model image and the photographed image, and the photographing condition is estimated again based on the subject area. The subject region in the identification target image can be extracted with high accuracy by regenerating the subject model image from the shooting condition re-estimated and fitting it with the shooting image.

Third Embodiment
Next, a third embodiment of the present invention will be described. In the present embodiment, the image processing apparatus 20 detects a registered object registered in advance by the user, in addition to the subject area using the subject model. By detecting a registered object in addition to the subject region, it is possible to perform image processing and imaging assistance using both the subject region and the registered object. Hereinafter, a third embodiment of the present invention will be described with reference to the drawings. In addition, the description is abbreviate | omitted about the structure already demonstrated by 1st, 2nd embodiment, the same code | symbol is attached | subjected.

  FIG. 5 is a schematic block diagram showing a functional configuration of the image processing apparatus 20 in the present embodiment. FIG. 5A is a block diagram when performing image processing, and FIG. 5B is a block diagram when performing imaging assistance. In FIG. 5A, in addition to the configuration of the first embodiment, a registered object detection unit 510 is added, and in FIG. 5B, a registered object detection unit 510 is added, and instead of the image processing unit 506. A photographing assist unit 511 is added. In the first and second embodiments, the subject area is extracted using the subject information stored in advance in the subject information storage unit. On the other hand, in the present embodiment, the user can detect information on an object registered at the time of shooting by registering information on other objects in advance.

  FIGS. 7C and 7D are flowcharts showing processing executed by the image processing apparatus 20 in the present embodiment. FIG. 7C is a flowchart when performing image processing using both a subject area and a registered object, and FIG. 7D is a flowchart when performing imaging assist. 7C and 7D, the processing contents in the photographed image acquisition step S310 to the subject area extraction step S350 are the same as the photographed image acquisition step S110 to the subject area extraction step S150 of the first embodiment. In FIG. 7C, in addition to the processing of the first embodiment, a registered object detection step S360 is added, and the processing content of the image processing step S370 is different. On the other hand, in FIG. 7D, in addition to the processing of the first embodiment, a registered object detection step S360 is added, and a photographing assist step S380 is added instead of the image processing step S370.

  In the flowchart of FIG. 7C, in the registered object detection step S360, the registered object detection unit 510 detects the registered object in the photographed image using the information of the registered object held in the registered object information holding unit. The detected position and area information of the registered object detected are transmitted to the image processing unit 506. In the image processing step S370, high image quality processing is performed based on the subject region extracted in the photo object region extraction step S350 and the detection information of the registered object detected in the registered object detection step S360. Specific processing content will be described later.

  In the case of FIG. 7D, the processing content in the registered object detection unit 510 is the same as that of FIG. 7C, but a photographing assist step S380 is added instead of the image processing step S370. In the shooting assistance step S380, the shooting assistance unit 511 performs shooting assistance for the user based on the subject region extracted in the subject region extraction step S350 and the detection information of the registered object detected in the registered object detection step S360.

  Next, processing contents of the registered object detection step S360 and the image processing step S370 will be more specifically described according to the flowchart of FIG. 7 (c). In the registered object detection step S360, an object or a person registered in advance by the user is detected. For example, when the user registers a picture of a child, a child area is detected other than the subject extracted using the subject model from the photographed image taken.

Here, the registration method of the registered object will be described first. FIG. 15 shows a schematic block diagram of a functional configuration of the registered object image setting unit 21 in the present embodiment. A flowchart showing processing by the registered object image setting unit 21 is shown in FIG. In the registered object image acquisition step T110, the registered object image photographed by the photographing unit 500 is acquired. Here, although the registration object image is acquired by photographing with the photographing unit, an image photographed in advance by the photographing unit may be registered. Further, in order to improve the detection accuracy of the registered object, it is desirable to register a plurality of images. Next, in the registered object information setting step T120, the information of the registered object acquired in the registered object image acquiring step T110 is set. For example, when the user registers a picture of a child, a tag of “child” is attached using a user interface or the like. In addition, tags such as "dog" and "car" are attached. FIG. 17 is a diagram showing an example of setting of a registered object area in the present embodiment. As shown in FIGS. 17A and 17B, the user may set a registered object area 702 for the registered object 701. In addition, although the image of the registered object may be held in the registered object information holding unit 515, the feature amount may be acquired and held from the image of the registered object or the registered object region. As an example of the feature amount, statistics of color features and texture features in an image (region) may be used. For example,
Each component of RGB, HSV, Lab, YCbCr color space Gabor filter It is assumed that a filter response of LoG is used. The color feature is 12 dimensions of 4 (color space) × 3 (components). Also, with respect to the filter response, the number of dimensions corresponds to the number of Gabor filters and LoG filters. In order to characterize an image (region), a statistic is obtained from feature amounts obtained for each pixel in the image (region). The statistic to be used is assumed to use four of mean, standard deviation, skewness and kurtosis. Skewness indicates the degree of asymmetry of the distribution, and kurtosis is a statistic that indicates how densely the distribution is close to the mean. Therefore, the color feature has 48 dimensions of 4 (color space) × 3 (components) × 4 (statistics), and the number of dimensions of the texture features is (number of filter responses) × 4 (statistics). The feature amount and the registered object information may be held in the registered object information holding unit.

  Next, a method of detecting a registered object from a photographed image according to the set information of the registered object will be described. The feature amount set in the registered object information setting step T120 is acquired from the photographed image. When the area of the registered object is set, an area of the same size may be cut out from the image and the feature amount may be acquired from the area. The acquired feature amount is compared with the registered feature amount, and if the similarity is equal to or more than a threshold value, it is determined that the object is a registered object. If it is determined that the object is a registered object, the information on the position and area of the registered object is used in the next image processing step S370. In the present embodiment, the feature amounts are compared to determine whether the object is a registered object, but when a face or the like is registered, a face authentication technique as shown in Non-Patent Document 4 may be used.

  In the image processing step S370, the image quality enhancement processing is performed on the photographed image using the subject region extracted and detected in the subject region extraction step S350 and the registered object detection step S360. Specifically, the white balance is adjusted in accordance with the subject area and the registered object area, or the development parameter is adjusted so as to maximize the contrast of the area. As in the first embodiment, unnecessary areas are removed or areas are divided for areas other than the subject area and the registered object area, and image quality improvement processing according to the category is performed.

  Next, according to the flowchart of FIG. 7D, details of processing in the case of performing imaging assistance for presenting imaging parameters and composition to the user instead of performing the image quality enhancement processing on the captured image will be described.

  In the shooting assist step S380, shooting assist is performed on the user using the subject region extracted and detected in the subject region extraction step S350 and the registered object detection step S360, and the registered object detection information. Specifically, the camera position, the angle, or the position of the registered object is presented to the user so that the user can shoot the subject and the registered object with a suitable composition according to the subject region and the registered object region and the position thereof. When three-dimensional model fitting is used in the process of the subject region extraction step S350, the three-dimensional position and orientation of the subject and the camera are obtained in detail, so that the camera position can be accurately assisted. Also, from the three-dimensional position of the subject and the approximate three-dimensional position of the registered object, it is possible to present the user with imaging parameters that bring the two regions into the depth of field. Besides, as in the first embodiment, from the color information of the subject and the registered object, parameters suitable for photographing it in the current environment may be presented.

  As described above, according to the present embodiment, the image processing apparatus 20 acquires a three-dimensional model of an object from the imaging conditions at the time of imaging, and generates an object model image. The subject region in the photographed image is extracted by fitting the generated subject model image and the photographed image. Furthermore, it is possible to detect an object registered in advance by the user, and to perform the high image quality processing based on the object detection result and the subject region. Also, imaging assistance can be performed for the user based on the object detection result and the subject area.

Fourth Embodiment
Next, a fourth embodiment of the present invention will be described. In the present embodiment, after extracting the subject area, the extracted subject area is compared with the held subject model to detect a temporal change of the subject. If the difference between the subject and the model is large, the corresponding part of the subject model is updated, or the corresponding part of the subject in the photographed image is changed to the corresponding part of the subject model. As a result, it is possible to update the model or to generate a captured image using the subject model when the subject is deteriorated. Hereinafter, a fourth embodiment of the present invention will be described with reference to the drawings. In addition, the description is abbreviate | omitted about the structure already demonstrated by the 1st-3rd embodiment, and the same code | symbol is attached | subjected.

  FIG. 6 is a schematic block diagram showing a functional configuration of the image processing apparatus 20 in each embodiment, and FIG. 6 (a) is a schematic block diagram of the present embodiment. In addition to the configuration of the first embodiment, a subject information evaluation unit 512 is added to the image processing apparatus 20 of the present embodiment. Specific processing contents of the subject information evaluation unit will be described later.

  A flowchart showing image processing by the image processing apparatus 20 of the present embodiment is shown in FIG. The processing contents in the photographed image acquisition step S410 to the subject area extraction step S450 of the present embodiment are the same as the photographed image acquisition step S110 to the subject area extraction step S150 in the first embodiment. In the present embodiment, a subject information evaluation step S460 is additionally provided. In addition, the processing content of the image processing step S470 is changed.

  In the subject information evaluation step S460, the subject information evaluation unit 512 compares the information of the subject area extracted in the subject area extraction step S450 with the information of the subject held in the subject information holding unit 507, and determines the current state of the subject. evaluate. The evaluation result is sent to the image processing unit 506. In the image processing step S470, the image quality enhancement process is performed on the captured image using the subject region extracted in the subject region extracting step S450 and the current subject information evaluated in the subject information evaluation step S460.

  Next, processing contents of the subject information evaluation step S460 and the image processing step S470 will be specifically described according to the flowchart of FIG.

  In the subject information evaluation step S460, the information on the subject area extracted in the subject area extraction step S450 is compared with the information on the subject held in the subject information holding unit 507 to evaluate the current state of the subject. Specifically, the texture of each object plane on the photographed image is compared with the texture of each object plane held in the object information holding unit based on the three-dimensional position / posture information of the object. Then, if there is a greatly different face or part of the face, it is replaced with color information / texture information of the corresponding part of the subject model held in the subject model holding unit in the next image processing step S460. Alternatively, the subject model information is updated according to the current state of the subject. About the processing content of other image processing processes, since it is the same as that of the first to third embodiments, the description will be omitted.

  As described above, according to the present embodiment, the image processing apparatus 20 acquires a three-dimensional model of an object from the imaging conditions at the time of imaging, and generates an object model image. The subject region in the photographed image is extracted by fitting the generated subject model image and the photographed image. Based on the subject region extraction result, the surface reflection characteristic information of the subject model is compared with the subject region in the photographed image to detect a difference from the model. This makes it possible to update the model or to generate a photographed image using the subject model when the subject is deteriorated.

Fifth Embodiment
Next, a fifth embodiment of the present invention will be described. In the present embodiment, a configuration in which a subject model can be added will be described. Hereinafter, a fifth embodiment of the present invention will be described with reference to the drawings. In addition, the description is abbreviate | omitted about the structure already demonstrated by the 1st-4th embodiment, and the same code | symbol is attached | subjected.

  A schematic block diagram showing a functional configuration of the image processing apparatus 20 in the present embodiment is shown in FIG. In the image processing apparatus 20 of the present embodiment, a subject information adding unit 513 is added to the configuration of the first embodiment. Specific processing contents of the subject information adding unit will be described later.

  FIG. 7F shows a flowchart showing the processing executed at the time of the image processing of this embodiment. The processing contents in the photographed image acquisition process S510 to the subject area extraction process S550 of the present embodiment are the same as the photographed image acquisition process S110 to the subject area extraction process S150 of the first embodiment. Further, in the present embodiment, a subject information addition step S560 is additionally provided. In the subject information adding step S560, the subject information adding unit 513 adds subject information to the subject information holding unit based on the information of the subject region extracted in the subject region extracting step S550. The added subject information is used in the process of enhancing the image quality of other captured images.

  Next, the processing content of the subject information adding step S560 will be described more specifically according to the flowchart of FIG. 7 (f). In the subject information addition step S560, subject information is added based on the information extracted in the subject region extraction step S550. The information to be added is the three-dimensional position / posture of the camera, the environment and light source information of shooting the subject, shooting parameters, texture information of each surface of the subject, and the like. By holding them in the subject information holding unit, the information can be used when photographing is performed next time or later.

  As described above, according to the present embodiment, the image processing apparatus 20 acquires a three-dimensional model of an object from the imaging conditions at the time of imaging, and generates an object model image. The subject region in the photographed image is extracted by fitting the generated subject model image and the photographed image. By additionally registering the extracted information of the subject area in the subject information holding unit of the image processing apparatus, more subject information can be used for the image quality improvement process. Furthermore, by applying such a method, it is possible to effectively increase the data on the position and attitude of the camera where the user often takes the subject and shooting parameters, and it is possible to improve the accuracy of the image quality improvement process. .

Other Embodiments
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or storage medium, and one or more processors in a computer of the system or apparatus read and execute the program. Processing is also feasible. It can also be implemented by a circuit (eg, an ASIC) that implements one or more functions.

501 Photographed image acquisition unit 502 Photographing condition acquisition unit 503 Subject information acquisition unit 504 Subject model image generation unit 505 Subject region extraction unit 506 Image processing unit 507 Subject information holding unit

Claims (14)

A first acquisition unit that acquires a photographed image of a subject photographed by the photographing unit;
A second acquisition unit configured to acquire position information and orientation information of the photographing unit when the acquired photographed image is photographed as a photographing condition;
Generation means for generating a model image of the subject based on the acquired imaging conditions;
Extracting means for extracting the area of the subject from the photographed image based on the generated model image;
An image processing apparatus comprising:   The image processing apparatus according to claim 1, wherein the extraction unit extracts the area of the subject from the photographed image by fitting the area of the subject included in the model image and the photographed image. .   The second acquisition unit may further acquire at least one of environmental information, light source information, and imaging parameters of the imaging unit when the acquired captured image is captured as the imaging condition. The image processing apparatus according to claim 1, wherein the image processing apparatus is characterized by: The information processing apparatus further includes third acquisition means for acquiring information on the subject based on the acquired imaging conditions.
The image processing apparatus according to any one of claims 1 to 3, wherein the generation unit generates the model image based on the acquired imaging condition and information on the subject. The third acquisition unit further acquires, as information on the subject, information on surface reflection characteristics of the subject,
The image processing apparatus according to claim 4, wherein the generation unit generates the model image further based on information on the acquired surface reflection characteristic.   The image processing apparatus according to claim 4, further comprising an addition unit that adds information on the subject.   The image processing apparatus according to any one of claims 1 to 6, further comprising image processing means for performing image processing on the area of the subject extracted from the photographed image. The apparatus further comprises detection means for detecting an object registered by the user from the photographed image,
8. The image processing apparatus according to claim 7, wherein the image processing unit further performs image processing on the area of the detected object.   9. The display device according to claim 1, further comprising: a presentation unit configured to present at least one of a photographing parameter of the photographing unit and a composition at the time of photographing based on the area of the subject extracted from the photographed image. The image processing apparatus according to any one of the above.   The image processing apparatus according to any one of claims 1 to 9, wherein the generation unit generates model images of a plurality of the subjects. The image processing apparatus further comprises estimation means for estimating at least one of the imaging conditions based on the result of extraction of the region of the subject by the extraction means.
11. The apparatus according to any one of claims 1 to 10, wherein the extraction unit extracts the region of the subject from the photographed image based on the model image generated by the generation unit based on the estimated photographing condition. An image processing apparatus according to any one of the preceding claims.   The image processing according to any one of claims 1 to 11, further comprising: evaluation means for evaluating temporal change of the subject based on a result of the extraction means extracting a region of the subject. apparatus. Acquiring a photographed image in which a subject is photographed by the photographing means;
Acquiring position information and orientation information of the photographing unit when the acquired photographed image is photographed as a photographing condition;
Generating a model image of the subject based on the acquired shooting conditions;
Extracting an area of the subject from the captured image based on the generated model image;
An image processing method comprising:   The program for functioning a computer as each means of the image processing apparatus of any one of Claims 1-12.