JP2021026320A - Image processing device, control method thereof and program - Google Patents

Abstract

To solve such a problem that a reconstructed three-dimensional shape of a subject becomes unnatural when overlapping three-dimensional models with each other.SOLUTION: An image processing device comprises: detection means which detects a prescribed subject region from an input image; three-dimensional model acquisition means which acquires plural types of three-dimensional models being information obtained by modeling the three-dimensional shape of at least a portion of the prescribed subject region; three-dimensional model arrangement means which arranges the plural types of three-dimensional models acquired by the three-dimensional model acquisition means on the basis of a result of detection by the detection means; and blending processing means which performs blending processing on a boundary of the three-dimensional models when the plural types of three-dimensional models arranged by the three-dimensional model arrangement means overlap with each other. The plural types of three-dimensional models are associated with importance for each type. The blending processing means increases the processing amount of the blending processing as the importance associated with the three-dimensional model is low.SELECTED DRAWING: Figure 3

Description

The present invention relates to image processing using a three-dimensional model.

Conventionally, there is known a technique of irradiating a subject in an image after shooting with light from a virtual light source to rewrite the subject. As a result, the user can add an arbitrary shadow to the subject. For example, in Patent Document 1, the three-dimensional shape of the subject is expressed by extracting the feature points of the subject and applying a three-dimensional model that reproduces the three-dimensional shape of each feature point. As a result, the influence of virtual light source irradiation based on the three-dimensional shape can be reflected.

Japanese Unexamined Patent Publication No. 2006-163871

However, when using a plurality of three-dimensional models such as a model for each organ of the subject, it is necessary to consider not only the accuracy of each model but also the relationship between the models. For example, when models overlap, the accuracy of the boundary may become a problem.

One of the image processing devices according to the present invention is a detection means for detecting a predetermined subject area from an input image, and a plurality of types of three-dimensional models which are information modeling at least a part of the three-dimensional shape of the predetermined subject area. A three-dimensional model acquisition means for acquiring a three-dimensional model, a three-dimensional model arrangement means for arranging a plurality of types of three-dimensional models acquired by the three-dimensional model acquisition means based on the result of detection by the detection means, and the three-dimensional model arrangement means. When a plurality of types of three-dimensional models are overlapped with each other, the three-dimensional models have a familiarization processing means for adapting to the boundary between the three-dimensional models, and the plurality of types of three-dimensional models are associated with importance for each type. The familiarization processing means is characterized in that the lower the importance associated with the three-dimensional model, the larger the processing amount of the familiarization processing.

According to the present invention, even when a plurality of three-dimensional models are used, the three-dimensional shape can be appropriately expressed.

It is a block diagram which shows the structure of the digital camera which concerns on embodiment of this invention. It is a block diagram which shows the structure of the image processing part of the digital camera which concerns on embodiment of this invention. It is a block diagram which shows the structure of the rewriting processing part 203 which concerns on embodiment of this invention. It is a block diagram which shows the structure of the subject three-dimensional shape reproduction part 302 which concerns on 1st Embodiment of this invention. It is a figure which shows the characteristic of the importance which concerns on embodiment of this invention. It is a figure which shows the method of superimposing the face model which concerns on embodiment of this invention. It is a figure which shows the calculation of the reflection component by rewriting which concerns on embodiment of this invention. It is a block diagram which shows the structure of the subject three-dimensional shape reproduction part 302 which concerns on 2nd Embodiment of this invention. It is a figure which shows the characteristic of the flatness evaluation value which concerns on 2nd Embodiment of this invention.

<First Embodiment>
Hereinafter, the first embodiment of the present invention will be described with reference to the drawings.

In this embodiment, a digital camera will be described as an example of the image processing device. However, the present invention is not limited to digital cameras, and can be applied to mobile terminals such as personal computers and server computers, so-called smartphones and tablet terminals. It can also be applied to surveillance cameras, in-vehicle cameras, industrial / industrial / medical devices, or computer devices for directly connecting or controlling them via a network.

FIG. 1 is a block diagram showing a configuration of a digital camera according to the present embodiment.

The digital camera in this embodiment includes an optical system 101, an image sensor 102, an A / D conversion unit 103, an image processing unit 104, a recording unit 105, a control unit 106, a memory 107, and a display unit 109. The optical system 101 includes a focus lens, an aperture, and a shutter. This optical system drives the focus lens during shooting to focus the subject and adjusts the exposure amount by controlling the aperture and shutter. The image sensor 102 is a photoelectric conversion element such as a CCD or CMOS that converts the amount of light of a subject imaged in the optical system 101 into an electric signal by photoelectric conversion. The A / D conversion unit 103 digitizes the input electric signal. The digitized image signal is processed by the image processing unit 104 for simultaneous processing, white balance correction processing, focusing degree calculation processing, rewriting processing, gamma processing, and the like, and is output to the recording unit 107. The recording unit 105 converts the image information output from the image processing unit 104 into an image format such as JPEG and records it. The control unit 106 controls the operation of the entire imaging device and image processing device of the present embodiment. For example, the target exposure amount is calculated by the optical system 101 from the brightness of the subject immediately before shooting. In addition, a predetermined evaluation value is calculated based on the captured image, and the parameters of the image processing performed by the image processing unit 104 are determined. The memory 107 stores information used by the image processing unit 104, settings at the time of shooting, lighting conditions, and the like, and outputs the information to the image processing unit 104 as needed. The operation unit 108 is a portion where the user gives an operation instruction to the image pickup apparatus. The display unit 109 is, for example, a liquid crystal display installed on the back surface of the camera, and displays a screen for assisting an operation at the time of shooting, an image stored in the recording unit 107, or the like.

In the present embodiment, the image processing unit 104 performs image processing described later under the control of the control unit 106, but the hardware configuration is not limited to this embodiment. For example, one piece of hardware may function as various means according to an input signal or a program, or a plurality of hardware may cooperate to function as one means. Further, a part of the processing may be realized by using a circuit instead of the processing according to the so-called software program.

Further, the display unit 109 and the recording unit 105 do not necessarily have to be possessed by the digital camera itself, and may be configured to connect an external display or an external recording medium. It suffices that the digital camera has at least means for performing display control and recording control.

Next, a detailed description of the image processing unit 104 will be given with reference to FIG.

FIG. 2 is a block diagram showing the configuration of the image processing unit 104 according to the present embodiment. The image processing unit 104 includes a simultaneous processing unit 201, a white balance correction processing unit 202, a rewriting processing unit 203, and a gamma processing unit 204.

The processing flow of the image processing unit 104 will be described. First, the video signal of the Bayer array converted by the A / D conversion unit 103 is input to the image processing unit 104. The input video signal to the image processing unit 104 is input to the simultaneous processing unit 201, and the input RGB signals of the Bayer array are subjected to simultaneous processing to generate color signals R, G, and B. The generated color signals of R, G, and B are input to the white balance correction processing unit 202, and the RGB color signal is gained based on the white balance gain value calculated by the control unit 107 to adjust the white balance. The color signal RGB whose white balance has been adjusted is output to the rewriting processing unit 203. The rewriting processing unit 203 receives the color signal RGB from the white balance correction processing unit 202. An image in which virtual light is applied to the color signals R, G, and B is generated. The details of the rewriting process will be described later. The color signals R_out, G_out, and B_out after the rewriting process are output to the gamma processing unit 205. The gamma processing unit 204 performs gamma processing in response to the input of the rewritten signals R_out, G_out, and B_out, and outputs the color signals Rg, Gg, and Bg after gamma to the recording unit 107.

Subsequently, a detailed description of the processing of the rewriting processing unit 203 will be given with reference to FIG.

FIG. 3 is a block diagram showing the configuration of the rewriting processing unit 203. Rewriting is a type of virtual lighting process that imparts the effect of irradiating an image with virtual light, which is light emitted from a virtual light source. The rewriting processing unit 203 includes a subject detection image generation unit 301, a subject three-dimensional shape reproduction unit 302, a virtual light source diffuse reflection component calculation unit 303, and a virtual light source addition processing unit 304.

First, the rewriting processing unit 203 receives input of color signals R, G, and B from the white balance correction processing unit 202 as an input image.

The subject detection image generation unit 301 uses the color signals R, G, and B input from the white balance correction processing unit 202 as an image signal format for maintaining the detection performance of the subject detection unit that constitutes the subject stereoscopic shape reproduction unit 302. Convert to. The subject stereoscopic shape reproduction unit 302 maintains the detection performance when the image signal is gamma-processed and the predetermined image size is satisfied. Therefore, the subject detection image generation unit 301 acquires the color signals R, G, and B input from the white balance correction processing unit 202, and performs gamma processing and resizing processing on the color signals R, G, and B. Generates a subject detection signal S with. The generated subject detection signal S is output to the subject three-dimensional shape reproduction unit 302.

Here, the subject detection image generation unit 301 generates the subject detection signal S by performing gamma processing and resizing processing, but the purpose is to maintain the detection performance of the subject detection unit constituting the subject three-dimensional shape reproduction unit 302. Other processing may be used.

The subject three-dimensional shape reproduction unit 302 calculates the subject three-dimensional shape information P. A method of obtaining the subject three-dimensional shape information P will be described with reference to FIG.

FIG. 4 is a block diagram showing the configuration of the subject three-dimensional shape reproducing unit 302. The subject stereoscopic shape reproduction unit 302 includes a first subject detection unit 401, a second subject detection unit 402, a stereoscopic model acquisition unit 403, a familiarization setting determination unit 404, a stereoscopic model acquisition unit 405, a familiarization setting determination unit 406, and a familiarization processing. It is composed of a unit 407, an alignment processing unit 408, and a familiarization processing unit 409. In this embodiment, a person's face will be described as an example of a subject whose three-dimensional shape is desired to be reproduced.

First, the subject detection image generated by the subject detection image generation unit 301 is input to the first subject detection unit 401 and the second subject detection unit 402. The subject detected by the first subject detection unit is a part of the subject that constitutes the subject detected by the second subject detection unit. In the present embodiment, the second subject detection unit 402 detects the face of a person, and the first subject detection unit 401 detects the nose of a person, which is an organ forming a part of the face of the person. The first subject detection unit 401 and the second subject detection unit 402 acquire information on the coordinates, size, and posture of the detected subject, respectively, and the three-dimensional model acquisition unit 403, the familiarization setting determination unit 404, and the three-dimensional model acquisition unit 405. , Output to the familiar setting determination unit 406.

The three-dimensional model acquisition units 403 and 405 are the recording unit 105 or the memory 107 from the information based on the detection results of the area, size, and posture of the detected subject input from the first subject detection unit 401 and the second subject detection unit 402, respectively. A similar model is selected and acquired from the three-dimensional models stored in advance in. The three-dimensional model to be acquired may be a polygon model that is a set of three-dimensional vertex data on a certain coordinate space, or image data rendered with a normal vector as a pixel value, as long as it is a model of a three-dimensional shape. Good. The number of polygons of the three-dimensional model and the resolution of the rendered image are independent for each model type. That is, in the present embodiment, the resolution of the three-dimensional model of the face and the resolution of the three-dimensional model of the nose are set separately. This is because the effect of rewriting changes greatly depending on the accuracy of the three-dimensional model for an undulating shape such as the nose, so it is desirable to use a three-dimensional model having a higher resolution than the entire face. The three-dimensional model acquisition unit 403 acquires a three-dimensional model of the shape of the nose of a person, and outputs the acquired three-dimensional model to the alignment processing unit 408. The three-dimensional model acquisition unit 405 acquires a model of a person's face, adapts the acquired three-dimensional model, and outputs the acquired three-dimensional model to the processing unit 407.

The familiarization setting determination units 404 and 406 determine parameters related to the familiarization process (smoothing process) for the image. The acclimation process will be described later, but for example, a process of applying a smoothing filter or the like to an image is performed. By performing the fitting process on the three-dimensional shape of the subject, it is possible to prevent the failure to detect the three-dimensional shape and the occurrence of an unnatural image processing effect based on the wrong three-dimensional shape while maintaining the accuracy of the rough three-dimensional shape. Can be done.

The familiarization setting determination units 404 and 406 are based on the importance related to the first subject detection unit 401 and the second subject detection unit 402 and the area, size, and posture of the detected subject, respectively, and the familiarization amount SMOOTH_VALUE and the familiarization area. Determine SMOOTH_AREA. That is, the acclimation amount is an index showing the strength of the acclimation process, and the larger the acclimation amount, the larger the smoothing processing amount, that is, the degree of acclimation with the peripheral pixels. In addition, the blending region indicates which region of the image is to be blended.

First, a method for determining the familiar amount SMOOTH_VALUE will be described. The amount of acclimation SMOOTH_VALUE is determined by the method shown in Equation 1.
SMOOTH_VALUE
= STD_VALUE x SIZE_VALUE ÷ P_VALUE
... (Equation 1)

Note that STD_VALUE is a preset standard familiarization amount, and P_VALUE is the importance of the subject. Equation 1 has a characteristic that the higher the importance of the subject, the smaller the amount of fitting, and the larger the size of the subject, the larger the amount of fitting.

By defining Equation 1 as described above, the following two effects can be expected. One is the effect of maintaining finer stereoscopic information by suppressing the amount of familiarization for a subject having a high importance P_VALUE with respect to the standard amount of familiarization. The other is that when the size of the subject, SIZE_VALUE, is large, the number of polygons and the rendering resolution of the original 3D model are insufficient for the subject by increasing the amount of familiarization with respect to the standard amount of familiarization. It is an effect to prevent.

Next, a method of determining the familiarization area will be described. When a 3D model is applied to a detected subject (3D model placement processing), some fitting error may occur depending on the detection accuracy of the subject. Since the influence of the error is easily noticeable in the vicinity of the contour line of the fitted model, the range of a predetermined distance from the contour line of the subject area in the captured image is determined as the familiar area. Here, when fitting processing is performed on a model in which a plurality of models are superimposed by the alignment superimposition processing unit 408 in the subsequent stage of the familiarization setting determination unit as in the familiarization setting determination unit 404, depending on the detection accuracy of the subject. There is a possibility that the model will be superimposed on a position different from the apparent overlapping position. In consideration of such a case, the area around the boundary portion where a plurality of types of models overlap each other may be used as a familiar area. Further, the familiar area is set as the peripheral area of the contour line, but when it is difficult to extract the contour, the entire model may be used as the familiar area.

Here, the importance will be described. The importance has a value of, for example, 1 to 256, and can be arbitrarily set by the user for each subject (face and nose in the example of FIG. 4). Alternatively, the control unit 106 may assign a predetermined importance according to the priority (high / medium / low, etc.) for each subject set by the user operation.

As another example, it may be determined based on the number of polygons in the 3D model, the resolution of the rendered normal image, or the orientation of the virtual light source illuminating the model. Furthermore, the final importance may be calculated by weighting and adding these items. Hereinafter, these examples will be described.

FIG. 5A is a characteristic used when determining the importance based on the number of polygons in the three-dimensional model. The horizontal axis is the number of polygons in the 3D model, and the vertical axis is the importance. The threshold Th_p is a predetermined upper limit of the number of polygons, and a three-dimensional model having a number of polygons exceeding Th_p always takes a maximum value of 256.

FIG. 5B is a characteristic used when determining the importance based on the normal resolution of the three-dimensional model. The horizontal axis is the normal resolution of the 3D model, and the vertical axis is the importance. The threshold value Th_r is an upper limit value of a predetermined normal image resolution, and a normal image having a resolution exceeding Th_r always takes a maximum value of 256.

FIG. 5C is a characteristic used when determining the importance based on the irradiation direction of the virtual light and the angle formed by the representative normal of the three-dimensional model. In the present embodiment, the mode normal in the three-dimensional model is the representative normal of the three-dimensional model. The horizontal axis is the angle formed by the irradiation direction of virtual light and the representative normal, and the vertical axis is the importance. Th_a is a predetermined upper limit of the formed angle, and when the angle is smaller than Th_a, the maximum value is always 256. Here, the angle to be formed shall take a value of −90 degrees to 90 degrees.

The determined familiarization settings are output to the familiarization processing units 409 and 407, respectively.

The acclimation processing units 407 and 409 perform acclimation processing of the depth component of the three-dimensional model based on the acclimation amount SMOOTH_VALUE calculated by the acclimation amount calculation units 406 and 404. The familiarization processing unit 407 fits the face model to the detected face position of the subject, and performs the familiarization processing around the contour line of the face. Then, the acclimation processing unit 409 superimposes the nose model on the face model that has been acclimated by the acclimation processing unit 407, and performs the acclimation processing on the superimposed region of the face and the nose model and around the contour line thereof.

Here, the acclimation process is, for example, a process that gives the effect of blurring or smoothing. In the present embodiment, the acclimation process is a low-pass filter process such as a Gaussian filter. Do more. The familiarization processing unit 407 performs the familiarization processing on the three-dimensional model acquired by the three-dimensional model acquisition unit 405, and outputs the three-dimensional model after being familiarized with the alignment superimposition processing unit 408.

Here, the acclimation processing units 407 and 709 control the number of taps of the low-pass filter based on the acclimation amount SMOOTH_VALUE, but the number of trials of the acclimation process may be controlled based on the acclimation amount SMOOTH_VALUE.

The alignment superimposition processing unit 408 aligns the three-dimensional model input from the three-dimensional model acquisition unit 403 and the familiarized three-dimensional model input from the familiarization processing unit 407 based on the coordinate information associated with each. I do. This will be described with reference to FIG. It is a schematic diagram about superimposition of models, (a) is a front view of a nose model which is a three-dimensional model of the first subject, and (b) is a front view of a face model which is a second three-dimensional model. (C) is a front view of a model in which each is superimposed. Further, (a') (b') (c') are views of each from the upper surface. By matching the coordinates of each model, they are superimposed, but as shown in (d), if there is a model that cannot be seen from the camera 100, that part is erased to make one model. Then, the superimposed model is adapted and output to the processing unit 409.

Here, an example of superimposing the three-dimensional models on top of each other has been described. However, the images obtained by rendering the normal information of the three-dimensional models of the first and second subjects are aligned and superposed to generate an image showing the normal information. You may. When generating an image, the normal information that cannot be seen from the camera 100 is not taken into consideration, and only the normal that can be seen from the camera 100 is generated as an image. The image showing the normal information generated in this way may be familiarized and output to the processing unit 409.

The acclimation processing unit 409 performs acclimation processing on the three-dimensional model input from the alignment superimposition processing unit 408 based on the acclimation processing set value determined by the acclimation setting determination unit 404. The acclimation processing method is the same as that of the acclimation processing unit 407. The three-dimensional model SUBJECT_MODEL after acclimatization is output to the virtual light source diffuse reflection component calculation unit 303. By doing this. The three-dimensional shape information of the subject for calculating the reflection component of the virtual light is output.

In the subject three-dimensional shape reproduction unit 302, two subject detection units are prepared in the present embodiment, but the number of detection units may be increased or decreased according to the resolution of importance and the number of subjects to be detected.

Next, the process of calculating the diffuse reflection components Rd, Gd, and Bd in the virtual light source diffuse reflection component calculation unit 303 will be described. How to obtain the diffuse reflection component will be described with reference to FIG. 7.

FIG. 7 is a diagram showing the positional relationship between the camera 100, the SUBJECT_MODEL 701, and the virtual light source 702 at the time of shooting, and the reflection characteristics of the virtual light source.

First, the virtual light source diffuse reflection component calculation unit 303 acquires the normal information N from the three-dimensional model SUBJECT_MODEL input from the subject three-dimensional shape reproduction unit 302.

The diffuse reflection component at the horizontal pixel position H1 (the vertical pixel position is omitted for simplification of explanation) of the captured image taken by the camera 100 is proportional to the inner product of the normal N1 at the camera coordinates H1 and the direction vector L1 of the virtual light source. However, the value is inversely proportional to the square of the distance K1 between the virtual light source and the subject position. The diffuse reflection component intensity Pd by the virtual light source can be represented by (Equation 2), and the diffuse reflection component Rd, Gd, and Bd for each color can be represented by (Equation 3).

Here, N in (Equation 2) is the three-dimensional normal vector of the subject, and K is the distance between the virtual light source and the subject. kd is the diffuse reflectance of the subject. α and L are the strength of the virtual light source and the three-dimensional direction vector of the virtual light source, respectively, which are arbitrarily determined.

Further, R, G, and B in (Equation 3) are color signals R, G, and B input from the white balance correction unit 202. Rw, Gw, and Bw are parameters indicating the colors of an arbitrarily determined virtual light source.

The virtual light source addition processing unit 304 performs a process of adding the diffuse reflection components Rd, Gd, and Bd calculated by the virtual light source diffuse reflection component calculation unit 303 to the input color signals R, G, and B. The color signals R_out, G_out, and B_out after irradiation with the virtual light source can be expressed as (Equation 4).
R_out = R + Rd
G_out = G + Gd
B_out = B + Bd
... (Equation 4)

The color signals R_out, G_out, and B_out calculated in this way after irradiation with the virtual light source are output to the gamma processing unit 205. In this way, the three-dimensional shape of the subject can be acquired and an arbitrary virtual light source can be irradiated.

<Second embodiment>
Next, a second embodiment according to the present invention will be described.

In the present embodiment, the processing in the subject three-dimensional shape reproducing unit 302 is different from that in the first embodiment. In the present embodiment, the configuration of the subject three-dimensional shape reproducing unit 302 of the first embodiment described in FIG. 4 is changed as shown in FIG. FIG. 8 is a diagram showing a configuration of a subject three-dimensional shape reproducing unit 302 according to the present embodiment. In addition to inputting subject information from the first subject detection unit 401 and the second subject detection unit 402, the familiarization setting determination units 804 and 806 of FIG. 8 of the subject detection image S from the subject detection image generation unit 301. It was configured to receive input.

The familiar setting determination units 804 and 806 will be described. First, the amount of acclimation SMOOTH_VALUE and the acclimation region SMOOTH_AREA are determined as in the first embodiment. Next, the flatness evaluation value FLAT_VALUE of SMOOTH_AREA is calculated by referring to the region corresponding to the familiar region SMOOTH_AREA of the subject detection image S. The flatness evaluation value FLAT_VALUE is calculated by a method using a high-pass filter. For example, first, the absolute value of the value obtained by applying a high-pass filter such as [-1, 0, 1] horizontally and vertically to the luminance signal of the subject detection image S is calculated. Next, the flatness evaluation value FLAT_VALUE is obtained so that the higher the absolute value calculated as shown in FIG. 9, the closer to 0. The threshold value TH is a predetermined flatness upper limit value, and when the flatness upper limit is exceeded, FLAT_VALUE is set to 0. Using this flatness evaluation value FLAT_VALUE, the familiar amount SMOOTH_VALUE'after applying the flatness evaluation value is calculated using (Equation 5).
SMOOTH_VALUE'= SMOOTH_VALUE / FLAT_VALUE
... (Equation 5)

The SMOOTH_VALUE'calculated in this way is output to the fitting processing units 407 and 409, and the fitting processing of the three-dimensional model is performed. For example, a region having a high flatness evaluation value, that is, a region having a low contrast is presumed to be a region of the same model, so the amount of familiarization is reduced. By controlling the amount of acclimation corresponding to the flatness evaluation value in this way, it is possible to reproduce a suitable three-dimensional model.

<Other embodiments>
The present invention is also realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiment is supplied to the system or device via a network or various storage media, and the computer (or CPU, MPU, etc.) of the system or device reads the program. This is the process to be executed.

Claims (11)

A detection means that detects a predetermined subject area from an input image,
A three-dimensional model acquisition means for acquiring a plurality of types of three-dimensional models, which is information modeling at least a part of the three-dimensional shape of a predetermined subject area.
A three-dimensional model arrangement means for arranging a plurality of types of three-dimensional models acquired by the three-dimensional model acquisition means based on the result of detection by the detection means.
When a plurality of types of three-dimensional models arranged by the three-dimensional model arranging means overlap each other, the three-dimensional model has a acclimation processing means for acclimating the boundary portion between the three-dimensional models.
The importance of the plurality of types of three-dimensional models is associated with each type, and the familiarization processing means increases the processing amount of the familiarization processing as the importance associated with the three-dimensional model decreases. An image processing device characterized by this. It also has a virtual lighting means that gives the effect of irradiating the image with virtual light.
The image processing apparatus according to claim 1, wherein the virtual lighting means determines the effect of virtual light applied to the input image based on a three-dimensional model that has been subjected to the familiarization processing by the familiarization processing means. The image processing apparatus according to claim 1 or 2, wherein the three-dimensional model is information in which at least a part of the three-dimensional shape of the predetermined subject area is modeled by using normal information. The image processing apparatus according to any one of claims 1 to 3, wherein the importance associated with the three-dimensional model is higher as the number of polygons in the three-dimensional model is larger. The image processing apparatus according to any one of claims 1 to 4, wherein the importance associated with the three-dimensional model is higher as the image resolution at the time of rendering the three-dimensional model is larger. The image processing apparatus according to any one of claims 1 to 3, wherein the importance associated with the three-dimensional model can be arbitrarily set by the user. The image according to any one of claims 3 to 5, wherein the importance associated with the three-dimensional model is based on the angle formed by the representative normal of the three-dimensional model and the irradiation direction of virtual light. Processing equipment. A detection means that detects a predetermined subject area from an input image,
A three-dimensional model acquisition means for acquiring a plurality of types of three-dimensional models, which is information modeling at least a part of the three-dimensional shape of a predetermined subject area.
A three-dimensional model arrangement means for arranging a plurality of types of three-dimensional models acquired by the three-dimensional model acquisition means based on the result of detection by the detection means.
When a plurality of types of three-dimensional models arranged by the three-dimensional model arranging means overlap each other, the three-dimensional model has a acclimation processing means for acclimating the boundary portion between the three-dimensional models.
The blending processing means is characterized in that the lower the contrast of the region corresponding to the boundary between the models in the region of the input image, the larger the processing amount of the blending treatment with respect to the region. Processing equipment. It is a control method of an image processing device.
A detection process that detects a predetermined subject area from the input image,
A three-dimensional model acquisition step of acquiring a plurality of types of three-dimensional models, which is information modeling at least a part of the three-dimensional shape of a predetermined subject area,
A three-dimensional model placement step of arranging a plurality of types of three-dimensional models acquired in the three-dimensional model acquisition step based on the detection result in the detection step,
When a plurality of types of three-dimensional models arranged in the three-dimensional model arrangement step overlap each other, it has a familiarization processing step of fitting the three-dimensional models to the boundary portion between the three-dimensional models.
The importance of the plurality of types of three-dimensional models is associated with each type, and in the familiarization processing step, the lower the importance associated with the three-dimensional model, the larger the processing amount of the familiarization processing. A control method characterized by that. It is a control method of an image processing device.
A detection process that detects a predetermined subject area from the input image,
A three-dimensional model acquisition step of acquiring a plurality of types of three-dimensional models, which is information modeling at least a part of the three-dimensional shape of a predetermined subject area,
A three-dimensional model placement step of arranging a plurality of types of three-dimensional models acquired in the three-dimensional model acquisition step based on the detection result in the detection step,
When a plurality of types of three-dimensional models arranged in the three-dimensional model arrangement step overlap each other, it has a familiarization processing step of fitting the three-dimensional models to the boundary portion between the three-dimensional models.
In the acclimation processing step, the lower the contrast of the region corresponding to the boundary between the models in the region of the input image, the larger the processing amount of the acclimation processing with respect to the region. Method. A program that can be executed by a computer, which causes the computer to function as the image processing device according to any one of claims 1 to 8.

Images