JP5590680B2 - Image composition apparatus, image composition method, and image composition program - Google Patents

Description

  The present invention relates to a technique for synthesizing an optically consistent video from a plurality of videos taken of subjects with different light source environments.

When synthesizing a plurality of shaded 3D images into one image, if images with different light source positions with respect to the subject are synthesized as they are, contradictions arise in the way the light strikes each subject, resulting in an uncomfortable image. Therefore, in order to generate an optically consistent video, it is desirable to calculate the light source position with respect to the subject in the video and correct the plurality of video so that the light source positions are the same before combining them. For this purpose, it is necessary to calculate the light source position in the video.
As a method of calculating such a light source position, there are a method of capturing an image so that the light source is reflected on the camera and a method of estimating the light source position from an image in which the light source is not directly reflected.
As the former method, there is a method of estimating a relative light source position by using a fisheye lens or an omnidirectional camera and capturing an image including a light source in the same manner as a subject (for example, see Non-Patent Document 1).
As the latter method, there is a method in which a metal sphere is placed in the same space as the subject, the space is imaged with a camera, and a light source position is estimated based on a light source reflected on the metal sphere (see Non-Patent Document 2, for example). This method is a method using a characteristic that the region with the highest illuminance on the spherical surface indicates the direction of the light source (for example, see Non-Patent Document 3). For example, FIG. 13A is an example of an image in which a sphere irradiated by a light source from the upper right front (broken arrow) direction is captured as a subject. FIG. 13B shows a luminance distribution on the line segment AB in FIG. The position of the line segment AB on the video plane is estimated by clustering the contour pixels of the selected subject area. In such a distribution, the region with the highest luminance is considered to be the region with the highest illuminance. Therefore, the light source position can be calculated based on the direction of the region with the highest luminance in the sphere on the video (the direction of the broken line arrow).

Atsuko Okura, Atsushi Kawakami, Katsushi Ikeuchi, “Estimation and Evaluation of Diffuse Reflectance of Outdoor Objects by Simultaneous Shooting of Light Source Environment and Object”, Journal of Information Processing Society of Japan, Computer Vision and Image Media Vol. 2 No. 1, May 2009, pp. 32-41 Debevec, P.E. “Rendering Synthetic Objects into Real Scenes: Bridging Traditional and Image-based Graphics with Global Illumination and High Dynamic Range Photography”, Proceedings of SIGGRAPH 98, 1998, pp. 189-198 Jorge Lopez-Moreno, Sunil Hadap, Erik Reinhard, Diego Gutierrez, “Composing images through light source detection”, Computers & Graphics, vol.34, no.6, 2010, pp. 698-707

  However, the methods described in Non-Patent Documents 1 and 2 described above must perform imaging by a special method, and require complicated calibration in advance. For example, in the method of Non-Patent Document 1, it is assumed that a device such as a fish-eye camera or an omnidirectional camera is used, and the light source position cannot be estimated unless the image is captured using such a camera. In the method of Non-Patent Document 2, the light source position cannot be estimated unless an image including a metal sphere is captured in advance in the imaging space. That is, in the above-described method, the light source position cannot be estimated based on an image captured without satisfying such a condition. Therefore, it is desirable to estimate the position of the light source with respect to the subject in the video image even if such special conditions are not satisfied.

  The present invention has been made in view of the circumstances as described above, and does not require complicated calibration or special devices, and synthesizes an image by estimating the light source direction from an image in which the light source is not directly reflected. A video composition device, a video composition method, and a video composition program are provided.

  In order to solve the above-described problems, the present invention is a video composition device that synthesizes an optically consistent video from a plurality of video images of subjects with different light source environments, and includes light source information including a light source position. Is included in a known video storage unit that stores a known video including a subject that is previously associated, a video input unit that receives an input video to be combined with the known video, and an input video that is input to the video input unit. A region selection unit that selects a plurality of subject regions, a spherical region determination unit that determines whether or not a subject included in the subject region selected by the region selection unit is a spherical surface, and a spherical surface determination unit that determines whether the subject is a spherical surface A light source position calculation unit that calculates a light source direction for each of a plurality of subject areas determined to be and calculates a light source position with respect to the subject based on the calculated plurality of light source directions; A light source position adjustment unit that corrects based on the light source position calculated by the calculation unit, and a video composition unit that synthesizes a known video whose light source position is corrected by the light source position adjustment unit with an input video. .

  In addition, the present invention is characterized by including a light source position correction unit that corrects an estimation result for the light source position calculated by the light source position calculation unit by using the characteristics of the neighboring region on the video.

The present invention further includes a spherical region reevaluation unit that performs reevaluation of the spherical region using another image including the same subject region as the spherical region with respect to the spherical region obtained by the spherical region determination unit. It is characterized by that.

  In addition, the present invention includes a known video storage unit that stores a known video including a subject in which light source information including a light source position is associated in advance, and optically captures a plurality of videos obtained by imaging subjects with different light source environments. Video synthesizing method of video synthesizing apparatus that synthesizes video with consistent consistency, video input step for inputting input video to be synthesized with known video, and selection of multiple subject areas included in input video A region selection step, a spherical region determination step for determining whether or not a subject included in the selected subject region is a spherical surface, and a light source direction for each of the plurality of subject regions determined to be spherical. A light source position calculating step for calculating a light source position with respect to the subject based on the calculated plurality of light source directions, and a known image by calculating the light source position calculated in the light source position calculating step. A light source position adjustment step of correcting, based on a known image obtained by correcting the light source position, characterized by comprising a video synthesizing step of synthesizing the input image.

  In addition, the present invention corrects the estimation result between the light source position calculating step and the light source position adjusting step by using the characteristics of the neighboring area on the video with respect to the light source position calculated in the light source position calculating step. A light source position correcting step is provided.

  Further, the present invention provides a spherical surface using another image including the same subject area as the spherical area with respect to the spherical area obtained in the spherical area determining step between the spherical area determining step and the light source position calculating step. A spherical region re-evaluation step for re-evaluating the region is provided.

  In addition, the present invention includes a known video storage unit that stores a known video in which a light source position is associated in advance, and synthesizes an optically consistent video from a plurality of videos obtained by imaging subjects with different light source environments. A video composition program for causing a computer of the video composition device to execute the above-described video composition method.

  As described above, according to the present invention, a video synthesizing apparatus that synthesizes an optically consistent video from a plurality of video images of subjects with different light source environments is a known video in which light source positions are associated in advance. Are selected, a plurality of subject areas included in the input video input are selected, a determination is made as to whether or not the subject included in the selected subject area is a spherical surface, and a plurality of determinations are made that the subject is a spherical surface. For each subject area, the light source position with respect to the subject in the subject area is calculated, the known video is corrected based on the calculated light source position, and the known video with the corrected light source position is combined with the input video. It is possible to synthesize an image by estimating the light source direction from an image in which the light source is not directly reflected without requiring a special calibration or a special device.

It is a block diagram which shows the structural example of the video composition apparatus by one Embodiment of this invention. It is a figure explaining the determination process of the spherical area by one Embodiment of this invention. It is a figure explaining the range of the light source direction of the estimation object by one Embodiment of this invention. It is a figure explaining a mode that a light source direction is estimated by one Embodiment of this invention. It is a figure explaining a mode that the light source position by one Embodiment of this invention is estimated. It is a flowchart which shows the operation example of the video composition apparatus by one Embodiment of this invention. It is a flowchart explaining the detail of the light source information estimation process by one Embodiment of this invention. It is a block diagram which shows the structural example of the video composition apparatus by the 2nd Embodiment of this invention. It is a flowchart which shows the operation example of the video synthesizing | combining apparatus by the 2nd Embodiment of this invention. It is a figure explaining the mechanism which correct | amends light source information by the 2nd Embodiment of this invention. It is a block diagram which shows the structural example of the video composition apparatus by the 3rd Embodiment of this invention. It is a flowchart which shows the operation example of the video synthesizing | combining apparatus by the 3rd Embodiment of this invention. It is a figure explaining the spherical surface characteristic utilized with a light source direction estimation technique.

Hereinafter, an embodiment of the present invention will be described with reference to the drawings.
<First Embodiment>
FIG. 1 is a block diagram showing a configuration of a video composition device 20 according to the first embodiment of the present invention. The video composition device 20 is a computer device that synthesizes optically consistent video from a plurality of video images of subjects with different light source environments to generate a composite video. The video composition device 20 includes a video input unit 1, a video output unit 2, a video / data storage unit 3, a region selection unit 4, a spherical region determination unit 5, a light source information estimation unit 6, and a light source information adjustment unit. 7 and a video composition unit 8.

The video input unit 1 receives an input video to be synthesized with a known video stored in advance in the video / data storage unit 3. For example, an image captured in various environments such as outdoors and indoors is input to the image input unit 1 from an input device such as a camera. The video input unit 1 stores the inputted input video in the video / data storage unit 3.
The video output unit 2 outputs the composite image generated by the video composition device 20 to an output device such as a display, and displays the image.

  The video / data storage unit 3 stores information used for each process in the video composition device 20. For example, the video / data storage unit 3 stores in advance a known video to be combined with the input video input to the video input unit 1. The known video is information including a subject whose shape and light source position are known and light source information including these is associated in advance. The known video is, for example, a video created by CG (computer graphics) or a video in which distance information based on the position of the input device is obtained by means such as a distance sensor. The video / data storage unit 3 includes a video input from the video input unit 1, a region selection unit 4, a spherical region determination unit 5, a light source information estimation unit 6, a light source information adjustment unit 7, and a video composition unit described later. Data and video obtained by 8 etc. are stored. For example, if the video composition device 20 is a personal computer, the video / data storage unit 3 is a so-called memory or hard disk.

  The area selection unit 4 reads the input video input to the video input unit 1 and stored in the video / data storage unit 3, and selects a plurality of subject areas included in the read input video. For the selection of the subject area, for example, an input for specifying the contour portion of the subject using an external input device such as a mouse or a digitizer may be accepted, or various subjects may be accumulated in advance as learning data, The subject region may be selected by performing collation by matching or the like. In this way, the region selection unit 4 selects a plurality of types of subject regions from the input video stored in the video / data storage unit 3.

  The spherical area determination unit 5 uses the illuminance distribution of the subject area selected by the area selection unit 4 to determine whether or not the subject included in the area is a spherical surface. For example, as shown in FIG. 2A, the spherical area determination unit 5 obtains an illuminance value distribution parallel to the light source direction on the video. Distribution 1 and distribution 2 shown in FIG. 2 (b) are examples of the illuminance value distribution corresponding to the broken line in FIG. 2 (a), and the subject is, for example, a depression like an apple core region. If there is, there are a plurality of peaks as in distribution 1, and if it is spherical (convex), the illuminance value distribution has only one peak as in distribution 2. Therefore, when there are two or more peaks in the illuminance value distribution in the light source direction, the spherical region determination unit 5 determines that the region is not a spherical surface, and when the number of peaks is one, the region is Determined to be spherical.

  That is, when the region selected from the video is not a perfect sphere due to the presence of the depression, there are a plurality of regions having high local illuminance, and it is difficult to calculate an accurate light source position. For example, if the subject is an apple, the number of peaks is plural due to the depression around the core of the apple. If the same processing is performed in such a case, it is considered that the estimation accuracy of the light source direction is deteriorated.

  The light source information estimation unit 6 is a light source position calculation unit that calculates a light source position with respect to the subject in the subject region for each of a plurality of subject regions that are determined to be spherical by the spherical region determination unit 5. First, calculation processing of the light source direction by the light source information estimation unit 6 will be described. The light source direction is the normal direction of the light source with respect to the subject. FIG. 3 is an example in which a three-dimensional coordinate system is superimposed and displayed on a subject in an image. The x axis is a direction perpendicular to the image surface, the y axis is a horizontal direction on the image surface, and the z axis is a vertical image on the image surface. Represents the direction. The light source direction calculated in the present embodiment corresponds to the near side from the image plane (yz plane), and is indicated by a combination of the rotation angle φ in the x-axis direction and the rotation angle θ in the z-axis direction.

  The rotation angle φ in the x-axis direction is the light source direction projected onto the yz plane, and the rotation angle θ in the z-axis direction is the light source direction projected onto the xy plane in front of the image plane. Specifically, the light source information estimation unit 6 performs clustering (for example, Fuzzy K-means) based on the luminance of the contour pixel of the subject region selected by the region selection unit 4, and displays the direction with the luminance peak in the yz plane. Calculated as the upper light source direction (angle φ). In the case of the image shown in FIG. 3, the upper right (dashed arrow) direction is the light source direction.

  Further, as shown in FIG. 4, the light source information estimation unit 6 corresponds to the peak position of the pixel value distribution in the light source direction on the image plane of the region determined to be spherical by the spherical region determination unit 5 described later. The angle θ is calculated as the direction of the light source. Further, the light source information estimation unit 6 calculates the light source intensity based on the relationship between the light source intensity and the light source direction as shown in the following formula (1).

  In Expression (1), L is determined by the light source intensity, Ω is determined by the normal vector n and the light source direction ω with respect to the light source and the constant C related to the reflectance of the object from the position where the object exists on the image. Here, since the pixel value P of the coordinates on the image corresponding to the position of the subject is known, the light source intensity L can be calculated.

  The light source information estimation unit 6 calculates the light source position based on the angle φ indicating the light source direction calculated in this way, the angle θ, and the light source intensity L. 5A is a view of the subject as viewed from above, FIG. 5B is a view (video) as seen from the front, and each segment with an arrow indicates a light source direction estimated from each subject. The light source information estimation unit 6 calculates the intersection of the line segments indicating the light source direction as the light source position. In FIG. 5B (front view), since the line segments indicating the light source directions from the two subjects do not intersect, the middle point (indicated by a black circle) where the distance between the two types of line segments is the shortest is indicated as the light source position. Calculate as In this way, the light source direction and the light source position are calculated. Here, for example, in the case of a real image, since the light source direction estimated from each subject includes an error, the direction (positive / negative) of the light source direction may be reversed. In such a case, among the three components in the light source direction, one type (for example, the least accurate z direction) or an average of a plurality of types of components may be obtained, or the light source direction estimated from a plurality of subjects. Of these, estimated values that differ greatly from the others may be excluded and averaged.

The light source information adjustment unit 7 corrects the light source information of the known video stored in the video / data storage unit 3 based on the light source direction and the light source position calculated by the light source information estimation unit 6.
The video synthesis unit 8 synthesizes the known video whose light source position has been corrected by the light source information adjustment unit 7 with the input video stored in the video / data storage unit 3.

Next, an operation example of the video composition device 20 according to the present embodiment will be described. FIG. 6 is a flowchart illustrating an operation example of each unit included in the video composition device 20.
First, the video input unit 1 inputs a video and stores it in the video / data storage unit 3 (step S1). Then, the area selection unit 4 selects a plurality of subject areas (step S2). For each subject area obtained in step S2, the light source information estimation unit 6 estimates the light source direction (angle φ) on the image plane (step S3).

  Then, the spherical area determination unit 5 determines whether or not the subject area is a spherical surface (step S4). When there are two or more peaks in the illuminance value distribution in the light source direction (angle φ), the local region is set as an out-of-spherical region and excluded from the light source direction estimation process. When the number of peaks is one, the local region is determined to be a spherical region. Then, the light source information estimation unit 6 estimates light source information for the subject area determined to be a spherical area (step S5). Details of the processing in step S5 will be described with reference to the flowchart of FIG.

  FIG. 7 shows a flow until the light source direction and the position of the light source are estimated for the subject area determined to be a spherical area. The light source information estimation unit 6 calculates the rotation angle θ in the z-axis direction as the light source direction (step S51). Further, the light source intensity L is calculated by the above-described equation (1) (step S52). Then, the light source information estimation unit 6 calculates the light source position based on the angle φ, the angle θ, and the light source intensity L (step S53).

  Returning to FIG. 6, the light source information adjustment unit 7 reads a known video to be synthesized from the video / data storage unit 3 (step S <b> 6). It is assumed that the known video has distance data acquired by a device such as a distance sensor whose position of each subject is known. For a known video having subject position information, the light source information adjustment unit 7 generates a video in which the light source information is adjusted to match the light source information obtained by the light source information estimation unit 6 (step S7). If the known video read in step S6 is CG, a light source is arranged at the light source position obtained in step S53, and rendering is performed to generate a video. On the other hand, if the known video read in step S6 is not CG, a video with the light source intensity obtained in step S52 set to 0 is generated and a light source is placed at the light source position as a result of step S53. Generate video for The video synthesizing unit 8 synthesizes the known video whose light source information is adjusted with the input video (step S8) and stores it in the video / data storage unit 3. The video output unit 2 outputs the composite video stored in the video / data storage unit 3.

<Second Embodiment>
Next, a second embodiment of the present invention will be described with reference to the drawings. FIG. 8 is a block diagram showing the configuration of the video composition device 20 of this embodiment. The video composition device 20 according to the second embodiment includes a light source information correction unit 9 in addition to the units included in the video composition device 20 according to the first embodiment. Hereinafter, the light source information correction unit 9 will be mainly described. In particular, the present embodiment corrects an estimation error that occurs when an actual image is used using a highlight area on the image as a clue. The light source information correction unit 9 corrects the position of the light source estimated by the light source information estimation unit 6 using a highlight area in the video.

  FIG. 9 is a flowchart showing an operation example of the video composition device 20 according to the present embodiment. Here, light source information correction processing (step S9) is added to the flow shown in FIG. Since the processing from step S1 to S6 and from step S6 to S8 is the same as that in the first embodiment, description thereof is omitted. In the light source information correction step (step S9), the presence or absence of a highlight region is searched for in the vicinity of the peak position of the illuminance value distribution obtained in the estimation of the light source information in step S5. If there is, the point with the highest illuminance in that area is set as the position on the image corresponding to the peak of the pixel value distribution. In FIG. 10A, X is a position on the image corresponding to the peak of the illuminance value obtained from the spherical object. First, it is searched whether there is a highlight region having an illuminance higher than the illuminance at the position X in the vicinity. In this search, for example, a search is made as to whether or not there is an illuminance higher than the illuminance at the position X in the range of the neighborhood n pixels square. Here, n is a numerical value such as 3 or 5. If there is an n-pixel square in the vicinity that is higher than the illuminance at the position X, the peak position of the pixel value obtained as the position X is replaced with the position with the highest illuminance in the neighboring highlight area. (FIG. 10 (b)).

<Third Embodiment>
Next, a third embodiment of the present invention will be described with reference to the block diagram of FIG. 11 and the flowchart of FIG. Since the difference between the third embodiment and the first embodiment is only that the spherical area reevaluation unit 10 is added in FIG. 11, the spherical area reevaluation unit 10 will be mainly described below. In the third embodiment, for example, when there are a plurality of images of the same subject at different light source positions such as an outdoor scene image by a fixed camera, a spherical region determination unit for the same subject region obtained in each video frame By integrating these results, it is more accurately determined whether or not the subject area is a spherical area.

  In the flowchart of FIG. 12, the spherical area reevaluation (step S10) is performed between the determination of the spherical area (step S4) and the estimation of the light source information (step S5) in the first embodiment. Other processes are the same as those in the first embodiment. Specific processing in step S10 will be described below. In the target subject area of the frame i, the mask area Mi (x, y) of the pixel (x, y) determined as the spherical area in step S4 is set as in the following expression (2).

  That is, when it is determined to be a spherical area, 1 is set to the mask, and when it is determined to be an aspherical area, 0 is set to the mask. When the average value of the mask in the same pixel exceeds a predetermined threshold th, the corresponding area is determined as a spherical area.

It is obvious that the target frame here may be all frames including the subject or may have a specific interval. Based on the result of this process, the process proceeds to the next light source information estimation (step S5).
In the above-described spherical region reevaluation step, the description has been given for a scene where the camera is fixed and only the light source position moves. However, even if the camera or the subject moves, the local region between the subjects If the correspondence is established, it is also possible to obtain the determination results by Expressions (2) and (3) based on the determination result of the spherical area for the same subject area included in a plurality of frames.

  As described above, according to the present embodiment, a plurality of spherical regions included in an image are detected, and the light source is in the normal direction of the high luminance region according to the distribution of the luminance of the pixels in the spherical region. The light source direction is calculated as follows, and the light source position is calculated based on the plurality of light source directions calculated for each of the plurality of spherical regions. This makes it possible to estimate the light source direction and light source position from multiple images with different light source environments and adjust the light source direction, position, and light source intensity of the subject in the image to synthesize an image that achieves optical consistency. be able to. Therefore, it is possible to calculate the light source position from an image in which the light source is not directly reflected without requiring complicated calibration or a special device.

  In addition, for example, by estimating and adjusting the light source information in each video scene for input video captured in various environments such as outdoors and indoors, it is possible to obtain a known video with optical consistency that has no sense of incongruity in shadows etc. Can be synthesized.

  Note that a program for realizing the function of the processing unit in the present invention is recorded on a computer-readable recording medium, and the program recorded on the recording medium is read into a computer system and executed to perform video composition. May be. Here, the “computer system” includes an OS and hardware such as peripheral devices. The “computer system” includes a WWW system having a homepage providing environment (or display environment). The “computer-readable recording medium” refers to a storage device such as a flexible medium, a magneto-optical disk, a portable medium such as a ROM and a CD-ROM, and a hard disk incorporated in a computer system. Further, the “computer-readable recording medium” refers to a volatile memory (RAM) in a computer system that becomes a server or a client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line. In addition, those holding programs for a certain period of time are also included.

  The program may be transmitted from a computer system storing the program in a storage device or the like to another computer system via a transmission medium or by a transmission wave in the transmission medium. Here, the “transmission medium” for transmitting the program refers to a medium having a function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line. The program may be for realizing a part of the functions described above. Furthermore, what can implement | achieve the function mentioned above in combination with the program already recorded on the computer system, and what is called a difference file (difference program) may be sufficient.

  DESCRIPTION OF SYMBOLS 1 ... Video input part, 2 ... Video output part, 3 ... Video | data storage part, 4 ... Area | region selection part, 5 ... Spherical area | region determination part, 6 ... Light source information estimation part, 7 ... Light source information adjustment part, 8 ... Video | video Composition unit, 9 ... Light source information correction unit, 10 ... Spherical region reevaluation unit, 20 ... Video composition device

Claims (9)

A video composition device that synthesizes optically consistent video from a plurality of video images of subjects with different light source environments,
A known video storage unit that stores a known video including a subject associated with light source information including a light source position in advance;
A video input unit to which an input video to be combined with the known video is input;
An area selection unit for selecting a plurality of subject areas included in the input video input to the video input unit;
A spherical region determination unit that determines whether or not a subject included in the subject region selected by the region selection unit is a spherical surface;
A light source position calculation unit that calculates a light source direction for each of a plurality of subject regions determined by the spherical region determination unit to be a spherical surface, and calculates a light source position with respect to the subject based on the calculated plurality of light source directions When,
A light source position adjustment unit that corrects the known image based on the light source position calculated by the light source position calculation unit;
A video synthesizing unit that synthesizes the known video whose light source position is corrected by the light source position adjusting unit with the input video ,
The spherical area determination unit obtains the illuminance value distribution of the subject area parallel to the light source direction on the image, and determines that the area is not spherical when there are a plurality of peaks in the illuminance value distribution in the light source direction. If the number of peaks of the illuminance value distribution has only one image synthesis device that region and wherein should be characterized as a spherical surface. The light source position calculation unit calculates a direction with a luminance peak as a first light source direction by performing clustering based on the luminance of the contour pixel of the subject region selected by the region selection unit, and the spherical region The angle corresponding to the peak position of the pixel value distribution in the light source direction on the image plane of the area determined to be spherical by the determination unit is calculated as the second light source direction, and is based on the relationship between the light source intensity and the light source direction. 2. The image synthesizing apparatus according to claim 1, wherein a light source intensity is calculated, and a light source position is calculated based on the first light source direction, the second light source direction, and the light source intensity. To the light source position calculated by the light source position calculation section, to claim 1 or 2, further comprising a light source position correcting unit for correcting the estimation result by utilizing the characteristics of neighboring areas on the image The video composition apparatus described. A spherical region reevaluation unit that performs reevaluation of a spherical region using another image including the same subject region as the spherical region with respect to the spherical region obtained by the spherical region determination unit, The video composition device according to any one of claims 1 to 3 . A known video storage unit that stores a known video including a subject in which light source information including a light source position is associated in advance, and an optically consistent video from a plurality of videos obtained by imaging subjects with different light source environments A video composition method of a video composition device for synthesizing
A video input step in which an input video to be combined with the known video is input;
An area selection step for selecting a plurality of subject areas included in the input video,
A spherical region determination step for determining whether or not a subject included in the selected subject region is a spherical surface;
A light source position calculating step of calculating a light source direction for each of a plurality of subject areas determined to be spherical, and calculating a light source position for the subject based on the calculated plurality of light source directions;
A light source position adjusting step for correcting the known video based on the light source position calculated in the light source position calculating step;
A video synthesis step of synthesizing the known video with the corrected light source position with the input video;
Equipped with a,
In the spherical area determination step, the illuminance value distribution of the subject area is obtained in parallel with the light source direction on the image. If there are a plurality of peaks in the illuminance value distribution in the light source direction, it is determined that the area is not spherical, and the light source direction If the number of peaks of the illuminance value distribution has only one image synthesis method that region and wherein should be characterized as a spherical surface. In the light source position calculating step, clustering is performed based on the luminance of the contour pixel of the subject region selected in the region selecting step to calculate a direction with a luminance peak as a first light source direction, and the spherical region The angle corresponding to the peak position of the pixel value distribution in the light source direction on the image plane of the region determined to be spherical by the determination step is calculated as the second light source direction, and based on the relationship between the light source intensity and the light source direction. 6. The video composition method according to claim 5, wherein a light source intensity is calculated, and a light source position is calculated based on the first light source direction, the second light source direction, and the light source intensity. Between the light source position calculating step and the light source position adjusting step,
To the light source position calculated at the light source position calculation step, to claim 5 or 6, further comprising a light source position correction step of using the characteristics of the neighboring region to correct the estimation results on video The video composition method described. Between the spherical area determination step and the light source position calculation step,
A spherical region re-evaluation step for re-evaluating the spherical region with respect to the spherical region obtained in the spherical region determining step using another image including the same subject region as the spherical region. The video composition method according to any one of claims 5 to 7 . A computer of a video synthesizer comprising a known video storage unit storing a known video in which light source positions are associated in advance, and synthesizing a video having optical consistency from a plurality of videos taken of subjects having different light source environments A video composition program for executing the video composition method according to any one of claims 5 to 8 .

Images