JP5772684B2 - Image processing apparatus, image processing method, and program - Google Patents

Description

  The present invention relates to an image processing apparatus, an image processing method, and a program, and more particularly to an image processing method and a program for generating a composite image from a plurality of image frames.

  An imaging device such as a video camera may be used to image a moving object such as a person or an object. At this time, the user may want to confirm the trajectory of the moving object with a single composite still image. For example, when a person who is playing golf is imaged, it may be desired to confirm the swing trajectory with a single composite still image.

  Patent Document 1 discloses a moving image clipping device that can obtain a synthesized image including a moving locus of a moving object even when a moving image including a background is targeted. The moving image cutout apparatus samples N frames, binarizes the absolute value of the phase-corresponding pixel difference between the sampled frames, and generates a mask frame corresponding to each frame. Then, the moving image clipping device generates an accumulated mask frame in which the values of the pixels of the mask frame are accumulated. Further, the moving image clipping device generates a reference mask frame obtained by binarizing the accumulated mask frame using a predetermined threshold. Thereafter, the moving image clipping device generates a moving object trajectory image by performing image composition using a mask frame corresponding to each frame and a reference mask frame.

JP 2004-334818 A

  In the technique described in Patent Literature 1, a mask frame and a reference mask frame that have been binarized using a predetermined threshold are generated. That is, in the mask frame and the reference mask frame, only a binary value indicating whether or not to perform the clipping process for each pixel is set. Then, the region where the binary value is “1” is cut out from each frame, and the locus image is generated by completely overwriting the cut-out image region over the initial frame.

  However, in this method, if the threshold setting is not appropriate, there is a risk that a non-moving object will be cut out and overwritten completely, and as a result, the moving object's trajectory cannot be grasped or misunderstood. A certain image) may be generated.

An image processing apparatus according to an aspect of the present invention includes:
A difference calculation unit that calculates a difference value of corresponding pixels between the image frames of N (N is an integer of 4 or more) image frames;
A binarization unit that generates a plurality of binarized frames indicating the magnitude of the difference value of each pixel by a binary level value by comparing the difference value calculated by the difference calculation unit with a predetermined condition;
A cumulative image generating unit that generates a summed frame obtained by summing the level values in the plurality of binarized frames for each of the N image frames;
An image synthesis unit that generates a synthesized image obtained by synthesizing the N image frames based on a synthesis ratio of each pixel of each of the N image frames determined from a level value set in the integrated frame;
The value range of the composition ratio of each pixel of each of the N image frames is three or more.

In the above-described image processing apparatus according to one aspect of the present invention,
The image processing apparatus may further include a key frame generation unit that calculates a value obtained by subtracting a first threshold value from a level value of each pixel in the integration frame as a synthesis ratio of each pixel of each of the N image frames.

In the above-described image processing apparatus according to one aspect of the present invention,
The image composition unit may use the level value of each pixel of the integrated frame as it is as a composition ratio of each pixel of the N image frames.

In the above-described image processing apparatus according to one aspect of the present invention,
The image composition unit may receive attention frame information indicating an image frame to be noted among the N image frames, and determine a composition order of the N image frames based on the attention frame information. Good.

In the above-described image processing apparatus according to one aspect of the present invention,
The difference calculation unit calculates a difference between a luminance value and a color difference value of corresponding pixels between the image frames,
The binarization unit may generate the binarized frame by performing a condition determination using both a luminance value and a color difference value.

An image processing method according to an aspect of the present invention includes:
A difference calculating step of calculating a difference value of corresponding pixels between the image frames of N (N is an integer of 4 or more) image frames;
A binarization step for generating a plurality of binarized frames indicating the magnitude of the difference value of each pixel by a binary level value by comparing the difference value calculated in the difference calculation step with a predetermined condition;
A cumulative image generating step of generating a summed frame obtained by summing the level values in the plurality of binarized frames for each of the N image frames;
An image synthesis step for generating a synthesized image obtained by synthesizing the N image frames based on a synthesis ratio of each pixel of each of the N image frames determined from the level value set in the integrated frame,
The value range of the composition ratio of each pixel of each of the N image frames is three or more.

The program according to one aspect of the present invention is:
On the computer,
A difference calculating step of calculating a difference value of corresponding pixels between the image frames of N (N is an integer of 4 or more) image frames;
A binarization step for generating a plurality of binarized frames indicating the magnitude of the difference value of each pixel by a binary level value by comparing the difference value calculated in the difference calculation step with a predetermined condition;
A cumulative image generating step of generating a summed frame obtained by summing the level values in the plurality of binarized frames for each of the N image frames;
An image that is determined from the level value set in the integrated frame and that generates a combined image by combining the N image frames based on a combining ratio of each pixel of the N image frames having a value range of 3 or more. A synthesis step;
Is to execute.

  According to the present invention, it is possible to provide an image processing apparatus, an image processing method, and a program capable of grasping the trajectory of a moving object, that is, capable of generating a composite image with little failure.

1 is a block diagram showing an overall configuration of an image processing apparatus 1 according to a first embodiment. FIG. 3 is a diagram illustrating an example of moving image data according to the first embodiment. It is a figure which shows the production | generation concept of the binarization frame of the binarization part 40 concerning Embodiment 1. FIG. It is a figure which shows the integrating | accumulating frame which the integrating | accumulating part 50 concerning Embodiment 1 produced | generated. FIG. 6 is a diagram illustrating a generation process of an image composition process performed by the image composition unit 60 according to the first embodiment. FIG. 3 is a block diagram illustrating an overall configuration of an image processing apparatus according to a second embodiment. It is a figure which shows the key frame production | generation process by the key frame production | generation part 70 concerning Embodiment 2. FIG. It is a figure which shows the production | generation process of the image composition process by the image composition part 60 concerning Embodiment 2. FIG. It is a figure which shows the key frame production | generation process by the key frame production | generation part 70 concerning Embodiment 2. FIG. It is a conceptual diagram which shows operation | movement of the moving image clipping device concerning patent document 1. FIG. It is a figure which shows the moving image used as the object of synthetic | combination image generation. FIG. 10 is a diagram illustrating an example of generating an accumulated frame according to the second embodiment. FIG. 10 is a diagram showing an example of key frame generation according to the second exemplary embodiment. It is a figure which shows the composition ratio in the case of image composition. It is a figure which shows the composition ratio in the case of image composition. 10 is a graph showing a composite image generated by the method according to the second embodiment and the method according to Patent Document 1.

<Embodiment 1>
Embodiments of the present invention will be described below with reference to the drawings. FIG. 1 is a block diagram showing a configuration of an image processing apparatus 1 according to the present embodiment. The image processing apparatus 1 is disposed in an imaging apparatus such as a video camera capable of capturing a moving image. The image processing apparatus 1 includes an image storage unit 10, a selection unit 20, a difference calculation unit 30, a binarization unit 40, an integration unit 50, and an image composition unit 60. The image processing apparatus 1 is an apparatus that extracts image frames of moving image data accumulated in advance and generates a composite image corresponding to a difference between the image frames. For example, the image processing apparatus 1 generates a composite image indicating a trajectory of a thrown ball or a golf swing. In the following description, an example in which the image processing apparatus 1 generates a composite image based on four image frames will be described. However, the present invention is not necessarily limited to this. For example, a composite image is generated from a large number of image frames such as 10 images. May be. That is, the image processing apparatus 1 generates a composite image from N (N is an integer of 4 or more) image frames. The image processing apparatus 1 may appropriately change the number of image frames to be combined in accordance with the nature of the subject.

  The image storage unit 10 stores moving image data for each image frame. The image storage unit 10 may be configured by, for example, a DRAM (Dynamic Random Access Memory) or the like. Further, the image storage unit 10 may be configured by a USB (Universal Serial Bus) memory that can be attached to and detached from the image processing apparatus 1. The image size of each image frame is the same. Each image frame holds an RGB value for each pixel, and the RGB value can be converted into luminance and color difference using a known conversion formula.

  The selection unit 20 selects four image frames to be used for generating a composite image from the image storage unit 10 and supplies them to the difference calculation unit 30 and the image composition unit 60. In the following description, the selection unit 20 selects four image frames shown in FIG.

  FIG. 2 is a diagram showing four image frames related to moving image data. Each image frame relates to moving image data in which a circular moving object moves. In each image frame, the captured moving object is indicated by diagonal lines, and the position of the moving object in other image frames is indicated by broken lines. The background color is the same color. The imaging order of each image frame is image frame 1 (Fr1), image frame 2 (Fr2), image frame 3 (Fr3), and image frame 4 (Fr4). Therefore, the moving body moves to the right as time advances, and the moving speed is gradually decelerated.

  The difference calculation unit 30 is a processing unit that extracts a difference value between corresponding pixels between the four supplied image frames. First, the difference calculation unit 30 extracts all sets of image frames. That is, the difference calculation unit 30 extracts 12 combinations (N × (N−1)) ((Fr1, Fr2) to (Fr4, Fr1)). And the difference extraction part 30 calculates the difference of the image signal of the pixel corresponding to each combination. Here, the difference extraction unit 30 calculates both the color difference difference value and the luminance difference value of the corresponding pixels. The difference calculation unit 30 supplies the calculated difference value to the binarization unit 40.

  The binarization unit 40 compares each difference value calculated by the difference calculation unit 30 with a predetermined condition, thereby setting each pixel to a binary level value (1 for the pixel satisfying the predetermined condition and 0 for the other pixels). The binarized frames (Δ (Fr1, Fr2) to Δ (Fr4, Fr1)) shown are calculated. The binary level value set for each pixel indicates whether or not the pixel difference from the corresponding pixel of the image frame to be compared is large. Here, the binarization unit 40 performs binarization in consideration of both the color difference difference value and the luminance difference value of each pixel. For example, binarization may be performed in accordance with whether or not the absolute value of the multiplication value obtained by multiplying the color difference difference value and the luminance difference value is equal to or greater than a threshold value. Also, the absolute value of the color difference difference value may be compared with a certain threshold value, the absolute value of the luminance difference value may be compared with another threshold value, and binarization may be performed depending on whether or not both are equal to or greater than the threshold value. good. That is, the binarization unit 40 may perform binarization using a predetermined condition that at least one region satisfies a condition when different image frames are compared.

  Note that the binarization unit 40 may perform binarization processing using only one of the luminance difference value and the color difference difference value, although the accuracy is lowered.

  FIG. 3 is a diagram illustrating binarized frames (Δ (Fr1, Fr2) to Δ (Fr4, Fr1)) generated by the binarization unit 40. For example, in Δ (Fr1, Fr2), the level values of the two circular areas are 1, and the level values of the other areas are 0. The binarization unit 40 calculates three types of binarized frames for each image frame (Fr1 to Fr4). The binarization unit 40 supplies all the calculated binarized frames (Δ (Fr1, Fr2) to Δ (Fr4, Fr1)) to the integration unit 50.

The integrating unit 50 uses the binarized frames (Δ (Fr1, Fr2) to Δ (Fr4, Fr1)) to integrate frames (SumΔ (Fr1) to SumΔ (Fr4)) for each image frame (Fr1 to Fr4). Is calculated. FIG. 4 is a conceptual diagram showing integrated frames (SumΔ (Fr1) to SumΔ (Fr4)). The integrating unit 50 integrates the level values (1 or 0) in the binarized frames (Δ (Fr1, Fr2) to Δ (Fr4, Fr1)) relating to the image frames (Fr1 to Fr4) for each pixel. For example, as shown in the following calculation formula, the integrating unit 50 sets the level values of pixels corresponding to Δ (Fr1, Fr2), Δ (Fr1, Fr3), and Δ (Fr1, Fr4) for frame 1 (Fr1). The integrated frame SumΔ (Fr1) is calculated by obtaining the integrated value of.
SumΔ (Fr1) = Δ (Fr1, Fr2) + Δ (Fr1, Fr3) + Δ (Fr1, Fr4)

  The integration unit 50 calculates the integration frames SumΔ (Fr2) to SumΔ (Fr4) for the other image frames (Fr2 to 4) by the same method. The integration unit 50 supplies the calculated integration frames SumΔ (Fr1) to SumΔ (Fr4) to the image synthesis unit 60.

  The image synthesis unit 60 synthesizes the four image frames (Fr1 to Fr4) supplied from the selection unit 20 using the integration frames SumΔ (Fr1) to SumΔ (Fr4) supplied from the integration unit 50. A composition procedure by the image composition unit 60 will be described with reference to FIG. FIG. 5 is a diagram showing an image composition process by the image composition unit 60. For convenience of explanation, region names are described in the initial frame (Fr1) in FIG. 5 (regions A1 to A5). “Bg” in FIG. 5 is an abbreviation of background and indicates a background color.

  The range of the level value of each pixel in the integration frames SumΔ (Fr1) to SumΔ (Fr4) is 0 to 3. The image composition unit 60 uses this level value as a composition ratio. Specifically, the composition ratio of the level 0 pixels is 0%, the composition ratio of the level 1 pixels is 33.3%, the composition ratio of the level 2 pixels is 66.7%, and the composition ratio of the level 3 pixels is 100. %.

  First, the image composition unit 60 selects an image frame 1 (Fr1) as an initial image frame, and synthesizes each image frame with the initial image frame (Fr1). The image composition unit 60 refers to the integrated frame SumΔ (Fr2) related to the image frame 2 (Fr2), and synthesizes the image frame 2 (Fr2) according to the level value of each pixel. No processing is performed for level 0 pixels. For the level 1 pixel, the RGB value of the image frame 2 (Fr2) is 33.3 (33.33333-, the same applies hereinafter)%, and the RGB value of the location of the image frame 1 (Fr1) is 66.7 (66.666). -, The same shall apply hereinafter) For level 2 pixels, the RGB value of image frame 2 (Fr2) is combined at a rate of 66.7%, and the RGB value at the location of image frame 1 (Fr1) is combined at a rate of 33.3%. For the level 3 pixel, the RGB value of the original pixel (in this case, the RGB value of the image frame 1 (Fr1)) is replaced with the RGB value of the image frame 2 (Fr2). For example, for the area A1, the RGB value of the image frame 1 (Fr1) is multiplied by 66.7% and the RGB value of the image frame 2 (Fr2) (that is, the background color) is multiplied by 33.3%. Add the values together. The RGB value of the image frame 1 (Fr1) in the area A1 is (120, 150, 180), and the RGB value (RGB value of the background color) of the image frame 2 (Fr2) in the area A1 is (90, 90, 90). In this case, the RGB value of the area A1 is (110, 130, 150). In this way, the image processing unit 60 generates a composite image frame (Fr1 + Fr2) of the initial image frame (Fr1) and the image frame 2 (Fr2).

  The image composition unit 60 generates a composite image frame (Fr1 + Fr2 + Fr3) of the composite image frame (Fr1 + Fr2) and the image frame 3 (Fr3) by the same method. For example, for the region A1, the RGB value of the composite image frame (Fr1 + Fr2) is combined at a rate of 66.7%, and the RGB value of the image frame 3 (Fr3) is combined at a rate of 33.3%. As a result, the composition ratio of the area A1 is such that the RGB value of the image frame 1 (Fr1) is 44.4% and the RGB value of the background color is 55.5%.

  The image synthesizing unit 60 synthesizes the synthesized image frame (Fr1 + Fr2 + Fr3) and the image frame 4 (Fr4) by the same method, and generates a final synthesized image (Fr1 + Fr2 + Fr3 + Fr4). The composition ratio of each region of the generated composite image is as shown in FIG. The image composition unit 60 outputs the generated composite image to an arbitrary storage device or display device (liquid crystal display or the like). The user confirms the trajectory of the moving object by referring to the composite image.

  In the above description, the image compositing unit 60 performs the image frame compositing process (combining in order from Fr1) in time-series order, but the present invention is not limited to this, and the compositing order is not limited. For example, the image composition unit 60 may perform composition in the order of image frame 4 (Fr4), image frame 3 (Fr3), image frame 2 (Fr2), and image frame 1 (Fr1). Similarly, the image composition unit 60 may perform composition in the order of image frame 2 (Fr2), image frame 4 (Fr4), image frame 3 (Fr3), and image frame 1 (Fr1).

  Note that the image composition unit 60 may receive image frame information to be focused on (frame-of-interest information) from the user, and may determine the order of image frame synthesis according to the frame-of-interest information. Details will be described below.

  The user inputs the frame information of interest using an input device (such as a mouse or a keyboard) integrated with the image processing device 1 or an input device connectable to the image processing device 1. For example, when generating a composite image of a golf swing, the user inputs an identifier of an image frame related to the follow-through when the user wants to confirm the follow-through in a focused manner. Alternatively, the user may input the order of abstract image frames such as “first half”, “middle board”, and “second half” among all the image frames to be synthesized. In this case, the image composition unit 60 identifies one image frame to be focused on based on user input (“first half”, “middle board”, “second half”).

  The image composition unit 60 determines the composition order of the image frames based on the input frame information of interest. For example, when nine image frames (Fr1 to Fr9) are combined and the eighth image frame (Fr8) in time series order is set as the frame-of-interest information, the image combining unit 60 The image frame (Fr8) is synthesized last, and the synthesis order of the image frames is determined so that the synthesis is performed earlier as the image frame is spaced apart from the eighth image frame. That is, the image composition unit 60 determines the order of image frames as Fr1, Fr2, Fr3, Fr4, Fr5, Fr6, Fr7 (or Fr9), Fr9 (or Fr7), and Fr8. The image composition unit 60 performs composition processing based on the determined composition order.

  As described above, the image composition unit 60 determines the composition order of the image frames according to the frame information of interest, so that the composition order of the image frames that the user wants to confirm most is the last. That is, after the synthesis of the image frame to be confirmed is completed, there is no possibility that another image frame is synthesized. Thereby, it is possible to obtain a composite image in which an image frame to be focused on is clearly displayed.

  The image processing apparatus 1 according to the present embodiment has been described above. Next, the effect of the image processing apparatus 1 according to the present embodiment will be described using the conventional technique (Patent Document 1) as a comparative example.

  When the technique of Patent Document 1 is used, it is very important to set a threshold value when binarizing from a difference value and a threshold value when generating a reference mask frame from an accumulated mask frame. If this threshold value is set incorrectly, an unintended area is cut out from a certain image frame, and an overwrite process using the area is performed. As a result, it is possible to generate a broken composite image in which the trajectory of the moving object cannot be grasped or misidentified.

  On the other hand, the image processing apparatus 1 according to the present embodiment performs image composition using the integrated frames. The image processing apparatus 1 targets four or more image frames. For this reason, the range of the composition ratio of each pixel defined in the generated integrated frame is three or more. The image composition unit 60 uses the composition ratio of each pixel as it is for image composition without binarization. That is, the image composition unit 60 does not represent a binary value indicating whether or not each image frame is to be extracted, but a composition ratio indicating how much the pixel value (RGB value in the above example) is used for the composition of each image frame. Is used for synthesis. Thereby, the image processing apparatus 1 can avoid performing an overwriting process using the region by cutting out an unintended region from the image frame. Therefore, it is possible to avoid generating a composite image having a failure that makes it impossible to grasp the trajectory of the moving object or misidentifies it.

  In the above description, the example in which the geometric moving object simply moves (FIGS. 2 to 5) has been described. However, the natural image to be synthesized by the image processing apparatus 1 is a change in moving direction or moving speed of the moving object. Is very complex. For example, when generating a composite image of a golf swing, the movement amount of the driver head is large, but the movement amount of the golfer's body axis is small. For this reason, when the technique described in Patent Document 1 is used, advanced know-how is required to set the above-described threshold value, and it is considered that there are actually many cases where a composite image having a failure is actually generated. On the other hand, the image processing apparatus 1 according to the present embodiment, even when image frames related to such complex natural images are to be combined, has a certain range in the combination ratio, so that the combination with less failure is performed. An image can be generated.

  Furthermore, the composite image generation processing according to the present embodiment does not require calculation using a complicated algorithm. For this reason, the processing load on the image processing apparatus 1 is small, and a composite image can be generated in a short time.

<Embodiment 2>
The image processing apparatus 1 according to the present embodiment is characterized by performing a level shift process on the integrated frame. The image processing apparatus 1 according to the present embodiment will be described while referring to differences from the first embodiment. In the following description, processing units that are assigned the same reference numerals and are not specifically mentioned perform the same processing as in the first embodiment. Also in the present embodiment, the four image frames shown in FIG.

  FIG. 6 is a block diagram showing a configuration of the image processing apparatus 1 according to the present embodiment. The image processing apparatus 1 further includes a key frame generation unit 70 in addition to the configuration shown in FIG. The integrating unit 50 supplies the integrated frames (SumΔ (Fr1) to SumΔ (Fr4)) corresponding to the generated image frames (Fr1 to Fr4) to the key frame generating unit 70 instead of the image synthesizing unit 60.

  The key frame generation unit 70 subtracts (level shifts) the supplied integrated frames (SumΔ (Fr1) to SumΔ (Fr4)) using a predetermined threshold (hereinafter referred to as a first threshold). (KeyΔ (Fr1) to KeyΔ (Fr4)) are generated. Hereinafter, the concept of key frame generation will be described with reference to FIG.

  FIG. 7A is an integration frame (SumΔ (Fr1) to SumΔ (Fr4)) shown in FIG. The key frame generation unit 70 generates a key frame (KeyΔ (Fr1) to KeyΔ (Fr4)) obtained by subtracting the first threshold value from the level value of the integrated frame (SumΔ (Fr1) to SumΔ (Fr4)). Specifically, the key frame generation unit 70 generates key frames (KeyΔ (Fr1) to KeyΔ (Fr4)) by subtracting the first threshold value from the level value having a value equal to or higher than the first threshold value as the level value. FIG. 7B shows key frames (KeyΔ (Fr1) to KeyΔ (Fr4) generated by subtracting 1 as a first threshold value from the integrated frames (SumΔ (Fr1) to SumΔ (Fr4)) shown in FIG. The key frame generation unit 70 supplies the generated key frames (KeyΔ (Fr1) to KeyΔ (Fr4)) to the image composition unit 60.

  Here, the first threshold value used by the key frame generation unit 70 is set to a positive integer such that the range of level values in the key frame (KeyΔ (Fr1) to KeyΔ (Fr4)) is three or more. For example, in the example of FIG. 7, the first threshold is set to 1 and 2 or more is not set.

  The image composition unit 60 performs image frame composition processing using the supplied key frames (KeyΔ (Fr1) to KeyΔ (Fr4)). The synthesis method is the same as in the first embodiment. FIG. 8 is a diagram showing an image composition process using the key frames (KeyΔ (Fr1) to KeyΔ (Fr4)) shown in FIG. 7B.

  The range of the level value of each pixel in the key frame (KeyΔ (Fr1) to KeyΔ (Fr4)) is 0-2. This level value is used as the composition ratio. That is, the synthesis ratio of level 0 pixels is 0%, the synthesis ratio of level 1 pixels is 50%, and the synthesis ratio of level 2 pixels is 100%. Since the synthesizing process is the same as that in FIG. 5, its details are omitted.

  The composition ratio of each region of the generated composite image (Fr1 + Fr2 + Fr3 + Fr4) is as shown in FIG. The image composition unit 60 outputs the generated composite image to an arbitrary storage device or display device (liquid crystal display or the like).

  Next, the significance of key frame generation will be described with reference to FIG. FIG. 9 is a diagram illustrating the concept of integrated frame creation and key frame creation. In FIG. 9, for ease of explanation, explanation is made using three image frames (FrA, FrB, FrC).

  The binarization unit 30 generates a binarized frame Δ (FrA, FrB) by the above-described method. At this time, as shown in FIG. 9, although the binarized frame is generated based on the image frame (FrA), the binarized frame Δ (FrA, FrB) is changed to the image frame (FrB). 1 (large difference) is set in the specific area. Similarly, in the binarized frame Δ (FrA, FrC), 1 (the difference is large) is set in a region unique to the image frame (FrC). In the integrated frame SumΔ (FrA) obtained by integrating these binarized frames, 1 is set in a region (region surrounded by a square in the drawing) peculiar to the image frame (FrB, FrC) to be compared.

  The key frame generation unit 70 subtracts 1 (level shift) as a first threshold value from the integrated frame SumΔ (FrA) to generate a key frame KeyΔ (FrA). By this subtraction, it is possible to reduce the influence of the specific region of the comparison target image frame (FrB, FrC). That is, the key frame Δ (FrA) can accurately represent only a specific region of the image frame (FrA) as compared with the integrated frame SumΔ (FrA). By performing image composition using the key frame Δ (FrA) generated in this way, a more accurate composite image can be obtained. For example, when the composite image generated by the method of Embodiment 1 (Fr1 + Fr2 + Fr3 + Fr4 in FIG. 5) and the composite image generated by the method of Embodiment 2 (Fr1 + Fr2 + Fr3 + Fr4 in FIG. 8) are compared, the latter composite image This is a trajectory image in which a circular moving body is displayed more clearly.

  Subsequently, the composite image generated by the image processing apparatus 1 according to the present embodiment is compared with the composite image generated by the moving image clipping apparatus described in Patent Document 1. First, the processing outline of the moving image clipping device described in Patent Document 1 will be described again with reference to FIG.

  The moving image clipping device generates a composite image using a mask frame binarized using a threshold value (FIG. 10A). FIG. 10B shows an image composition process by the moving image clipping device. This moving image clipping device uses 100% of the pixel value of the image frame as the composition source, or uses 100% of the pixel value of the image frame to be synthesized (overwriting). Therefore, a good composite image can be obtained when the above-described threshold setting is appropriate, but an unintended composite image is generated when it is inappropriate.

  Hereinafter, the composite image generated by the image processing apparatus 1 according to the present embodiment and the composite image generated by the moving image clipping apparatus described in Patent Document 1 are compared using a simple example. Consider moving image data in which a rectangular object having a width of 4 as shown in FIG. The composite image is generated from ten image frames. For the sake of clarity, it is considered that pixel values of 1 are set for rectangular objects and 0 are set for other areas, not RGB values.

  FIG. 12A shows the moving state of the rectangular object. As shown in the figure, the rectangular object moves to the right by one square for each image frame. FIG. 12B shows the generated integrated frames SumΔ (Fr1) to SumΔ (Fr10). FIG. 13A is a diagram showing key frames (KeyΔ (Fr1) to KeyΔ (Fr10)) generated by subtracting 3 as the first threshold value from the integrated frame shown in FIG. 12B. ) Is a diagram showing key frames (KeyΔ (Fr1) to KeyΔ (Fr10)) generated by subtracting 4 as the first threshold value from the integrated frame shown in FIG. 12B.

  FIG. 14A shows a composition ratio of each image frame used by the image composition unit 60 according to the present embodiment calculated from the key frames (KeyΔ (Fr1) to KeyΔ (Fr10)) shown in FIG. k1 to k10,% display). Since the key frame (KeyΔ (Fr1) to KeyΔ (Fr10)) shown in FIG. 13 (a) has a range of 6, dividing each column by 6 displays the composition ratio (k1 to k10,% display) of each image frame. ) Is calculated. FIG. 14B shows the composition ratio according to Patent Document 1 calculated using the key frames (KeyΔ (Fr1) to KeyΔ (Fr10)) shown in FIG. As for the composition ratio, 100% is set in a column having a value larger than 0, and 0% is set in a column having 0.

  FIG. 15A shows a composite ratio (k1 to k10,% display) of each image frame according to the present embodiment calculated from the key frames (KeyΔ (Fr1) to KeyΔ (Fr10)) shown in FIG. 13B. ). FIG. 14B shows the composition ratio according to Patent Document 1 calculated using the key frames (KeyΔ (Fr1) to KeyΔ (Fr10)) shown in FIG. 13B.

  Note that the initial image frame is assumed to be image frame 1 (Fr1). Therefore, in FIGS. 14 and 15, 100% is set in each column of k1. 12 to 15, the inside of the thick line frame indicates the position of the rectangular object. The synthesis order is a time-series order (Fr1 → Fr2 →... → Fr10).

  First, a synthesis process when the first threshold value is 3 (when 3 is subtracted from the level value of each image pixel in the integrated frame) will be considered. As is apparent from FIG. 14B, the technique of Patent Document 1 completely performs the overwriting process even for an area that should not be overwritten (for example, 100% is set in the column outside the thick line frame of the composition ratio k2). Have been). That is, the pixel value (0) of the background color is set in the area where the pixel value (1) of the rectangular area is to be set. On the other hand, in the method according to the present embodiment, as shown in FIG. 14A, pixel values of an image frame are mixed at a certain ratio. At this time, the mixing ratio of the pixel values corresponding to the rectangular object is high, and the mixing ratio of the background color is low. Therefore, it is possible to obtain a composite image in which a trajectory along which the rectangular object moves can be confirmed.

  Next, a synthesis process when the first threshold value is 4 (when 4 is subtracted from the level value of each pixel of the integrated frame) will be considered. As is apparent from FIG. 15B, in the technique of Patent Document 1, the composition ratio other than the rectangular area is 0%. Therefore, a desired composite image can be obtained. On the other hand, in the method according to the present embodiment, as shown in FIG. 15A, the pixel values in the rectangular area of each image frame are mixed at a certain ratio, although the accuracy is not 100%. Therefore, although it is not a composite image in which the rectangular object is clearly displayed, it is possible to obtain a composite image in which the locus of movement of the rectangular object can be confirmed.

  Next, a composite image generated by the method according to the present embodiment and a composite image generated by the method described in Patent Document 1 are shown in a graph format. FIG. 16 is a graph showing pixel values of a composite image generated for the moving image data shown in FIG. The horizontal axis of the graph indicates the horizontal coordinate of the composite image, and the vertical axis indicates the pixel value.

  FIG. 16A is a graph showing a composite image generated by the method according to the present embodiment (generated using the composite ratio shown in FIG. 14A or the composite ratio shown in FIG. 15A). . A broken line (EXP) indicating an expected value is also displayed in the graph. FIG. 16B is a graph showing a composite image generated by the method described in Patent Document 1 (generated using the composite ratio shown in FIG. 14B or the composite ratio shown in FIG. 15B). .

As a confirmation, the calculation method of the calculated value of each coordinate in the method according to the present embodiment will be described below. For example, when the first threshold is 3 and the horizontal axis is 5, the following calculation using the composition ratio shown in FIG. 14A is performed to calculate the final calculated value (0.33).
1 × 1 = 1
1 * (1- (1/2)) + 1 * (1/2) = 1
1 * (1- (1/2)) + 1 * (1/2) = 1
1 * (1- (1/2)) + 1 * (1/2) = 1
1 × (1− (1/6)) + 0 × (5/6) = (5/6)
(5/6) × (1− (1/6)) + 0 × (5/6) = (25/36)
(25/36) × (1− (1/6)) + 0 × (5/6) = (125/216)
(125/216) × (1- (1/6)) + 0 × (5/6) = (625/1296)
(625/1296) × (1- (1/6)) + 0 × (5/6) = (3125/7777)
(3125/7777) × (1− (1/6)) + 0 × (5/6) = (15625/46656) = 0.3349

  As is clear from comparison between the two graphs ((a) and (b)) in FIG. 16, the method described in Patent Document 1 can obtain a good composite image when the threshold is appropriate. However, when the threshold value is not appropriate, the shape of the graph is extremely collapsed, that is, the pixel value of each pixel of the generated composite image becomes an unintended value. On the other hand, in the method according to this embodiment, even if the threshold value is not appropriate, the shape of the graph does not collapse extremely. it can.

  Although the present invention has been described with reference to the above embodiment, the present invention is not limited to the configuration of the above embodiment, and can be made by those skilled in the art within the scope of the invention of the claims of the claims of the present application. It goes without saying that various modifications, corrections, and combinations are included. For example, in the above description, RGB is assumed as the color space. However, the present invention is not necessarily limited to this, and the present invention can be applied to other color spaces (YUV, CMY, etc.).

  Each processing of the selection unit 20, the difference calculation unit 30, the binarization unit 40, the integration unit 50, the image synthesis unit 60, and the key frame generation unit 70 causes a CPU (Central Processing Unit) to execute a computer program. Can also be realized. In this case, the computer program can be stored using various types of non-transitory computer readable media and supplied to the computer. Non-transitory computer readable media include various types of tangible storage media. Examples of non-transitory computer-readable media include magnetic recording media (for example, flexible disks, magnetic tapes, hard disk drives), magneto-optical recording media (for example, magneto-optical disks), CD-ROMs (Read Only Memory), CD-Rs, CD-R / W and semiconductor memory (for example, mask ROM, PROM (Programmable ROM), EPROM (Erasable PROM), flash ROM, RAM (random access memory)) are included. The program may also be supplied to the computer by various types of transitory computer readable media. Examples of transitory computer readable media include electrical signals, optical signals, and electromagnetic waves. The temporary computer-readable medium can supply the program to the computer via a wired communication path such as an electric wire and an optical fiber, or a wireless communication path.

  In addition to the case where the function of the above-described embodiment is realized by the computer executing the program that realizes the function of the above-described embodiment, this program is not limited to the OS ( A case where the functions of the above-described embodiment are realized in cooperation with an operating system or application software is also included in the embodiment of the present invention. Furthermore, the present invention is also applicable to the case where the functions of the above-described embodiment are realized by performing all or part of the processing of the program by a function expansion board inserted into the computer or a function expansion unit connected to the computer. It is included in the embodiment.

DESCRIPTION OF SYMBOLS 1 Image processing apparatus 10 Image storage part 20 Selection part 30 Difference calculation part 40 Binarization part 50 Integration part 60 Image composition part 70 Key frame generation part

Claims (7)

A difference calculation unit that calculates a difference value of corresponding pixels between the image frames of N (N is an integer of 4 or more) image frames;
A binarization unit that generates a plurality of binarized frames indicating the magnitude of the difference value of each pixel by a binary level value by comparing the difference value calculated by the difference calculation unit with a predetermined condition;
A cumulative image generating unit that generates a summed frame obtained by summing the level values in the plurality of binarized frames for each of the N image frames;
An image synthesis unit that generates a synthesized image obtained by synthesizing the N image frames based on a synthesis ratio of each pixel of each of the N image frames determined from a level value set in the integrated frame;
The image processing apparatus according to claim 1, wherein a value range of a composition ratio of each pixel of each of the N image frames is three or more.   The key frame generation part which calculates the value which subtracted the 1st threshold from the level value of each pixel in the integration frame as a composition ratio of each pixel of each of the N image frames, further comprising: Item 8. The image processing apparatus according to Item 1.   The image processing apparatus according to claim 1, wherein the image composition unit uses the level value of each pixel of the integrated frame as it is as a composition ratio of each pixel of the N image frames.   The image composition unit receives attention frame information indicating an image frame to be noted among the N image frames, and determines a composition order of the N image frames based on the attention frame information. The image processing apparatus according to claim 1, wherein the image processing apparatus is characterized in that: The difference calculation unit calculates a difference between a luminance value and a color difference value of corresponding pixels between the image frames,
The image according to any one of claims 1 to 4, wherein the binarization unit generates the binarized frame by performing a condition determination using both a luminance value and a color difference value. Processing equipment. A difference calculating step of calculating a difference value of corresponding pixels between the image frames of N (N is an integer of 4 or more) image frames;
A binarization step for generating a plurality of binarized frames indicating the magnitude of the difference value of each pixel by a binary level value by comparing the difference value calculated in the difference calculation step with a predetermined condition;
A cumulative image generating step of generating a summed frame obtained by summing the level values in the plurality of binarized frames for each of the N image frames;
An image synthesis step for generating a synthesized image obtained by synthesizing the N image frames based on a synthesis ratio of each pixel of each of the N image frames determined from the level value set in the integrated frame,
An image processing method characterized in that a value range of a composition ratio of each pixel of each of the N image frames is three or more. On the computer,
A difference calculating step of calculating a difference value of corresponding pixels between the image frames of N (N is an integer of 4 or more) image frames;
A binarization step for generating a plurality of binarized frames indicating the magnitude of the difference value of each pixel by a binary level value by comparing the difference value calculated in the difference calculation step with a predetermined condition;
A cumulative image generating step of generating a summed frame obtained by summing the level values in the plurality of binarized frames for each of the N image frames;
An image that is determined from the level value set in the integrated frame and that generates a combined image by combining the N image frames based on a combining ratio of each pixel of the N image frames having a value range of 3 or more. A synthesis step;
A program for running

Images