JP6516478B2 - Image processing apparatus and control method of image processing apparatus - Google Patents

Description

  The present invention relates to an image processing apparatus and a control method of the image processing apparatus, and more particularly to a technique suitable for use in extracting moving objects in a moving image.

  Background subtraction is one of the techniques for extracting moving objects. This is a method of obtaining a difference between an image in which the moving body is not captured and an image in which the moving body is captured, and extracting an area in which the difference is equal to or more than a threshold as a subject. In Patent Document 1, an image of a section in which the presence of a moving part is not detected in continuous-shot or moving-image captured images is stored as a background image, and a difference between the background image and a series of images including subjects is stored. A configuration for extracting a subject using it is described.

  In addition, as a method of extracting a moving object, subject tracking may be used. This is a method of extracting a moving object by obtaining corresponding feature points or areas in an image between adjacent frames. Patent Document 2 describes a configuration for extracting a moving object by extracting the color gamut of an image of several frames and determining whether the position of the extracted color gamut has changed.

JP, 2010-283637, A JP, 2000-348185, A

  However, in order to use the background subtraction method as described in the above-mentioned Patent Document 1, it is necessary to take an image without a mobile object. Even if there are cases where there are no images with no moving objects in the captured moving image, a method of using the frame images of the moving images to create a moving object-removed image from which the moving objects have been removed will be considered. Be

  At this time, a moving object removed image can be created by selecting a frame other than the moving object, that is, a frame in which the background is recorded, for each pixel. However, for example, if there is a pixel for recording the moving object over the entire frame, for example, the moving object is periodically moved around a certain position, the moving object can not be removed in principle. Furthermore, if there are pixels that alternately record the moving object and the background, and if the frame in which the moving object is recorded is selected by mistake, removal of the moving object fails.

Furthermore, in the method of tracking the moving object as described in Patent Document 2, since the color gamut representing the moving object is not determined, for example, the moving object moves in front of a stationary object having a certain unique color range. In the case of crossing, there is a possibility that a stationary object is erroneously detected as a moving object.
In view of the above-mentioned problems, the present invention can extract a moving object even when the moving object crosses in front of a stationary object having a unique color gamut or when an image in which a subject is not photographed is not photographed. The purpose is to

The image processing apparatus according to the present invention is an image processing apparatus for extracting a moving object in a moving image, obtaining a combination of sections with high reproducibility in the moving image, selecting one frame from the section with high reproducibility, A moving object is generated by applying a pixel value in a frame to each pixel to generate a moving object removed image without a moving object, and a difference between the moving object removed image and an image in each frame of a moving image. The target pixel is classified as a moving object by comparing the feature amount of the target pixel adjacent to the moving object seed extracted by the extracting unit and the moving object seed extracted by the extracting unit, and the classification is further performed. and a classifying means for classifying the pixels that have been classified as not a mobile at the time of completion and the background, the generating means, in each pixel of a frame of the section of video, threshold fluctuation feature amounts less and The combination of the sections with high reproducibility is obtained based on the number of times the frame continues intermittently, and the classification unit classifies the pixel which is not the mobile as the background when the invasion from the mobile is finished. Furthermore, the moving object is invaded from the background, and the extracting unit extracts the moving object from the moving image based on the classification by the classification unit.

  According to the present invention, it is possible to correctly extract a moving object in a moving image.

It is a block diagram showing an example of composition of an imaging device concerning an embodiment of the present invention. It is a flow chart which shows an example of the whole processing procedure in mobile's extraction concerning an embodiment. It is a flowchart which shows an example of a creation processing procedure of a mobile body removal image concerning an embodiment. It is a flow chart which shows an example of extraction processing procedure of a mobile concerning an embodiment. It is a flowchart which shows an example of the invasion processing procedure from the mobile body seed which concerns on embodiment. It is a flow chart which shows an example of an invasion processing procedure from a background seed concerning an embodiment. It is a figure which shows the example of mobile body removal image creation which concerns on embodiment. It is a figure which shows the example of the moving body removal image creation which concerns on embodiment, and the example of a background recording candidate area. It is a figure which shows the example of the background recording candidate area which concerns on embodiment. It is a figure which shows the example of selection of the background recording area which concerns on embodiment. It is a figure which shows the example of the background recording area which concerns on embodiment. It is a figure which shows the example of the mobile body extraction process which concerns on embodiment. It is a figure which shows the example of mobile body removal image creation which concerns on embodiment. It is a figure which shows the example of the mobile body removal image which concerns on embodiment. It is a figure which shows the example of the invasion from the mobile body seed which concerns on embodiment. It is a figure which shows the example of the invasion from the background seed which concerns on embodiment.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the attached drawings.
First Embodiment
FIG. 1 is a block diagram showing a basic configuration of an imaging apparatus 100 which is an example for realizing the image processing apparatus of the present invention.
The imaging apparatus 100 may be any electronic device having a camera function, such as a digital camera, a camera such as a digital video camera, a mobile phone with a camera function, a computer with a camera, and the like.

  The optical system 101 includes a lens, a shutter, and an aperture, and causes the light from an object to form an image on the image sensor 102 under the control of the CPU 103. An imaging element 102 such as a CCD image sensor or a CMOS image sensor converts light imaged through the optical system 101 into an image signal.

The CPU 103 implements the functions of the imaging device 100 by controlling the respective units constituting the imaging device 100 in accordance with the input signal or a program stored in advance.
The primary storage unit 104 is, for example, a volatile device such as a RAM, stores temporary data, and is used for the work of the CPU 103. Further, the information stored in the primary storage unit 104 may be used by the image processing unit 105 or may be recorded on the recording medium 106.

  The recording medium 106 records, for example, data of an image obtained by shooting, which is stored in the primary storage unit 104. The recording medium 106 is removable from the imaging apparatus 100, for example, like a semiconductor memory card, and the recorded data can be attached to a personal computer or the like to read out the data. In other words, the imaging apparatus 100 has a mounting and demounting mechanism and a reading and writing function of the recording medium 106.

The secondary storage unit 107 is, for example, a non-volatile storage device such as an EEPROM, stores a program (firmware) for controlling the imaging apparatus 100 and various setting information, and is used by the CPU 103.
The display unit 108 displays a viewfinder image at the time of shooting, displays a shot image, and displays a GUI image for interactive operation. The operation unit 109 is an input device group that receives a user's operation and transmits input information to the CPU 103, and may be, for example, a button, a lever, a touch panel or the like, or an input device using voice or a line of sight.

  Note that the imaging apparatus 100 according to the present embodiment has a plurality of patterns of image processing to be applied to a captured image by the image processing unit 105, and the pattern can be set from the operation unit 109 as an imaging mode. The image processing unit 105 performs image processing called so-called development processing, as well as adjusting the color tone according to the shooting mode. The CPU 103 may realize at least a part of the functions of the image processing unit 105 as software.

  FIG. 2 shows the entire flow of mobile object extraction in this embodiment, and FIGS. 3, 4, 5, and 6 show the flow of creation of a mobile object removed image, the flow of mobile object extraction, and the flow from mobile object seeds, respectively. Invasion flow and background Invasion flow from the seed is shown.

The flowcharts in FIGS. 2 to 6 are realized by the CPU 103 developing a program stored in the secondary storage unit 107 in a work memory area of the primary storage unit 104 and executing the program to control each unit.
In the following, the present embodiment for extracting a moving object in a moving image will be described using the example of FIG.
First, in S201, the CPU 103 creates a moving body-removed image from which a moving body has been removed by selecting a section in which the background is recorded (hereinafter, background recording section) for each pixel of the moving image.

A series of flow of frame selection for creating a moving object eliminated image at a certain pixel will be described using an example of temporal variation of RGB values in FIG. 7.
In S301, the CPU 103 substitutes 1 for the variable i, in S302 the CPU 103 substitutes 1 for the variable j, and in S303, the section in which the background may be recorded (hereinafter, background recording) Select candidate interval). When background is recorded, single color information is continuously recorded, and the procedure for obtaining such a section will be described below.

First, for each of the RGB values, the CPU 103 calculates the difference between adjacent frames, and obtains the absolute value thereof (FIG. 8A).
Next, the CPU 103 obtains a section in which a frame in which the difference absolute value of each of the RGB values is less than or equal to the first threshold continues for only a second threshold or more, and selects the section as a background recording candidate section. FIGS. 8B and 9 show background recording candidate sections selected by the above-described selection method. The first threshold and the second threshold are stored in advance in the CPU 103, respectively.

  Next, in step S304, the CPU 103 selects a section in which the background is actually recorded (hereinafter referred to as a background recording section) from the background recording candidate sections selected in step S303. First, the CPU 103 extracts RGB values in the central frame of the background recording candidate section as a representative value of each candidate section.

  Next, based on the combination of the representative values described above, the CPU 103 obtains a section having the combination with the highest degree of reproducibility, and selects this section as a background recording section. The reproduction value vote (s) in a certain section s has an initial value of 0, and 1 is added when all the expressions 1 are satisfied. In addition, suppose that t takes the section numbers other than the section s which is paying attention.

| candi (s, R)-candi (t, R) | <thr3
| candi (s, G)-candi (t, G) | <thr3 formula 1
| candi (s, B)-candi (t, B) | <thr3
Here, candi (s, R), candi (s, G) and candi (s, B) respectively represent R, G and B of representative values in the section s, and thr 3 represents a third threshold, respectively.
The degree of reproduction of each candidate section in the example of FIG. 9 is as shown in FIG. From FIG. 10, as shown in FIG. 11, sections 1, 3 and 5 are selected as background recording sections.

  In step S305, the CPU 103 determines RGB pixel values to be applied to the target pixel. An arbitrary frame is selected from among the background recording sections selected in S304, and the RGB values in the frame are used as the background information in the pixel of interest is recorded.

In step S306, the CPU 103 increments the variable j by 1. In step S307, it is determined whether or not the processing of the total number of rows has been performed. If the processing is not completed, the process returns to step S303. Go to
In step S308, the CPU 103 increments the variable i by 1. In step S309, the CPU 103 determines whether all rows have been processed. If the processing is not completed, the process returns to step S302. The creation process of the body removed image is ended.

  With regard to the selection method of selecting an arbitrary frame from the background recording section, any means may be used, but it is desirable to select a more central frame in consideration of changes in ambient light and the like. Therefore, in this case, among the background recording sections, the section located at the center in time is focused, and the RGB values in the center frame of this section are used.

  Also, in the present embodiment, the section in which the same color is repeatedly recorded is selected as the background recording section, but, for example, each section is selected for a longer recording time, and each section is not used. The background recording section may be selected using the recording time in the section. Furthermore, although the center frame of the center section is used for the background recording section selected in the present embodiment, other evaluations such as using the RGB values in the frame at the center time of the moving image recording sections are used. The frame to be used may be selected according to the standard. Furthermore, instead of using a combination of RGB values in a certain frame, it is also possible to separately obtain the mode value or median value of each of the RGB values in the background recording section and use this.

In S201, the CPU 103 repeatedly performs the above-described procedure for each pixel (S301 to S309) to create a moving body removed image.
As described above, a frame having a combination of feature amounts with high reproducibility is selected from time variation of feature amounts of each pixel, and a moving body removed image is generated from the combination of feature amounts in the frame. The feature amount is at least one of color information, luminance value, and distance information.

  FIG. 12 shows an image of one frame in a moving image in which a jairo rober moves in front of a dart drop, and illustrates temporal variation of RGB values in each pixel of (1), (2) and (3) in FIG. 13 (a), 13 (b) and 13 (c) respectively.

From Fig. 13 (a), although the pixel and the background of the yellowtail are alternately recorded periodically in the pixel of Fig. 12 (1), the color of the background is recorded more continuously, so At the pixel, it is possible to correctly select the frame in which the background is recorded.
Next, as shown in FIG. 13B, the pixel shown in FIG. 12B records the object on the fulcrum fulcrum (hereinafter referred to as the fulcrum on the fulcrum) over the entire frame. As described above, it is not possible to remove the moving object for the pixels where the moving object is always recorded.

  Furthermore, although the object on the fulcrum and the background are alternately recorded periodically as in the pixel (1) in the pixel (3) of FIG. 12 from FIG. 13 (c), the object on the fulcrum It is recorded more continuously. Therefore, in this pixel, the frame on which the object is recorded on the supporting point is erroneously selected. Thus, although it can not be said that the mobile-body-removed image has removed the mobile body as much as possible, a method for correctly extracting the mobile body even in such a case will be described later.

FIG. 14 shows a moving body removed image created in the procedure of S201 for a moving image including the image of FIG.
From FIG. 14, although a part of the moving body can be removed, it is not possible to remove the portion on the fulcrum such as the object performing periodic motion.

It returns to description of the flowchart of FIG.
In S202, the CPU 103 substitutes 1 into the variable n, and in S203, the CPU 103 performs mobile object extraction processing. The moving object extraction process will be described with reference to the flowchart of FIG.
In steps S401 and S402, the CPU 103 creates a seed necessary for mobile object extraction. Here, a “moving body seed” or a “background seed” is created from each pixel constituting an image of a target frame. First, at S401, a mobile object seed is created. Next, in step S402, a background seed is created.

  In S401, the CPU 103 creates a mobile unit seed. The mobile seed refers to a part or all of all the pixels constituting the mobile. A pixel whose difference absolute value between the moving body removed image created in S201 and the image of a frame (hereinafter, a target frame) for which the moving body is desired to be extracted is equal to or more than a third threshold is created as a moving body seed.

Next, in step S402, the CPU 103 creates a background seed. Background seed refers to a pixel in which background information is recorded over the entire frame. Whether or not background information has been recorded over all frames, the absolute value of the difference of the time variation of the RGB value of the pixel of interest is determined, and the calculated difference absolute value becomes less than the first threshold over all frames. Determine the pixel as the background.
Finally, a pixel which has not been determined as a mobile object seed or a background seed is considered to be indeterminate.

  Next, in S403 and S404, the CPU 103 extracts a mobile using the “mobile seed”, “background seed” and “indeterminate” created in S401 and S402. The feature quantities of the target pixel and its surrounding pixels are compared, and if they are sufficiently similar, the process of making the target pixel belong to the same group as the surrounding pixels (here, mobile or background) will be described below. Described as "invasion".

First, in step S403, the CPU 103 invades the indeterminate state from the mobile object seed. A process procedure of classifying a pixel of interest as a moving object in which a moving object seed invades an indefinite state is shown in the flowchart of FIG.
The CPU 103 _{adds the} target pixel to the moving body in step S502 when the formula (2) is satisfied with the indeterminate pixel I _{indet of} interest and the moving object pixel I _move among the four neighboring pixels adjacent thereto. Do. Here, thr4 represents a fourth threshold.
| I _indet (R)-I _move (R) | <thr4
| I _indet (G)-I _move (G) | <thr4 equation 2
| I _indet (B)-I _move (B) | <thr4

  In step S503, the CPU 103 determines whether all the unfixed pixels adjacent to the moving body pixel have been added to the moving body in step S502. The CPU 103 repeats steps S501 to S503 until all indefinite pixels adjacent to the moving body pixel are not added to the moving body.

FIGS. 15 (a), (b) and (c) show an example of the process of invasion of a mobile object seed to an indefinite pixel.
From FIG. 15, a part of the background is misjudged as a moving object because there is a pixel included in the moving object seed although it is correctly the background. In addition, it is not possible to invade a region having a color that is not similar to the color of the moving object, and remains indefinite.
Here, in S504, the CPU 103 determines that the pixel that has become indeterminate at the end of the invasion from the moving body seed is not a moving body, and adds the above-mentioned indeterminate pixel to the background seed.

  For the reason described above, as shown in FIG. 15C, there may be pixels that are determined to be moving objects, although they are correctly background. In order to correctly determine these pixels as the background, in S404, the CPU 103 invades the moving object from the background seed.

The processing procedure for invading the mobile unit from the background seed is shown in the flowchart of FIG.
In step S602, the CPU 103 adds the target pixel to the background in step S602 when the expression 103 is satisfied with the moving object pixel I _move to be focused and the background pixel I _back among the four neighboring pixels adjacent thereto. . Here, thr5 represents a fifth threshold. Although thr4 and thr5 may have the same value, thr5 may be set to a value smaller than thr4. This is that when the edge of the moving object is blurred, such as when the image of the frame of interest is resized, the moving object is accidentally invaded through an ambiguous edge when it invades from the background. It is to avoid.
I _move (R)-I _back (R) | <thr5
I _move (G)-I _back (G) | <thr5 equation 3
I _move (B)-I _back (B) | <thr5

In step S603, the CPU 103 determines whether all the moving object pixels adjacent to the background pixel have been added to the background in step S602. The CPU 103 repeats steps S601 to S603 until all moving object pixels adjacent to the background pixel are not added to the background.
FIGS. 16 (a) and 16 (b) show the process of invasion from the background seed to the moving object pixel. Further, FIG. 16 (c) shows a diagram after the invasion from the background seed, that is, a state in which the mobile object has been extracted.

In step S204, the CPU 103 increments n by 1, and determines in step S205 whether all frames have been processed. If the processing is not completed, the process returns to step S203 to perform the next frame processing. If the process is completed, the process proceeds to S206.
The above procedure (S203 to S205) is repeated for all frames of the input moving image. In this way, it is possible to extract the moving object even when the image in which the moving object is not captured is not captured or when the moving object crosses in front of a stationary object having a unique color gamut.

  Furthermore, in S206, the CPU 103 creates a certain still image (hereinafter, representative still image). The CPU 103 can create a cinema graph by combining the moving object extracted in the present embodiment with the representative still image.

(Other embodiments)
The present invention supplies a program that implements one or more functions of the aforementioned embodiments to a system or apparatus via a network or storage medium, and one or more processors in a computer of the system or apparatus read and execute the program. Can also be realized. It can also be implemented by a circuit (eg, an ASIC) that implements one or more functions.

100 image pickup apparatus 101 optical system 102 image pickup element 103 CPU
104 primary storage unit 105 image processing unit 106 recording medium 107 secondary storage unit 108 display unit 109 operation unit

Claims (5)

An image processing apparatus for extracting a moving object in a moving image,
A moving object removed image in which a moving object is not shown by finding a combination of sections having high reproducibility in a moving image, selecting one frame from the section having high reproducibility, and applying the pixel value in that frame to each pixel Generating means for generating
An extraction unit that extracts a mobile object seed that is a part of a mobile object using a difference between the mobile object removed image and an image in each frame of a moving image;
The target pixel is classified as a moving object by comparing the moving body seed extracted by the extraction means with the feature amount of the adjacent target pixel, and the pixels classified as not moving objects at the time of classification are further added And classification means for classifying
The generation means obtains a combination of high-reproduction intervals based on the number of frames in which a feature amount variation continues below a threshold intermittently in each pixel of a frame in a certain interval of a moving image.
The classification means classifies the pixel which is not a mobile as the background when the invasion from the mobile is finished, and further, invades the mobile from the background,
The image processing apparatus, wherein the extraction unit extracts a moving object from a moving image based on the classification by the classification unit. The image processing apparatus according to claim 1 , wherein the feature amount is at least one of color information, a luminance value, and distance information. The invasion The image processing apparatus according to claim 1 or 2, characterized in that is to classify the pixel of interest to the mobile or background. The mobile device further comprises a combining unit that combines the moving object extracted in the image of each frame into a still image,
The image processing apparatus according to any one of claims 1 to 3, wherein the combining unit creates a cinema graph. A control method of an image processing apparatus for extracting a moving object in a moving image, comprising:
A moving object removed image in which a moving object is not shown by finding a combination of sections having high reproducibility in a moving image, selecting one frame from the section having high reproducibility, and applying the pixel value in that frame to each pixel Generating step to generate
Extracting a moving body seed which is a part of the moving body using a difference between the moving body removed image and an image in each frame of a moving image;
The target pixel is classified as a moving object by comparing the moving body seed extracted in the extraction step with the feature amount of the adjacent target pixel, and the pixels classified as non-moving objects at the time of classification are background And a classification process to classify
In the generation step, in each pixel of a frame of a section in a moving image, a combination of sections with high reproducibility is determined based on the number of times in which a frame whose feature amount variation is equal to or less than a threshold continues intermittently.
In the classification step, when the invasion from the moving body is finished, pixels which are not the moving body are classified as the background, and further, the invasion from the background to the moving body is performed,
In the extraction step , a moving body is extracted from a moving image based on the classification in the classification step.

Images