JP2021040265A - Image processing device, imaging apparatus, image processing method, and program - Google Patents

Abstract

To provide a device that automatically creates a composite image that conforms to a user's intention when creating a locus composite image.SOLUTION: An image processing device has acquisition means for acquiring subject information from a plurality of images, extracting means for extracting feature areas from the subject information based on a predetermined rule, and compositing means for creating a composite image representing a locus of the subject in the plurality of images using at least some of the plurality of images. The composite image includes the feature areas.SELECTED DRAWING: Figure 3

Description

The present invention relates to an image processing device that creates a composite image from a plurality of images.

It may be necessary to easily grasp the trajectory of the subject from a plurality of photographs taken by continuous shooting or the like. Patent Document 1 discloses a technique of extracting the same subject from a plurality of images by using an image processing device and creating a composite image in which the subject appears a plurality of times.

Japanese Unexamined Patent Publication No. 2014-164652

However, using the technique disclosed in Patent Document 1, synthesis cannot be performed in the following scenes. For example, there is a case where a locus composition is performed on a state in which a high jumping player is jumping, and an image when the player is at the highest position is used for the composition. The technique described in Patent Document 1 cannot automatically create such a composite image.

The present invention has been made in view of the above problems, and an object of the present invention is to provide an image processing device that automatically creates a composite image in consideration of the movement of a subject when performing trajectory synthesis. ..

In order to solve the above problems, the present invention comprises an acquisition means for acquiring subject information from a plurality of images, an extraction means for extracting a feature region from the subject information based on a predetermined rule, and the plurality of images. An image processing apparatus comprising: a compositing means for creating a composite image representing a locus of a subject in the plurality of images using at least a part of the images, and the composite image includes the feature region. I will provide a.

According to the configuration of the present invention, it is possible to provide an image processing device that creates a composite image based on the movement of a subject when performing trajectory synthesis.

It is a block diagram which shows the structure of the digital camera as an image processing apparatus in embodiment of this invention. It is a figure for demonstrating the method of the locus synthesis in embodiment of this invention. It is a flowchart for demonstrating the locus synthesis in embodiment of this invention. It is a figure for demonstrating the display of the display part in embodiment of this invention.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

FIG. 1 is a block diagram showing a configuration of a digital camera 100 as an image processing device according to the present embodiment.

The shutter 101 is a mechanical shutter having an aperture function, the imaging unit 22 includes an imaging element that converts an optical image into an electric signal, and the A / D converter 23 converts the output of an analog signal from the imaging unit 22 into a digital signal. Convert. In the present embodiment, the image sensor of the image pickup unit 22 has a Bayer array in which the pixels of R, G, and B are regularly arranged, but the present invention is not limited to this. The system control unit 50 controls the image pickup unit 22, the image pickup lens 103, and the shutter 101.

In addition to the mechanical shutter, the shutter 101 can also use an electronic shutter whose storage time is controlled by controlling the reset timing of the image sensor.

The image processing unit 24 performs various correction processing such as predetermined pixel interpolation processing and shading correction, white balance processing, γ correction processing, and color conversion for the data from the A / D converter 23 or the data from the memory control unit 15. Perform processing etc. Further, the electronic zoom function is realized by cutting out the image and performing the scaling process by the image processing unit 24. The shading correction process corrects the shading caused by the characteristics of the image pickup lens 103 and the aberration of the image pickup unit 22 with respect to the data from the A / D converter 23 or the data from the memory control unit 15. , Correct the brightness level in the screen. In the white balance (WB) process, the white balance process for adjusting the white reference in the screen to white is performed on the image data after the shading correction process. In the present embodiment, the shading correction is a correction that applies a gain for each pixel according to the two-dimensional coordinates (positions) of the image sensor of the image pickup unit 22, and the white balance processing is the correction of R, G1, G2, of the Bayer arrangement. This is a process of applying different gains for each B. Further, the image synthesizing unit 54 includes a synthesizing processing circuit for synthesizing a plurality of image data. Further, the image synthesizing unit 54 may be integrally configured with the image processing unit 24.

Further, in the image processing unit 24, a predetermined calculation process is performed using the captured image data, and the system control unit 50 performs control such as exposure control and distance measurement control based on the obtained calculation result, and the TTL method is used. AF processing, AE processing, EF processing, etc. are performed.

The memory control unit 15 controls the memory 32.

The data of the A / D converter 23 is written to the memory 32 via the image processing unit 24 and the memory control unit 15, or the data of the A / D converter 23 is directly written to the memory 32 via the memory control unit 15.

The display unit 28 displays image data for display written in the memory 32 and the like.

The display unit 28 can realize the live view display and the electronic finder function by sequentially acquiring and displaying the image data sequentially captured by repeatedly exposing and reading the image in the image capturing unit 22 at a predetermined cycle.

Further, the display unit 28 can arbitrarily turn on / off the display according to the instruction of the system control unit 50, and when the display is turned off, the power consumption of the digital camera 100 can be significantly reduced. ..

The memory 32 is a memory for storing captured still images and moving images, and has a storage amount necessary for storing a predetermined number of still images and moving images for a predetermined time.

The memory 32 can also be used as a work area for the system control unit 50.

The system control unit 50 controls the entire digital camera 100. The system control unit 50 realizes the processing in each step of the present embodiment by executing the program recorded in the non-volatile memory 56. RAM or the like is used as the system memory 52. In the system memory 52, constants and variables for the operation of the system control unit 50, a program read from the non-volatile memory 56, and the like are expanded. The system control unit 50 also controls the display by controlling the memory 32, the display unit 28, and the like.

The system timer 53 is a time measuring unit that measures the time used for various controls.

The non-volatile memory 56 is composed of a flash memory or the like. The program code executed by the system control unit 50 is written in the non-volatile memory 56, and the program code is executed while being read sequentially. Further, the non-volatile memory 56 is provided with an area for storing system information and an area for storing user setting information, and various information and settings can be read out and restored at the next startup.

The operation unit 70 includes various buttons, a touch panel, and the like, and includes a shutter switch (SW1, SW2), a menu button, a set button, a macro button, a flash setting button, a single shooting / continuous shooting / self-timer switching button, and the like. Further, the operation unit 70 may be provided with a menu move + (plus) button, a menu move- (minus) button, an image quality selection button, an exposure compensation button, a date / time setting button, and the like.

The shutter switch SW1 by operating the operation unit 70 turns on during the operation of the shutter button, and operations such as AF (autofocus) processing, AE (autoexposure) processing, and AWB (auto white balance) processing start.

The shutter switch SW2 is operated by operating the operation unit 70 to turn on when the operation of the shutter button is completed, and a series of operations such as exposure processing, development processing, compression / decompression processing, and recording processing are started.

The power supply unit 30 includes a primary battery such as an alkaline battery or a lithium battery, a secondary battery such as a NiCd battery, a NiMH battery, or a Li battery, an AC adapter, and the like. The power supply control unit 80 controls the power supply unit 30.

The connector interfaces with a recording medium such as a memory card or a hard disk, and connects to a recording medium such as a memory card or a hard disk.

The subject extraction unit 55 extracts a subject region from an image by, for example, a well-known color region division method. The subject extraction unit 55 extracts the pixels whose movement amount is equal to or more than the threshold value, and extracts the subject area by grouping the adjacent pixels whose movement amount is equal to or more than the threshold value. The subject area extracted from the subject extraction unit 55 is combined by the image composition unit 54. However, depending on the ISO sensitivity, the threshold value for comparison with the amount of change may be changed in consideration of the fluctuation of the signal level due to noise. Further, here, when the subject area is not fixed by the tripod, the subject area may be simply extracted to exclude the subject area, and then the image may be aligned between the frames. In addition, a method of extracting a subject area by extracting a person by machine learning or by dividing each frame into multiple parts and specifying a divided area including a moving body area based on a motion vector obtained in each divided area. There may be.

The head position detection unit 57 detects the head from the subject area extracted by the subject extraction unit 55 and calculates the coordinates. The head position detection unit 57 may also be used by the subject extraction unit 55.

The motion amount detection unit 58 reads out each image that has been continuously photographed and temporarily stored in the memory 32 to obtain the motion amount detection amount. For the motion amount detection amount, the difference between two images before and after within the time taken continuously in a time series is obtained. However, if the movement of the subject between two images before and after within the captured time is small, the area where the subject areas overlap is large, and only a part of the subject may be detected as a moving object. Images with different positions of subjects separated by more than one frame may be compared.

FIG. 2 is a diagram for explaining trajectory synthesis in the present embodiment.

In the example of the situation described below, it is assumed that a person slides down by a snowboard. The imager 230 takes an image of the subject 201 (a person gliding on a snowboard). The positions of the subject 201 when continuous imaging is performed 10 times at a shutter speed of 1/100 second are indicated by positions 211 to 220. Therefore, from each image obtained by continuous imaging, the subject extraction unit 55 detects a moving object 201 shown at positions 211 to 220, and the image synthesizing unit 54 synthesizes the extracted moving object into one still image. Perform processing. When synthesizing images, pixels may be overwritten or weighted addition may be performed. By weighting addition, the image can be made so that the background can be seen through. Further, a plurality of frame images used for compositing may be composited after being aligned with each other. As a result, in the locus synthesis, one still image having the form shown in FIG. 2 is generated.

Next, the method of trajectory synthesis in this embodiment will be described. FIG. 3 is a flowchart for explaining the locus synthesis in the present embodiment.

First, in step S301, the system control unit 50 acquires a candidate image to be combined. For example, in the case of the present embodiment, it is assumed that the captured image is stored in the recording medium 200. The captured image in which the captured image is stored in the recording medium 200 by operating the operation unit 70 is read out via the recording medium interface 18 and stored in the memory 32. The image data stored in the memory 32 is read again by the memory control unit 15 and the image processing unit 24, converted into image data for display and written to the memory 32, and then by the display unit 28 via the memory control unit 15. Is displayed. Further, by operating the image movement button included in the operation unit 70, a plurality of candidate images to be combined are selected. In this embodiment, a candidate image may be acquired by imaging, although the image is selected from the images already captured. For example, a method of performing continuous shooting imaging and acquiring an image captured immediately after continuous shooting as a candidate image to be combined may be used.

Next, in step S302, the system control unit 50 acquires subject information for each of the candidate images. In the case of the present embodiment, the subject extraction unit 55 calculates the subject area for each of the candidate images, and the system control unit 50 calculates the evaluation value for the subject area and uses it as the subject information. For example, the system control unit 50 calculates the evaluation value using at least one of the area of the subject area, the aspect ratio of the subject area, and the position of the head with respect to the subject area. Here, when the area of the subject area is used, the number of pixels determined to be the subject area is used as the subject information. However, when the subject area has a complicated shape, it may be calculated by drawing a rectangle surrounding the subject area and calculating with the area instead. When using the aspect ratio of the subject area, the aspect ratio may be calculated by dividing the vertical length of the rectangle surrounding the subject area by the horizontal length. Further, when the position of the head with respect to the subject area is used, the position of the head calculated by the head position detecting unit 57 may be calculated by the relative coordinates with respect to the rectangle surrounding the subject area and used as an evaluation value. However, the subject information of the head position may be calculated by another method such as calculation using whether the head is present in the upper half or the lower half of the rectangle surrounding the subject area as an index.

Further, in step S303, the system control unit 50 compares the subject information of all the images and extracts the feature area. In the case of this embodiment, the feature image is automatically determined based on the evaluation value calculated in step S202. The subject area of the image having the maximum evaluation value or the image having the minimum evaluation value may be extracted as a feature area. Alternatively, an image in which the evaluation value changes to a predetermined value or more from the previous image may be extracted, and in this case, the feature area becomes a plurality of features. By the above processing, for example, an image of a person jumping and spreading his hands wide, an image of landing and shrinking his body, an image of a cartwheel with his head down Characteristic images can be extracted. In addition, the feature region may be extracted by using subject information other than the above-mentioned evaluation value.

Finally, in step S304, the image synthesizing unit 54 synthesizes so that the feature image is included. For example, when the feature images are the first, ninth, and thirteenth of the candidate images, the thinning interval may be unified to four, and the fifth image may be added to compensate. Further, the feature image may be fine-tuned when the thinning interval cannot be made constant.

An example of adjusting the thinning interval will be described below.

FIG. 4 is a diagram for explaining the display of the display unit 28 in the present embodiment. The screen 401 displays the composite image synthesized in step S304. The slide bar 402 is used for adjusting the thinning interval. Bar 403 displays images having feature regions in chronological order. The bar 403 indicates the position of the image having the feature region in time series. Each time the user touches the slide bar 402, the image synthesizing unit 54 creates a composite image again. For example, if the candidate images are 41 images, the feature images are the 21st and 41st images, and the thinning interval is 10 images, the 1st, 11th, 21st, and 31st images are combined. , 41st sheet is synthesized. In that state, when the user operates the slide bar 402 and sets the thinning interval to 5, for example, the 1st, 6th, 11th, 16th, 21st, and 26th sheets, The 31st, 36th, and 41st sheets are recombined.

The slide bar 402 is also used to change an image having a feature area. In this case, each time the user touches the slide bar 402 to change the image having the feature area, the image synthesizing unit 54 creates a composite image again.

As described above, the present embodiment can be used to automatically create an image of trajectory composition according to the user's intention.

(Other embodiments)
Although the above embodiment has been described based on the implementation using a digital camera, the present embodiment is not limited to the digital camera. For example, it may be carried out by a portable device having a built-in image sensor, or may be a network camera capable of capturing an image.

In the present invention, a program that realizes one or more functions of the above-described embodiment is supplied to a system or device via a network or a storage medium, and one or more processors in the computer of the system or device program. It can also be realized by the process of reading and operating. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

22 Imaging unit 28 Display unit 50 System control unit 100 Digital camera 101 Shutter 102 Barrier

Claims (12)

An acquisition method for acquiring subject information from multiple images,
An extraction means for extracting a feature region from the subject information based on a predetermined rule,
It has a compositing means for creating a composite image representing a trajectory of a subject in the plurality of images by using at least a part of the plurality of images.
An image processing device characterized in that the composite image includes the feature region. The image processing apparatus according to claim 1, wherein the compositing means creates the composite image using an image having the feature region among the plurality of images. The image processing apparatus according to claim 1 or 2, wherein the predetermined rule relates to a subject area in the image. The image processing apparatus according to claim 3, wherein the extraction means extracts the feature area based on the area or aspect ratio of the subject area. The image processing apparatus according to claim 3, wherein the extraction means extracts the feature region based on the position of the head of the subject region. The image processing apparatus according to claim 3, wherein the extraction means extracts the feature region based on a difference in the subject regions of the images adjacent to each other in time series. It also has a changing means to change the settings based on the user's instructions.
The image processing apparatus according to any one of claims 1 to 6, wherein when the changing means changes the setting, the compositing means creates the composite image again. The image processing apparatus according to claim 7, wherein the changing means changes the thinning of the image used for the composite image. The image processing apparatus according to claim 7, wherein the changing means changes the feature region. An imaging means that captures multiple images and
An acquisition means for acquiring subject information from the plurality of images, and
An extraction means for extracting a feature region from the subject information based on a predetermined rule,
It has a compositing means for creating a composite image representing a trajectory of a subject in the plurality of images by using at least a part of the plurality of images.
The composite image is an imaging device including the feature region. The acquisition step to acquire subject information from multiple images,
An extraction step of extracting a feature area from the subject information based on a predetermined rule, and
It has a compositing step of creating a composite image representing a trajectory of a subject in the plurality of images using at least a part of the plurality of images.
An image processing method characterized in that the composite image includes the feature region. A computer program that operates an image processing device on a computer.
The acquisition step to acquire subject information from multiple images,
An extraction step of extracting a feature area from the subject information based on a predetermined rule, and
Using at least a part of the plurality of images, a compositing step of creating a composite image representing the trajectory of the subject in the plurality of images is performed.
The composite image is a program characterized by including the feature region.

Images