JP2012010133A - Image processing apparatus and image processing program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To properly determine a highlight period and to set it.SOLUTION: When setting a highlight period which indicates an essential point of a series of images into the series of images by processing a series of images consecutively photographed, characteristic quantities per image is extracted (S21), variation quantities per unit time of characteristic quantities per the image is detected (S22), and the highlight period is determined (S23) based on the variation quantities.

Description

  The present invention relates to an image processing apparatus and an image processing program.

  When playing back videos taken with a digital video camera, the most prominent scenes, the most brilliant scenes, and scenes that show off are detected as priority scenes in order to grasp the contents effectively in a short time. There is known an image processing apparatus that displays the image (for example, see Patent Document 1).

JP 2007-134771 A

  However, the above-described conventional image processing apparatus has a problem that a scene detected as a highlight scene is not necessarily a scene requested by the user, and an appropriate highlight scene is not determined.

  When setting a highlight period that represents the main points of a series of images by processing a series of consecutively shot images, the feature quantity for each image is extracted and the feature quantity for each image The amount of change per unit time is detected, and the highlight period is determined based on the amount of change.

  According to the present invention, the highlight period can be appropriately determined and set.

The figure which shows the structure of the digital camera carrying the image processing apparatus of one embodiment Flow chart showing highlight period setting program during video recording Flow chart showing highlight period setting program during video playback Flowchart showing highlight determination subroutine The figure which shows the example of highlight judgment at the time of video shooting Flow chart showing special reproduction processing program The figure which shows an example of special reproduction processing The figure which shows the other example of special reproduction processing The figure which shows the other example of special reproduction processing

  An embodiment in which the image processing apparatus of the present invention is mounted on a digital camera capable of shooting a moving image will be described.

  FIG. 1 is a diagram illustrating a configuration of a digital camera according to an embodiment. It should be noted that illustration and description of camera devices and apparatuses that are not directly related to the present invention are omitted. The photographing optical system 1 forms a subject image on the image pickup surface of the image pickup device 2. In addition to the photographic lens and the diaphragm, the photographic optical system 1 includes a lens, a diaphragm drive control device, a memory for storing optical system information, and the like. The image sensor 2 is an image sensor such as a CCD or a CMOS, and converts a subject image into an image signal. The A / D converter 3 converts an analog image signal into a digital image signal. The signal processing circuit 4 performs various processes such as a clamp process, a gain process, a pixel interpolation process, and a color conversion process on the image signal. The buffer memory 5 is a memory for temporarily storing images.

  The compression recording circuit 6 compresses the image signal in a predetermined format and records it on a recording medium 7 such as a memory card. The special reproduction processing circuit 8 adds a special effect to the highlight period in the moving image. Details of this special reproduction processing will be described later. The read / expand circuit 9 reads and expands an image compressed and recorded on the recording medium 7. The display control circuit 10 displays an image including a moving image on the monitor 11. The control device 12 includes a CPU 12a, performs various calculations and sequence control of the camera, determines a highlight period from the moving image, and performs special reproduction processing on the highlight period. The operation unit 13 is an operation member for operating the camera, and includes a release switch, a menu switch, a cross key switch, a highlight switch, a reproduction switch, and the like.

  2 to 4 and 6 are flowcharts showing a control program for highlight period setting and special reproduction processing of the image processing apparatus according to the embodiment. These programs are executed by the CPU 12a of the control device 12. The operation of the embodiment will be described with reference to these flowcharts.

  FIG. 2 shows a highlight period setting program at the time of moving image recording. While the release switch of the operation unit 13 is turned on and a moving image is being shot and recorded, the CPU 12a of the control device 12 repeatedly executes the program shown in FIG. In step 1, it is determined whether or not the highlight switch of the operation unit 13 has been operated. The highlight switch of the operation unit 13 is a switch that is operated by the photographer assuming that this is a highlight scene at the time of photographing, and for the photographer himself to set a highlight period. Note that even when the highlight mode, which will be described in detail later, is not set, the operation of the highlight switch has priority, and when the highlight switch is operated, the highlight period is set even when the highlight mode is not set. When the highlight switch is on, the process proceeds to step 2, and a highlight flag is set for the image taken when the highlight switch is on, and the image is recorded on the recording medium 7 together with the image data.

  Here, the highlight period is the most prominent image, most detailed, in order to grasp the contents of a series of images effectively in a short time in a series of images taken by moving image shooting or continuous shooting of still images. It is set as an image or a group of images representing the main points of a series of images, such as a certain image or an image of a show.

  When the highlight switch of the operation unit 13 is not turned on, the process proceeds to step 3 to determine whether or not the highlight mode is set. The image processing apparatus according to the embodiment has a mode in which a highlight period is automatically determined and a highlight flag is set, that is, a highlight mode, when a moving image is captured and recorded. In the highlight mode, the menu screen of the operation unit 13 is operated to display a menu screen on the monitor 11, and the cross key of the operation unit 13 is operated to set and cancel the highlight mode. If the highlight mode is set, the process proceeds to step 4 to execute a highlight determination subroutine shown in FIG. 4 to determine a highlight period and set a highlight flag. Details of this highlight determination processing will be described later. When the highlight mode is not set, step 4 is skipped. In step 5, it is determined whether or not the release switch of the operation unit 13 is turned on, that is, whether or not the moving image shooting and recording operation is continued. If it is not repeated, the process is terminated.

  FIG. 3 shows a highlight period setting program at the time of moving image reproduction. While the playback switch of the operation unit 13 is turned on and the moving image playback operation is being performed, the CPU 12a of the control device 12 repeatedly executes the program shown in FIG. In step 11, it is determined whether or not the highlight switch of the operation unit 13 has been operated. As described above, the highlight switch of the operation unit 13 is a switch for the photographer himself to set the highlight period, and priority is given to the operation of the highlight switch even when the highlight mode is not set. Even for a moving image for which a highlight period is not set at the time of recording and recording a moving image, the highlight period is set again by operating the highlight switch of the operation unit 13 while the moving image is being reproduced on the monitor 11. be able to. When the highlight switch is on, the process proceeds to step 12, and a highlight flag is set for the image reproduced when the highlight switch is on, and the image is recorded on the recording medium 7 together with the image data.

  When the highlight switch of the operation unit 13 is not turned on, the process proceeds to step 13 to determine whether or not the above-described highlight mode is set. The image processing apparatus according to the embodiment has a mode in which a highlight period is automatically determined and a highlight flag is set when a moving image is reproduced, that is, a highlight mode. In the highlight mode, the menu screen of the operation unit 13 is operated to display a menu screen on the monitor 11, and the cross key of the operation unit 13 is operated to set and cancel the highlight mode. When the highlight mode is set, the process proceeds to step 14, and the highlight determination subroutine shown in FIG. 4 is executed to determine the highlight period and set the highlight flag. Details of this highlight determination processing will be described later. If the highlight mode is not set, step 14 is skipped. In step 15, it is determined whether or not the playback switch of the operation unit 13 is turned on, that is, whether or not the video playback operation is continued. If so, the process returns to step 11 to repeat the above-described processing. If it is not continued, the process is terminated.

  FIG. 4 shows a highlight determination subroutine. In step 21, the feature amount of the image is extracted for each frame image in the moving image. As the feature amount, for example, a motion vector amount in an image indicating the motion of the main subject may be detected. Further, the spatial frequency in the image indicating the focus state with respect to the main subject may be detected. Alternatively, the luminance, shape, hue, etc. of the main subject in the image may be detected and used as the feature amount. Note that, as the main subject in the image, for example, a person's face may be recognized as a main subject by face recognition processing, or a subject indicating a defocus amount closest to the focus detection result may be used as the main subject. The above-described feature amounts are detected for the frame images that are shot and recorded in time series in moving image shooting, and stored in the buffer memory 5 in order.

  In the subsequent step 22, the amount of change in the feature amount per unit time is detected. The feature amount for each frame image stored in the buffer memory 5 is read, and the change amount per unit time of the feature amount in the continuous images is obtained. In step 23, it is determined whether or not the amount of change of the feature amount per unit time exceeds a preset threshold value. If the amount of change of the feature amount exceeds the threshold value, the process proceeds to step 24. A highlight flag is set for the image currently being processed, and is recorded on the recording medium 7 together with the image data. If the change amount of the feature amount is equal to or less than the threshold value, step 24 is skipped.

  FIG. 5 shows an example of highlight determination in the case where a moving image of a competition scene is taken. FIG. 5 (a) shows scenes 0 to 4 at the start, running and goal in the competition, and FIG. 5 (b) shows the amount of change per unit time of the motion vector amount of the main subject in each scene, that is, the runner. Show. In the scene 0 at the start, since there is almost no movement of the runner, the motion vector amount is almost zero. In scene 1 immediately after the runner starts, the amount of motion vector of the runner increases rapidly. In the running scene 2, the amount of motion vector of the runner is almost constant and changes little. In scene 3 immediately after the goal, the amount of motion vector of the runner decreases rapidly. In the scene 4 in which the runner moves to a predetermined place after the goal, the runner's motion vector amount is a minute amount corresponding to the walking movement.

  In this embodiment, a highlight period is a period in which the amount of change per unit time of the motion vector amount of the main subject exceeds a preset threshold value. However, when the amount of change per unit time of the motion vector amount exceeds the threshold, the main subject has already started moving, and when the amount of change per unit time of the motion vector amount falls below the threshold Since then, the main subject has continued to move. Therefore, in this embodiment, as shown in FIG. 5B, a period including a predetermined period before and after a period in which the amount of change per unit time of the motion vector amount of the main subject exceeds the threshold value is included. Set to highlight period.

  FIG. 6 shows a special reproduction processing program. When the menu switch of the operation unit 13 is operated and the special reproduction process is selected by the cross key, the CPU 12a of the control device 12 executes the special reproduction process program. In step 31, it is determined whether or not a highlight flag is set for each frame image while reproducing a moving image on the monitor 11. If there is a frame image for which the highlight flag is set, the process proceeds to step 32 to perform special reproduction processing. Details of the special reproduction process will be described later. In step 33, it is determined whether or not the reproduction of the moving image is continued. When the reproduction is continued, the process returns to step 31 to repeat the above-described processing, and when the reproduction is not continued, the processing is terminated.

  FIG. 7 shows an example of special reproduction processing. In this example, the main subject is extracted from the image within the highlight period by the above-described method, the brightness of the main subject is increased, the brightness of the background is decreased, and an effect of making the main subject stand out is added.

  FIG. 8 shows another example of special reproduction processing. In this example, in order to perform slow motion playback during the highlight period, a new frame is generated based on the images of the frames before and after the captured moving image frame, thereby enabling smooth slow motion playback. In FIG. 8, (a) shows normal playback frames (1), (2), (3), (4),..., (B) based on the previous and next frame images between normal playback frames. A new frame is created and added. Further, FIG. 8C is a diagram in which a new frame is generated and added based on the previous and next frame images between the playback frames shown in FIG. 8B, which is smoother than the 1/2 playback shown in FIG. 8B. Slow motion playback can be realized.

  FIG. 9 shows another example of special reproduction processing. In this example, in order to perform slow motion playback during the highlight period, images with a high shutter speed are acquired between moving image frames during moving image shooting and recording, and slow motion playback is performed using all these image frames. In FIG. 9, moving images are taken at a normal shutter speed, for example, 1/60 to 1/250 [sec], and normal image frames (1), (2), (3), (4),. In the meantime, a plurality of still images are taken at a high shutter speed, for example, 1/1000 [sec]. During normal playback, only normal image frames (1), (2), (3), (4),... Are played back. During slow motion playback during the highlight period, normal image frames and high-speed shutters are played back. Play back both still images taken at speed. By performing slow motion playback using a frame image shot at such a high shutter speed, it is possible to perform sharp and smooth slow motion playback without blurring of the outline.

  In the above-described embodiment, an example in which a highlight period is automatically set or manually set for a moving image has been described. However, a highlight period is set for a large number of still images taken continuously (continuous shooting). Automatic detection or manual setting can also be performed.

  In the embodiment described above, an example in which the image processing apparatus of the present invention is mounted on a digital camera has been shown. However, the image processing apparatus of the present invention may be a highlight setting dedicated device, or the image processing of the present invention. The device may be mounted on a moving image playback device such as a DVD. Alternatively, the image processing program of the present invention is executed on a personal computer, and the highlight period is automatically set or manually set for a moving image or continuous shot image captured in the personal computer, and the above-described special reproduction is performed during the highlight period. Processing may be performed.

  In the above-described embodiments and their modifications, all combinations of the embodiments and the modifications are possible.

  According to the above-described embodiment and its modifications, the following operational effects can be achieved. First, when processing a series of consecutively captured images and setting a highlight period that represents the main points of the series of images in the series of images, a feature amount for each image is extracted, and for each image, Since the change amount of the feature amount per unit time is detected and the highlight period is determined based on the change amount, the highlight period can be appropriately determined and set.

  In addition, according to the embodiment, since the main subject is extracted from the image and the motion vector amount of the main subject is detected as the feature amount, it is possible to appropriately highlight without being affected by the background change. A period can be set.

  Furthermore, according to the embodiment, since the highlight switch for manually setting the highlight period in any period in the series of images is provided, the highlight period is not automatically set by the image processing apparatus. The highlight period can be arbitrarily set by the user even in a place where the highlight is not performed.

  According to one embodiment, the highlight period can be set not only when shooting a series of images but also when playing back a series of images. You can set or add a highlight period.

  According to an embodiment, as a computer program that executes a process of processing a series of consecutively shot images and setting a highlight period that represents a point of the series of images in the series of images, Since there is an extraction step for extracting the feature amount for each image, a detection step for detecting the change amount per unit time of the feature amount for each image, and a determination step for determining the highlight period based on the change amount. The highlight period can be appropriately determined and set on an image reproduction device or a computer.

2; Image sensor, 11; Monitor, 12; Control device, 12a; CPU, 13;

Claims (6)

An image processing apparatus that processes a series of images taken continuously and sets a highlight period that represents a main point of the series of images in the series of images,
Extracting means for extracting a feature amount for each image;
Detecting means for detecting a change amount per unit time of the feature amount for each image;
An image processing apparatus comprising: a determination unit that determines the highlight period based on the change amount. The image processing apparatus according to claim 1.
The image processing apparatus, wherein the extraction unit extracts a main subject from the image and detects a motion vector amount of the main subject as the feature amount. The image processing apparatus according to claim 1 or 2,
An image processing apparatus comprising: an operation member for manually setting the highlight period in an arbitrary period in the series of images. In the image processing device according to any one of claims 1 to 3,
An imaging means for capturing the series of images by moving image shooting or continuous shooting of still images,
An image processing apparatus, wherein the highlight period is set when the series of images is captured. In the image processing device according to any one of claims 1 to 4,
Display means for displaying the image;
An image processing apparatus, wherein the highlight period is set when the display unit reproduces the series of images. An image processing program that causes a computer to execute a process of processing a series of images that are continuously captured and setting a highlight period in the series of images that represents key points of the series of images,
An extraction step of extracting a feature amount for each image;
A detection step of detecting a change amount per unit time of the feature amount for each image;
An image processing program causing a computer to execute a determination step of determining the highlight period based on the amount of change.

Images