JP2010178318A - Imaging device and imaging method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To properly extract a representative image useful for a user from consecutive images. <P>SOLUTION: A digital camera 1 includes an imaging element 12 and continuously performs imaging to generate a plurality of image data, and obtains consecutive images. In the digital camera 1, a motion detecting unit 15 detects motion between captured images constituting the consecutive images and detects a moving object appearing in the consecutive images. A consecutive shooting scene classifying unit 281 identifies a change of state of the moving object in the consecutive images based on detection results by the motion detecting unit 15, and classifies the consecutive images into consecutive shooting scenes. A consecutive image dividing unit 283 divides the consecutive images at positions corresponding to the consecutive shooting scenes having been classified by the consecutive shooting scene classifying unit 281, and extracts the captured image at the divided position as a representative image. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to a photographing apparatus and a photographing method that perform continuous photographing processing to acquire continuous images.

  2. Description of the Related Art Conventionally, there has been known a photographing apparatus such as a digital camera that has a continuous photographing mode (continuous shooting mode) for continuously photographing a subject in a short time. In this continuous shooting mode, captured images are sequentially generated at a predetermined shooting interval, and finally a continuous image composed of a predetermined number of continuous shooting images is obtained.

  By the way, the photographed images photographed in the continuous shooting mode are sequentially browsed by the user, and all or a part selected as necessary is stored. For this reason, when the number of continuous shots increases, the user must spend time to view the captured image, and the work burden for selecting and storing a desired image increases.

  In order to solve this type of problem, various techniques for extracting a representative image (representative image) from a plurality of consecutively photographed images have been proposed as techniques for presenting an image useful for the user. ing. For example, in Patent Document 1, a feature amount of an image is extracted and evaluated, and the best image worth printing is automatically extracted. In Patent Document 2, attached information such as digital camera status information at the time of shooting is stored in association with an image, and a single print in which images suitable for printing are arranged by referring to the attached information at the time of printing. Data is being acquired. In Patent Document 3, a plurality of pieces of image data are analyzed to detect the movement of a subject, and image data to be stored is selected based on the detected movement.

JP 2007-88594 A JP 2007-81973 A JP 2004-180076 A

  By the way, the continuous shooting mode provided in the photographing apparatus is normally used when photographing a moving subject (moving body). Therefore, in the continuous image obtained in the continuous shooting mode, which captured image is useful depends on how the moving body moves in the continuous image. For this reason, for example, in a technique for calculating a feature amount of a single image as in Patent Document 1, and a technique for extracting an image based on camera state information at the time of shooting as in Patent Document 2, In some cases, the representative image could not be extracted properly. Further, in the method of Patent Document 3, although the movement between adjacent images is taken into consideration, how the moving body moves as a whole continuous image is not taken into consideration.

  The present invention has been made in view of the above, and an object thereof is to provide a photographing apparatus and a photographing method capable of appropriately extracting a representative image useful for a user from continuous images.

  In order to solve the above-described problems and achieve the object, an imaging apparatus according to the present invention includes a continuous image acquisition unit that performs continuous imaging processing to acquire continuous images composed of image data of a plurality of captured images. , For each of the plurality of captured images, detecting a movement between adjacent captured images to detect a moving body in the captured image, and based on a detection result by the moving body detection unit A change discriminating unit that discriminates a state change of the moving body in the continuous image, and an image extracting unit that extracts a representative image from the continuous image based on a discrimination result by the change discriminating unit. It is characterized by providing.

  In the imaging device according to the present invention, in the above invention, the change determination unit includes at least one of a change in a moving direction of the moving body, a change in a moving speed of the moving body, and a change in the size of the moving body. It is characterized by discriminating one or the other.

  In the imaging device according to the present invention, in the above invention, the image extraction unit divides the continuous image at a position corresponding to a determination result by the change determination unit, and at least the captured image at the divided position is It is extracted as a representative image.

  In the imaging device according to the present invention, in the above invention, the change determination unit determines a state in which the moving direction of the moving body changes regularly according to a predetermined change pattern, and the image extraction unit When the regular change is discriminated by the change discriminating unit, the continuous image is divided at a position where the moving direction of the moving body has changed.

  In the imaging device according to the present invention, in the above invention, the change determination unit is configured such that the moving body moves along a predetermined direction without changing the direction or the moving direction of the moving body is irregular. The changed state is determined, and the image extracting unit determines whether the moving body starts moving from the moving start position of the moving body when the change determining section determines a change along the predetermined direction or the irregular change. The continuous image is divided at an intermediate position on the movement path to the movement end position.

  In the imaging device according to the present invention, in the above invention, the change determination unit is configured such that the moving body moves along a predetermined direction without changing the direction or the moving direction of the moving body is irregular. The changed state is determined, and when the change determining unit determines a change along the predetermined direction or the irregular change, the moving body is closest to the center of the shooting angle of view. The continuous image is divided at the positions.

  In the imaging device according to the present invention, in the above invention, the moving body detection unit is configured to be able to detect a plurality of moving bodies, and the change determination unit has a state in which the moving bodies overlap each other. The overlapped state is determined, and the image extraction unit divides the continuous image at a position where the moving bodies overlap each other.

  In the imaging device according to the present invention, in the above invention, the moving body detection unit is configured to be able to detect a plurality of moving bodies, and the change determination unit is any of the plurality of moving bodies. A state in which the moving body starts moving from the position of one moving body is determined as a split state, and the image extraction unit divides the continuous image at a position where the moving body is split. And

  In the imaging device according to the present invention, in the above invention, the image extraction unit further divides the continuous image at at least one of a movement start position of the moving body and a movement end position of the moving body. It is characterized by that.

  Further, the imaging device according to the present invention is characterized in that, in the above invention, the image extraction unit extracts a predetermined number of captured images adjacent to the captured images at the divided positions as the representative images. To do.

  In the imaging device according to the present invention, in the above invention, the image extracting unit divides the moving section of the moving body into one or more sections according to a determination result by the change determining section, At least one section is divided.

  Moreover, the imaging device according to the present invention is characterized in that, in the above-mentioned invention, a display processing unit that performs processing for displaying the representative image on a display unit is provided.

  In the imaging device according to the present invention, in the above invention, the display processing unit displays each representative image in a list or in order of shooting when a plurality of the representative images are extracted by the image extraction unit. It is characterized in that the processing is performed.

  In addition, the imaging method according to the present invention includes a continuous image acquisition step of continuously performing imaging processing to acquire a continuous image composed of image data of a plurality of captured images, and the plurality of captured images adjacent to each other. A moving body detecting step of detecting movement between the picked-up images and detecting a moving body in the picked-up image; and a detection result of the moving body detecting step based on a detection result of the moving body. A change determining step for determining a state change; and an image extracting step for extracting a representative image from the continuous image based on a determination result obtained by the change determining step.

  According to the present invention, motion detection is performed between adjacent captured images to detect a moving object that appears in a continuous image, and a state change of the detected moving object in the continuous image is determined, thereby determining the state of the continuous image. A representative image can be extracted from the inside. Therefore, a representative image useful for the user can be appropriately extracted.

  DESCRIPTION OF EMBODIMENTS Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the drawings. In this embodiment, a case where the photographing apparatus of the present invention is applied to a digital camera will be described as an example. Note that the present invention is not limited to the embodiments. Moreover, in description of each drawing, the same code | symbol is attached | subjected and shown to the same part.

  Here, the digital camera of this embodiment includes a continuous shooting mode as one of the shooting modes. In this continuous shooting mode, for example, the continuous shooting is started at the timing when the start of continuous shooting is instructed, and the shot images are sequentially generated at a predetermined shooting interval, so that it is finally composed of a predetermined number of shot images. Acquire continuous images. Note that the number of continuous shots is determined according to a user operation, for example. For example, a configuration may be adopted in which continuous shooting is terminated in response to an operation input during the continuous shooting mode, or a continuous image may be acquired in accordance with the number of continuous shooting operations input in advance. In the present embodiment, a captured image that is estimated to be useful to the user is extracted as a representative image from the continuous images acquired in the continuous shooting mode, and presented to the user. In the present embodiment, it is assumed that a subject (moving body) that moves is included in the continuous shot image.

(Embodiment 1)
In the first embodiment, a state change of a moving body that appears in the acquired continuous image is determined, and the continuous shooting scene is classified. Then, continuous images are divided according to the classified continuous shooting scenes, and representative images are extracted based on the division positions. Hereinafter, a case where the continuous shooting scene is classified into five scenes a to e will be described. The illustrated continuous shooting scene classification method is an example, and can be set as appropriate.

  For example, when a moving body moves in a continuous image along a predetermined direction without changing its direction (when the moving direction is a constant direction), the continuous shooting scene is set as a scene a. If the moving direction of the moving body is irregular and not stable (when the moving direction is unstable), the continuous shooting scene is set as a scene b. In addition, when it is determined that the moving direction is regularly changed according to a predetermined change pattern, the continuous shooting scene is set as a scene c. Further, when a plurality of moving objects are detected in the continuous image and it is determined that these moving objects overlap each other, the continuous shooting scene is set as a scene d. When a plurality of moving objects are detected in the continuous image and it is determined that one moving object is divided into two or more, the continuous shooting scene is set as a scene e.

  FIG. 1 is a diagram showing an example of continuous images classified into the scene a, and shows five captured images I11 to I19 constituting the continuous image in chronological order (imaging order). In FIG. 1, a bird O1 is shown as a moving object, and a bird (moving object) O1 that has stopped on a branch of a tree starts to flapping, jumps off the branch, and is shown out of the frame. Here, the moving body O1 is moving on a substantially straight line from the lower right to the upper left in the image. As described above, in the continuous shooting scene in which the moving body O1 moves along a predetermined direction, the captured image I11 that captures the state in which the moving body O1 is stopped and the captured image I19 that captures after the moving body O1 is out of frame are displayed. In comparison, it can be estimated that the photographed image I13 at the position where the moving body O1 starts moving, the photographed image I15 taken during the movement, and the photographed image I17 at the position out of the frame are useful for the user.

  FIG. 2 is a diagram illustrating another example of the continuous images classified into the scene a, and shows the five captured images I21 to I29 constituting the continuous image in time series order (imaging order). FIG. 2 shows a car O2 as a moving body, and shows a state where the car (moving body) O2 that appears from the right side moves toward the left side and goes out of the frame. In the example of FIG. 2, the captured image I21 at the position where the moving body O2 has started moving (in this case, the frame-in position), the captured image I25 positioned in the middle of the screen, particularly near the center of the screen, and the position out of the frame. It can be estimated that the captured image I29 is useful to the user.

  Therefore, for a continuous image in which the continuous shooting scene is classified into the scene a, the movement start position of the moving body, the intermediate position on the movement path of the moving body (hereinafter referred to as “movement center position”), and the movement end position ( The representative image is presented to the user by dividing the continuous image using the frame-out position as a division position and extracting the captured image at the division position as the representative image.

  According to this, for example, in the example of FIG. 1, the captured image I13 at the start of movement of the moving body O1, the captured image I15 in the middle of movement (moving center position), and the captured image I17 at the frame-out position can be presented as representative images. it can. Therefore, the user can grasp the content of the continuous image from the presented representative image. Then, for example, it is possible to select one of these representative images and browse the captured images whose shooting order is adjacent to each other in order from the selected representative image. Accordingly, since it is not necessary to sequentially browse all the captured images constituting the continuous image from the top, the state in which the moving body O1 as shown in the captured image I11 stays on the tree branch continues for a long time. Even if it exists, a desired picked-up image can be selected easily and a user's work burden can be reduced.

  FIG. 3 is a diagram illustrating an example of continuous images classified into the scene c, and shows the five captured images I31 to I39 constituting the continuous image in time series order (imaging order). In FIG. 3, a dolphin O3 is shown as a moving object, and a state where the dolphin (moving object) O3 that has emerged from the water surface moves along a parabola and passes through a ring at the apex position of the parabola and then dives on the water surface. Yes. In this example, it is presumed that the photographed image I31 at the movement start position and the photographed image I39 at the movement end position (disappearance position in FIG. 3) as well as the photographed image I35 through which the moving body O3 passes are useful for the user. .

  Therefore, for a continuous image in which the moving direction of the moving object is regularly changed according to a predetermined change pattern and the continuous shooting scene is classified into the scene c, the moving start position of the moving object and the change of the moving direction of the moving object are changed. The continuous image is divided by using the position (the vertex position if the moving body O3 illustrated in FIG. 3 moves along the parabola), the movement end position or the frame-out position as a division position, and the captured image at the division position is representative. Extract as an image. In addition, although the case where the moving body moved along a parabola is illustrated here, the present invention is not limited to this, and an appropriate change pattern is detected and a continuous image is divided at a position where the moving direction changes. As good as Further, when the direction changes at a plurality of positions, one or a plurality of positions may be selected from the positions to be divided positions, or all positions may be set as the divided positions.

  FIG. 4 is a diagram illustrating an example of the captured images classified into the scene d, and shows the five captured images I41 to I49 constituting the continuous image in time series order (imaging order). FIG. 4 shows a situation where a person O41, which is a moving body, batting a ball O43, which is another moving body. In this example, the captured image I41 at the movement start position (batting operation start position) of the moving object (person) O41, the captured image I43 at the movement start position (frame-in position) of the moving object (ball) O43, and the moving object. In addition to the shot image I47 that is the frame-out position of the (ball) O43, the shot image I45 at the moment of batting, that is, the position where the moving body (bat held by the person) O41 and the moving body (ball) O43 overlap is useful for the user. It is estimated to be.

  Therefore, for a continuous image in which a plurality of moving objects overlap in the continuous image and the continuous shooting scene is classified into the scene d, the movement start position, the overlapping position, the movement end position, or the frame-out position of the moving object. Is divided as a division position, and a captured image at the division position is extracted as a representative image. Examples of the state in which the moving bodies overlap each other include the case where the moving bodies intersect as well as the case where the moving bodies contact each other as illustrated.

  Although not shown, examples of the continuous image classified into the scene b include a continuous image in which a person who is ice skating as a moving body is captured. For a continuous image in which the continuous shooting scene is classified into the scene b, for example, similarly to the continuous image classified into the scene a, the movement start position, the movement center position, the movement end position, or the frame-out position of the moving object is divided into positions. Then, the continuous image is divided, and the captured image at the division position is extracted as a representative image.

  Moreover, as a continuous image classified into the scene e, the continuous image etc. which copied the skyrocket as a moving body are mentioned, for example. For a continuous image in which the moving object is split in the continuous image and the continuous shooting scene is classified into the scene e, for example, the moving start position, the split position, the moving end position, or the frame-out position of the moving object is set. A continuous image is divided as a division position, and a captured image at the division position is extracted as a representative image.

  FIG. 5 is a schematic block diagram illustrating a configuration example of the digital camera 1 according to the first embodiment. As shown in FIG. 5, the digital camera 1 includes an imaging optical system 11, an imaging element 12 as a continuous image acquisition unit, an AFE (Analog Front End) 13, a frame memory 14, a motion detection unit 15 as a moving object detection unit, Image processing unit 16, face detection unit 17, recording medium I / F 18, recording medium holding unit 19, recording medium 20, video encoder 21, display driver 22, display unit 23, video signal output terminal 24, operation unit 25, RAM 26, A ROM 27, a controller 28, and the like are provided.

  The imaging optical system 11 includes an imaging lens, a diaphragm, a shutter, and the like, and forms an incident subject image on the imaging element 12. The image pickup device 12 is a solid-state image pickup device such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor), for example, and receives a light beam from a subject via the image pickup optical system 11 and photoelectrically converts it into frame units. Image data (analog electrical signal) is obtained. The AFE 13 performs analog signal processing such as CDS (Correlated Double Sampling) and AGC (Automatic Gain Control) on the image data obtained by the image sensor 12, and then performs A / D conversion processing to convert it into a digital electrical signal. To do. The image data digitized by the AFE 13 is output to the frame memory 14 and the motion detection unit 15 and is temporarily recorded in the RAM 26.

  The frame memory 14 is used as a working memory by the motion detection unit 15. The frame memory 14 has an area for holding image data for two frames, and in the continuous shooting mode, the image data of the latest captured image (latest image) generated this time and the previously generated captured image (previous image) ) Image data.

  The motion detection unit 15 is for detecting a motion between captured images generated during the continuous shooting mode based on the image data from the AFE 13, and sequentially captures images at a continuous shooting timing for each predetermined shooting interval. The motion between images is detected by obtaining a motion vector between captured images taken in from the element 12 and output from the AFE 13.

  FIG. 6 is a diagram illustrating an example of the moving object detection area Ea and the registered moving object area Eb-1 set for the field angle range Ev of the captured image. 7 (a) and 7 (b) show the moving body detection area Ea and the registered moving body areas Eb-2 and Eb− that are set in the field angle range Ev of the captured images sequentially captured following FIG. It is a figure which shows an example of 3, respectively. In the first embodiment, as shown in FIG. 6, the motion detection unit 15 sets a moving body detection area Ea indicated by an alternate long and short dash line, for example, for almost the entire field angle range (subject range) Ev of the previous image. 6, FIG. 7 (a), and FIG. 7 (b), if a moving body has already been detected and registered, the current state of the moving body (registered moving body) is indicated. Registered mobile object areas Eb-1, Eb-2, and Eb-3 are set at the positions. Here, the moving body detection area Ea is fixedly set with respect to the field angle range Ev. On the other hand, as shown in FIGS. 6, 7 (a), and 7 (b), the registered mobile area Eb- 1, Eb- 2, Eb- 3 was executed with the previously generated captured image. It is set to the position of the registered moving body determined according to the result of motion detection.

  Then, the motion detection unit 15 detects a mobile body that appears in the latest image by performing motion detection on the mobile body detection area Ea, and also registers the mobile body areas Eb-1, Eb-2, and Eb-3. The position of the registered moving body in the latest image is tracked by performing motion detection on the target. Specifically, pattern matching is performed with the latest image for the moving body detection area Ea set for the previous image, and registered moving body areas Eb-1, Eb-2, and Eb-3 set for the previous image. Perform pattern matching with the latest image. Here, in order to improve the accuracy of pattern matching, a plurality of macroblocks are set in the mobile object detection area Ea and the registered mobile object areas Eb-1, Eb-2, and Eb-3, and the motion vector is set for each macroblock. To do so. For example, in the examples shown in FIG. 6, FIG. 7A, and FIG. 7B, 70 macroblocks Ba are set for the moving object detection area Ea. On the other hand, as schematically shown outside the angle of view range Ev in FIG. 6 or as shown in FIGS. 7A and 7B, the registered mobile body areas Eb-1, Eb-2, Eb For -3, the number of macroblocks Bb corresponding to the size is set. For example, six macroblocks Bb are set in the registered mobile area Eb-1 in FIG. 6 and the registered mobile area Eb-2 in FIG. 7A, and the registered mobile area in FIG. 7B. Four macroblocks are set in Eb-3. The number of macroblocks set in the registered mobile area may be the same. In addition, the size of the macro block set in the moving object detection area or the registered moving object area can be set as appropriate.

  Here, the case where the registered mobile body areas Eb-1, Eb-2, and Eb-3 are set for one registered mobile body and the registered mobile body is tracked is illustrated. Each time a moving object is detected by motion detection performed on the area Ea, the detected moving object is registered as a registered moving object, and the motion detecting unit 15 registers each registered moving object. Set the mobile area and perform motion detection individually.

  As shown in FIG. 5, the face detection unit 17 detects the face by processing the image data digitized by the AFE 13 and recorded in the RAM 26. For example, the face detection unit 17 detects the position of the face in the photographed image (more specifically, in the latest image) by the template matching method, and detects the position of the facial feature points such as eyes, nose, and mouth. To detect the size, orientation, angle, etc. of the face. By the face detection by the face detection unit 17, for example, the presence / absence of a face in the captured image, the face position, the face size, the face orientation, the face angle (tilt), and the like are obtained as face detection results. When a plurality of faces are included in the captured image, a face detection result is obtained for each face.

  The image processing unit 16 reads the image data once recorded in the RAM 26, performs various image processing on the image data, and performs processing for converting the image data into image data suitable for recording or display. For example, when recording image data of a photographed image or displaying recorded image data, image data compression processing or decompression processing based on a JPEG (Joint Photographic Experts Group) method or the like is performed. The image data image-processed by the image processing unit 16 is output to the recording medium I / F 18 and recorded on the recording medium 20, or is output to the video encoder 21 and displayed on the display unit 23.

  The video encoder 21 sends the image data converted for display to the display driver 22. For example, in the shooting mode, an image captured from the image sensor 12 for each frame and image-processed by the image processing unit 16 is switched and displayed on the display unit 23 in units of frames to display a live view image. On the other hand, in the reproduction mode, the captured image read from the recording medium 20 and image-processed by the image processing unit 16 is displayed on the display unit 23. The video encoder 21 outputs image data for display to an external device connected to the video signal output terminal 24 as necessary. The display unit 23 is for displaying various setting information of the digital camera 1 in addition to the captured image and the live view image, and is realized by a display device such as an LCD (Liquid Crystal Display) or an EL display (Electroluminescence Display). Is done.

  The recording medium I / F 18 performs writing of image data converted for recording, reading of recorded image data, and the like with respect to the recording medium 20 that is detachably held by the recording medium holding unit 19. . The recording medium 20 is a memory card such as an xD-picture card (registered trademark) or a compact flash (registered trademark) card.

  The operation unit 25 performs various operations by the user, such as a setting operation of various shooting modes including a continuous shooting mode and a playback mode, an instruction to start shooting, an instruction to start and stop continuous shooting, and an operation to set shooting conditions. This is for receiving an operation signal to the controller 28 and is realized by a button switch, a dial, various sensors, etc. to which various functions are assigned.

  The ROM 27 records various camera programs for operating the digital camera 1 and realizing various functions of the digital camera 1, data used during the execution of the camera program, and the like in advance. The RAM 26 is used as a working memory for the image processing unit 16 and the controller 28. For example, image data from the AFE 13 is temporarily recorded, and a continuous image is obtained for work when generating a live view image to be displayed on the display unit 23 or when recording a captured image on the recording medium 20. The photographed image selected as the stored image from the continuous images is used for work when recording on the recording medium 20.

  The controller 28 reads out and executes a camera program from the ROM 27 in accordance with an operation signal from the operation unit 25 and performs operation control and memory control of each unit constituting the digital camera 1 to control the entire operation of the digital camera 1. To control. In addition, processing such as AF (automatic focus), AE (automatic exposure), and AWB (automatic white balance) is performed. The controller 28 includes a continuous shooting scene classification unit 281 as a change determination unit, a continuous image division unit 283 as an image extraction unit, an importance calculation unit 285, and a saved image selection processing unit 287 as a display processing unit. Including. The continuous shooting scene classification unit 281 classifies continuous shooting scenes of continuous images. The continuous image dividing unit 283 divides a continuous image at a position corresponding to a continuous shooting scene, and extracts a captured image at the divided position as a representative image. The importance calculation unit 285 calculates the importance of each captured image constituting the continuous image. The stored image selection processing unit 287 presents a representative image extracted from the continuous images to the user, and performs processing for selecting a captured image (stored image) to be stored in the recording medium 20 according to the user operation.

  Next, a processing procedure performed by the digital camera 1 will be described. FIG. 8 is a flowchart for explaining a processing procedure performed by the digital camera 1 according to the first embodiment, and shows a processing procedure of the digital camera 1 in the continuous shooting mode.

  In the continuous shooting mode, as shown in FIG. 8, the continuous shooting mode is in a standby state until an instruction to start continuous shooting is received from the user. When continuous shooting is started (step a1: Yes), the process proceeds to shooting processing (step a3). That is, shooting processing is sequentially performed at a continuous shooting timing every predetermined shooting interval, and image data of a shot image is generated. The generated image data of the photographed image is recorded in the RAM 26 in association with, for example, an image number assigned in the photographing order.

  Subsequently, the motion detection unit 15 performs a motion detection process (step a5). In this motion detection process, the motion detection unit 15 performs motion detection between the latest captured image (latest image) generated this time and the previously generated captured image (previous image) in step a3, and moves in the latest image. The body is detected and the registered moving body in the latest image is tracked. Note that the motion detection process in step a5 is not performed after the first shooting process immediately after the start of continuous shooting because there is no previous image. FIG. 9 is a flowchart for explaining the detailed processing procedure of the motion detection process.

  In the motion detection process, the motion detection unit 15 first determines the presence or absence of a registered mobile object. Here, the moving body is registered in step b13 in FIG. 9 or in step c9 of the face detection process described later with reference to FIG. For this reason, there is no registered moving body in the motion detection process performed first after the continuous shooting is started (step b1: No), and the process proceeds to step b7.

  On the other hand, when there is a registered moving body (step b1: Yes), the motion detection unit 15 sets a registered moving body area for the registered moving body and executes motion detection between the latest image and the previous image. (Step b3). At this time, when a plurality of registered mobile bodies are registered, a registered mobile body area is set for each registered mobile body and motion detection is performed. Specifically, the motion detection unit 15 sets a registered mobile body area at the previously calculated position of the registered mobile body (position in the previous image), and executes motion detection with the latest image. Then, the motion detection unit 15 calculates the position of the registered moving body (current position in the latest image) according to the motion detection result, and stores it in the RAM 26 together with the image number of the latest image (step b5). Then, it moves to step b7.

  In step b7, the motion detection unit 15 performs motion detection in the moving object detection area between the latest image and the previous image. Specifically, the motion detection unit 15 sets a moving body detection area in almost the entire range of the angle of view, and performs motion detection with the latest image. If no moving object is detected as a result of the motion detection here (step b9: No), the process returns to step a5 in FIG. 8, and then proceeds to step a7. On the other hand, when detecting the moving body (step b9: Yes), the motion detecting unit 15 determines whether the detected moving body is a registered moving body. If the detected mobile body is a registered mobile body that has already been registered (step b11: Yes), the process returns to step a5 in FIG. 8, and then proceeds to step a7. On the other hand, if the detected moving body is not registered (step b11: No), the motion detecting unit 15 registers it as a registered moving body, and stores it in the RAM 26 together with the current position, the image number of the latest image, and the like (step). b13). And it returns to step a5 of FIG. 8, and moves to step a7 after that.

  In step a7, the face detection unit 17 performs face detection processing. In this face detection process, face detection in the latest image generated this time in step a3 is performed. FIG. 10 is a flowchart for explaining the detailed processing procedure of the face detection processing.

  In the face detection process, the face detection unit 17 first performs face detection in the latest image (step c1). If no face is detected as a result of the face detection here (step c3: No), the process returns to step a7 in FIG. 8, and then proceeds to step a9. On the other hand, when detecting the face (step c3: Yes), the face detection unit 17 stores the face detection result in the RAM 26 together with the image number of the latest image (step c5).

  Subsequently, the face detection unit 17 determines whether or not the detected face is a registered moving body. Specifically, the position of the detected face is compared with the current position (position in the latest image) of the registered mobile body, and if it matches, it is determined as a registered mobile body, and if it does not match, it is not a registered mobile body. Is determined. If the detected face is a registered moving body (step c7: Yes), the process returns to step a7 in FIG. 8, and then proceeds to step a9. On the other hand, when the detected face is not a registered moving body (step c7: No), the face detecting unit 17 registers it as a registered moving body and stores it in the RAM 26 together with the current position and the image number of the latest image (step c9). . Then, the process returns to step a7 in FIG. 8, and then proceeds to step a9.

  In step a9, it is determined whether or not continuous shooting is to be ended. Until continuous shooting is ended (step a9: No), the processing returns to step a3 and is repeated. When the continuous shooting is to be ended (step a9: Yes), the process proceeds to step a11. For example, continuous shooting is ended in response to an instruction to end continuous shooting by the user. Alternatively, continuous shooting is terminated when a predetermined number of continuous shot images have been generated.

  In step a11, it is determined whether or not the number of continuous shots exceeds a predetermined number. If the number of continuous shots is less than or equal to the predetermined number (step a11: No), the process is terminated. On the other hand, when the number of continuous shots exceeds the predetermined number (step a11: Yes), the continuous shot scene classification unit 281 performs a continuous shot scene classification process (step a13). FIG. 11 is a flowchart showing a detailed processing procedure of continuous shooting scene classification processing.

  In the continuous shooting scene classification process, the continuous shooting scene classification unit 281 first determines the presence or absence of a moving object detected in a continuous image. When a moving body is detected in a continuous image, that is, when there is a moving body registered as a registered moving body as a result of motion detection or face detection performed each time a captured image is generated (step d1: Yes), a moving body state analysis process is performed (step d3). FIG. 12 is a flowchart showing a detailed processing procedure of the moving body state analysis processing.

  In the moving object state analysis process, first, the continuous shooting scene classification unit 281 analyzes the movement start position of the moving object (step e1). The processing here is performed for each mobile body when there are a plurality of mobile bodies registered as registered mobile bodies. For example, a captured image that is first detected and registered as a registered moving body is set as a movement start position, and an image number is acquired as a movement start image number.

  Subsequently, the continuous shooting scene classification unit 281 analyzes the moving direction of the moving object (step e3). The processing here is performed for each mobile body when there are a plurality of mobile bodies registered as registered mobile bodies. For example, a change in the moving direction of the moving body is determined according to the moving path of the moving body determined by the result of motion detection performed using the moving body as a registered moving body or the face detection result of the moving body. And when the state which the moving body is moving along the predetermined | prescribed direction is discriminate | determined, a moving direction is made constant. If the moving direction of the moving body is not determined irregularly and an unstable state is determined, the moving direction is made unstable. Then, when it is determined that the moving direction of the moving body has regularly changed according to a predetermined change pattern, the image number of the captured image whose moving direction changes is acquired as the direction change image number. For example, when the moving path of the moving body O3 can be approximated to a quadratic curve as illustrated in FIG. 3, the captured image at the apex (extreme value) position is taken as a captured image whose moving direction changes, The image number is acquired as the direction change image number.

  Subsequently, the continuous shooting scene classifying unit 281 determines the number of moving objects detected in the continuous image. When the number of moving bodies is one (step e5: No), the process moves to step e11. On the other hand, when there are a plurality of detected moving objects (step e5: Yes), the continuous shooting scene classification unit 281 analyzes the overlapping state of these moving objects (step e7). For example, based on the result of motion detection performed for each mobile object as a registered mobile object and the face detection result for that mobile object, the state in which the positions of these mobile objects overlap is determined, and the image number of the determined captured image Is acquired as an overlapping image number.

  Subsequently, the continuous shooting scene classification unit 281 analyzes the division state of the moving object (step e9), and then proceeds to step e11. For example, a state in which another moving body starts moving from the position of one of the moving bodies as a starting point based on the result of motion detection performed for each moving body as a registered moving body or the face detection result for the moving body And the image number of the determined captured image is acquired as a split image number.

  In step e11, the continuous shooting scene classification unit 281 analyzes the frame-out position of the moving object. The processing here is performed for each mobile body when there are a plurality of mobile bodies registered as registered mobile bodies. For example, the moving body moves to the end of the field angle range, and a captured image outside the field angle range is set as the frame-out position, and the image number is acquired as the frame-out image number.

  Subsequently, the continuous shooting scene classification unit 281 analyzes the movement end position of the moving object (step e13). The processing here is performed for each mobile body when there are a plurality of mobile bodies registered as registered mobile bodies. For example, a captured image in which the moving body is stopped or a captured image in which the moving body cannot be tracked is set as the movement end position, and the image number is acquired as the movement end image number. Then, the process returns to step d3 in FIG. 11, and then proceeds to step d5.

  In step d5, the continuous shooting scene classification unit 281 determines the moving direction of the moving object according to the result of the moving object state analysis process in step d3. Then, the continuous shooting scene classifying unit 281 sets (ON) a constant direction flag for a moving body having a constant movement direction (step d5: Yes) (step d7). On the other hand, the continuous shooting scene classifying unit 281 is a moving body whose moving direction is not constant (step d5: No), and a moving body whose moving direction is not unstable (moving whose moving direction changes regularly according to a predetermined change pattern). Body) (step d9: No), the direction change flag is set (step d11). The continuous shooting scene classifying unit 281 sets a direction unstable flag (step d13) for a moving object whose movement direction is unstable (step d9: Yes).

  Subsequently, the continuous shooting scene classification unit 281 determines the overlapping of the moving objects according to the result of the moving object state analysis process in Step d3. Then, if there are overlapping moving bodies (step d15: Yes), the continuous shooting scene classification unit 281 sets an overlapping flag (step d17), and sets a continuous shooting scene of continuous images as the scene d (step d19). . When there is no overlap of moving objects (step d15: No), the continuous shooting scene classifying unit 281 subsequently determines the division of the moving objects. Then, if there is a division of the moving body (step d21: Yes), the continuous shooting scene classification unit 281 sets a division flag (step d23), and sets the continuous shooting scene of the continuous image to the scene e (step d25). . When there is no division of the moving body (step d21: No), the continuous shooting scene classification unit 281 determines the direction unstable flag. If the direction instability flag is ON (step d27: Yes), the continuous shooting scene classification unit 281 sets the continuous shooting scene of the continuous image to the scene c (step d29). When the direction unstable flag is OFF (step d27: No), the continuous shooting scene classifying unit 281 determines the direction change flag. If the direction change flag is ON (step d31: Yes), the continuous shooting scene classification unit 281 sets the continuous shooting scene of the continuous image to the scene b (step d33). When the direction change flag is OFF (step d31: No), the continuous shooting scene classification unit 281 sets the continuous shooting scene of the continuous image to the scene a (step d35). Note that the continuous shooting scene of the continuous image is set to the scene a in this step d35 when the direction flag is ON.

  Subsequently, the continuous shooting scene classification unit 281 determines whether or not the moving object is out of frame according to the result of the moving object state analysis process in step d3. Then, the continuous shooting scene classification unit 281 sets a frame-out flag (step d39) for a moving object that has been out-of-frame (step d37: Yes).

  Subsequently, the continuous shooting scene classification unit 281 determines whether or not the moving object has finished moving in accordance with the result of the moving object state analysis process in step d3. Then, the continuous shooting scene classification unit 281 sets a movement end flag (step d43) for the moving body that has finished moving (step d41: Yes). And it returns to step a13 of FIG. 8, and moves to step a15 after that.

  In step a15, the continuous image division unit 283 performs continuous image division processing. FIG. 13 is a flowchart illustrating a detailed processing procedure of the continuous image division processing according to the first embodiment.

  In the continuous image dividing process, the continuous image dividing unit 283 first acquires the movement start image number of the moving object (step f1). Subsequently, the continuous image dividing unit 283 determines a movement end flag. If the movement end flag is ON (step f3: Yes), the continuous image dividing unit 283 acquires the movement end image number (step f5), and obtains a value obtained by subtracting the movement start image number from the movement end image number. It is determined whether or not the calculated value exceeds a predetermined value. If it exceeds the predetermined value, that is, if the number of photographed images from the start of movement of the moving body to the end of movement is greater than the predetermined number (step f7: Yes), step f17 is entered. Move. If it does not exceed the predetermined value (step f7: No), the process proceeds to step f15.

  On the other hand, if the movement end flag determined in step f3 is OFF (step f3: No), the continuous image dividing unit 283 subsequently determines a frame-out flag. If the frame-out flag is ON (step f9: Yes), the continuous image dividing unit 283 acquires the frame-out image number (step f11), and obtains a value obtained by subtracting the movement start image number from the frame-out image number. It is determined whether or not the calculated value exceeds a predetermined value. If the predetermined value is exceeded, that is, if the number of photographed images from the start of movement of the moving object to the frame-out is greater than the predetermined number (step f13: Yes), the process proceeds to step f17. . If it does not exceed the predetermined value (step f13: No), the process proceeds to step f15. Also, when the frame-out flag determined in step f9 is OFF (step f9: No), the process proceeds to step f15.

  In step f15, the continuous image dividing unit 283 calculates a value obtained by subtracting the movement start image number from the final image number, and determines whether the calculated value exceeds a predetermined value. When the predetermined value is exceeded, that is, when the number of photographed images from the start of movement of the moving body to the end of continuous shooting is greater than the predetermined number (step f15: Yes), step f17 Move on. If it does not exceed the predetermined value (step f15: No), the process returns to step a15 in FIG. 8, and then proceeds to step a17.

  In step f17, the continuous image dividing unit 283 determines the scene set by the continuous shooting scene classification process in step a13 in FIG. 8, and branches the process according to the scene. That is, the continuous image dividing unit 283 performs the scene a / c dividing process when the continuous shooting scene is the scene a or the scene c (step f19). And it returns to step a15 of FIG. 8, and moves to step a17 after that. If the continuous shooting scene is the scene b, the continuous image dividing unit 283 performs a scene b dividing process (step f21). And it returns to step a15 of FIG. 8, and moves to step a17 after that. If the continuous shooting scene is the scene d, the continuous image dividing unit 283 performs a scene d dividing process (step f23). And it returns to step a15 of FIG. 8, and moves to step a17 after that. If the continuous shooting scene is the scene e, the continuous image dividing unit 283 performs a scene e dividing process (step f25). And it returns to step a15 of FIG. 8, and moves to step a17 after that. Here, each process of step f19-step f25 is demonstrated sequentially.

  FIG. 14 is a flowchart showing a detailed processing procedure of the scene a / c division processing. In the scene a / c division process, the continuous image division unit 283 first sets the first division position as the movement start image number (step g1). Subsequently, the continuous image dividing unit 283 calculates the movement center position of the moving body (step g3). That is, the intermediate position on the movement path from the movement start position of the moving body to the movement end position or the frame-out position is calculated as the movement center position. Then, the continuous image dividing unit 283 sets the second division position as the movement center image number (step g5).

  Subsequently, the continuous image dividing unit 283 determines a movement end flag. If the movement end flag is ON (step g7: Yes), the continuous image dividing unit 283 sets the third division position as the movement end image number (step g9), and proceeds to step g17. On the other hand, when the movement end flag is OFF (step g7: No), the continuous image dividing unit 283 subsequently determines the frame-out flag. If the frame-out flag is ON (step g11: Yes), the continuous image dividing unit 283 sets the third division position as the frame-out image number (step g13), and proceeds to step g17. If the frame-out flag is OFF (step g11: No), the continuous image dividing unit 283 does not set the third division position (step g15). Then, the continuous image dividing unit 283 extracts the captured images at the set first division position, second division position, and third division position as representative images (step g17), and then returns to step f19 in FIG.

  FIG. 15 is a flowchart showing a detailed processing procedure of the scene b division processing. In the scene b dividing process, the continuous image dividing unit 283 first sets the first division position as the movement start image number (step h1). Subsequently, the continuous image dividing unit 283 acquires a direction change image number (step h3). Then, the continuous image dividing unit 283 sets the second division position as the direction change image number (step h5).

  Subsequently, the continuous image dividing unit 283 determines a movement end flag. If the movement end flag is ON (step h7: Yes), the continuous image dividing unit 283 sets the third division position as the movement end image number (step h9), and proceeds to step h17. On the other hand, when the movement end flag is OFF (step h7: No), the continuous image dividing unit 283 subsequently determines the frame-out flag. If the frame-out flag is ON (step h11: Yes), the continuous image dividing unit 283 sets the third division position as the frame-out image number (step h13), and proceeds to step h17. If the frame-out flag is OFF (step h11: No), the continuous image dividing unit 283 does not set the third division position (step h15). Then, the continuous image dividing unit 283 extracts the captured images at the set first division position, second division position, and third division position as representative images (step h17), and then returns to step f21 in FIG.

  FIG. 16 is a flowchart showing a detailed processing procedure of the scene d division processing. In the scene d division processing, the continuous image division unit 283 first sets the first division position as the movement start image number (step i1). Subsequently, the continuous image dividing unit 283 acquires a divided image number (step i3). Then, the continuous image dividing unit 283 sets the second division position as a divided image number (step i5).

  Subsequently, the continuous image dividing unit 283 determines a movement end flag. If the movement end flag is ON (step i7: Yes), the continuous image dividing unit 283 sets the third division position as the movement end image number (step i9), and proceeds to step i17. On the other hand, when the movement end flag is OFF (step i7: No), the continuous image dividing unit 283 subsequently determines the frame-out flag. If the frame-out flag is ON (step i11: Yes), the continuous image dividing unit 283 sets the third division position as the frame-out image number (step i13), and proceeds to step i17. If the frame-out flag is OFF (step i11: No), the continuous image dividing unit 283 does not set the third division position (step i15). Then, the continuous image dividing unit 283 extracts the captured images at the set first division position, second division position, and third division position as representative images (step i17), and returns to step f23 of FIG.

  FIG. 17 is a flowchart showing a detailed processing procedure of the scene e division processing. In the scene e division processing, the continuous image division unit 283 first sets the first division position as the movement start image number (step j1). Subsequently, the continuous image dividing unit 283 acquires an overlapping image number (step j3). Then, the continuous image dividing unit 283 sets the second division position as an overlapping image number (step j5).

  Subsequently, the continuous image dividing unit 283 determines a movement end flag. If the movement end flag is ON (step j7: Yes), the continuous image dividing unit 283 sets the third division position as the movement end image number (step j9), and proceeds to step j17. On the other hand, when the movement end flag is OFF (step j7: No), the continuous image dividing unit 283 subsequently determines the frame-out flag. If the frame-out flag is ON (step j11: Yes), the continuous image dividing unit 283 sets the third division position as the frame-out image number (step j13), and proceeds to step j17. If the frame-out flag is OFF (step j11: No), the continuous image dividing unit 283 does not set the third division position (step j15). Then, the continuous image division unit 283 extracts the captured images at the set first division position, second division position, and third division position as representative images (step j17), and returns to step f25 in FIG.

  When the continuous image dividing process is finished, the importance calculating unit 285 subsequently performs the moving object importance calculating process as shown in FIG. 8 (step a17). FIG. 18 is a flowchart showing a detailed processing procedure of the moving object importance degree calculation processing. In this priority scoring process, a loop A process is performed for all captured images constituting the continuous image (step k1 to step k25). Hereinafter, the captured image to be processed in the loop A is referred to as a processed image.

  That is, first, the importance level calculation unit 285 determines whether or not there is a moving object in the processed image. When the moving body is not detected in the processed image, that is, when the movement of the registered moving body is not detected in the processed image (step k3: No), the process proceeds to step k11.

  On the other hand, when the moving object is detected in the processed image, that is, when the movement of the registered moving object is detected in the processed image (step k3: Yes), the importance calculating unit 285 The position of the moving body is scored (step k5). For example, it is estimated that the importance is higher as it is located at the center of the angle of view range, and the importance calculation unit 285 determines the scoring value as the position of the moving object detected in the processed image is closer to the center of the angle of view range. Set larger.

  Subsequently, the importance level calculation unit 285 scores the speed change of the moving object (step k7). For example, it is estimated that the higher the moving speed of the moving body, the higher the importance. The importance calculation unit 285 calculates the moving speed of the moving body based on the displacement of the registered moving body with reference to the position of the registered moving body in several captured images generated before the processed image, for example. And the importance calculation part 285 sets a scoring value large, so that the movement speed is quick.

  Subsequently, the importance calculation unit 285 scores a change in the direction of the moving object (step k9). For example, it is estimated that the greater the change in the moving direction of the moving body, the higher the importance. For example, the importance calculation unit 285 refers to the position of the moving body in several captured images generated before the processed image, and determines the change in the moving direction based on the displacement of the position. Then, the importance calculation unit 285 sets the scoring value larger as the determined change in the moving direction is larger.

  In subsequent step k11, the importance calculation unit 285 determines the presence or absence of a face in the processed image. When there is no face in the processed image, that is, when a face is not detected as a result of face detection in the processed image (step k11: No), the process proceeds to step k21.

  On the other hand, when there is a face in the processed image, that is, when a face is detected as a result of face detection in the processed image (step k11: Yes), the importance calculation unit 285 scores the size of the face. (Step k13). Larger faces are more important than smaller faces. Therefore, for example, the importance calculation unit 285 sets the scoring value larger as the face is larger.

  Subsequently, the importance calculation unit 285 scores the face position (step k15). The closer the face position is to the center of the field angle range, the higher the importance. On the other hand, when the position of the face is located at the end of the angle of view range, the face is likely to be framed out in the next photographed image, so the importance is low. Therefore, for example, the importance calculation unit 285 sets the scoring value larger as the face position is closer to the center of the field angle range.

  Subsequently, the importance calculation unit 285 scores the face orientation (step k17). For example, the importance is estimated to be lower as the face is away from the front direction, and the importance calculation unit 285 reduces the scoring value when the face is facing the front, and the face direction is the front. The scoring value is set larger as it gets out of the range.

  Subsequently, the importance calculation unit 285 scores the face angle (step k19). For example, it is estimated that the importance is higher as the face is not tilted with respect to the vertical direction of the processed image, and the importance calculation unit 285 decreases the scoring value when the face is not tilted, and the face angle is larger. Set the scoring value higher.

  Subsequently, the importance calculation unit 285 weights each scoring value (step k21). The weighting performed here is, for example, a weighting factor in advance for each scoring item of moving body position, moving body speed change, moving body direction change, face size, face position, face direction and face angle. Set and multiply each scoring value by a weighting factor. The weighting factor can be appropriately set according to the importance of the scoring item. For example, if the user attaches importance to the captured image in which the moving object is shown in the center of the angle of view range, the weighting coefficient for the position of the moving object and the position of the face can be set large. Therefore, it is possible to set the importance of a captured image having a high scoring value high.

  Then, the importance level calculation unit 285 calculates the sum of the weighted scoring values as the importance level of the processed image (step k23). If all the captured images constituting the continuous image are processed as a processed image and the process of loop A is performed, the process returns to step a17 in FIG. 8, and then proceeds to step a19. The above-described moving object importance degree calculation process is an example, and the scoring method and scoring items are not limited to this, and can be set as appropriate.

  Subsequently, the saved image selection processing unit 287 performs processing for displaying the representative image extracted as a result of the continuous image division processing in step a15 on the display unit 23 and displaying a notification of the selection request for the divided scenes. (Step a19). For example, thumbnails of representative images may be displayed as a list, and divided scenes may be selected by accepting a representative image selection operation, or representative images may be sequentially switched and displayed one by one according to a user operation. It is good also as a structure which selects a division scene by accepting.

  Then, when the divided scene is selected in response to the notification of the selection request (step a21: Yes), the saved image selection processing unit 287 subsequently performs a process of displaying the captured image of the selected divided scene. And a process for displaying a notification of a selection request for a stored image (step a23). At this time, the stored image selection processing unit 287 is, for example, when a captured image between the selected representative image and a representative image with a subsequent shooting order, for example, a representative image at the first division position is selected. The captured image at the first divided position and the captured image between the first divided position and the second divided position are selected as the captured images of the divided scene. Then, the saved image selection processing unit 287 rearranges the selected captured images in order of importance based on the result of the moving object importance calculation process in step a17. Then, the stored image selection processing unit 287 displays a list of thumbnails of the captured images selected in the rearranged order, and accepts an operation for selecting the captured image to be stored. According to this, it is possible to present to the user in order from the most important one among the captured images of the selected divided scenes. Therefore, the user can efficiently browse the captured images even when the number of captured images of the divided scene is large, and the work burden can be further reduced.

  A configuration may be adopted in which a predetermined number of photographed images having a higher importance are selected from among the photographed images of the divided scenes, displayed in a list, and a selection of saved images is accepted. Moreover, it is good also as a structure which displays a list in order of imaging | photography, without rearranging by importance. Here, the thumbnails of the captured images of the selected divided scene are displayed as a list, but the captured images of the selected divided scene are sequentially switched and displayed one by one in accordance with the user operation, and the selected captured image is selected. It is good also as a structure which receives operation. At this time, the captured images of the divided scenes may be rearranged in the order of importance and switched and displayed in the rearranged order based on the result of the importance calculation process in step a17.

  If the saved image is selected in response to the selection request notification (step a25: Yes), the saved image selection processing unit 287 performs the saving process (step a27), and proceeds to step a29.

  In step a29, the saved image selection processing unit 287 monitors the presence / absence of a split scene change operation. Then, when there is an operation for changing the divided scene (step a29: Yes), the saved image selection processing unit 287 returns to step a19 to notify the selection request for the divided scene again and performs the above-described processing. On the other hand, when there is no operation for changing the divided scene (step a29: No), the saved image selection processing unit 287 determines whether or not to save the image depending on the presence or absence of the save completion operation. In the meantime (step a31: No), the process returns to step a25 to perform the above-described processing. On the other hand, when the storage is completed (step a31: Yes), this process is finished.

  As described above, according to the first embodiment, motion detection is performed between captured images constituting a continuous image to detect the movement of a moving body that appears in the continuous image, and the motion detection results are continuously generated based on the motion detection result. By determining the state change of the moving body in the image, the representative image can be extracted from the continuous image. Specifically, the continuous images are classified into continuous shooting scenes based on the change in the state of the moving object, and the continuous images are divided at positions corresponding to the classified continuous shooting scenes. Can be extracted as According to this, since the representative image can be extracted depending on how the moving body moves in the continuous image, it is possible to appropriately extract the representative image useful for the user from the continuous image. Further, the extracted representative image can be displayed on the display unit 23 and presented to the user. Therefore, it is possible to reduce the work burden on the user when selecting a captured image to be saved from the continuous images.

  In Embodiment 1 described above, when a continuous shot scene of continuous images is classified as scene a, the representative image is extracted with the movement center position of the moving object as the division position. On the other hand, the continuous image may be divided at a position where the moving body is closest to the center of the shooting angle of view. According to this, it is possible to extract a photographed image in which the moving body appears near the center as a representative image.

  Moreover, although the case where a continuous image of continuous images is classified into the scene c and the representative image is extracted with the change position in the moving direction as the division position is illustrated, the present invention is not limited to this. For example, when the moving direction of the moving body changes regularly according to a predetermined change pattern, the distance from the moving start position to the moving end position or frame-out position of the moving body is set to a predetermined number. It is good also as dividing and extracting the picked-up image of the divided position as a representative image.

  Further, when the moving body is a face, the state change of the moving body is determined by taking into consideration the face position, face size, face orientation, face angle (tilt), etc. detected by the face detection unit 17. You may make it do.

  Further, the number of photographed images to be presented to the user as representative images may be set in advance. Then, when more than the set number of representative images are extracted, a set number of representative images may be selected from the extracted representative images. At this time, using the importance value calculated for each captured image by the importance calculation unit 285, a set number of representative images may be selected in order from the representative images having the highest importance.

  In the first embodiment described above, the case where the captured image at the position determined by the continuous image dividing unit 283 is extracted as the representative image has been described. However, the present invention is not limited to this. For example, a predetermined number of photographed images adjacent to each other on the basis of the divided positions may be extracted as representative images.

  In the first embodiment described above, the representative image extracted by the continuous image dividing unit 283 is displayed on the display unit 23 and presented to the user, and the selection operation of the saved image is accepted. May be automatically stored and recorded on the recording medium 20. In this case, as described above as a modified example, a predetermined number of photographed images adjacent to each other on the basis of the division position may be extracted as representative images, and the extracted representative images may be stored. According to this, the representative image and, for example, several photographed images before and after the representative image can be automatically recorded on the recording medium 20.

  Further, when a plurality of moving bodies are detected in the continuous image, for example, one or a plurality of moving bodies are selected according to a user operation, and motion detection is performed on the selected moving bodies to determine the state change. It may be. According to this, extraction of a representative image focusing on a desired moving body among a plurality of moving bodies can be realized.

(Embodiment 2)
FIG. 19 is a diagram for explaining the principle of extracting a representative image according to the second embodiment. As an example of a moving path (change in the position of the moving object) of the moving object shown in each captured image constituting the continuous image, a parabola is used. The moving path of the moving body which draws and moves is shown.

  In the second embodiment, the movement speed is analyzed based on the movement (movement amount) of the moving body between the captured images. Subsequently, the moving section of the moving body is divided into, for example, a section where the moving speed is equal to or higher than a predetermined reference speed and a section where the moving speed is less than the reference speed, and each section is equally divided into a predetermined number. . The reference speed may be set in advance as a fixed value, or may be set variably according to a user operation, for example. And the picked-up image of the division position equally divided in each section is extracted as a representative image. In FIG. 19, the moving section of the moving body is divided into a section A and a section C in which the moving speed is equal to or higher than the reference speed, and a section B in which the moving speed is less than the reference speed. Then, the sections A, B, and C are equally divided into five representative images I51 to I55 from the section A, five representative images I61 to I65 from the section B, and five representative images I71 to 71 from the section C. It shows how I75 is extracted. 20-1 to 5 are diagrams showing representative images I51 to I55 extracted from the section A in FIG. 19, and FIGS. 21-1 to 5 are representative images I61 to I65 extracted from the section B in FIG. FIGS. 22-1 to 5 are diagrams illustrating representative images I71 to I75 extracted from the section C in FIG.

  In addition, although the case where the same number of photographed images is extracted from each of the sections A, B, and C is illustrated here, the number of divisions may be set individually. For example, the number of divisions in a section with a high moving speed may be set larger than the number of divisions in a section with a low moving speed. Further, the number of photographed images in the section may be counted and equally divided by the number of divisions corresponding to the counted number. Further, it is possible to divide only a section having a high movement speed and not to divide a section having a low movement speed. On the contrary, it is possible to divide only a section having a low movement speed and not to divide a section having a high movement speed. Although not shown, when there is no section where the moving speed of the moving body is equal to or higher than the reference speed, the entire moving section of the moving body is equally divided and, for example, five captured images are extracted.

  FIG. 23 is a diagram illustrating a configuration of the controller 28b of the digital camera according to the second embodiment. The digital camera of the second embodiment can be realized by replacing the controller 28 with the controller 28b shown in FIG. 23 in the configuration of the digital camera 1 shown in FIG. 5 in the first embodiment. In FIG. 23, the same reference numerals are given to the same components as those in the first embodiment.

  As shown in FIG. 23, in the second embodiment, the controller 28b includes a moving body state analyzing unit 282b as a change determining unit, a continuous image dividing unit 283b as an image extracting unit, an importance calculating unit 285, a display And a stored image selection processing unit 287 as a processing unit. The moving body state analysis unit 282b analyzes the state of the moving body that has appeared in the continuous image and determines the state change. In the second embodiment, the moving body state analyzing unit 282b analyzes the moving start position, the frame-out position, the moving end position, and the moving speed of the moving body. The continuous image dividing unit 283b divides the continuous image at a position corresponding to the moving speed of the moving body, and extracts the captured image at the divided position as a representative image.

  FIG. 24 is a flowchart for explaining a processing procedure performed by the digital camera according to the second embodiment, and shows a processing procedure of the digital camera in the continuous shooting mode. In FIG. 24, the same reference numerals are assigned to the same processing steps as those in the first embodiment.

  In the second embodiment, when the number of continuous shots is greater than the predetermined number (step a11: Yes), the moving body state analysis unit 282b performs a moving body state analysis process (step l13). FIG. 25 is a flowchart showing a detailed processing procedure of the moving body state analysis processing.

  In the moving body state analyzing process, first, the moving body state analyzing unit 282b analyzes the movement start position of the moving body (step m1). For example, a captured image that is first detected and registered as a registered moving body is set as a movement start position, and an image number is acquired as a movement start image number.

  Subsequently, the moving body state analyzing unit 282b analyzes the moving speed of the moving body (step m3). For example, the moving body state analyzing unit 282b calculates the moving amount of the moving body between the captured images as the moving speed based on the motion detection result of the moving body between the captured images constituting the continuous image.

  Subsequently, the moving body state analyzing unit 282b analyzes the frame-out position of the moving body (step m5). For example, the moving body moves to the end of the field angle range, and a captured image outside the field angle range is set as the frame-out position, and the image number is acquired as the frame-out image number.

  Subsequently, the moving body state analyzing unit 282b analyzes the moving end position of the moving body (step m7). For example, a captured image in which the moving body is stopped or a captured image in which the moving body cannot be tracked is set as the movement end position, and the image number is acquired as the movement end image number. Then, the process returns to step l13 in FIG. 24, and then proceeds to step l15.

  In step l15, the continuous image dividing unit 283b performs a continuous image dividing process. FIG. 26 is a flowchart illustrating a detailed processing procedure of continuous image division processing.

  In the continuous image dividing process, the continuous image dividing unit 283b first acquires the movement start image number of the moving object (step n1). Then, the continuous image dividing unit 283b sets the first captured image of the section (moving section) in which the moving body is moving in the continuous image as the movement start image number (step n3). Subsequently, the continuous image dividing unit 283b determines a movement end flag. If the movement end flag is ON (step n5: Yes), the continuous image dividing unit 283b sets the last captured image in the movement section as the movement end image number (step n7), and proceeds to step n15.

  On the other hand, if the movement end flag determined in step n5 is OFF (step n5: No), the continuous image dividing unit 283b subsequently determines a frame-out flag. If the frame-out flag is ON (step n9: Yes), the continuous image dividing unit 283b sets the last captured image in the movement section as the frame-out image number (step n11), and proceeds to step n15.

  If the frame-out flag determined in step n9 is OFF (step n9: No), the last captured image number of the moving section is set as the final image number of the continuous image (step n13), and the process proceeds to step n15.

  In step n15, the continuous image dividing unit 283b determines a change in the moving speed of the moving object. Then, if the moving speed between any of the captured images is equal to or higher than the predetermined reference speed (step n15: Yes), the continuous image dividing unit 283b and the reference speed between the interval where the moving speed is equal to or higher than the reference speed. First, a section where the moving speed is equal to or higher than the reference speed is equally divided into a predetermined number to set a division position (step n17). Subsequently, the continuous image dividing unit 283b equally divides the section in which the moving speed is less than the reference speed into a predetermined number and sets the division position (step n19). Then, the process proceeds to step n23. On the other hand, when the moving speed is not equal to or higher than the reference speed (step n15: No), the entire moving section is equally divided into a predetermined number to set the dividing position (step n21). Then, the process proceeds to step n23.

  In step n23, the captured image at the division position is extracted as a representative image. Then, the process returns to step l15 in FIG. 24, and then proceeds to step a17.

  As described above, according to the second embodiment, motion detection is performed between captured images constituting a continuous image to detect the movement of a moving body that appears in the continuous image, and the motion detection results are continuously detected based on the motion detection result. A change in the moving speed can be determined as a change in the state of the moving body in the image. Then, by dividing the continuous image at the division position according to the moving speed of the moving body, the captured image at the divided position can be extracted as the representative image. For example, the moving section of the moving object is divided into a section where the moving speed is equal to or higher than a predetermined reference speed and a section where the moving speed is less than the reference speed, and the section is equally divided to set a division position and extract a representative image. Can do. Therefore, a representative image can be extracted according to a change in the moving speed of the moving body in the continuous image and presented to the user.

  In the first embodiment described above, it is determined whether the overlapping state or split state of the moving body, whether the moving direction of the moving body is constant or unstable, or whether the moving body changes regularly according to a predetermined change pattern. The continuous images were classified into continuous shooting scenes, and continuous images were divided at positions corresponding to the continuous shooting scenes. On the other hand, in the second embodiment, a change in moving speed is determined as a change in the state of the moving body, and a continuous image is divided at positions corresponding to the determined change in moving direction. On the other hand, a change in the size of the moving body may be determined, and the continuous image may be divided at a position corresponding to the determined change in size. For example, in the same manner as in the application example of the second embodiment, the moving body is divided into a section having a size equal to or larger than a predetermined reference size and a section that does not satisfy the reference size, and the section is divided into equal parts. It is good also as setting and extracting a representative image. Further, when there is no section where the size of the moving body is equal to or larger than the reference size, the entire moving section of the moving body may be equally divided to set the division position.

  In this case, similarly to the second embodiment, the section in which the size of the moving object is large and the section in which the size of the moving object is small may be divided by different numbers of divisions. For example, the number of divisions in a large size section may be set larger than the number of divisions in a small size section. Further, the number of photographed images in the section may be counted and equally divided by the number of divisions corresponding to the counted number. Further, only a section having a large size may be divided, and a section having a small size may not be divided. On the contrary, it is possible to divide only a small-sized section and not divide a large-sized section.

  Further, the division position may be set by determining the state change of the moving body by combining at least two or more of the change in the moving direction of the moving body, the change in the moving speed, and the change in the magnitude. Further, at this time, when the moving body is a face, a change in at least one of the face position, the face size, the face direction, and the face angle (tilt) detected by the face detection unit 17 is taken into account. The change in the state of the moving body may be determined in consideration of the above.

  Further, the image processing unit 16, the continuous shooting scene classifying unit 281 constituting the controller 28 or 28b, the moving object state analyzing unit 282b, the continuous image dividing units 283 and 283b, the importance calculating unit 285, and the saved image selection processing unit 287 are: The configuration may be realized by hardware, or may be realized as software by executing a predetermined program. When implemented as software, for example, a program for realizing part or all of the processing shown in FIGS. 8 to 18 or the processing shown in FIGS. 24 to 26 is recorded in the ROM 27. Then, the controller 28, 28b reads out and executes this program, so that the continuous shooting scene classification unit 281, the moving body state analysis unit 282b, the continuous image division units 283, 283b, the importance calculation unit 285, and the saved image selection processing unit. The configuration of 287 may be realized.

  In the above-described embodiment, an example in which the photographing apparatus of the present invention is applied to a digital camera has been described. However, the application target is not limited to a digital camera, and is applied to a camera attached to a mobile phone or a camera attached to a PC. May be.

It is a figure which shows an example of a continuous image. It is a figure which shows the other example of a continuous image. It is a figure which shows the other example of a continuous image. It is a figure which shows the other example of a continuous image. 1 is a schematic block diagram illustrating a configuration example of a digital camera according to Embodiment 1. FIG. It is a figure which shows an example of a mobile body detection area and a registration mobile body area. It is a figure which shows the other example of a mobile body detection area and a registration mobile body area. 4 is a flowchart illustrating a procedure of processing performed by the digital camera according to the first embodiment. It is a flowchart explaining the detailed process sequence of a motion detection process. It is a flowchart explaining the detailed process sequence of a face detection process. It is a flowchart which shows the detailed process sequence of a continuous shooting scene classification | category process. It is a flowchart which shows the detailed process sequence of a moving body state analysis process. 3 is a flowchart illustrating a detailed processing procedure of continuous image division processing according to the first embodiment. It is a flowchart which shows the detailed process sequence of a scene a / c division process. It is a flowchart which shows the detailed process sequence of a scene b division process. It is a flowchart which shows the detailed process sequence of a scene d division | segmentation process. It is a flowchart which shows the detailed process sequence of a scene e division | segmentation process. It is a flowchart which shows the detailed process sequence of a mobile body importance degree calculation process. FIG. 10 is a diagram for explaining a principle of extracting a representative image according to the second embodiment. It is a figure which shows an example of the representative image extracted from the area A of FIG. It is a figure which shows the other example of the representative image extracted from the area A of FIG. It is a figure which shows the other example of the representative image extracted from the area A of FIG. It is a figure which shows the other example of the representative image extracted from the area A of FIG. It is a figure which shows the other example of the representative image extracted from the area A of FIG. It is a figure which shows an example of the representative image extracted from the area B of FIG. It is a figure which shows the other example of the representative image extracted from the area B of FIG. It is a figure which shows the other example of the representative image extracted from the area B of FIG. It is a figure which shows the other example of the representative image extracted from the area B of FIG. It is a figure which shows the other example of the representative image extracted from the area B of FIG. It is a figure which shows an example of the representative image extracted from the area C of FIG. It is a figure which shows the other example of the representative image extracted from the area C of FIG. It is a figure which shows the other example of the representative image extracted from the area C of FIG. It is a figure which shows the other example of the representative image extracted from the area C of FIG. It is a figure which shows the other example of the representative image extracted from the area C of FIG. 6 is a diagram illustrating a configuration of a controller of a digital camera according to Embodiment 2. FIG. 10 is a flowchart illustrating a procedure of processing performed by the digital camera according to the second embodiment. It is a flowchart which shows the detailed process sequence of a moving body state analysis process. 10 is a flowchart illustrating a detailed processing procedure of continuous image division processing according to the second embodiment.

1 Digital Camera 11 Imaging Optical System 12 Image Sensor 13 AFE
14 frame memory 15 motion detection unit 16 image processing unit 17 face detection unit 18 recording medium I / F
DESCRIPTION OF SYMBOLS 19 Recording medium holding | maintenance part 20 Recording medium 21 Video encoder 22 Display driver 23 Display part 24 Video signal output terminal 25 Operation part 26 RAM
27 ROM
28, 28b Controller 281 Continuous shooting scene classification unit 282b Moving object state analysis unit 283, 283b Continuous image division unit 285 Importance calculation unit 287 Saved image selection processing unit

Claims (14)

A continuous image acquisition unit that continuously performs shooting processing to acquire a continuous image composed of image data of a plurality of captured images;
For each of the plurality of captured images, a moving body detection unit that detects a movement between adjacent captured images and detects a moving body in the captured images;
Based on the detection result by the moving body detection unit, a change determination unit that determines a state change of the moving body in the continuous image;
Based on the determination result by the change determination unit, an image extraction unit that extracts a representative image from the continuous image;
An imaging apparatus comprising:   The change determination unit determines at least one of a change in a moving direction of the moving body, a change in a moving speed of the moving body, and a change in the size of the moving body. The photographing apparatus described.   The image extraction unit divides the continuous image at a position corresponding to a determination result by the change determination unit, and extracts at least a photographed image at the divided position as the representative image. Shooting device. The change determining unit determines a state in which the moving direction of the moving body is regularly changed according to a predetermined change pattern;
The said image extraction part divides | segments the said continuous image in the position where the moving direction of the said mobile body changed, when the said regular change is discriminate | determined by the said change discrimination | determination part. Shooting device. The change determining unit determines a state in which the moving body moves along a predetermined direction without changing a direction or a state in which the moving direction of the moving body changes irregularly,
The image extraction unit includes a movement path from a movement start position of the moving body to a movement end position of the moving body when the change determination section determines a change along the predetermined direction or the irregular change. The photographing apparatus according to claim 3, wherein the continuous image is divided at an upper intermediate position. The change determining unit determines a state in which the moving body moves along a predetermined direction without changing a direction or a state in which the moving direction of the moving body changes irregularly,
The image extraction unit, when the change determination unit determines a change along the predetermined direction or the irregular change, the moving body displays the continuous image at a position closest to the center of the shooting angle of view. The photographing apparatus according to claim 3, wherein the photographing apparatus is divided. The moving body detection unit is configured to be able to detect a plurality of moving bodies,
The change determination unit determines the overlapping state of the moving bodies as an overlapping state,
The imaging apparatus according to claim 3, wherein the image extraction unit divides the continuous image at a position where the moving bodies overlap each other. The moving body detection unit is configured to be able to detect a plurality of moving bodies,
The change determining unit determines, as a split state, a state in which another moving body starts moving, starting from the position of any one of the plurality of moving bodies.
The imaging apparatus according to claim 3, wherein the image extraction unit divides the continuous image at a position where the moving body is divided.   The imaging apparatus according to claim 3, wherein the image extraction unit further divides the continuous image at at least one of a movement start position of the moving body and a movement end position of the moving body.   The image capturing apparatus according to claim 3, wherein the image extraction unit extracts a predetermined number of photographed images whose photographing order is adjacent to the photographed image at the divided position as the representative image.   The said image extraction part divides | segments the inside of the movement area of the said mobile body into one or more areas according to the discrimination | determination result by the said change discrimination | determination part, and divides | segments the inside of at least 1 area of the said area. 3. The photographing apparatus according to 3.   The photographing apparatus according to claim 1, further comprising a display processing unit that performs a process of displaying the representative image on a display unit.   The said display process part performs the process which displays each representative image in a list or in order of imaging | photography in order, when the said image extraction part extracts the said representative image in multiple numbers. Shooting device. A continuous image acquisition step of performing continuous shooting processing to acquire a continuous image composed of image data of a plurality of captured images;
For each of the plurality of captured images, a moving object detection step of detecting a movement between adjacent captured images and detecting a moving object in the captured images;
Based on the detection result of the moving body detection step, a change determination step for determining a state change of the moving body in the continuous image;
Based on the determination result by the change determination step, an image extraction step of extracting a representative image from the continuous image;
A photographing method characterized by comprising:

Images