JP5678775B2 - Image processing apparatus and image processing method - Google Patents

Description

  The present invention relates to an image processing apparatus and an image processing method such as a video camera and a digital video camera, and more particularly to an image processing apparatus and image processing for extracting an optimal still image from moving images or still image data continuously or continuously acquired. Regarding the method.

  In a moving image recording / playback apparatus such as a conventional video camera, a picture can be recorded (extracted) as a still image by pressing a shutter button during playback or playback pause for the playback video. A moving image is a continuum of still images, and it is difficult to select a characteristic still image of the moving image from a large number of still images. Therefore, there is a technique for automatically selecting a still image representative of the moving image from the moving image data and extracting it as a still image.

  For example, in Patent Document 1, when a representative image is extracted from a moving image, the number of representative images is determined according to the number of scenes in the moving image, so that the operation of batch reference of moving image contents is inefficient. In moving images for the purpose of avoiding a situation that would be difficult, or extracting representative images that are insufficient as information for collectively grasping the contents of moving images A representative image extraction method is disclosed.

  Further, in Patent Document 2, the still image extraction unit extracts a still image from a period excluding a period in which the interframe difference value included in the image processing information obtained by processing the captured image is equal to or greater than a threshold value. A still image extraction device is disclosed that extracts an image having a high evaluation value by extracting an image having a high evaluation value based on knowledge of still image extraction that extracts or outputs a higher evaluation value as the inter-frame difference value is smaller Yes.

  Further, Patent Document 3 discloses a camera operation information acquisition unit that captures camera operation information obtained when a photographer operates a camera when capturing a moving image, and an image that captures image processing information obtained by processing the captured image. Processing information acquisition means, shooting state information acquisition means for capturing shooting state information during shooting obtained by processing a signal from the sensor, and from when the photographer performs a shooting start operation until the shooting end operation A still image extracting means for extracting at least one still image from the captured moving images, and at least one of image processing information from the image processing information acquiring means and shooting state information from the shooting state information acquiring means. There is disclosed a still image extraction device that extracts a still image by a still image extraction unit based on information and camera operation information from a camera operation information acquisition unit.

Japanese Patent No. 3936666 Japanese Patent No. 3525493 Japanese Patent No. 3404803

  However, the conventional function for selecting the optimum still image from the moving image is to cut out one or more optimum still images suitable for the conditions for one moving image file, for example, intermittently at the same location. When shooting a movie, there will be multiple movie files shot in the same scene, and even if you use the above-mentioned conventional optimum still image selection function, multiple similar still images of the same scene will be generated. It will end up. Such still image selection cannot be said to be in line with the user's wishes, and there is a problem in that it is necessary to make further selection again by the user's own hands.

  The present invention has been made to solve such problems, and is an image processing apparatus and image that can extract an optimum still image from a moving image or a group of still images while taking position information into consideration. It is an object to provide a processing method and a program.

  The image processing apparatus according to the present invention includes a storage unit that stores a plurality of shooting files consisting of either moving images, continuous shots, or continuously acquired still image groups, and each of the plurality of shooting files stored in the storage unit, An optimal still image extracting unit that extracts an optimal still image according to a predetermined standard or algorithm, and a shooting position when the optimal still image is captured from each of the plurality of optimal still images extracted by the optimal still image extracting unit, or A spatial information extraction unit that extracts spatial information including information on at least one of the shooting directions, and the plurality of optimum still images are grouped into one or more groups based on the spatial information extracted by the spatial information extraction unit A grouping unit configured to generate a group, and the optimum still image extracting unit includes a grouping unit that includes the grouping unit when the grouping unit includes a plurality of optimum still images. Per-loop is characterized in that to extract an optimal still image again.

  Further, the storage unit stores N (N is a natural number of 2 or more) or more shooting files, and the grouping unit is the nth (n is 2) extracted by the optimum still image extraction unit. As described above, based on the spatial information of the optimal still image extracted from the (N-1) or less natural number) shooting file and the spatial information of the optimal still image extracted from the (n + 1) th shooting file, both optimal still images It is possible to determine whether or not the images are included in the same group, and it is possible to reliably perform grouping by sequentially comparing the spatial information of each captured file.

  Still further, the optimum still image extracting unit generates a plurality of groups by the grouping unit, and when the optimum still image is extracted again for each group, the first information consisting of spatial information and the predetermined information The optimum still image can be extracted using any one or more of the second information including the reference value or the feature value used in the reference or algorithm, and the optimum still image extraction unit extracts the optimum still image again. In this case, the optimum still image extraction process can be performed from information already calculated.

  In addition, a mode determination unit that extracts a change amount of the spatial information of at least a part of a frame image or a still image included in each shooting file and determines a mode based on the magnitude of the change amount of the spatial information is provided. When the amount of change in the spatial information is larger than a predetermined threshold, the optimum still images are extracted from the optimum still image extracting unit and then grouped by the grouping unit, and again by the optimum still image extracting unit as necessary. When the first mode for extracting the optimum still image is selected and the amount of change in the spatial information is smaller than a predetermined threshold, the optimum still image is extracted by the optimum still image extracting unit after grouping by the grouping unit The second mode can be selected, and by determining the mode based on the amount of change in the spatial information, a mode suitable for the amount of change can be set. .

  Further, the mode determination unit may change or add to the magnitude of the amount of change in the spatial information, and may be any one or more of the number of shooting files and the number of frame images or still images included in all shooting files. Based on the information, the first or second mode can be selected, and the mode can be set in consideration of not only the amount of change but also the overall processing speed.

  Furthermore, the mode determination unit calculates an average value, a maximum value, and a minimum value of the amount of change in the spatial information in each frame image or still image included in each shooting file for each shooting file, The larger one of the difference between the average value and the maximum value or the minimum value is obtained as a representative value, and when the representative value calculated for each shooting file is larger than a predetermined threshold, the first mode is selected, When the representative value calculated for each shooting file is smaller than a predetermined threshold, the second mode can be selected, and the position representing the shooting file from the average value and the maximum and minimum values. A representative value as information can be determined, and the mode can be determined accordingly.

  The optimum still image extraction unit may extract an optimum still image based on a feature value obtained from a frame image constituting the moving image or a still image constituting the group of still images continuously or continuously acquired. The optimum still image extraction process can be performed based on feature considerations.

  Further, the optimum still image extraction unit extracts one frame image according to the predetermined standard or algorithm to make the optimum still image, or extracts two or more frame images and synthesizes them to combine the optimum still image. An image can be generated, and two or more frame images are combined to generate a single still image, whereby various noises included in the frame image can be reduced.

  An image processing method according to the present invention includes a predetermined reference or algorithm from each of the plurality of shooting files stored in a storage unit that stores a plurality of shooting files consisting of moving images, continuous shooting, or continuously acquired still image groups. The optimum still image extracting step for extracting the optimum still image according to the method, and from the plurality of optimum still images extracted in the optimum still image extracting step, the shooting position or shooting direction when the optimum still image is shot is selected. A plurality of optimum still images are grouped into one or more groups based on a spatial information extraction step for extracting spatial information including at least one of the information and the spatial information extracted in the spatial information extraction step In the optimum still image extraction step, a plurality of optimum still images are included in the group grouped in the grouping step. If it is characterized in that to extract an optimal still image for each said group again.

  A program according to the present invention is for causing a computer to execute the above-described image processing.

  According to the present invention, it is possible to provide an image processing apparatus, an image processing method, and a program capable of extracting an optimal still image from a moving image or a group of still images while taking position information into consideration.

It is a block diagram which shows the structure of the imaging device 1a which concerns on Embodiment 1 of this invention. It is a flowchart which shows the optimal still image extraction method concerning Embodiment 1 of this invention. It is a flowchart which shows the optimal still image extraction method concerning Embodiment 2 of this invention. It is a flowchart which shows the optimal still image extraction method concerning Embodiment 3 of this invention. It is a block diagram which shows the structure of the imaging device 1b which concerns on Embodiment 4 of this invention. It is a flowchart which shows the optimal still image extraction method concerning Embodiment 4 of this invention.

  DESCRIPTION OF EMBODIMENTS Preferred embodiments of an imaging apparatus and an imaging system according to the present invention will be described below with reference to the drawings. The dimensions, materials, and other specific numerical values shown in the embodiments are merely examples for facilitating understanding of the invention, and do not limit the present invention unless otherwise specified. In the present specification and drawings, elements having substantially the same function and configuration are denoted by the same reference numerals, and redundant description is omitted, and elements not directly related to the present invention are not illustrated. To do.

  The imaging device according to the present embodiment is an image processing device such as a video camera or a digital still camera, for example, and has a function of cutting out an optimal still image from a moving image or a group of still images acquired continuously or continuously. is there. Then, instead of simply cutting out an optimal still image from a movie or still image group 1 file, use spatial information including the position or shooting direction obtained from the GPS function, etc., and shoot in the same scene and in the same direction. A moving image or still image group is discriminated as one moving image still image group and grouped, and an optimal still image is selected from each group according to an existing method, that is, a predetermined standard or algorithm. Or, first, the optimal still image is extracted for each file by the existing method, the spatial information of the extracted optimal still image is obtained, the spatial information of the optimal still image obtained from one file and the other files in the same manner When the spatial information of the obtained optimum still image is compared and it can be determined that they are the same, either one is selected as the optimum still image. By the above processing, it is possible to prevent a plurality of still images acquired in the same situation (position / direction) from being cut out, and to extract only characteristic still images. Embodiments of the present invention will be described below.

<1> Embodiment 1 of the present invention
As described above, the present invention prevents the extraction of a plurality of still images acquired in the same situation (position / direction) from a moving image or a group of still images, and still images having spatial characteristics. In order to facilitate understanding of the present invention, first, the overall configuration of the imaging apparatus according to the embodiment of the present invention will be described.

  FIG. 1 is a block diagram showing an overall configuration of an imaging apparatus 1a according to Embodiment 1 of the present invention. The imaging device 1a according to the present embodiment can capture a moving image and a still image. The present invention can be used in other electronic devices such as a digital still camera, a mobile phone, a PHS (Personal Handyphone System), and a PDA (Personal Digital Assistant) as long as it can capture a moving image. .

  The imaging apparatus 1 a includes an imaging unit 120 having a zoom lens 101, a focus lens 102, a diaphragm 103, and an imaging element 104. The zoom lens 101 is moved along the optical axis LA by a zoom actuator (not shown). Similarly, the focus lens 102 moves along the optical axis LA by a focus actuator (not shown). The diaphragm 103 operates by being driven by a diaphragm actuator (not shown).

  Photographing using the imaging apparatus 1a is performed according to the following procedure. The image sensor 104 photoelectrically converts light that has passed through the zoom lens 101, the focus lens 102, and the aperture 103 to generate an analog image signal of the subject.

  After the analog image signal processing unit 105 amplifies the analog image signal, the A / D conversion unit 106 converts the amplified signal into image digital data. The image input controller 107 takes in the image digital data output from the A / D conversion unit 106 and stores it in the line memory 203 and the main memory 204 via the bus 200.

  When capturing a moving image, the digital image data output from the A / D conversion unit 106 is stored in the main memory 204 via the bus 200.

  When acquiring a still image, a still image frame (frame image) is acquired from the moving image acquired by the image sensor 104 at an arbitrary timing, and frame data that is a part of the frame image is stored in the line memory 203. For example, when acquiring all 60 frames of a moving image of 60 frames per second as still images, moving images from the image input controller 107 may be sequentially stored in the line memory 203, and still images are acquired at arbitrary timing. A configuration (for example, a still image acquisition unit) is not necessary. When a still image is acquired at an arbitrary timing according to a user instruction, for example, a predetermined number of frame images are acquired from a moving image based on the timing when the user presses a shutter button (not shown) of the imaging device 1a, and the frame data The still image acquisition unit can be configured by a switch or the like that performs a switch operation so that the image can be input to the line memory 203.

  By processing as described above, when the imaging apparatus 1 captures a moving image and a still image at the same time, after the still image data is directly compressed by the line memory 203, the still image data after the compression is the main image. Since the data can be stored in the memory 204, the amount of data stored in the main memory 204 can be reduced. As a result, the bus bandwidth of the main memory 204 can be ensured, and for example, compressed still image data such as still image frames acquired 60 frames per second can be written to the main memory 204.

  The digital signal processing unit 108 fetches image digital data stored in the main memory 204 based on a command from the central control unit 100 via the bus 200, performs predetermined signal processing, and includes a luminance signal and a color difference signal. Generate data.

  The digital signal processing unit 108 also performs various digital corrections such as offset processing, white balance adjustment processing, gamma correction processing, RGB interpolation processing, noise reduction processing, contour correction processing, color tone correction processing, and light source type determination processing.

  The multiplexing unit 112 multiplexes the image digital data and audio digital data stored in the main memory 204 to generate stream data.

  The compression / decompression processing unit 201 performs predetermined compression processing on the data stored in the main memory 204 in accordance with an instruction from the central control unit 100 via the bus 200 to generate compressed data. Further, in accordance with a command from the central control unit 100, the compressed data stored in the card-type recording medium 302 or the like is subjected to decompression processing in a predetermined format to generate uncompressed data.

  Note that in the imaging apparatus 1a of the present embodiment, a compression method conforming to the JPEG standard is used for still images, and MPEG2 standard or AVC / H. A compression method conforming to the H.264 standard is adopted.

  The sound / image processing unit 202 performs predetermined image processing on the digital data read from the main memory 204 in accordance with instructions from the central control unit 100 via the bus 200. For example, image data for various processing such as a menu image and an OSD image is generated, and the image data is superimposed on the original image digital data read from the main memory 204 and output to the liquid crystal monitor 304. With this output, the image displayed on the liquid crystal monitor 304 is a combination of various image data.

  The flash ROM 206 stores various setting information relating to the operation of the imaging apparatus 1a, such as user setting information. The main memory 204 is used as a calculation work area of the central control unit 100 and also as a temporary storage area for image data. The VRAM 205 is used as a temporary storage area for image data for display.

  The media control unit 207 controls data writing to and data reading from the card-type recording medium 302 through the card I / F 301 in accordance with instructions from the central control unit 100.

  The position sensor 208 demodulates a GPS signal transmitted by three or more GPS (Global Positioning System) satellites (not shown), and the time when the source GPS satellite transmits a GPS signal (transmission time), The position of the GPS satellite is acquired. Then, the position sensor 208 extracts the time (reception time) at which the GPS signal is received based on the time information acquired by the clock unit 210, and determines each GPS satellite from the extracted reception time and the acquired transmission time. The relative distance from the imaging device 1a is calculated, and the plane position (latitude and longitude information) of the imaging device 1a at the relative distance calculated from each position of the GPS satellite is derived based on the three-point positioning method. To do.

  Here, the position sensor 208 according to the present embodiment uses the GPS signal to derive the planar position of the imaging device 1a, but the position derivation method is not limited to that using the GPS signal. For example, a plane position is derived based on a three-point positioning method using distances from at least three radio towers whose plane positions are fixed, or an accelerometer and a triaxial accelerometer are mounted on the imaging device 1a. The plane position can be derived internally using the integral value of, or various existing plane position derivation methods can be applied.

  The azimuth angle sensor 209 is configured using a geomagnetic sensor, a plurality of GPS elements, and the like, and detects an optical axis azimuth angle at which the optical axis of the imaging unit 120 faces on a horizontal plane.

  Information regarding the plane position detected by the position sensor 208 and the optical axis azimuth angle detected by the azimuth sensor 209 is added as additional information in the stream generated by the multiplexing unit 109. For example, H.M. In H.264 / AVC, the information can be added in SEI (Supplemental Enhancement Information). By assigning the information in the SEI, the information can be held for each picture. On the other hand, the information may be given as metadata separately from the stream. When given as metadata, there is an advantage that information for each picture can be obtained without decoding the stream.

  The liquid crystal monitor 304, the speaker 305, the operation unit 306, and the input / output terminal 307 are connected to the input / output I / F 303. The liquid crystal monitor 304 displays various images such as a captured image and various menu images temporarily recorded in the main memory 204, for example. The speaker 305 outputs sound temporarily recorded in the main memory 204, for example. The operation unit 306 includes an operation key including a relay switch and a power switch (not shown), a cross key, a joystick, a touch panel superimposed on the liquid crystal monitor 304, and the like. Accept.

  The input / output terminal 307 is connected to a television monitor (not shown), a PC (Personal Computer), or the like.

  The central control unit 100 is configured by a semiconductor integrated circuit including a CPU (Central Processing Unit), a ROM storing various programs, a RAM as a work area, and the like. The overall operation of the imaging apparatus 1a, such as still image extraction processing, is comprehensively controlled.

<1-1> Configuration of Optimal Still Image Extracting Device Next, the central control unit 100 in the imaging device 1a according to the present embodiment will be described in detail. The central control unit 100 according to the present embodiment includes an optimum still image extraction unit 401, a spatial information extraction unit 402, and a grouping unit 403. The central control unit 100 is connected to a face image information registration unit 404 in which face image information is registered.

  The central control unit 100 acquires a plurality of moving images stored in the card type recording medium 302 and the like as shooting files via the main memory 204 and the card I / F 301. In this specification, the moving image data stored as one file by the user is referred to as a moving image file, and a group of still images continuously acquired or continuously acquired as a single file, which will be described later, is described below. It is called a still image group file, and a moving image file and a still image group file are collectively called a shooting file. In the present embodiment, a method for extracting an arbitrary number of optimum still images from a plurality of shooting files, particularly moving image files, will be described.

  In the present embodiment, first, the optimum still image extraction unit 401 extracts an optimum still image from each of the plurality of captured files acquired as described above according to a predetermined standard or algorithm. The method for extracting the optimum still image is not particularly limited. For example, when a user registers a face image of a family member or friend in the face image information registration unit 404 and a face image that matches the registered face image is detected in each frame image constituting the moving image data, it is extracted. If more than one are extracted, some or all of each information such as the number of registered face images detected, the size of the face image / position on the screen, focus information / brightness information of the image, smile level, etc. The frame image determined to be optimal is extracted as one optimal still image per file. In this case, one frame image may be extracted as an optimum still image, or two or more frame images may be extracted and combined to generate an optimum still image. By generating one still image from a plurality of frame images, various noises included in the frame image can be removed.

  In the present embodiment, the central control unit 100 is described as performing face image detection. However, as shown in FIG. 1, a face image detection unit 410 including a face image information registration unit (not shown) is provided. A face image detection process may be performed by the face image detection unit 410 and the detection result may be sent to the optimum still image extraction unit 401 of the central control unit 100. In addition to detecting a face image, other objects such as pets may be registered and added to the detection target.

  The spatial information extraction unit 402 includes spatial information including information on at least one of a shooting position and a shooting direction when the optimum still image is shot from each of the plurality of optimum still images extracted by the optimum still image extraction unit 401. To extract. As described above, the imaging apparatus 1a according to the present embodiment includes the position sensor 208 and the orientation sensor 209. In the present embodiment, the positional information acquired by the position sensor 208 and the direction information acquired by the direction sensor 209 are collectively referred to as spatial information.

  Here, in the present embodiment, when shooting moving image data, the position information acquired by the position sensor 208 and the direction information acquired by the azimuth sensor 209 are added to each frame image. . The spatial information extraction unit 402 reads the assigned spatial information.

  In addition, the spatial information may be provided not for each frame but for each of one to a plurality of frame images or one to a plurality of GOPs at a timing at which position information and direction information are acquired. For example, when spatial information is given to the first frame image of each GOP, the spatial information of each frame image included in the first GOP is changed from the spatial information given to the first and second first frame images. It may be obtained by averaging the time. Further, when extracting the optimum still image, only the frame image to which the spatial information is added may be used.

Next, the grouping unit 403 groups a plurality of optimum still images extracted one by one from a plurality of moving image files into one or more groups based on the spatial information extracted by the spatial information extraction unit 402. In this case, for example, when the grouping unit 403 acquires latitude, longitude, and direction (angle) as the spatial information, the latitude / longitude exists in a certain area, and the direction is within a predetermined angle. In this case, it can be determined that they are the same group. A certain area may be simply divided every 1 to several m ² according to the accuracy of the obtained position information, and the area may be a rectangular area.

  Regarding the direction, for example, a certain object is photographed by changing the angle, and the user decides in advance whether to measure a reference value of how much the angle is different from that of a different still image. Also good. Also, for example, if there are 20 video files and you want to obtain 5 optimal still images from them, 20 optimal still images need to be grouped into 5 groups. By setting to obtain a still image, the grouping unit 403 dynamically increases or decreases the reference value of the area or angle so as to group into a desired number of groups. Also good. In this case, a plurality of frame images (optimal still images) are included in one or more of the five groups.

  Specifically, the grouping unit 403 is the n-th (nth) extracted by the optimum still image extraction unit 401 when N (N is a natural number of 2 or more) shooting files are stored in the imaging apparatus 1a. Is based on the spatial information of the optimum still image extracted from the 2nd and (N-1) or less natural number) shooting file and the spatial information of the optimal still image extracted from the (n + 1) th shooting file. By determining whether or not both optimal still images are included in the same group, all N shooting files can be grouped.

  When a plurality of optimum still images are included in the group grouped by the grouping unit 403, the optimum still image extracting unit 401 again extracts one optimum still image for each group. In this case, the optimum still image extraction unit 401 may re-extract the optimum still image according to an existing standard or algorithm similar to or different from the above. In this case, it is possible to use the feature amount, reference value, etc. (second information) of each frame image obtained by the optimum still image extraction unit 401 at the time of the first optimum still image extraction. Alternatively, the spatial information (first information) extracted by the spatial information extraction unit 402 may be used to extract what is determined to be more optimal. Alternatively, the optimal still image may be re-extracted by using the first and second information together.

<1-2> Optimal Still Image Extraction Method Next, an optimal still image extraction method in the imaging apparatus 1a according to the present embodiment will be described. FIG. 2 is a flowchart showing the optimum still image extraction method according to this embodiment. As shown in FIG. 2, first, the central control unit 100 reads a moving image file stored in the card type recording medium 302 or the like via the main memory 204 or the card I / F 301 (step S101). The moving image file in this case is usually a single moving image file, but may be composed of a plurality of moving image files designated as one file by the user. Next, the optimum still image extraction unit 401 extracts one optimum still image from one moving image file by an existing method such as detection of an object (step S102). As described above, the registration information of the face image information registration unit 404 may be used to determine whether or not the image is an optimum still image according to the presence or absence of the registered face image.

  Next, the spatial information extraction unit 402 acquires the spatial information of the frame image (optimal still image) extracted for each moving image file (step S103). At this time, each extracted optimum still image has information on the feature amount used in the optimum still image extraction process. The feature amount information and the spatial information acquired by the spatial information extraction unit 402 are combined to obtain image information of the optimum still image. The image information of these optimum still images is sent to the spatial information extraction unit 402.

  Next, the grouping unit 403 determines whether or not there is an optimum still image extracted from another moving image file in steps S101 to S103 in a storage area (not shown) in the central control unit 100 (step S104). If there is an optimum still image that has already been extracted (hereinafter also referred to as an existing optimum still image) (step S104: Yes), the grouping unit 403 displays the image information of the existing optimum still image and the current optimum still image. The image information, that is, the position / direction information (spatial information) and the feature amount are compared (step S105). When comparing the spatial information, first, the positional information is extracted and compared. If the positional information cannot be regarded as being within the same range (same group), it is not necessary to compare the other spatial information. When the position information matches, the direction information is then compared to determine whether the directions can be regarded as being in the same group.

  As described above, when it is determined that any one or more of the existing optimum still images and the current optimum still image are in the same group (step S106: Yes), the feature amounts of both are compared, If the feature amount of the optimum still image is larger than the feature amount of the existing optimum still image (step S108: Yes), the existing optimum still image in the storage area is overwritten with the current optimum still image (step S110). .

  In this case, if it is determined that the feature amount is larger than that of two or more existing optimum still images, the data of two or more existing optimum still images are deleted and replaced with the data of the current optimum still image. Then, the central control unit 100 determines whether or not there is a next moving image file (step S111), and if there is a moving image file, repeats the processing from step S101.

  On the other hand, if there is no optimum still image extracted from another moving image file in the storage area, the processing of the next moving image file is performed (step S111). If it is not determined in step S106 that the spatial information of both of them is within the same range in the compared optimal still images, the newly extracted optimal still image is also stored as the optimal still image (step S107). If there is a new moving image file, the processing from step S101 is repeated (step S111). As a result of comparing the feature values in step S108, if the optimum still image stored originally has a larger feature value, the optimum still image currently being processed is deleted as invincible (step S109). If there is a new moving image file, the processing from step S101 is repeated (step S111).

  As described above, in the first embodiment of the present invention, the spatial information extraction unit 402 acquires the spatial information for the optimal still image extracted by the existing optimal image extraction method, and the optimal still image is spatially similar. It is determined whether or not shooting has been performed. Therefore, when a plurality of video files are created with the same angle, the extracted spatial information matches within a predetermined range, and these are grouped and a more optimal still image is obtained again by the optimal image extraction process. It is possible to extract a plurality of still images acquired in the same situation (position / direction) as described above.

  Also, even when the position and direction information of each frame image in the moving image file has changed greatly, the optimum still image is extracted by the optimum still image extraction unit 401, so that the optimum still image position information is determined as one. . For this reason, it is possible to reliably extract optimum still images having different positions and directions by comparing the spatial information of the grouping unit 403.

<2> Embodiment 2 of the present invention
Next, a second embodiment of the present invention will be described. In the first embodiment described above, the optimum still image is once extracted by the existing method, and then grouped by the spatial information to extract the optimum still image again. However, in this embodiment, first, the space is extracted. The optimum still image is extracted after grouping by information.

  The configuration of the imaging apparatus according to the present embodiment is the same as that of the imaging apparatus 1a shown in FIG. 1, and the central control unit 100 includes an optimum still image extraction unit 401, a spatial information extraction unit 402, and a grouping unit 403. Prepare. The detailed description is the same as that of the first embodiment and is omitted.

  Next, the still image extraction method according to this embodiment will be described. FIG. 3 is a flowchart showing the optimum still image extraction method according to this embodiment. As shown in FIG. 3, first, as in step S101, the central control unit 100 reads one moving image file stored in the card type recording medium 302 or the like via the main memory 204 or the card I / F 301 ( Step S201). Next, the spatial information extraction unit 402 acquires the position information and direction information of the moving image file, and calculates the feature amount M therefrom (step S202).

  Here, although the spatial information in each moving image file will be described later, simply, the spatial information in the head, center, or last frame image of each moving image file, or an average value thereof can be used. Alternatively, the average value of the spatial information of all the frame images may be calculated. As described above, when the spatial information is given only to a specific position (specific frame image), one of them is used. One or an average value may be used.

  Further, the method of determining the feature value M here is not particularly defined, but the feature value M is a value such that when the calculated value is within a certain range, all the results within that range become M. For example, when the calculated value is in the range of 1.5 to 2.4, the feature value M obtained by rounding off is 2. That is, if the value of the feature amount calculated from the frame image is within the range of 1.5 to 2.4, the feature amount M = 2 of the frame image is obtained. When grouping based on this feature amount, for example, it can be grouped for each value of the feature amount M, or for every certain range such as feature amount M = 0 to 5, 6 to 10,. Grouping is also possible.

Here, an example of a method for calculating the feature amount M will be described.
(1)
The feature quantity M is an array m [5] consisting of five elements, where m [0] is latitude, m [1] is longitude, m [2] is altitude, m [3] is direction information, m [4] Is a face feature amount. The latitude, longitude, and altitude are obtained in the GPX format used in general GPS, and the accuracy can be obtained up to 1/1000000 units. The direction shall be obtained with a built-in compass, and shall be obtained in units of 360 degrees. The face feature amount is obtained from a face image detection unit (not shown) provided in the face image detection unit 410 or the central control unit 100, and represents the face feature amount detected in each frame image.

(2)
For each value, a setting value that defines how far the error is set to the same value is m ′ [5]. For ease of explanation, numerical examples are given only for latitude.

(3)
A value obtained by rounding m [5] according to m ′ [5] is defined as M [5].
A case where M [0] is obtained from m [0] and m ′ [0] will be described.
m [0] = 35.427441
When m ′ [0] = 1/10000, M [0] is obtained by rounding off 1 / 1000th place. That is, M [0] = 35.44274 is obtained.

(4)
Similarly, M is obtained for other elements, and finally an array M [5] is output.

  After the feature amount M is calculated as described above, a group is determined according to the value of the feature amount M (step S203). Here, the number of groups to be grouped may be specified by the user, or the lower limit or upper limit of the number may be designated. If the amount of change in the feature amount M exceeds a predetermined range, the number of groups is set to another group. Such a standard may be provided. Depending on the value of the feature value M, for example, 10 moving image files may be grouped as 10 groups as they are, or may be grouped into groups of less than 10.

  Next, when there is a moving image file not registered in the group (step S204: Yes), the processing from step S201 is repeated. When all the moving image files have been registered as a group, each group is regarded as one moving image file, and the optimum still image extracting process is performed by the optimum still image extracting unit 401 according to the existing optimum still image extracting method as in step S102. To extract one optimum still image for each group (step S205). As described above, the optimum still image extraction unit 401 may extract one frame image to obtain an optimum still image, or extract two or more frame images and combine them to generate an optimum still image. May be. By generating one still image from a plurality of frame images, various noises included in the frame image can be removed.

  As described above, in the second embodiment of the present invention, the spatial information extraction unit 402 uses the spatial information to group moving image files. In other words, conventionally, for example, 10 moving image files are simply regarded as 10 files and optimal still images are extracted, but in the present embodiment, 10 moving image files are spatially different. Whether it is a file or not is determined again according to the spatial information, and it is determined whether or not there are 10 files that are also spatially different. As a result, when it is determined that the files are the same in space, they are grouped and regarded as one file, and thus are acquired in the same situation (position / direction) as in the first embodiment. It is possible to prevent a plurality of still images from being cut out.

  In the first embodiment, the process is a three-stage process of optimum still image extraction, grouping, and optimum still image extraction. In the present embodiment, the two-stage process of grouping and optimum still picture extraction is performed. Therefore, compared with the first embodiment, the entire processing amount of the optimum still image extraction process may be reduced depending on the number of moving image files, the number of frame images constituting the moving image file, and the like. However, as will be described later, when the spatial information changes significantly in one moving image file, such as when shooting while moving, the extracted information about the optimum still image space and the representative space of the moving image file The information may be different, and in that case, it may be preferable for the user to extract the optimum still image by the method of the first embodiment.

<3> Embodiment 3 of the present invention
Next, a third embodiment of the present invention will be described. In the first and second embodiments described above, the method for extracting an optimal still image from a moving image file has been described. However, in this embodiment, a still image that is not a moving image file but is continuously shot or continuously acquired at high speed. A case will be described in which a group is regarded as one file for processing.

  The configuration of the imaging apparatus according to the present embodiment is the same as that of the imaging apparatus 1a shown in FIG. 1, and the central control unit 100 includes an optimum still image extraction unit 401, a spatial information extraction unit 402, and a grouping unit 403. Prepare. The detailed description is the same as that of the first embodiment and is omitted.

  Next, the still image extraction method according to this embodiment will be described. FIG. 4 is a flowchart showing the optimum still image extraction method according to the present embodiment. As shown in FIG. 4, first, as in step S101, the central control unit 100 is saved as one file stored in the card type recording medium 302 or the like via the main memory 204 or the card I / F 301. One still image group continuously or continuously acquired is read (step S201).

  Next, an existing optimum still image extraction process is performed on the read still image group file to extract an optimum still image (step S302). When the processing is completed for one still image group file, it is determined whether there is an optimum still image extracted from another still image group file.

  Next, the grouping unit 403 determines whether or not there is an optimal still image extracted from another moving image file in steps S301 and S302 in a storage area (not shown) in the central control unit 100 (step S303). If there is an optimum still image that has already been extracted (hereinafter also referred to as an existing optimum still image) (step S303: Yes), the grouping unit 403 displays the image information of the existing optimum still image and the current optimum still image. Image information, that is, position / direction information (spatial information) and feature quantity are compared (step S304). If it is determined that the spatial information of one or more of the existing optimum still images and the current optimum still image are within the same range (step S305: Yes), the feature values of both are compared and the current optimum still image is compared. If the feature amount of the image is larger than the feature amount of the existing optimum still image (step S306: Yes), the existing optimum still image in the storage area is overwritten with the current optimum still image (step S307). Then, the central control unit 100 determines whether or not there is a next still image group file (step S310), and if there is a still image group file, repeats the processing from step S301.

  If there is no optimum still image extracted from another moving image file in the storage area, the next still image group file is processed (step S310). In step S305, if it is determined that the spatial information of the two is not within the same range in the compared optimal still images, the newly extracted optimal still image is also stored as the optimal still image (step S308). If there is a new still image group file, the processing from step S301 is repeated (step S310). Further, as a result of comparing the feature amounts in step S306, if the optimum still image stored originally has a larger feature amount, the optimum still image currently being processed is deleted as invincible (step S309), and similarly If there is a new still image group file, the processing from step S301 is repeated (step S310).

  The processes in steps S303 to S310 described above correspond to steps S101 to S111 in the first embodiment shown in FIG. The difference is whether the shooting file from which the optimum still image is extracted is a moving image or a still image group.

  Here, in the case where the shooting file is a continuous still image group that is continuously shot at high speed as in the present embodiment, unlike a still image (frame image) extracted from a moving image, a position is previously stored for each still image. It is possible to add spatial information such as information and direction information. In the present embodiment, for example, when the data captured by the imaging unit 120 is compressed by the compression / expansion processing unit 201 and stored in the main memory 204, the position information from the position sensor 208 and the direction information from the direction sensor 209 are obtained. Acquired and assigned to each still image. Thereby, in the case of a moving image, for example, when spatial information is given at a predetermined frame interval, it is necessary to calculate and give the spatial information of the extracted still image (frame image) based on the spatial information. By providing spatial information in advance at the time of shooting, it is not necessary to calculate the spatial information of the extracted still image.

  For example, only position information may be given at the time of shooting, and the direction information may be acquired and compared as necessary, that is, when the position information matches. As a result, the process of giving direction information to all the still images is reduced.

  As described above, in the third embodiment of the present invention, as in the first and second embodiments, since the optimum still image is extracted using the spatial information, it is acquired in the same situation (position / direction). In the same manner as in the first embodiment, even when the position and direction information of each still image are greatly changed in the continuous still image group file, it is possible to prevent a plurality of still images that have been cut out. Optimal still images with different positions and directions can be extracted with certainty.

  Furthermore, since the spatial information is preliminarily assigned to each still image included in the still image group, the processing for acquiring and giving the spatial information as in step S103 is unnecessary, and when comparing the spatial information in step S304, The spatial information extraction unit 402 only needs to read and compare the spatial information given to each still image, and the processing in the spatial information extraction unit 402 can be reduced.

<4> Embodiment 4 of the present invention.
Next, a fourth embodiment of the present invention will be described. In Embodiments 1 to 3 described above, basically, as in Embodiments 1 and 3, the optimum still images are extracted from the captured file by the existing optimum still image extraction process, and then grouped by spatial information. Then, there are two methods: a method of extracting an optimum still image, and a method of first grouping the shooting files by spatial information and then extracting the optimum still image by an existing optimum still image extraction process as in the second embodiment. Explained. Hereinafter, the mode for extracting the optimum still image by the methods of the first and third embodiments is referred to as a first mode, and the method for extracting the optimum still image by the method of the second embodiment is referred to as a second mode.

  In this embodiment, it is automatically determined which method of the first mode or the second mode is used to perform the optimum still image extraction. For example, when the optimum still image is extracted from the above-described embodiment, particularly the moving image file shown in the first and second embodiments, the optimum still image extraction is performed using any of the first and second modes. In order to determine whether to perform the determination, a criterion as to whether or not the spatial information of the shooting file is uniquely determined can be used.

<4-1> Configuration of Optimal Still Image Extracting Device FIG. 5 is a block diagram showing a configuration of an imaging device 1b according to Embodiment 4 of the present invention. As illustrated in FIG. 5, the imaging apparatus 1b according to the present embodiment newly adds a mode determination unit to the central control unit 100 in order to determine which of the first and second modes is used. The point having 405 is different from the first embodiment.

  The mode determination unit 405 extracts the amount of change in the spatial information of at least a part of the frame image or still image included in each shooting file, and if the amount of change in the spatial information is greater than a predetermined threshold value, When the mode is selected and the amount of change in the spatial information is smaller than a predetermined threshold, the second mode is selected.

  Specifically, the mode determination unit 405 calculates an average value, a maximum value, and a minimum value of the amount of change in spatial information in each frame image or still image included in each shooting file for each shooting file. The larger one of the differences between the average value and the maximum value or the minimum value is obtained as the representative value A. When the representative value A calculated for each shooting file is larger than the predetermined threshold T, the first mode is selected. When the representative value A selected for each captured file is smaller than the predetermined threshold T, the second mode can be selected.

  Here, in the present embodiment, the mode determination unit 405 is described as selecting a mode according to the amount of change in spatial information, but instead of the amount of change in spatial information, Alternatively, the first or second mode can be selected based on one or more information of the number of shooting files and the total number of frame images or still images included in all shooting files. That is, the first mode includes three stages of extraction processing, grouping, and extraction processing, and the second mode includes two stages of grouping and extraction processing. The total processing time of either process depends on the number of frame images or still images to be extracted in the extraction process. Grouping depends on the number of files to be grouped or the number of frame images or still images. It is also possible to determine the speed of the processing time according to these conditions and select the first mode or the second mode based on this.

<4-2> Optimal Still Image Extraction Method Next, an optimal still image extraction method in the imaging apparatus 1b according to the present embodiment will be described. FIG. 6 is a flowchart showing the optimum still image extraction method according to this embodiment. In this example, it is assumed that the shooting file is made up of a moving image. For example, when a moving image is shot with an imaging device with GPS, the GPS information is updated per unit time and embedded in the moving image stream. When a moving image is shot while moving, the GPS information may change greatly even within the same moving image (one moving image file). Therefore, as shown in FIG. 6, first, an arbitrary spatial information change amount threshold value T is set (step S401). The threshold T may be obtained in advance by experimentation or the like, and when the first mode is recommended, the threshold T is decreased and the second mode is recommended. In such a case, the threshold value T may be increased or changed dynamically.

  Next, for all moving image files, all or part of the spatial information of all frame images included in each moving image file is referred to (step S402), and for each moving image file, the average value, maximum value, Ask for. Then, the difference between the average value and the maximum value and the minimum value is obtained, and the larger one of them is determined as the representative value A indicating the representative position information of the moving image (step S403).

  Then, the representative value A and the threshold value T of all moving image files are compared (step S404). If at least one representative value A> threshold value T among all the moving image files, it is determined that the representative value A of the moving images cannot be grouped, and the first mode shown in FIG. After extracting the still image, the optimum still image is extracted by a mode for grouping using spatial information such as position information (step S405). On the other hand, if the representative value A <threshold value T of all moving image files, the representative value A of each moving image file is set (step S406). Then, optimum still image extraction is performed in the second mode shown in FIG. That is, after grouping with the representative value A of each moving image file, the optimum still image extraction is performed in the mode for performing the optimum still image extraction (step S407).

  As described above, in the fourth embodiment of the present invention, the mode for extracting the optimum still image can be dynamically changed according to the amount of change in the position information of each moving image file. In addition, since the representative value A is calculated as the information on the amount of change in the position information, when the second mode is determined, the representative value A can be used for grouping as it is. For example, if the position information has changed significantly in one video file, the position information of the optimum still image may differ from the representative position information of the video file, and the same situation (position / direction) In this way, it may not be possible to accurately determine whether or not the image has been shot, but in this way, it is possible to match the optimal still image with the representative position information of the video file by using the amount of change in the position information Therefore, it is possible to accurately determine whether or not the image is taken in the same situation. As described above, the user can automatically extract an optimum still image for each scene from an arbitrary moving image file group.

  It should be noted that the present invention is not limited to the above-described embodiments, and various modifications can be made without departing from the scope of the present invention.

  For example, the above-described optimum still image extraction processing, spatial information extraction processing, grouping processing, and mode determination processing can be realized by causing a CPU to execute a computer program. In this case, the computer program can be stored using various types of non-transitory computer readable media and supplied to the computer. Non-transitory computer readable media include various types of tangible storage media. Examples of non-transitory computer-readable media include magnetic recording media (for example, flexible disks, magnetic tapes, hard disk drives), magneto-optical recording media (for example, magneto-optical disks), CD-ROMs (Read Only Memory), CD-Rs, CD-R / W and semiconductor memory (for example, mask ROM, PROM (Programmable ROM), EPROM (Erasable PROM), flash ROM, RAM (random access memory)) are included. The program may also be supplied to the computer by various types of transitory computer readable media. Examples of transitory computer readable media include electrical signals, optical signals, and electromagnetic waves. The temporary computer-readable medium can supply the program to the computer via a wired communication path such as an electric wire and an optical fiber, or a wireless communication path.

DESCRIPTION OF SYMBOLS 1a Imaging device 1b Imaging device 100 Central control part 120 Imaging part 201 Compression / decompression processing part 203 Line memory 204 Main memory 206 Flash ROM
208 Position sensor 209 Direction sensor 301 Card I / F
302 Card Type Recording Medium 401 Optimal Still Image Extraction Unit 402 Spatial Information Extraction Unit 403 Grouping Unit 404 Face Image Information Registration Unit 405 Mode Determination Unit 410 Face Image Detection Unit

Claims (8)

A storage unit for storing a plurality of shooting files consisting of either a moving image or a series of still images acquired continuously or continuously;
An optimal still image extraction unit that extracts an optimal still image from each of the plurality of shooting files stored in the storage unit according to a predetermined standard or algorithm;
Spatial information extraction for extracting, from each of the plurality of optimum still images extracted by the optimum still image extraction unit, spatial information including information on at least one of a photographing position and a photographing direction when each optimum still image is photographed. And
A grouping unit that groups the plurality of optimum still images into one or more groups based on the spatial information extracted by the spatial information extraction unit;
The optimum still image extraction unit, when a plurality of optimum still images are included in the group grouped by the grouping unit, extracts one optimum still image for each group again. apparatus. The storage unit stores N (N is a natural number greater than or equal to 2) or more shooting files,
The grouping unit includes spatial information of the optimal still image extracted from the nth (n is a natural number of 2 or more and (N−1) or less) imaging file extracted by the optimal still image extraction unit, The image processing apparatus according to claim 1, wherein it is determined whether or not both optimal still images are included in the same group based on the spatial information of the optimal still image extracted from the (n + 1) th shooting file. The optimal still image extracting unit generates a plurality of groups by the grouping unit, and re-extracts an optimal still image for each group, the first information including spatial information, and the predetermined reference or algorithm The image processing apparatus according to claim 1, wherein the optimum still image is extracted using at least one of the second information including the reference value and the feature amount used in step 1. A mode determining unit that extracts a change amount of the spatial information of at least a part of a frame image or a still image included in each shooting file, and determines a mode based on a magnitude of the change amount of the spatial information;
When the amount of change in the spatial information is larger than a predetermined threshold, the optimum still images are extracted from the optimum still image extracting unit, then grouped by the grouping unit, and optimized again by the optimum still image extracting unit as necessary. Select the first mode for extracting still images,
When the amount of change in the spatial information is smaller than a predetermined threshold value, the second mode in which the optimum still image is extracted by the optimum still image extracting unit after being grouped by the grouping unit is selected. Item 8. The image processing apparatus according to Item 1 . In addition to or in addition to the amount of change in the spatial information, the mode determination unit may include information on one or more of the number of shooting files and the number of frame images or still images included in all shooting files. The image processing apparatus according to claim 4 , wherein the first mode or the second mode is selected based on the first mode. The mode determination unit calculates, for each shooting file, an average value, a maximum value, and a minimum value of the amount of change in the spatial information in each frame image or still image included in each shooting file, and the average value and The larger one of the differences from the maximum value or the minimum value is obtained as a representative value, and when the representative value calculated for each shooting file is larger than a predetermined threshold, the first mode is selected and each shooting file is selected. The image processing apparatus according to claim 4 , wherein the second mode is selected when the representative value calculated every time is smaller than a predetermined threshold. Optimal to extract the optimum still image from each of the plurality of shooting files stored in the storage unit storing a plurality of shooting files consisting of moving images, continuous shooting or a group of still images acquired continuously. A still image extraction process;
Spatial information for extracting, from each of the plurality of optimum still images extracted in the optimum still image extraction step, spatial information including information on at least one of a photographing position and a photographing direction when the optimum still image is photographed. An extraction process;
A grouping step of grouping the plurality of optimum still images into one or more groups based on the spatial information extracted in the spatial information extraction step;
In the optimum still image extracting step, when a plurality of optimum still images are included in the group grouped in the grouping step, one optimum still image is extracted again for each group. Image processing method. Optimal to extract the optimum still image from each of the plurality of shooting files stored in the storage unit storing a plurality of shooting files consisting of moving images, continuous shooting or a group of still images acquired continuously. Still image extraction processing,
Spatial information for extracting, from each of the plurality of optimum still images extracted in the optimum still image extraction process, spatial information including information on at least one of a photographing position and a photographing direction when the optimum still image is photographed. Extraction process,
A program for causing a computer to execute a grouping process for grouping the plurality of optimum still images into one or more groups based on the spatial information extracted in the spatial information extraction process,
In the optimum still image extraction process, when a plurality of optimum still images are included in the group grouped by the grouping process, one optimum still image is extracted again for each group. program.

Images