JP2020170998A - Image processing apparatus, imaging apparatus, image processing method, and program - Google Patents

Abstract

To solve the problem in which it is difficult to grasp the entire moving image in a case of using one frame of the moving image as a representative image.SOLUTION: An image processing apparatus comprises selection means for allowing a user to select plural frames out of a moving image; synthesis means for generating a synthetic image on the basis of the frames; and control means which uses the synthetic image as the representative image of the moving image.SELECTED DRAWING: Figure 6

Description

The present invention relates to an image processing device that creates a representative image of a moving image.

An image processing device such as a digital camera may be provided with a function of generating a representative image (thumbnail image) from the moving image so that the user can easily recognize the content of the moving image. For example, Patent Document 1 describes the content of generating a representative image using one frame of a moving image.

Japanese Patent Application Laid-Open No. 8-251540

However, in the technique described in Patent Document 1, since a specific one frame is used as a representative image, it may be insufficient to grasp the content of the moving image. For example, in a moving image in which a moving object moves along a certain trajectory, if a specific frame is used as a representative image, the trajectory of the moving object cannot be grasped from the representative image.

The present invention has been made in view of the above problems, and an object of the present invention is to provide an image processing device that creates a representative image that can represent the contents of a plurality of frames of a moving image.

In order to solve the above problems, the present invention describes a selection means in which a user selects a plurality of frames from a moving image, a compositing means for generating a composite image from the plurality of frames, and the composite image as a representative of the moving image. Provided is an image processing apparatus characterized by having a control means used as an image.

According to the configuration of the present invention, when generating a representative image of a moving image, it is possible to provide an image processing device that generates a representative image that can represent the contents of a plurality of frames of the moving image.

It is a block diagram which shows the hardware structure of the image pickup apparatus in embodiment of this invention. It is a figure for demonstrating the format of the moving image file in embodiment of this invention. It is a figure for demonstrating the structural example of the data of the still image file in embodiment of this invention. It is a figure for demonstrating an example of the screen to be displayed on the display part 130 in embodiment of this invention. It is a figure for demonstrating the generation of the representative image in embodiment of this invention. It is a figure for demonstrating composition of an image in embodiment of this invention. It is a figure for demonstrating an example of the screen to be displayed on the display part 130 after substituting the representative image in embodiment of this invention.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

FIG. 1 is a block diagram showing a hardware configuration of an imaging device according to the present embodiment. The image pickup device in the present embodiment is, for example, a digital camera as an example, but may be a mobile phone capable of taking an image.

The control unit 101 includes, for example, a CPU (MPU), a memory (DRAM, SRAM), a non-volatile memory (EEPROM), and the like, and executes various processes (programs) to control each block of the image pickup apparatus 100. Control data transfer between blocks. Further, the control unit 101 controls each block of the image pickup apparatus 100 in response to an operation signal from the operation unit 102 that receives an operation from the user. In addition, the control unit 101 analyzes the image obtained by the image processing unit 111 described later, and controls each block of the image pickup apparatus 100 according to the analysis result.

The operation unit 102 includes switches for inputting various operations related to imaging, such as a power button, a still image recording button, a moving image recording start button, a zoom adjustment button, and an autofocus button. It also includes a menu display button, an enter button, other cursor keys, a pointing device, a touch panel, and the like, and when the user operates these keys and buttons, an operation signal is transmitted to the control unit 101.

The bus 103 is a general-purpose bus for sending various data, control signals, instruction signals, and the like to each block of the image pickup apparatus 100.

The non-volatile memory 105 is a memory that can be electrically erased and recorded, and stores constants, programs, and the like for the operation of the control unit 101.

The image pickup unit 110 converts the optical image of the subject captured by the lens into an image signal by an image pickup element such as a CCD sensor or a CMOS sensor, controls the amount of light by an aperture, performs analog-to-digital conversion, and performs an analog-digital conversion to an image processing unit. Send to 111.

The image processing unit 111 performs image quality adjustment processing on the input digital image signal to adjust the white balance, color, brightness, etc. based on the set value. Then, the image signal processed by the image processing unit 111 is transmitted by the control unit 101 to the memory 104, the video output unit 150 described later, and the display control unit 131.

In the present embodiment, for example, the image pickup unit 110 has an "optical zoom" function and an "optical vibration isolation" function, and the image processing unit 111 has an "electronic zoom" function and an "electronic vibration isolation" function. ing. Here, the "optical zoom" function and the "electronic zoom" function are functions for enlarging the obtained image according to the operation of the user. Further, the "optical vibration isolation" function and the "electronic vibration isolation" function are functions for preventing image shake due to vibration of the image pickup apparatus 100 main body. Then, these functions are used simultaneously, alternately, or independently under the control of the control unit 101.

Here, the "optical zoom" function is a function of moving the lens of the imaging unit 110 in response to the operation of the zoom key of the operation unit 102 by the user to enlarge / reduce the optical image of the captured subject. .. Further, the "electronic zoom" function means that the image processing unit 111 enlarges an image obtained by cutting out a part of the image generated by the imaging unit 110 in response to the user operating the zoom key of the operation unit 102. This is a process for generating an image signal. In addition, the "optical vibration isolation" function calculates the amount of movement of the lens based on the value of the acceleration signal from the vibration detection unit (not shown) and moves the lens to prevent the acquired image from shaking. It is a function to do. Further, the "electronic vibration isolation" function prevents the image from shaking by adjusting the position of the image signal acquired by the imaging unit 110 based on the value of the acceleration signal from the vibration detection unit (not shown). It is a function. Further, the "electronic vibration isolation" function can also be realized by shifting the reading position of the image sensor of the image pickup unit 110 based on the value of the acceleration signal from the vibration detection unit (not shown).

Further, the audio input unit 120 collects (collects) the sound around the image pickup device 100 by, for example, a built-in omnidirectional microphone or an external microphone connected via the audio input terminal, and analog. It is digitally converted and transmitted to the voice processing unit 121. The voice processing unit 121 performs processing related to voice such as processing for optimizing the level of the input digital voice signal. Then, the voice signal processed by the voice processing unit 121 is transmitted to the memory 104 by the control unit 101. The memory 104 temporarily stores the image signal and the audio signal obtained by the image processing unit 111 and the audio processing unit 121.

The image processing unit 111 and the audio processing unit 121 read out the image signal and the audio signal temporarily stored in the memory 104, encode the image signal, encode the audio signal, and the like, and perform the compressed image signal and the compressed audio signal. Etc. are generated. The control unit 101 transmits these compressed image signals and compressed audio signals to the recording / playback unit 140.

The recording / reproducing unit 140 records the compressed image signal, the compressed audio signal, and other control data related to imaging on the recording medium 141, which are generated by the image processing unit 111 and the audio processing unit 121. When the audio signal is not compressed and encoded, the control unit 101 transmits the audio signal generated by the audio processing unit 121 and the compressed image signal generated by the image processing unit 111 to the recording / playback unit 140. It is recorded on the recording medium 141.

The recording medium 141 may be a recording medium built in the image pickup apparatus 100 or a removable recording medium. The recording medium 141 can record various data including a compressed image signal, a compressed audio signal, and an audio signal generated by the image pickup apparatus 100, and a medium having a capacity larger than that of the non-volatile memory 105 is generally used. .. For example, the recording medium 141 includes all types of recording media such as hard disks, optical disks, magneto-optical disks, CD-Rs, DVD-Rs, magnetic tapes, non-volatile semiconductor memories, and flash memories.

Further, the recording / reproducing unit 140 reads out and reproduces the compressed image signal, the compressed audio signal, the audio signal, various data, and the program recorded on the recording medium 141. Then, the control unit 101 transmits the read compressed image signal and compressed audio signal to the image processing unit 111 and the audio processing unit 121. The image processing unit 111 and the audio processing unit 121 temporarily store the compressed image signal and the compressed audio signal in the memory 104, decode them according to a predetermined procedure, and decode the decoded audio signal into the audio output unit 151. The signal is transmitted to the video output unit 150 and the display control unit 131. When the audio signal is uncompressed and recorded on the recording medium 141, the control unit 101 directly transmits the audio signal to the audio output unit 151.

The audio output unit 151 is composed of, for example, an audio output terminal, and transmits an audio signal in order to output audio from a connected earphone, speaker, or the like. Further, the audio output unit 151 may be a speaker built in the image pickup apparatus 100 and outputting audio related to the audio signal. The video output unit 150 is composed of, for example, a video output terminal, and transmits an image signal in order to display the video on a connected external display or the like. Further, the audio output unit 151 and the video output unit 150 may be one integrated terminal, for example, a terminal such as an HDMI (registered trademark) (High-Definition Multimedia Interface) terminal.

Further, the display control unit 131 causes the display unit 130 to display an image based on the image signal transmitted from the image processing unit 111, an operation screen (menu screen) for operating the image pickup apparatus 100, and the like. The display unit 130 may be any display device such as a liquid crystal display, an organic EL display, or electronic paper.

The communication unit 152 communicates between the image pickup device 100 and the external device, and transmits or receives data such as an audio signal, an image signal, a compressed audio signal, and a compressed image signal, for example. It also transmits and receives control signals related to imaging, such as imaging start and end commands, and other information. The communication unit 152 is, for example, a wireless communication module such as an infrared communication module, a Bluetooth (registered trademark) communication module, a wireless LAN communication module, a WirelessUSB, and a GPS receiver.

Next, the normal operation of the image pickup apparatus 100 of the present embodiment will be described.

When the user operates the power button of the operation unit 102, the image pickup apparatus 100 of the present embodiment is instructed to start by the operation unit 102 to the control unit 101. In response to this instruction, the control unit 101 controls a power supply unit (not shown) to supply power to each block of the image pickup apparatus 100.

When the power is supplied, the control unit 101 indicates from the operation unit 102, for example, which mode the mode changeover switch of the operation unit 102 is, for example, a still image imaging mode, a moving image imaging mode, or a playback mode. Confirm by signal.

In the still image imaging mode, the imaging device 100 takes an image by operating the still image recording button of the operation unit 102 in the imaging standby state, and the compressed image signal is recorded on the recording medium 141. Then, the imaging standby state is set again. In the moving image imaging mode, the imaging device 100 starts imaging by the user operating the moving image recording start button of the operation unit 102 in the imaging standby state, and during that time, the compressed image signal and the compressed audio signal or the audio signal are transmitted to the recording medium 141. Recorded. Then, when the user operates the moving image recording end button of the operation unit 102, the imaging is terminated, and the imaging standby state is set again. In the reproduction mode, the compressed image signal, the compressed audio signal, or the audio signal related to the file selected by the user is reproduced from the recording medium 141, the audio signal is output from the audio output unit 151, and the image is displayed on the display unit 130.

First, the still image imaging mode will be described. When the still image imaging mode is set by the operation unit 102, the control unit 101 first sets each block of the imaging device 100 to the imaging standby state as described above.

In the image pickup standby state, the image processing unit 111 transmits an image signal to the display control unit 131 and causes the display unit 130 to display an image related to the image signal. The user prepares for imaging while looking at the screen displayed in this way.

When the image pickup instruction signal is transmitted by the user operating the still image recording button of the operation unit 102 in the image pickup standby state, the control unit 101 transmits the image pickup control signal to each block of the image pickup apparatus 100 as follows. Control to make the operation.

The image pickup units 110 and 112 convert the optical image of the subject captured by the lens into an image signal by the image pickup element, analog-digitally convert it, and transmit it to the image processing unit 111. The image processing unit 111 processes the image quality adjustment processing (white balance, color, brightness, etc.) of the input digital image signal based on the set value. Then, the image signal processed by the image processing unit 111 is transmitted by the control unit 101 to the memory 104, the video output unit 150, and the display control unit 131. Here, the user can confirm the captured still image by viewing the image displayed on the display unit 130.

Then, the image processing unit 111 reads out the image signal temporarily stored in the memory 104, performs predetermined coding, generates a compressed image signal, and outputs the compressed image signal to the recording / reproducing unit 140. The recording / playback unit 140 writes the compressed image signal as a still image file to the recording medium 141 under the control of a file system such as UDF or FAT.

When the coding operation by the image processing unit 111 is completed, the control unit 101 transmits a control signal to each block of the image pickup apparatus 100 so as to shift to the image pickup standby state, and returns to the image pickup standby state.

Next, the moving image imaging mode will be described. When the moving image imaging mode is set by the operation unit 102, first, as described above, the control unit 101 causes each block of the imaging device 100 to be set to the imaging standby state.

In the image pickup standby state, the image processing unit 111 transmits an image signal to the display control unit 131 and causes the display unit 130 to display an image related to the image signal. The user prepares for imaging while looking at the screen displayed in this way.

When the user operates the moving image recording start button of the operation unit 102 in the imaging standby state to transmit the imaging start instruction signal, the control unit 101 transmits the imaging start control signal to each block of the imaging device 100. , Control to operate as follows.

The image pickup units 110 and 112 convert the optical image of the subject captured by the lens into an image signal by the image pickup element, analog-digitally convert it, and transmit it to the image processing unit 111. The image processing unit 111 processes the image quality adjustment processing (white balance, color, brightness, etc.) of the input digital image signal based on the set value. Then, the image signal processed by the image processing unit 111 is transmitted by the control unit 101 to the display control unit 131 and the memory 104. The display control unit 131 causes the display unit 130 to display an image related to the received image signal.

On the other hand, the audio input unit 120 digitally converts the analog audio signal obtained by the microphone and transmits the obtained digital audio signal to the audio processing unit 121. The audio processing unit 121 outputs an audio signal by performing processing for optimizing the level of the input digital audio signal and the like. The control unit 101 transmits the voice signal processed by the voice processing unit 121 to the memory 104.

Then, the image processing unit 111 and the audio processing unit 121 read the image signal and the audio signal temporarily stored in the memory 104 and perform predetermined coding to generate the compressed image signal, the compressed audio signal, and the like. Then, the control unit 101 synthesizes these compressed image signals and compressed audio signals, forms a data stream, and outputs the data stream to the recording / reproducing unit 140. The recording / playback unit 140 writes the data stream as one moving image file to the recording medium 141 under the control of a file system such as UDF or FAT. When the audio is not compressed, the control unit 101 outputs the audio signal generated by the audio processing unit 121 to the recording / playback unit 140 together with the compressed image signal generated by the image processing unit 111. Then, as described above, the recording / playback unit 140 writes the data stream as one moving image file to the recording medium 141 under the control of the file system such as UDF and FAT.

The above operation is continued during imaging.

Further, during imaging, the control unit 101 may perform the image pickup units 110, 112 and the image processing unit 111 according to the operation of the operation unit 102 by the user or the analysis result of the image signal generated by the image processing unit 111. , Various control signals are transmitted to the voice processing unit 121 and the like. For example, a control signal for moving the lens and adjusting the aperture is transmitted to the imaging units 110 and 112, and a control signal for adjusting the image and sound is transmitted to the image processing unit 111 and the sound processing unit 121.

Similarly, during imaging, the user operates the zoom key of the operation unit 102, so that the control unit 101 performs the "optical zoom" function of the imaging units 110 and 112 and the "electronic zoom" function of the image processing unit 111. Can be operated. Further, based on the acceleration signal detected by the vibration detection unit (not shown), the control unit 101 operates the "optical vibration isolation" function of the imaging units 110 and 112 and the "electronic vibration isolation" function of the image processing unit 111. are doing.

Then, when the user operates the moving image recording end button of the operation unit 102 to transmit the imaging end instruction signal to the control unit 101, the control unit 101 sends the imaging end control signal to each block of the imaging device 100. It is transmitted and controlled so that the following operations are performed.

The image processing unit 111 and the audio processing unit 121 stop transmitting the image signal and the audio signal to the memory 104, respectively. Then, the remaining image signal and audio signal stored in the memory 104 are read out and subjected to predetermined coding to generate a compressed image signal, a compressed audio signal, and the like.

The control unit 101 synthesizes these final compressed image signals and compressed audio signals to form a data stream, and outputs the data stream to the recording / reproducing unit 140. When the audio is not compressed, the control unit 101 outputs the audio signal generated by the audio processing unit 121 and the compressed image signal to the recording / playback unit 140.

The recording / playback unit 140 writes the data stream as one moving image file to the recording medium 141 under the control of a file system such as UDF or FAT. Then, when the supply of the data stream is stopped, the control unit 101 controls so as to perform the following operations in order to generate thumbnails.

The recording / reproducing unit 140 reads the compressed image signal of the first frame of the moving image file recorded on the recording medium 141 and transmits it to the image processing unit 111. The image processing unit 111 temporarily stores the compressed image signal in the memory 104 and decodes it in a predetermined procedure. Next, the image processing unit 111 performs predetermined coding for thumbnails on the obtained image signal to generate a compressed image signal for thumbnails. Then, the control unit 101 outputs the thumbnail compressed image signal to the recording / playback unit 140. Under the file system management of UDF, FAT, etc., the recording / playback unit 140 writes the compressed image for thumbnails to the recording medium 141 so as to be combined with the base moving image file, completes the moving image file, and performs the recording operation. Stop.

When the recording operation is stopped, the control unit 101 transmits a control signal to each block of the imaging device 100 so as to shift to the imaging standby state, and returns to the imaging standby state.

It is possible to store the copyright holder name and the creator name in the non-volatile memory 105 in response to the operation of the operation unit 102 by the user. Further, each time the operation unit 102 is operated, the copyright holder name and the creator name stored in the non-volatile memory 105 can be changed. The copyright holder name and creator name stored in the non-volatile memory 105 are recorded as metadata in a still image file or a moving image file according to the imaging process.

The still image file and the moving image file recorded in the still image imaging mode and the moving image imaging mode described above are assumed to be recorded according to DCF (Design rule for Camera File system).

Next, the playback mode will be described. When the reproduction mode is set by the operation unit 102, the control unit 101 transmits a control signal to each block of the image pickup apparatus 100 so as to shift to the reproduction state, and performs the following operations.

The recording / playback unit 140 reads a still image file composed of a compressed image signal recorded on the recording medium 141, or a moving image file composed of a compressed image signal and a compressed audio signal or an audio signal. The control unit 101 sends the read compressed image signal and compressed audio signal to the image processing unit 111 and the audio processing unit 121. If it is an uncompressed voice signal, the control unit 101 transmits the voice signal to the voice output unit 151.

The image processing unit 111 and the audio processing unit 121 temporarily store the compressed image signal and the compressed audio in the memory 104, and decode them in a predetermined procedure. Then, the control unit 101 transmits the decoded audio signal to the audio output unit 151, and the decoded image signal to the video output unit 150 and the display control unit 131. The display control unit 131 causes the display unit 130 to display the image related to the input image signal, and the audio output unit 151 displays the sound related to the input audio signal from the built-in speaker or the connected earphone or speaker. Output.

As described above, the image pickup apparatus 100 records and reproduces still images and moving images.

FIG. 2 is a diagram for explaining the format of the moving image file in the present embodiment. The format of the moving image file handled in this embodiment is MP4, and the image is H. 264, it is assumed that the audio is compressed by AAC.

In the MP4 format, the data recorded in the file is described inside a data structure called a "box" and is recorded in the file in units of boxes. The box consists of the following fields.
Size: A 4-byte field that represents the data size of the entire box, including the Size field Type: A 4-byte type identifier that represents the type of box.

Since the fields after the Type field are optional depending on the box, the description thereof is omitted here. In addition, it is possible to enclose the box inside the box, thereby realizing a nested structure.

ftyp201 is a file type box, which is a box indicating a file type. The movie 202 is a movie box, and stores information related to audio and video, including mvhd (movie header box) and truck (track box) described later. The mdat 203 is a movie data box, and stores actual data such as audio, video, and text.

A typical box in moov202 will be described. The uid 204 is a general-purpose unique identifier box in which an arbitrary identifier or data is stored. The stored data includes a thumbnail 230, which is an image obtained by cutting out one frame of a recorded moving image and reducing it. The udta 205 is a user data box, and stores metadata such as the model name of the image pickup device 100, the position information of the image pickup device 100 received by the communication unit 152, and the UTC time. In mvhd206, the creation date and time, modification date and time, time scale, Duraion, etc. of the moving image are described. The trak207 is a video trak, and includes the types of video codecs, stts (time sample box), stsc (sample chunk box), stss (synchronous sample box), stsz (sample size box), and stco (chunk offset) described later. Information about the video data in the mdat is stored. The trak 208 is an audio trak, and stores information about audio data in mdat such as a sampling frequency, the number of bits, and the number of channels. The trak 209 is a text trak, and stores information about the text data in mdat.

The tkhd210 stores the track creation date and time, Duration, ID, and the like. In the present embodiment, the ID of the video track is 1, the ID of the audio track is 2, and the ID of the text track is 3.

The edts211 is an edit box, and the playback start time of the track is described. For example, when it is desired to shift the start of audio reproduction later than the start of reproduction of video, it can be realized by describing a value in the audio edts.

Reference information between tracks is described in tref212. In the present embodiment, since the video track refers to the text track as a chapter, the text track ID 3 is described in the video tref, and since it is referred to as a chapter, it is described as'chap'as an identifier. Has been done. The same is true for audio tracks. There is no tref in the text track.

The stsc213 represents how many samples (frames in the case of video) a chunk is composed of. The stts214 stores the number of samples and the duration of each sample. The stss215 has a keyframe number and exists only in the video track. For example, when the GOP (Group of Pictures) is 3, it is described as 1, 4, 7 ..., when the GOP is 12, 1, 13, 25 ..., When the GOP is 15, it is described. It is described as 1, 16, 31 ... 216 is stsz, and the data size of each sample is stored. The stco217 stores the file offset of each chunk.

Wide218 is a 64-bit expansion box when the moving image file size is 4 GB or more.

The text chunks 219 and 220 are text chunks stored in mdat, and the title of each chapter is described as text data. The voice chunks 221 and 222 represent voice chunk data stored in mdat. The video chunks 223 and 224 represent the video chunk data stored in mdat.

As described above, by using the information stored in stsz and stco for video, it is possible to access any video frame or chapter in mdat.

FIG. 3 is a diagram for explaining a structural example of data of a still image file in the present embodiment.

The image file 301 has a marker (SOI) 302 indicating the start of the image file at the beginning, and then has an application marker (APP1) 303 corresponding to the header portion. The application marker (APP1) 303 is composed of the following.
Size (APP1 Lens) 303, application marker identification code (APP1 Identifier Code) 305, image data creation date and time (Date Time) 306, image data generation date and time (Data Time Original) 307, other imaging information 308, Thumbnail image (Thumbnail Data) 309.

The image data recorded in the image file 301 is composed of a quantization table (DQT) 310, a Huffman table (DHT) 311, a frame start marker (SOF) 312, a scan start marker (SOS) 313, and compressed data 314. .. Then, it is terminated by a marker (EOI) 315 indicating the end of the image file data.

FIG. 4 is a diagram for explaining an example of a screen displayed on the display unit 130 in the present embodiment. On the screen shown in FIG. 4, the thumbnail 230 stored in the uid 204 displays a moving image, and the thumbnail image 309 in FIG. 3 displays a still image. In FIG. 4, the moving image is shown to be a moving image by displaying icons representing a film on both sides of the image. Image 401 represents a moving image.

FIG. 5 is a diagram for explaining the generation of a representative image in the present embodiment. Hereinafter, the flow of the present embodiment will be described with reference to FIG.

In step S501, the user (user) of the imaging device 100 operates the operation unit 102 to instruct reproduction, and the recording / reproduction unit 140 reproduces the moving image. The moving image here may be a moving image previously stored in the recording medium 141, or may be a moving image captured by the imaging unit 110 immediately before.

In step S502, the control unit 101 determines whether or not the reproduction of the moving image is completed. In step S503, the user selects a frame of the moving image to be used for compositing.

FIG. 6 is a diagram for explaining the composition of images in the present embodiment. In FIG. 6, it is shown that the image 610 is synthesized by using the frames 601 to 604 of the moving image. In the moving image of FIG. 6, a person moving in front of the background is shown. While the moving image is being reproduced, NO is determined in step S502, the flow proceeds to step S503, and the user selects the frame 601 through the operation unit 102 in step S503. Next, the process returns to step S501, the reproduction of the moving image continues, and the process proceeds to step S502 again, and the control unit 101 determines whether or not the reproduction of the moving image is completed. If it is determined that the reproduction is not completed, the process proceeds to step S503, and the user selects the frame 602 to be used for the composition again. Thus, the user selects frames 601 to 604 to be used for composition until the reproduction is completed.

Next, in step S504, the image processing unit 111 synthesizes the frames 601 to 604. There are many methods of compositing, but as an example, the image processing unit 111 recognizes the human image and the background of the subject from each of the frames 601 to 604, and cuts out the subject from each image. Next, the image processing unit 111 creates a background of the composite image by using the background image from which the subject is cut out. Finally, the image processing unit 111 adds the cut-out subject to the background of the composite image to create a composite image.

The synthesis method described above is only an example, and for example, known trajectory synthesis or comparative bright synthesis can be used.

In step S505, the image processing unit 111 compresses the composite image created in step S504 to have the same size as the image 401 shown in FIG. In this way, the created composite image can replace the image 401 as a representative image of the moving image.

FIG. 7 is a diagram for explaining an example of a screen to be displayed on the display unit 130 after substituting the representative image in the present embodiment. In FIG. 7, it can be seen that the representative image 701 shows a composite image.

Further, the control unit 101 may provide a step of automatically determining the composite image before substituting the composite image with the representative image. For example, if the time of the first and last frames used in the composite image generated by the composite is close, it may be difficult to grasp the entire moving image even if the composite image is used. In such a case, the control unit 101 gives an appropriate presentation / warning to the user through the display unit 130. Alternatively, after generating the composite image, compare the first frame and the last frame used for the composite image, and if the difference is less than a predetermined threshold, grasp the entire moving image even if the composite image is used. May be difficult. Similarly, in such a case, the control unit 101 gives an appropriate presentation / warning to the user through the display unit 130. Alternatively, if the recording time of the moving image itself is short, the merit of using the composite image itself is small, and the control unit 101 may give a presentation / warning to the user through the display unit 130 so as not to recommend the composition.

The composite image generated in step S504 may be further subjected to image processing to create a representative image. For example, an image having an MPF (Multi-Picture-Format) format formulated by the Camera & Imaging Products Association may be used as a representative image.

By using the composite image created in the present embodiment as the representative image of the moving image, it is possible to grasp the movement of the subject over the entire moving image rather than using the image of one frame as the representative image.

(Other embodiments)
Although the above embodiment has been described based on the implementation using a digital camera, the present embodiment is not limited to the digital camera. For example, it may be carried out by a portable device having a built-in image sensor, or a network camera capable of capturing an image.

In the present invention, a program that realizes one or more functions of the above-described embodiment is supplied to a system or device via a network or a storage medium, and one or more processors in the computer of the system or device program. It can also be realized by the process of reading and operating. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

100 Imaging device 101 Control unit 102 Operation unit 103 Bus 104 Memory 105 Non-volatile memory

Claims (13)

A selection method for the user to select multiple frames from the moving image,
A compositing means for generating a composite image from the plurality of frames, and
An image processing apparatus comprising: a control means for using the composite image as a representative image of the moving image. The image processing apparatus according to claim 1, wherein the representative image is a thumbnail image. The image processing apparatus according to claim 1 or 2, wherein the control means uses an image that compresses the composite image as the representative image. The image processing apparatus according to any one of claims 1 to 3, wherein the compositing means recognizes a background and a subject from the plurality of frames, cuts out the subject, and generates the composite image. .. The image processing apparatus according to claim 4, wherein the composite image includes a plurality of cut-out subjects added to the composite image. The image processing apparatus according to any one of claims 1 to 5, wherein the compositing means generates the composite image when the plurality of frames satisfy predetermined conditions. The image processing apparatus according to any one of claims 1 to 5, further comprising a warning means for giving a warning when the plurality of frames do not satisfy a predetermined condition. The predetermined conditions are that the difference between the time of the first frame and the time of the last frame of the plurality of frames is smaller than the predetermined value, and that the first frame of the plurality of frames The image processing apparatus according to claim 6 or 7, wherein the difference between the last frames is smaller than or equal to a predetermined threshold value. The image processing apparatus according to claim 8, wherein the predetermined value is determined based on the reproduction time of the moving image. Has a display means
The image processing apparatus according to any one of claims 1 to 9, wherein the display means displays the representative image. Imaging means for capturing moving images and
A selection means for the user to select a plurality of frames from the moving image, and
A compositing means for generating a composite image from the plurality of frames, and
An image processing apparatus comprising: a control means for using the composite image as a representative image of the moving image. A selection step in which the user selects multiple frames from the moving image,
The compositing step of generating a composite image from the plurality of frames and
An image processing method comprising: a control step of using the composite image as a representative image of the moving image. A computer program that operates an image processing device on a computer.
A selection step in which the user selects multiple frames from the moving image,
The compositing step of generating a composite image from the plurality of frames and
A program characterized in that a control step of using the composite image as a representative image of the moving image is performed.

Images