JP2020088663A - Imaging apparatus - Google Patents

Abstract

To provide a UI that makes it easy to extract a still image from a video.SOLUTION: When an image difference with a plurality of frames is large, the difference is colored and cut out and displayed as one sheet. If the image difference is small, display a list of the difference areas side by side.SELECTED DRAWING: Figure 4

Description

The present invention relates to an image pickup apparatus using a solid-state image pickup element.

In digital cameras in recent years, the frame speed of continuous shooting has increased, and the number of pixels of moving images has increased, so that there is an increasing number of opportunities to select an arbitrary frame from a plurality of continuously shot frames.

There is also a function to let the camera choose the best shot by detecting the subject or scene, but in some cases you may want to choose your own.

When selecting one by one, it is common to display candidate images side by side among a plurality of frames and select from them. For example, in a medical image, it may be desired to compare the past image with the latest image. As described in (1) and (2), the images are displayed side by side with the past image, and the difference image between the past image and the latest image is displayed together.

Further, in subject recognition, face detection is performed as described in Patent Document 2, and face areas are cut out and arranged.

Japanese Patent Laid-Open No. 2008-536 JP, 2011-130113, A

However, a PC can use a monitor widely, and it is easy to enlarge and arrange them. However, since the display area is small in the camera body, it is easy to understand by enlarging or coloring the differences. There is a demand for a function that assists in selecting an image to be displayed.

Furthermore, there is a need to not only fit the difference area to the display area but also to use the display area efficiently according to the relationship between the subject of interest and the difference area.

In order to solve the above problems, an image pickup apparatus according to the present invention is, as described in claim 1, an image pickup unit that obtains image data from an image pickup element, a display unit that displays image data, and a plurality of images. It is characterized in that it has a calculation means for calculating the difference of data, determines the size of the difference area of the image by comparing with a preset threshold value, and changes the display method of the image data according to the size of the difference area. ..

According to the imaging device of the present invention, when extracting a still image from a moving image or selecting one still image from a plurality of images, the frame difference is displayed in an easy-to-understand manner, so that a desired frame can be easily obtained. ..

Configuration of the imaging device according to the present embodiment Circuit diagram for integrating the difference value in the image pickup apparatus according to the present embodiment The figure explaining operation|movement of 1st Embodiment Operation flow of the first embodiment The figure explaining operation|movement of 2nd Embodiment Operation flow of the second embodiment The figure explaining operation|movement of 3rd Embodiment Operation flow of the third embodiment

Hereinafter, modes for carrying out the present invention will be described with reference to the drawings.

In FIG. 1, 100 is an image processing apparatus. In the present embodiment, the configuration of a digital camera is shown as an example of the image processing apparatus 100.

Reference numeral 10 is a taking lens including a focus lens, and 12 is a mechanical shutter having a diaphragm function. Reference numeral 14 is an image pickup element for converting an optical image incident from an optical system such as the photographing lens 10 and shutter 12 into an electric signal, and 16 is an A/D (analog/digital) for converting an analog signal output of the image pickup element 14 into a digital signal. It is a converter.

A timing generation circuit 18 supplies a clock signal and a control signal to the image pickup device 14 and the A/D converter 16, and is controlled by the memory control circuit 22 and the system control circuit 50. In addition to the 12 mechanical shutters, the storage time can be controlled as an electronic shutter by controlling the reset timing of the 18 image sensor, and the electronic shutter can be used for shooting moving images.

An image processing circuit 20 performs predetermined pixel interpolation processing and color conversion processing on the data from the A/D converter 16 or the data from the memory control circuit 22. Further, the electronic zoom function is realized by cropping an image and performing a scaling process by the 20 image processing circuits. Further, the image processing circuit 20 performs a predetermined arithmetic process using the captured image data.

Based on the obtained calculation result, the system control circuit 50 performs AF (autofocus) processing, AE (automatic exposure) processing, and EF (flash light control) processing on the exposure control means 40 and the distance measurement control means 42. .. Further, the image processing circuit 20 also performs AWB (auto white balance) processing.

Further, the image processing circuit 20 includes a subject detection circuit. A known method such as pattern matching or feature point extraction can be applied as the subject detection method and is not particularly limited. Although a face is used as an example of the subject to be detected in the present embodiment, another subject (subject region) may be detected using a known subject detection method, and the subject to be detected is not limited. In the present embodiment, the subject detection circuit is a face detection circuit, which outputs the coordinates of the face area (area information), the coordinates of the eyes, etc. as the output of the detection result.

A memory control circuit 22 controls the A/D converter 16, the timing generation circuit 18, the image processing circuit 20, the memory 30, and the compression/expansion circuit 32. The data of the A/D converter 16 is written into the memory 30 via the image processing circuit 20 and the memory control circuit 22, or the data of the A/D converter 16 is directly written into the memory 30.

Reference numeral 28 denotes an image display unit including a TFT, LCD, etc. The display image data written in the memory 20 is displayed by the image display unit 28 via the memory control circuit 22. The electronic viewfinder function can be realized by sequentially displaying the captured image data using the image display unit 28. Further, the image display unit 28 can arbitrarily turn on/off the display according to an instruction from the system control circuit 50, and when the display is turned off, the power consumption of the image processing apparatus 100 can be significantly reduced. Can be done.

Reference numeral 30 denotes a memory for storing captured still images and moving images, which has a storage capacity sufficient to store a predetermined number of still images and moving images for a predetermined time. As a result, even in the case of continuous shooting or panoramic shooting in which a plurality of still images are continuously shot, a large amount of images can be written in the memory 30 at high speed.

The memory 30 can also be used as a work area of the system control circuit 50.

Reference numeral 31 is a non-volatile memory composed of a flash ROM or the like. The program code executed by the system control circuit 50 is written in the non-volatile memory 31, and the program code is executed while being sequentially read. In addition, an area for storing system information and an area for storing user setting information are provided in the non-volatile memory, and various information and settings are read out and restored at the next startup.

Reference numeral 32 denotes a compression/decompression circuit that compresses/decompresses image data by adaptive discrete cosine transform (ADCT) or the like, reads an image stored in the memory 30 and performs compression processing or decompression processing, and stores the processed data in the memory 30 Write in.

An exposure control unit 40 controls the shutter 12 having a diaphragm function, and also has a flash dimming function by interlocking with the flash 48.

Reference numeral 42 is a distance measuring control means for controlling focusing of the taking lens 10, and 44 is zoom control means for controlling zooming of the taking lens 10.

A flash 48 has an AF auxiliary light projection function and a flash light control function. The exposure control unit 40 and the distance measurement control unit 42 are controlled by using the TTL method, and the system control circuit 50 causes the exposure control unit 40 and the distance measurement to be performed based on the calculation result obtained by calculating the captured image data by the image processing circuit 20. The control means 42 is controlled.

A system control circuit 50 controls the entire image processing apparatus 100.

Reference numerals 60, 62, 64, 66, 70, and 72 are operation means for inputting various operation instructions of the system control circuit 50, and are a single unit such as a switch, a dial, a touch panel, pointing by visual line detection, a voice recognition device, or the like. It is composed of multiple combinations.

Here, a concrete description of these operating means will be given.

Reference numeral 60 denotes a mode dial switch, which can switch and set various function modes such as power-off, automatic shooting mode, shooting mode, HDR shooting mode, panoramic shooting mode, moving picture shooting mode, playback mode, and PC connection mode.

Reference numeral 62 denotes a shutter switch SW1, which is turned on during operation of the shutter button and instructs the start of operations such as AF (autofocus) processing, AE (automatic exposure) processing, and AWB (auto white balance) processing.

Reference numeral 64 denotes a shutter switch SW2, which is turned on when the operation of the shutter button is completed. In the case of flash photography, after the EF (flash pre-emission) process is performed, the image sensor 14 is exposed for the exposure time determined in the AE process.

In the case of flash photography, light is emitted during this exposure period, and the exposure control means 40 blocks light at the same time as the end of the exposure period, thus ending the exposure to the image sensor 14. Further, the start of a series of processes up to the recording process of writing the signal read from the image sensor 14 into the recording medium 200 is instructed. More specifically, a read process of writing image data in the memory 30 via the A/D converter 16 and the memory control circuit 22, a development process using calculation in the image processing circuit 20 and the memory control circuit 22, and a memory 30. The image data is read from and the compression/expansion circuit 32 performs compression.

A display changeover switch 66 can change the display of the image display unit 28. With this function, it is possible to save power by cutting off the current supply to the image display unit including a TFT LCD or the like when shooting is performed using the optical finder 104.

Reference numeral 70 denotes an operation unit including various buttons, a touch panel, a rotary dial, and the like, which includes a menu button, a set button, a macro button, a multi-screen playback page break button, a flash setting button, a single shooting/continuous shooting/self timer switching button, and the like. .. There are also menu move + (plus) button, menu move-(minus) button, playback image move + (plus) button, playback image-(minus) button, shooting image quality selection button, exposure compensation button, date/time setting button, etc. is there.

Reference numeral 72 denotes a zoom switch unit as a zoom operation unit that allows the user to instruct to change the magnification of the captured image. Hereinafter, it is also referred to as the zoom switch 72. The zoom switch 72 is composed of a tele switch for changing the image pickup angle of view to the telephoto side and a wide switch for changing the image pickup angle of view to the wide angle side. By using this zoom switch 72, it becomes a trigger for instructing the zoom control means 44 to change the imaging angle of view of the taking lens 10 and performing an optical zoom operation. The image processing circuit 20 also serves as a trigger for clipping an image and electronically changing the imaging angle of view by pixel interpolation processing or the like.

Reference numeral 86 denotes a power supply means including an alkaline battery primary battery, a NiCd battery, a NiMH battery, a secondary battery such as a Li-ion battery, and an AC adapter.

Reference numeral 90 is an interface with a recording medium such as a memory card or hard disk, and 92 is a connector for connecting with a recording medium such as a memory card or hard disk.

A protection unit 102 is a barrier that covers the image pickup unit including the lens 10 of the image processing apparatus 100 to prevent the image pickup unit from being soiled or damaged.

Reference numeral 104 denotes an optical viewfinder, which enables shooting using only the optical viewfinder without using the electronic viewfinder function of the image display unit 28.

A communication unit 110 has various communication functions such as USB, IEEE1394, LAN, and wireless communication.

Reference numeral 112 denotes a connector for connecting the image processing apparatus 100 to another device by the communication unit 110 or an antenna in the case of wireless communication.

Reference numeral 200 is a recording medium such as a memory card or a hard disk.

The recording medium 200 includes a recording unit 202 including a semiconductor memory and a magnetic disk, an interface 204 with the image processing apparatus 100, and a connector 206 for connecting to the image processing apparatus 100.

210 is a recording medium such as a memory card or a hard disk.

Each configuration in the image processing apparatus 100 described above may be configured by hardware, or a part or all thereof may be configured as a software module.

FIG. 2 is a circuit diagram for integrating the difference value in the image pickup apparatus of the present embodiment.

The first and second frames for calculating the difference are used as input images, and the difference circuit 201 calculates the frame difference. The difference value obtained for each pixel is integrated by the integrating circuit 202 to obtain an integrated value of the difference between the two frames. Further, the difference obtained by the difference circuit 201 is binarized by the binarization circuit 203 and integrated by the integration circuit 204 to obtain the number of pixels having the difference.

In addition, in integration, all pixels may be integrated, but it may be divided into a plurality of blocks and the difference presence/absence determination may be performed for each block by acquiring an integral value for each block.

<First Embodiment>
FIG. 3 is a diagram for explaining the operation of the first embodiment.

When the ratio of tightening the difference region is large with respect to the image, it is difficult to visually recognize the difference if the image is reduced too much. Therefore, as shown in FIG. 3A, one image is displayed and the difference area is colored to make it easier to recognize the difference in movement of the person. With this, a display suitable for selecting a desired pose is displayed.

On the other hand, when the ratio of tightening the difference area is small with respect to the image, the difference is cut out and enlarged for easy visual recognition as shown in FIG. Furthermore, by displaying a plurality of images for comparing the differences of a plurality of frames, a display suitable for selecting a change in facial expression is displayed.

FIG. 4 is an operation flow of the first embodiment.

Using the first frame and the second frame for calculating the difference as input images, frame difference detection is performed (401). The size of the frame difference is determined using a preset threshold value (402), and if the frame difference is large, the difference area is colored (403), and the difference area is cut out so that it fits (404), and one image is displayed. (405).

When the frame difference is small in the determination of the frame difference in 402, the frame difference is cut out so that the difference area fits (406), and it is determined whether or not the necessary number of images to be subjected to image comparison have been prepared (407). If so, a list is displayed (408). If it is determined in 407 that the required number of sheets has not been completed, in 406, the clipping of the image is repeated until the required number of sheets is completed.

<Second Embodiment>
FIG. 5 is a diagram for explaining the operation of the second embodiment.

In addition to obtaining difference information for multiple images, face detection is performed by pattern matching that considers the placement of facial parts such as eyes and mouth, and main subject detection is performed that divides regions based on image color luminance information and detects main regions. By using it, the operation is changed depending on the magnitude relation of the information.

FIG. 5A is an example in which the difference area is larger than the face area, and the difference area is displayed so that it fits in order to grasp the entire person including the limbs. Alternatively, the main subject area may be displayed so as to fit in the face area.

FIG. 5B is an example in which the face area is larger than the difference area, and is cut out in the difference area in order to grasp the facial expression. Alternatively, when it is desired to grasp the atmosphere of the entire face, the face region may be cut out as a region slightly larger than the difference region.

Further, depending on the scene, the difference area and the face area may not overlap, but in that case, the area is set to a wider area so as to include them and the face area is cut out.

FIG. 6 is an operation flow of the second embodiment.

Face detection 601 is performed using one of the two images for which difference detection is performed, and main subject detection 602 is performed to obtain a face area and a main subject area. Further, a frame difference detection 603 is performed to obtain a difference area.

The face area and the difference area are compared (604), and if there is a difference area in the face area, the cutout frame is set so that the difference area fits (605), and cutout 606 and display 607 are performed. In 604, the face area and the difference area are compared, and when there is no difference area in the face area, the cutout frame is set so that the area including the difference area and the face area fits (608), the cutout 606, and the display 607. Do.

<Third Embodiment>
FIG. 7 is a diagram for explaining the operation of the third embodiment.

A state before the user selects an image is shown in FIG. FIG. 7A shows that a plurality of images having different characteristics are presented from a moving image or a plurality of still images. Among the images that can be selected, a scene in which the ball is handled by the foot, a running scene, and a scene in which the ball is handled by the hand are presented as characteristic images. The image difference is the basis of the image selection at that time, and the difference from the first image or a specific image is repeatedly acquired for the images of the image group that can be an option. An image determined to have a large difference among these differences is presented as a candidate image. The user looks at the screen where a plurality of candidate images are presented and selects an image to be left as a candidate for image selection.

The state after the user selects an image is shown in FIG. FIG. 7B shows that a plurality of images having similar characteristics are presented among a moving image or a plurality of still images. By selecting a scene in which the user is running from the group of images that can be selected, images similar to the scene in which the user is running are presented side by side as candidate images. Specifically, the difference is acquired based on the image selected by the user, and the image with the small difference is presented as a candidate image. As a result, in the running scene, it is possible to easily select a favorite image depending on the facial expression and pose.

FIG. 8 is an operation flow of the third embodiment.

When the flow for image selection on the playback screen or the edit screen is entered, images with large image differences are displayed as a list of candidate images (801), and it is determined whether the user has selected one of these candidate images (802). ), if it is determined that the user has selected an image, an image having a small difference from the selected image is displayed as a list of candidate images (803). If the user has not selected an image in 802, the state is held until the user makes a selection.

100 image processing device, 10 shooting lens, 12 mechanical shutter,
14 image pickup devices, 16 A/D converter

Claims (8)

An image pickup means for obtaining image data from the image pickup element,
Display means for displaying image data,
Calculation means for calculating the difference between a plurality of image data,
An image pickup apparatus comprising: a threshold value that is compared with a preset threshold value to determine the size of a difference area of an image, and to change the display method of image data according to the size of the difference area. The image pickup apparatus according to claim 1, wherein when the difference area is large, one image data is displayed on the display means, and when the difference area is small, the plurality of image data are displayed side by side on the display means. When the difference area is large, the difference area of the image data displayed on the display means is colored and highlighted, and when the difference area is small, the image data arranged on the display means is cut out so as to include the difference area and displayed. The image pickup apparatus according to claim 2. An image pickup means for obtaining image data from the image pickup element,
Display means for displaying image data,
Calculation means for calculating the difference between a plurality of image data,
And a method of displaying image data, which is changed according to a positional relationship between a subject of interest and a differential region that are separately detected. When the subject of interest is included in the difference region, the subject is cut out and displayed so as to include the difference region, and when the subject of interest is included in the difference region, the subject of interest is cut out and displayed. The imaging device described. The imaging device according to claim 5, wherein the subject of interest is a face detected by face detection. The imaging device according to claim 5, wherein the subject of interest is the main subject detected by detecting the main subject. An image pickup means for obtaining image data from the image pickup element,
Display means for displaying image data,
Calculation means for calculating the difference between a plurality of image data,
When displaying multiple image data side by side, when there is no user selection, the image data with a large difference is displayed side by side, and when there is a user selection, the image data with a small difference is displayed side by side. Image pickup device.