JP5517192B2 - Electronic device, image processing method and program - Google Patents

Description

  The present invention relates to an electronic device having a photographing function, an image processing method for processing captured image data, and a program for image processing.

  Conventionally, there are various techniques for processing or processing an image captured using an electronic imaging device (digital camera) that captures a still image. For example, a digital camera is used to capture an image with a subject in the background, and then a background image with no subject in the background is captured to generate difference information from the background image and the image with the subject. One of them is an invention related to an image processing technique in which only the image is cut out (for example, Patent Document 1).

Japanese Patent Laid-Open No. 10-21408

  In a digital camera to which the technology disclosed in Patent Document 1 is applied, in order to cut out a subject such as an object or a person who is a main subject of photographing from a background image, after photographing the image containing the subject, It is necessary to perform an operation of moving the subject without taking the camera to shoot an image without the subject. However, if the background image is inappropriate in terms of contrast or if the first photographed image is unclear due to backlighting, cropping of the subject may fail, and the user will have to start over from the first photographing again. There is a problem that it is troublesome because it must be done.

  An object of the present invention is to provide an electronic device, an image processing method, and a program capable of reliably obtaining a clipped image of a subject in one try.

In order to achieve the above object, the invention described in claim 1
Imaging means;
Display means capable of displaying an image based on an imaging signal from the imaging means;
Subject specifying means capable of specifying any subject within the imaging range of the imaging means;
Subject recognition means for recognizing the contour of the subject designated by the subject designation means during the first imaging by the imaging means;
Display control means for displaying on the display means the contour recognized by the subject recognition means;
Subject extracting means for extracting an image of the subject specified by the subject specifying means based on the first captured image and the second captured image by the imaging means;
Subject discriminating means capable of discriminating a subject moved within an imaging range by the imaging means;
With
The subject discrimination means
A feature amount computing unit for extracting feature points in the second captured image;
By searching whether the feature point corresponding to where in said first captured image, before Symbol first aligning the captured image and the previous SL second captured image, aligned with the previous SL first from the difference between the captured image and the previous SL second captured image, anda block matching unit that determines the moved object,
The subject designating unit designates the subject determined to have moved by the subject determining unit ;
It is characterized by that.

The invention according to claim 2 is the invention according to claim 1 ,
The image obtained by the first imaging is a moving image.

The invention according to claim 3 is the invention according to claim 1 or 2 ,
The image processing apparatus further includes image combining means for generating a composite image by combining the subject image extracted by the subject extraction means and the selected background image.

The invention according to claim 4
An image processing method in an electronic device including an imaging unit and a display unit,
A subject designation step for designating any subject within an imaging range of the imaging means;
A subject recognition step of recognizing the contour of the subject designated by the subject designation step during the first imaging by the imaging means;
A subject contour display step for causing the display means to display the contour recognized by the subject recognition step;
A subject extraction step of extracting an image of the subject designated by the subject designation step based on a first captured image and a second captured image by the imaging means;
A subject discrimination step capable of discriminating a subject moved within an imaging range by the imaging means;
Including
Wherein the object determination step, the feature points are extracted in the second captured image, by extracting feature points is searched whether corresponding to where in said first captured image, before Symbol first captured image When the front Symbol to align the second captured image, from the difference between the aligned the previous SL first captured image and the previous SL second captured image, to determine the moved object,
In the subject specifying step, the subject determined to have moved in the subject determining step is specified.
It is characterized by that.

The invention described in claim 5
A microprocessor that controls the imaging means and the display means;
Subject designation means capable of designating any subject within the imaging range of the imaging means based on a signal from the input means;
Subject recognition means for recognizing the contour of the subject designated by the subject designation means during the first imaging by the imaging means;
Display control means for displaying the contour recognized by the subject recognition means on the display means;
Subject extracting means for extracting an image of the subject specified by the subject specifying means based on the first captured image and the second captured image by the imaging means;
Subject discrimination means capable of discriminating a subject moved within an imaging range by the imaging means;
Function as
As the object discriminating means, wherein to extract a second feature point from the captured image of, by the extracted the feature point is searched where to corresponding said first captured image, before Symbol first captured image When the front Symbol to align the second captured image, from the difference between the aligned the previous SL first captured image and the previous SL second captured image, to determine the moved object,
As the subject specifying means, the subject determined to have been moved by the subject determining means is specified ,
It is characterized by that.

  According to the present invention, in an electronic device having a photographing function and a function of generating a cut-out image, it is possible for a user to predict whether or not a subject will be cut out successfully, thereby cutting out the subject in one try. There is an effect that an image can be obtained reliably.

It is a block diagram which shows one Embodiment of the mobile telephone as an example of the electronic device to which this invention is applied. It is explanatory drawing which shows the structural example of the flag provided in the control part of the mobile telephone of embodiment. It is image explanatory drawing which shows an example of the image image | photographed by the imaging means, and the image cut out. It is a block diagram which shows schematic structure of the imaging part and image recognition part of the mobile telephone of FIG. It is a flowchart which shows the first half part of the control for the image processing performed in the mobile telephone of embodiment to which this invention is applied. It is a flowchart which shows the second half part of the control for the image processing performed in the mobile telephone of embodiment to which this invention is applied. It is a flowchart which shows the procedure of the extraction subject designation | designated process in the flowchart of FIG.

  DESCRIPTION OF EXEMPLARY EMBODIMENTS Hereinafter, preferred embodiments of the invention will be described with reference to the drawings.

  FIG. 1 is a block diagram of an embodiment of a mobile phone as an example of an electronic apparatus having an imaging function to which the present invention is applied.

  As shown in FIG. 1, the mobile phone 10 of this embodiment controls a whole device according to a program and performs a CPU (microprocessor) that performs image processing, and a read-only memory that stores a program executed by the CPU (11) and fixed data. ROM (read-only memory), CPU (11) working area, and control unit 11 including RAM (random access memory) that can be written and read at any time to provide a temporary data storage area, numeric keys, function keys, etc. A touch panel type display unit 13 having a liquid crystal panel 131 and a touch panel 132 for input, and a power source unit 14 for supplying a DC power source voltage to electronic components inside the device.

  In addition, the mobile phone 10 includes a telephone unit 15 that drives a speaker 121 or amplifies an audio signal from the microphone 122 and converts it into a digital signal, a filter, a power amplifier, and the like. A communication unit 16 for performing voice and data communication with the base station, a storage unit 17 including a writable nonvolatile memory for storing address book data, captured images, and the like, and a CCD (Charge Coupled Device) ) And CMOS (Complementary Metal-Oxide Semiconductor) and other image sensors and optical systems such as lenses, and an image recognition unit 19 that processes image data captured by the image capture unit 18. They are connected to each other via a bus 20.

FIG. 4 shows a schematic configuration of an imaging unit 18 and an image recognition unit 19 that processes image data captured by the imaging unit 18.
The imaging unit 18 includes a lens unit 181 including a plurality of zoom lenses, a focus lens, and a lens driving unit that moves the lens in the optical axis direction, an image sensor for photoelectric conversion, a driving circuit that scans the sensor, and a timing generation circuit. And an imaging control unit 183 that controls the lens driving unit and the image sensor to read out an image signal. The imaging control unit 183 also performs adjustment control of imaging conditions such as AE (automatic exposure processing), AF (automatic focusing processing), and AWB (automatic white balance).

  The image recognition unit 19 includes an image data generation unit 191 that generates image frames for each screen from the image signal output from the electronic imaging unit 182, and an image memory 192 that stores the image frames generated by the image data generation unit 191. From the feature amount calculation unit 193, the block matching unit 194, the image processing unit 195, the recording control unit 196, the display control unit 197 for driving and controlling the liquid crystal panel 131 of the touch panel type display unit 13, and the touch panel 132 of the display unit 13 An input detection unit 198 for detecting an input is provided. FIG. 4 shows a shutter button 121 and a selection determination button 122 among various input keys constituting the input operation unit 12.

  The image data generation unit 191 appropriately adjusts the gain for each of the RGB color components with respect to the analog value image signal output from the electronic imaging unit 182, and then performs sample holding by a sample hold circuit (not shown) to perform A / The digital signal is converted into digital data by a D converter (not shown), color processing including pixel interpolation processing and γ correction processing is performed by a color process circuit (not shown), and then the luminance signal Y and color difference signal data Cb of digital values are processed. , Cr (YUV data).

  The generated luminance signal Y and color difference signal data Cb and Cr are DMA-transferred to an image memory 192 used as a buffer memory by a DMA controller (not shown). The image memory 192 is configured by, for example, a DRAM or the like, and temporarily stores data processed by the feature amount calculation unit 193, the block matching unit 194, the image processing unit 195, the control unit (CPU) 11, and the like. The feature amount calculation unit 193 performs feature extraction processing for extracting feature points from the background image with reference to the background image. Specifically, the feature amount calculation unit 193 selects, for example, a predetermined number (or a predetermined number or more) of high feature block areas (feature points) based on the YUV data of the background image, The contents are extracted as a template (for example, a square of 16 × 16 pixels). Here, the feature extraction process is a process of selecting a feature having a high characteristic convenient for tracking from a large number of candidate blocks.

  When the block matching unit 194 generates the subject cutout image, the image f1 in which the target subject B as shown in FIG. 3A exists and the subject B as shown in FIG. 3C do not exist at the same position. Block matching processing for alignment with an image (hereinafter referred to as a background image) f3 is performed. Specifically, the block matching unit 194 corresponds to where in the subject existing image f1 the template extracted in the feature extraction process corresponds, that is, the template pixel value optimally matches in the subject existing image f1. Search for a position (corresponding region).

  The image processing unit 195 has a determination function for determining whether or not the subject B exists in the background based on a plurality of image frames (including moving images) stored in the image memory 192. The image processing unit 195 also includes an image f1 (FIG. 3A) in which the subject B exists from the image frames stored in the image memory 192 and a background image f3 in which the subject B does not exist at the same position (FIG. c)) based on the difference information of each corresponding pixel between the background image f3 and the subject presence image f1 is aligned using the template extracted by the block matching unit 194 It has a function of generating an image f4 (FIG. 3D) obtained by cutting out the subject B from the image f1. The image control unit 196 stores the clipped image f4 generated as described above in the recording unit 17.

  The display control unit 197 includes a VRAM, a VRAM controller, a digital video encoder, and the like, and performs control to read display image data temporarily stored in the image memory 192 and display the display image data on the display unit 13. Specifically, the digital video encoder periodically reads the luminance signal Y and the color difference signals Cb and Cr read from the image memory 192 and stored in the VRAM (not shown) from the VRAM via the VRAM controller. Based on these data, a video signal is generated and output to the display unit 13. Based on the video signal supplied from the display control unit 197, the display unit 13 displays a processed image such as a live view image, a captured image, or a cutout image on the display screen of the liquid crystal panel 131. The input detection unit 198 detects an input from the touch panel 132 and inputs a corresponding signal to the control unit (CPU) 11.

  In the present embodiment, the image recognizing unit 19 touches a subject to be cut out in an image frame f1 as shown in FIG. 3A displayed on the screen of the liquid crystal panel 131 by the image pickup signal from the image pickup unit 18. If designated, if the designated subject can be cut out in the subsequent processing, the contour is correctly displayed as in the image f2 in FIG. Therefore, if the user performs the first shooting operation after confirming the image, and then performs the second shooting with the subject moved from the shooting range, the image recognition unit 19 performs the first and second shooting. A subject image B is extracted from the subject existing image f1 based on the difference information of the corresponding pixels between images from the captured images f1 and f3 (see FIGS. 3A and 3C), and a cutout image f4 ( FIG. 3D) can be generated.

Next, a control procedure for image processing in the first embodiment of the mobile phone having an imaging function to which the present invention is applied will be described with reference to the flowcharts of FIGS.
As shown in FIG. 5, when the power is turned on, the control unit (CPU) 11 first clears the designation flag to “0” in the initial setting (step S1). In the designation flag, as shown in FIG. 2, “0” represents a state in which no subject to be extracted is designated, and “1” represents a state in which a subject to be extracted is designated. Thereafter, the mobile phone shifts to a standby state, which is a normal state (step S2). Then, it is determined whether there is an incoming call (step S3). When there is an incoming call, it is determined whether or not an off-hook operation has been performed (step S4). If it is determined in step S4 that there is no off-hook operation (= NO), the process returns to step S3. If it is determined in step S4 that there is an off-hook operation (= YES), call processing (step S5) is executed. Thereafter, it is determined whether or not an on-hook operation has been performed (step S6). When it is determined in step S6 that there is no on-hook operation (= NO), the process waits for the on-hook operation to be performed. After the execution, the process returns to step S3.

  On the other hand, when it is determined in step S3 that there is no incoming call (= NO), it is determined whether or not the extracted image shooting operation is set (step S8). If it is determined that there is no shooting operation setting (= NO), the process proceeds to step S22 in FIG. If it is determined in step S8 that the shooting operation has been set (= YES), it is determined whether or not the subject to be extracted has been designated on the touch panel 132 (step S9). The detailed procedure of the extracted subject designation process S9 will be described later with reference to FIG.

  Thereafter, it is determined whether or not the designation flag is “1” (step S10). If the subject is designated, the designation flag is set to “1” in the extracted subject designation process S9. If the designation flag is “0” (= NO), the process jumps to step S13. If the designation flag is “1” (= YES), a process of recognizing the contour of the designated subject (step S11) is performed. As shown in FIG. 3B, the recognized contour is displayed on the screen of the liquid crystal panel 131 (step S12). Therefore, if the displayed contour is incorrect, the user can know at an early stage that the subject cannot be extracted by this error message. As for the process of recognizing the contour of the subject in step S11, an image processing method for extracting the contour of the photographed object from the image data is conventionally known. In the present invention, the same contour extraction processing method is used in the present invention. Detailed description is omitted.

  After step S12, it is determined whether or not the first shooting operation (shutter button on operation) has been performed (step S13). Here, when it is determined that the first shooting operation is not performed (= NO), it waits for the first shooting operation to be performed. When the first shooting operation is performed (= YES), it is determined in the next step S14 whether or not moving image shooting is set. If it is determined that the moving image shooting is not performed (= NO), Image photographing processing is executed (step S15). On the other hand, when it is determined in step S10 that moving image shooting is set (= YES), moving image shooting processing is executed for a predetermined time (step S16). Subsequently, it is determined whether or not the user has performed a second shooting operation (step S17).

  If it is determined that the second shooting operation has not been performed (= NO), the system waits for the second shooting operation to be performed. When it is determined in step S14 that the second shooting operation has been performed (= YES), the second shooting process is executed (step S18). Next, an extraction process (step S19) for extracting a subject image from the first and second images is executed, a character is generated from the extracted subject image, and stored in the storage unit 17. (Step S20), and the designated flag is cleared to “0” (Step S21).

Further, in the present embodiment, the designation flag is cleared to “0” in step S21, and then the process returns to step S3. In step S3, it is determined that there is no incoming call, and the process proceeds to step S8, where the extracted image is shot. If it is determined that there is none (= NO), the process jumps to step S22 in FIG. Then, in this step, it is determined whether or not a composition operation has been performed. If it is determined that a composition operation has been performed (= YES), a user operation can select a background image stored in advance in the storage unit 17. After one is selected (step S23) and the character is further selected (step S24) and then the position of the character relative to the background image is adjusted (step S25), the background image and the character are combined and the combined image is stored. The process (step S26) memorize | stored in the part 17 is performed, and it returns to step S3.
If it is determined in step S22 that the compositing operation has not been performed (= NO), the process proceeds to step S27, where it is determined whether another operation such as a transmission operation or a background image shooting operation has been performed. If it is determined that no operation has been performed (= NO), the process returns to step S3. If it is determined that another operation has been performed (= YES), processing corresponding to the operation (step S28) is executed, and step S3 is performed. Return to.

  In the extracted subject designation process in step S9, as shown in FIG. 7, it is first determined whether or not a touch operation on the display screen (touch panel) has been performed (step S41). If it is determined that a touch operation on the screen has been performed (= YES), it is determined whether or not a recognizable subject exists at the touch position (step S42). If there is a recognizable subject (= YES), the recognized subject is designated as an extraction target in step S43, the designation flag is set to “1”, and the extraction subject designation processing is terminated. (Step S44). If it is determined in step S42 that there is no recognizable subject at the touch position (= NO), an error message “No subject that can be specified as an extraction target” is output and extracted in step S45. The subject designation process ends. Therefore, the user can know at an early stage that the subject cannot be extracted by this error message.

  On the other hand, if it is determined in step S41 that the touch operation on the screen has not been performed (= NO), it is determined whether or not there has been an operation using the cross key and the enter key, that is, a cursor movement and selection operation (step S46). If it is determined that there is an operation with the cross key and the enter key (= YES), it is determined whether or not a recognizable subject exists at the selected position (step S47). If a recognizable subject exists (= YES), the selected subject is designated as an extraction target in step S48, and then the designation flag is set to “1” and the extraction subject designation processing is terminated. And return (step S49). If it is determined in step S47 that there is no recognizable subject at the selected position (= NO), the process proceeds to step S45, an error message is output, and the extracted subject designation process is terminated.

  If it is determined in step S46 that there is no operation using the cross key and the enter key (= NO), it is determined whether there is a subject that has moved within the shooting range (step S50). That is, step S50 functions as a subject determination unit that can determine a subject that has moved within the imaging range of the imaging unit. If there is a moved subject (= YES), the moved subject is designated as an extraction target in step S51, the designation flag is set to “1”, and the extracted subject designation process is terminated and returned (step S51). S52). If it is determined in step S50 that no subject has moved (= NO), steps S51 and S52 are skipped and the process returns to the original.

  As described above, in this embodiment, in the subject extraction shooting mode, when it is determined that the subject is specified by the touch panel or the cross key and the enter key, or the subject is moved within the shooting range, the selection is performed. After the contour of the subject that has been displayed is displayed, if a shooting operation is performed twice while the subject is out of the shooting range, the specified subject is extracted from the first shot image and the second shot image. Is done. For this reason, it is possible to predict whether or not the clipping will be successful based on the contour of the subject or the error message displayed before the shooting operation is performed, and it will be understood that the clipping has failed after the second shooting operation is completed. There is an advantage that a clipped image of the subject can be reliably obtained in one try.

  Further, in this embodiment, since the subject to be extracted can be specified only by moving the subject within the photographing range, there is an advantage that the user's operation when generating the subject cutout image is simplified. Furthermore, the subject image can be easily extracted by using the moving image shooting as the first shooting. Further, by having a function of generating a composite image obtained by combining the extracted subject image and an arbitrary background image, a collage photo or a moving image can be easily created and enjoyed.

Although the invention made by the present inventor has been specifically described based on the embodiment, the present invention is not limited to the embodiment. For example, in the above-described embodiment, in the subject extraction shooting mode, when it is determined that the subject is specified by the touch panel, the subject is specified by the cross key and the enter key, or the subject is moved within the shooting range. Although it has been described that the moved subject is designated as an extraction target, and the subject is extracted through two shootings after the contour of the designated or selected subject is displayed, the three kinds of subject designation methods are not necessarily used. It is not necessary to have, and it is only necessary to have any two types or one type of subject designation method.
However, according to the subject designation method using the touch panel, the subject can be designated easily and reliably. Further, according to the subject designation method using the cross key and the enter key, it is not necessary to provide a touch panel on the display unit, so that an increase in cost can be suppressed. Furthermore, when it is determined that the subject has moved within the shooting range, the method of designating the moved subject as an extraction target is cheaper to apply to an electronic device that does not include the cross key and the enter key. Can be realized.

  Further, in the above-described embodiment, the two shootings are performed by the user's shooting operation, but only the first shooting is performed by the user's shooting operation, and there is a change such that the subject is out of the shooting range. Alternatively, the second shooting may be performed, and the subject image may be cut out from the two captured images. In the embodiment, the contour of the designated subject is displayed. Instead of displaying the contour, the brightness of the designated subject is changed or a display with a color filter is applied. You may make it perform.

  Further, in the above-described embodiment, the cellular phone as an example of an electronic device effective by applying the present invention has been described as an example. However, the present invention is not limited to the cellular phone, and any electronic device having an imaging function may be used. For example, it can be used not only for digital cameras but also for PDAs (Personal Digital Assistances) and notebook computers.

10 mobile phone 11 control unit (CPU; microprocessor)
DESCRIPTION OF SYMBOLS 12 Operation part 13 Display part 14 Power supply part 15 Telephone part 16 Communication part 17 Memory | storage part 18 Image pick-up part 19 Image recognition part 131 Liquid crystal panel (display means)
132 Touch panel (input means)
181 Lens unit 182 Electronic imaging unit (imaging means)
183 Imaging control unit 191 Image data generation unit 192 Image memory 193 Feature amount calculation unit 194 Block matching unit 195 Image processing unit 196 Recording control unit 197 Display control unit 198 Input detection unit

Claims (5)

Imaging means;
Display means capable of displaying an image based on an imaging signal from the imaging means;
Subject specifying means capable of specifying any subject within the imaging range of the imaging means;
Subject recognition means for recognizing the contour of the subject designated by the subject designation means during the first imaging by the imaging means;
Display control means for displaying on the display means the contour recognized by the subject recognition means;
Subject extracting means for extracting an image of the subject specified by the subject specifying means based on the first captured image and the second captured image by the imaging means;
Subject discriminating means capable of discriminating a subject moved within an imaging range by the imaging means;
With
The subject discrimination means
A feature amount computing unit for extracting feature points in the second captured image;
By searching whether the feature point corresponding to where in said first captured image, before Symbol first aligning the captured image and the previous SL second captured image, aligned with the previous SL first from the difference between the captured image and the previous SL second captured image, anda block matching unit that determines the moved object,
The subject designating unit designates the subject determined to have moved by the subject determining unit;
An electronic device characterized by that.   The electronic apparatus according to claim 1, wherein the image obtained by the first imaging is a moving image.   The electronic apparatus according to claim 1, further comprising an image composition unit that generates a composite image by combining the subject image extracted by the subject extraction unit and the selected background image. An image processing method in an electronic device including an imaging unit and a display unit,
A subject designation step for designating any subject within an imaging range of the imaging means;
A subject recognition step of recognizing the contour of the subject designated by the subject designation step during the first imaging by the imaging means;
A subject contour display step for causing the display means to display the contour recognized by the subject recognition step;
A subject extraction step of extracting an image of the subject designated by the subject designation step based on a first captured image and a second captured image by the imaging means;
A subject determination step of determining a subject that has moved within an imaging range by the imaging means;
Including
Wherein the object determination step, the feature points are extracted in the second captured image, by extracting feature points is searched whether corresponding to where in said first captured image, before Symbol first captured image When the front Symbol to align the second captured image, from the difference between the aligned the previous SL first captured image and the previous SL second captured image, to determine the moved object,
In the subject specifying step, the subject determined to have moved in the subject determining step is specified.
An image processing method. A microprocessor that controls the imaging means and the display means;
Subject designation means capable of designating any subject within the imaging range of the imaging means based on a signal from the input means;
Subject recognition means for recognizing the contour of the subject designated by the subject designation means during the first imaging by the imaging means;
Display control means for displaying the contour recognized by the subject recognition means on the display means;
Subject extracting means for extracting an image of the subject specified by the subject specifying means based on the first captured image and the second captured image by the imaging means;
Subject discrimination means capable of discriminating a subject moved within an imaging range by the imaging means;
Function as
As the object discriminating means, wherein to extract a second feature point from the captured image of, by the extracted the feature point is searched where to corresponding said first captured image, before Symbol first captured image When the front Symbol to align the second captured image, from the difference between the aligned the previous SL first captured image and the previous SL second captured image, to determine the moved object,
As the subject specifying means, the subject determined to have been moved by the subject determining means is specified,
A program characterized by that.