JP2006128754A - Imaging apparatus and imaging method - Google Patents

Abstract

An object is extracted from an acquired image and a composite photograph of the extracted object and a background image is acquired even in a device such as a digital camera that has few hardware resources.
When an extraction unit performs subject extraction, an imaging condition is controlled by an imaging condition control unit and an image suitable for subject extraction is acquired from the imaging unit. The extraction unit 103 extracts only a focused subject in the image through attention area extraction and contour extraction. At this time, depending on the extraction result of the extraction unit 103, feedback is further applied to change the imaging condition.
By synthesizing the extracted subject image with the second image as a background, a new composite photograph can be obtained.
[Selection] Figure 1

Description

  The present invention relates to an imaging apparatus that captures an image of a subject and acquires digital data, and more particularly to image generation in a digital device such as a digital camera or a camera-equipped mobile phone.

  In recent years, digital still cameras have exploded in popularity. Digital still cameras have the advantages of being able to see photos taken immediately, erasing them if they are not needed, and taking as many photos as the memory capacity allows. It is something that does not exist in the salt camera.

  However, the advantages listed here are not exhaustive of all the advantages of a digital camera. As digital still cameras that take advantage of further advantages, there are those shown in Patent Document 1 and Patent Document 2.

  FIG. 9 is a block diagram showing a conventional digital still camera described in Patent Document 1. In FIG.

  In FIG. 9, an imaging unit 901 converts a light signal into an electric signal and acquires first and second images (digital data). The acquired image is temporarily stored in the storage unit 902. A cut-out image generation unit 903 cuts out a partial image (cut-out image) having a cut-out pattern shape output from the cut-out pattern storage unit 908 from the first image, and outputs the cut-out image. Then, the image synthesizing unit 904 synthesizes a clipped image on the second image as a background to generate a synthesized image.

  The display unit 905 displays the first image, and further displays the clipping pattern by the display control unit 906 based on the instruction from the movement instruction unit 907 to determine the cutout portion of the first image.

  The cutout pattern storage unit 908 further includes a pattern generation unit 909, which can generate a pattern of an arbitrary shape and register it.

  FIG. 10 is a block diagram showing a conventional image input apparatus described in Patent Document 2. In FIG.

  In FIG. 10, the imaging unit 1002 includes a lens, a diaphragm, a shutter, and the like, and projects input light onto the imaging unit 1005. The imaging unit 1005 is a CCD, and converts an optical signal into an electrical signal. The low-frequency cutoff filter 1004 is between the two and blocks the low-frequency component of the optical signal.

In general, image information in a blurred area has a gentle change in luminance with respect to the operation direction and has a low frequency. Therefore, the image information can be cut using an optical or electrical low-frequency cutoff filter. As a result, the obtained image is obtained by extracting the focused subject.
JP 2004-173300 A JP-A-10-260462 JP 2002-224116 A JP 2003-334194 A Image Recognition / Understanding Symposium 2004 Proceedings I-93-94

  However, in the conventional configuration shown in Patent Document 1, only a partial image having a pattern shape registered in advance can be cut out. Therefore, the cut-out partial image and other images are not naturally adapted to the shape of the subject. If you look at the composite photo produced by, you can't deny the unnaturalness. It's just like combining a group photo with a face photo of a child who was absent on the day of taking a group photo at elementary and junior high school.

  On the other hand, in the configuration shown in Patent Document 2, a low frequency filter is newly required for the hardware configuration of a general digital camera. In addition, since it is not a configuration that can acquire the image again by applying feedback and changing the imaging conditions according to the state of the image data that has passed through the low frequency filter, the acquired image does not necessarily cut out only the subject from the background. Not necessarily.

  The present invention solves the above-described problems of the conventional invention, and does not require any special hardware resource, and uses a general digital camera hardware resource to extract a subject from an arbitrary image and extract the subject. An object of the present invention is to provide an imaging apparatus capable of immediately obtaining a composite photograph with an image and another image.

  In order to solve the above-described conventional problems, the imaging apparatus of the present invention acquires an extraction instruction unit that receives an instruction to extract an arbitrary subject in an image, and an image that is optimal for extracting the specified subject. Therefore, when the imaging condition control means for determining the imaging condition of the image and outputting to the imaging means, and when the extraction instruction means outputs the extraction instruction, the image taken by the imaging means is acquired and acquired. When the partial image including only the subject in the captured image is extracted and the subject cannot be extracted, an extraction unit that outputs to the imaging condition control unit so as to change the imaging condition, and the extraction instruction unit When the extraction instruction is output, the partial image extracted by the extraction unit is set as the foreground, the image captured by the imaging unit is set as the background, and the partial image is placed on the background image according to the instruction received by the synthesis instruction unit. And after determining the size, by combining the foreground image and the background image has a synthesizing means for outputting a composite image, and displays a composite image in which the combining means is generated on the display means.

  With this configuration, even if hardware resources such as a general digital camera are used, a subject can be accurately extracted from an arbitrary image, and a photograph in which the extracted image and another image are combined can be easily obtained. .

  According to the image pickup apparatus of the present invention, it is possible to cut out an object of any shape without depending on the cut-out pattern, even if the device has few hardware resources such as a digital still camera. Pictures can be easily obtained.

  Embodiments of the present invention will be described below with reference to the drawings.

(Embodiment 1)
FIG. 1 is a block diagram of an imaging apparatus according to Embodiment 1 of the present invention.

  In FIG. 1, an extraction instruction unit 101 is a unit that instructs to perform subject extraction, and controls normal photography or composite photography by turning on and off the extraction instruction unit 101.

  When the extraction instruction unit 101 is turned on, the imaging condition control unit 102 determines an imaging condition (focus, aperture value, etc.) and outputs it to the imaging unit 106 in order to acquire an image suitable for subject extraction.

  The imaging unit 106, the display unit 107, and the storage unit 108 are main components of a general digital still camera 109, and are controlled by a microcontroller (not shown) that performs overall control.

  When the extraction instruction unit 101 is turned on, the extraction unit 103 acquires an image photographed for subject extraction from the imaging unit, separates the subject and its background, and cuts out a partial image including only the subject. FIG. 2 is a block diagram describing further details of the extracting means. The extracting means 103 includes two blocks, an attention area extracting means 201 and a contour extracting means 202, and the attention area extracting means 201 detects the subject from the entire image. The contained rectangle is cut out, and then the outline extracting unit 202 recognizes the outline of the subject from the rectangle and cuts out as a partial image including only the subject.

  As the attention area extraction unit 201, a method described in Non-Patent Document 1 is known, and according to this method, a attention area in an image, that is, a rectangle including a subject can be cut out.

  Further, the method described in Patent Document 3 or Patent Document 4 is known as the contour extracting unit 202. With these techniques, the contour of the subject can be extracted and a partial image including only the subject can be cut out.

  When the extraction instruction unit 101 is on, the combination instruction unit 104 receives input from the user regarding the size of the partial image cut out by the extraction unit 103 and the position to be pasted, and issues an instruction to the combination unit 105.

  The synthesizing unit 105 synthesizes the two images according to the instruction of the synthesizing instruction unit 104 using the partial image extracted by the extracting unit 103 as the foreground and the image from the imaging unit 106 as the background, and outputs a synthesized image.

  The composite image is displayed on the display unit 107 and simultaneously stored in the storage unit 108. Alternatively, the composite image may be saved in the storage unit 108 when the save instruction button is pressed.

  FIG. 3 is an appearance of the digital still camera according to the present embodiment. To turn on the extraction instruction means, the dial 302 called a mode dial is turned, and the dial is set to the position where the extraction icon is drawn. When the extraction instruction is on, the position designation key can be used to determine the position where the partial image including the subject is placed in the compositing instruction means, and the position of the partial image is moved up, down, left, and right by operating the key. . Also, the zoom lever is moved left and right to determine the size of the partial image.

  Hereinafter, the movement of the imaging apparatus according to Embodiment 1 of the present invention will be described with reference to the chart of FIG.

  When the extraction instructing means is on, it is internally divided into an extraction mode 401 and a synthesis mode 414, which are executed sequentially.

  First, the user turns the mode dial 302 to turn on the extraction instruction unit 103 and sets it to a portion where an extraction icon is drawn. In step 402, the imaging apparatus opens a data path from the imaging unit to the extraction unit (step 403), and displays “+” in the center (step 404).

  Of course, “+” only displays the position of the subject to be extracted in the sense of “please place it here”, and it is self-evident that the design can be anything if it can convey the same intention. .

  The user determines the composition so that “+” is within the contour of the desired subject (step 405). Then, the shutter 301 is half-pressed (step 406).

  Then, the imaging condition control means 102 changes the imaging conditions so that an image optimal for subject extraction can be acquired, and acquires an image under those conditions (step 407).

  Then, a subject is extracted using the acquired image (step 408).

  If the subject cannot be extracted from the acquired image, the process returns to step 407, the imaging condition is changed again, and steps 407 to 408 are repeated (step 409).

  If the subject has been extracted, the extraction success is indicated by highlighting the contour of the subject (step 410).

  The user looks at the emphasized outline, and if the extracted subject is acceptable, the user fully presses the shutter (step 412). If the subject does not like the process, the process returns to step 405 and steps 405 to 410 are repeated.

  When the shutter is fully pressed, the imaging apparatus cuts out a partial image including only the extracted subject and temporarily stores it (step 413).

  Up to this point, the extraction mode 401 ends, and the process proceeds to the synthesis mode 414 as it is.

  When the composition mode 414 is entered, the data path from the imaging means 106 to the composition means 104 is opened (step 415).

  The synthesizing unit 105 displays an image obtained by using the image acquired from the imaging unit 106 as a background and the partial image extracted by the extracting unit 103 as a foreground (step 416).

  The user changes the position and size of the partial image while viewing the displayed picture. At this time, the position designation key 304 is used to change the position of the partial image. Further, the size is enlarged or reduced by using the zoom lever 303 and changed to a desired size (step 417).

  When the partial image is in the position / size that the user likes, the user fully presses the shutter 301 (step 418).

  Then, the imaging apparatus combines the background and the foreground, and generates a composite photograph (step 419).

  The generated composite photograph is stored in the storage unit 108.

  Through the above steps, the partial image including only the subject extracted from the first image is combined with the second image.

  When the contour of the subject can be extracted, it is presented to the user by highlighting the contour, but this is not limited to highlighting, and the extracted contour line can be displayed in a blinking manner. Alternatively, it may be a method of presenting by changing the color or whitening the subject itself.

  In the present embodiment, the important point is that the imaging condition control means 102 can control the imaging condition in order to obtain an optimal image for subject extraction. Further, depending on the extraction result, the imaging condition is again set by feedback. This is a changeable point. By changing the imaging conditions, it is possible to acquire an image that is easy for subject extraction. As a result, it is possible to reduce the image processing load and implement even poor hardware such as a digital still camera. .

  In general, when a picture is taken with the aperture wide open on the telephoto side, the depth of field becomes shallow, and the distance of the in-focus depth becomes short. FIG. 5A is an image with a deep depth of field, but when the aperture is opened and the depth of field is reduced, the depth of focus is reduced as shown in FIG. Only the subject is in focus and the background is blurred.

  With such an image, it is easy to extract a subject from the background with a relatively small amount of calculation, that is, it can be implemented even with scarce hardware such as a digital camera.

  The imaging apparatus according to Embodiment 1 of the present invention is effective in the following situations. For example, consider the case of taking a child to a zoo. If you want to take a two-shot photo of a child with a panda behind the panda house, the conventional camera can only capture the chance of the child and the panda entering one frame and press the shutter. However, it is a common story that the panda can not easily fit in one frame because it moves around, and that there are many visitors, so it is not possible to place the child in a position that can be two-shot with the panda. . Also, in front of the crocodile house, I want to take a two-shot picture with the child against the background of the crocodile, but I can not line up with the crocodile because it is physically dangerous or the child is scared to come close .

  In the situation as described above, with the imaging device according to the present invention, first, a picture of a child is taken, and a partial image obtained by extracting only the child is superimposed on a picture of a panda or crocodile. Two-shot photos can be easily taken in real time on the spot.

  The effective point of the present invention is also in this respect, and it is clear that the ability to easily obtain a composite photo in real time on the spot doubles the enjoyment of shooting and the enjoyment of viewing images. is there.

(Embodiment 2)
FIG. 6 is a block diagram of the imaging apparatus according to the second embodiment of the present invention. In FIG. 6, the same components as those in FIG.

  In FIG. 6, the storage unit 108 can also store the partial image extracted by the extraction unit 103. The partial image selection means 109 displays a list of partial images saved in the storage means 108 at the same magnification, or enlarges or reduces them, and allows the user to select one of them. When the user selects one partial image, it is read from the storage means 108 and input to the synthesizing means 105.

  The synthesizing unit 105 generates a composite photograph using the image captured by the imaging unit 106 as a background and the partial image read from the storage unit 108 and the partial image extracted by the extracting unit 103 as the foreground.

  The present embodiment is different from the first embodiment in that the partial image including the extracted subject can be stored in the storage unit 108.

  Therefore, the user can not only extract a subject in real time and generate a composite photo, but also use the pre-extracted subject as a foreground of the composite photo.

  According to the present embodiment, an animation or movie character is extracted as a partial image of a subject, or acquired from the outside and stored in the storage means 108, and an arbitrary composite photograph is selected by selecting it. It is certain that the enjoyment of taking pictures of users will increase.

(Embodiment 3)
In the third embodiment of the present invention, the communication unit 110 is further provided to enable communication with a third party through a public network through a composite image generated.

  FIG. 7 is a chart showing the movement of the imaging apparatus according to the present invention in the third embodiment. Hereinafter, a message exchange method according to the present embodiment will be described with reference to FIGS.

  The purpose of this imaging apparatus is to convey a simple message regardless of a Japanese input device such as a keyboard.

  As shown in FIG. 8A, the user writes a message to be transmitted with a pen on paper (step 701).

  When the extraction instruction unit is turned on (step 703), the imaging apparatus opens a data path from the imaging unit to the extraction unit (step 704), and displays “+” in the center of the screen (step 705).

  The user sets “+” to the center of the message (step 706) and presses the shutter halfway (step 707).

  Then, the imaging apparatus sets an imaging condition optimal for character extraction, and acquires an extraction image (step 708). As shown in FIG. 8B, only characters are extracted from the image (step 709). At this time, portions other than the characters are made transparent with an α value of 0. If the character cannot be extracted, the process returns to step 708, the shooting conditions are changed, and steps 708 to 710 are performed again.

  In order to cut out a character, it is desirable that the image has an emphasized outline of the character. Accordingly, in step 709, the imaging conditions are changed so that an image that emphasizes edges and a sharp image can be acquired.

  When the character is extracted, the outline of the character is highlighted (step 711). The user looks at it to determine whether the extraction is successful. In the case of failure, the process returns to Step 706 and Steps 706 to 712 are repeated again.

  If it is determined that the extraction is successful, the shutter is fully pressed (step 713).

  In response to this, the imaging device temporarily stores the partial image including the character string (step 714), and internally ends the extraction mode and shifts to the synthesis mode.

  Then, an image data path from the imaging means to the synthesizing means is opened (step 716), and as shown in FIG. 8D, a message is displayed on the display means 107 with the current image as the background. The displayed image is displayed (step 716).

  While viewing the display, the user changes the position and size of the message (step 718), and if it is OK, the shutter is fully pressed (step 719).

  When the shutter is fully pressed, the synthesizing unit generates the currently displayed image as a composite photograph (step 720), and stores the generated synthesized image in the storage unit (step 721).

  According to the present embodiment, it is possible to input a message without depending on a character input device such as a keyboard, and display it by superimposing it on, for example, an image of yourself, which can be transmitted by e-mail. Therefore, transmission of a message is possible with two actions of imaging → transmission.

  Therefore, it is not necessary to memorize a complicated operation procedure like a general mobile phone with a camera, and even a young child can use it.

  Generally, in a device such as a mobile phone with a camera, it is difficult to say that anyone can use it by inputting a message character string by key input. Also, even if a person can perform key input, considering the size of the key input load and the content of the message to be transmitted, it is often possible to stop transmission depending on the content of the message.

  The third embodiment of the present invention can solve this problem. It can be communicated by attaching a message to an arbitrary photograph, and can be realized with a few input procedures.

  The image pickup apparatus according to the present invention is useful as an image pickup apparatus such as a digital still camera for acquiring digital image data and a mobile phone with a digital still camera. It can also be applied to an image acquisition system that combines a PC and a camera.

1 is a block diagram of an imaging apparatus according to Embodiment 1 of the present invention. The block diagram of the extraction means in Embodiment 1 of this invention 1 is an external view of a digital still camera according to Embodiment 1 of the present invention. Chart showing operation of imaging apparatus according to Embodiment 1 of the present invention Schematic showing differences in images due to differences in depth of field Block diagram of an imaging apparatus according to Embodiments 2 and 3 of the present invention Chart showing operation of imaging apparatus according to Embodiment 3 of the present invention The figure which shows the message composition to the background image in Embodiment 3 of this invention Block diagram showing the configuration of a conventional imaging device Block diagram showing the configuration of a conventional imaging device

Explanation of symbols

DESCRIPTION OF SYMBOLS 101 Extraction instruction means 102 Imaging condition control means 103 Extraction means 104 Composition instruction means 105 Composition means 106 Imaging means 107 Display means 108 Storage means 201 Attention area extraction means 202 Contour extraction means

Claims (12)

An image pickup apparatus capable of displaying image data acquired by an image pickup means on a display means and further storing the image data in a storage means,
An extraction instruction means for receiving an instruction to extract an arbitrary subject in the image;
In order to obtain an optimal image for extracting the designated subject, an image capturing condition is determined,
Imaging condition control means for outputting to the imaging means;
If the extraction instruction means is outputting an extraction instruction,
Obtaining an image taken by the imaging means;
Extract a partial image containing only the subject in the acquired image,
If the subject could not be extracted, the imaging condition control means
Extraction means for outputting to change the imaging condition;
If the extraction instruction means is outputting an extraction instruction,
The partial image extracted by the extraction means is the foreground, the image taken by the imaging means is the background,
After determining the position and size of the partial image to be placed on the background image according to the instruction received by the compositing instruction means,
Composing means for synthesizing the foreground image and the background image and outputting the synthesized image,
An image pickup apparatus, wherein the composite image generated by the combining means is displayed on the display means and further stored in the storage means. The imaging unit according to claim 1, wherein the extraction unit includes:
When the extraction instruction means outputs an extraction instruction,
A region of interest extraction means for extracting an area including the subject in the image, using the extraction image captured by the imaging means as an input;
The region is input, the contour of the subject in the region is extracted, and contour extracting means for cutting out the subject is included.
An imaging apparatus, wherein a partial image including only a subject is extracted from an area extracted by the attention area extracting means. The imaging apparatus according to claim 1,
The imaging apparatus includes an aperture, a shutter speed, and sensitivity, and is controlled so as to reduce a depth of field of an image obtained by photographing these three parameters. The imaging apparatus according to claim 1,
The storage device stores the partial image output by the extraction unit. The imaging apparatus according to claim 4,
The imaging apparatus includes a partial image selection unit that displays a partial image stored in the storage unit and can select any one of them.
The synthesizing unit uses the partial image output from the partial image selecting unit as a foreground,
With the image taken by the imaging means as a background,
An image pickup apparatus that generates a composite image based on a composite position and size instruction from the composite instruction unit. The imaging apparatus according to claim 5, wherein
The synthesizing means includes a partial image extracted by the extracting means, and
A partial image selected by the partial image selection means;
With an image taken by the imaging means as input,
An imaging apparatus characterized by combining the three images. The imaging apparatus according to claim 1,
The imaging apparatus includes a communication unit capable of transmitting and receiving an e-mail attached with an image from an external network,
The communication means stores the received e-mail and attached image in the storage means,
The composition unit generates a composite image according to a composition position and a size output from the composition instruction unit with an image stored in the storage unit as a background and an image acquired by the imaging unit as a background. An imaging device. The imaging apparatus according to claim 1,
The image pickup apparatus, wherein the display unit highlights the contour line of the subject extracted by the extraction unit. The imaging apparatus according to claim 1,
The imaging apparatus, wherein the display means can change the color of the contour line of the subject extracted by the extraction means to an arbitrary color. The imaging apparatus according to claim 1,
The imaging device characterized in that the display means blinks the outline of the subject extracted by the extraction means. The imaging apparatus according to claim 1,
The image pickup apparatus characterized in that the display means informs the user of the area extracted by the extraction means by whitening the inside of the outline of the subject extracted by the extraction means. A method of combining two or more images,
Instructing to extract any subject in the image;
Determining an imaging condition of an image in order to obtain an optimal image for extracting a specified subject;
Extracting a partial image including only a subject from an image captured under the conditions determined in the step of determining the image capturing condition;
Inputting the position and size of the partial image on the image acquired by the imaging means;
Combining the captured image and the extracted partial image according to the parameters input in the input step;
Displaying the combined image generated in the combining step;
An imaging method including a step of storing the composite image generated in the composite step.