JP2013145291A - Imaging apparatus - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide means for suppressing the shift of a main object in a panoramic image, while suppressing the number of obtained unit images.SOLUTION: An imaging apparatus includes an imaging unit imaging an image of an object, an object recognition unit, an imaging control unit, and a panoramic synthesizing unit. The object recognition unit extracts a target to be recognized from the image obtained by the imaging unit. When an imaging range of the imaging unit is moved in a predetermined direction to perform panoramic photographing, the imaging control unit images a unit image by a plurality of times at such an imaging timing that a part of the imaging range is overlapped. The panoramic synthesizing unit joins a plurality of unit images obtained by panoramic photographing to each other to generate one panoramic image. Then, the imaging control unit decides the imaging timing in panoramic photographing, according to the size of the target to be recognized on the image.

Description

  The present invention relates to an imaging apparatus.

  Conventionally, a panoramic image having a wide viewing angle has been synthesized by capturing a plurality of unit images that are continuous in a predetermined direction (for example, the horizontal direction) and joining these unit images into one. . In such panoramic photography, there is also known a camera in which the camera side performs imaging at predetermined timings while the user moves the camera, and generates a panoramic image from unit images obtained by the imaging.

  In panoramic photography, if the main subject is located at the joint between unit images, the main subject may be misaligned in the combined panoramic image. For example, means for suppressing the shift of the main subject in the panoramic image by reducing the imaging interval of the unit images has been proposed.

Japanese Patent No. 4356621

  In the above prior art, it is necessary to acquire a large number of unit images at the time of panoramic photographing, so that the load on the camera becomes very large.

  In view of the above circumstances, there is provided means for suppressing a shift of a main subject in a panoramic image while suppressing the number of unit images to be acquired.

  An imaging device that is one embodiment of the present invention includes an imaging unit that captures an image of a subject, a subject recognition unit, an imaging control unit, and a panorama synthesis unit. The subject recognition unit extracts a recognition target from the image acquired by the imaging unit. The imaging control unit captures unit images a plurality of times at an imaging timing at which a part of the imaging range overlaps when panoramic imaging is performed by moving the imaging range of the imaging unit in a predetermined direction. The panorama synthesizing unit generates a single panoramic image by connecting a plurality of unit images acquired by panoramic shooting. And an imaging control part determines the imaging timing in panoramic photography according to the size of the recognition object on an image.

  According to the imaging apparatus of one aspect, it is possible to suppress the deviation of the main subject in the panoramic image while suppressing the number of unit images to be acquired.

1 is a block diagram illustrating a configuration example of an electronic camera according to a first embodiment. The figure which shows the operation example of the panoramic photography mode in 1st Embodiment. Overview of imaging timing in the panoramic shooting mode of the first embodiment The figure which shows the relationship between the size of the face in 1st Embodiment, and the duplication range of a unit image. The figure which shows the process example in the panorama composition in 1st Embodiment. The figure which shows the operation example of the panoramic photography mode in 2nd Embodiment. Overview of imaging timing in the panoramic shooting mode of the second embodiment The figure which shows the operation example of the panoramic photography mode in 3rd Embodiment. Overview of imaging timing in the panoramic shooting mode of the third embodiment

<Description of First Embodiment>
FIG. 1 is a block diagram illustrating a configuration example of the electronic camera (imaging device) according to the first embodiment. The electronic camera of the present embodiment has a panoramic shooting mode for generating a panoramic image as one of the shooting modes. In the panorama shooting mode, when the user moves the electronic camera in a predetermined direction (for example, the horizontal direction), the electronic camera captures a plurality of unit images continuous in the predetermined direction. The electronic camera in the panoramic shooting mode generates one continuous panoramic image by joining unit images in a predetermined direction.

  An electronic camera 11 shown in FIG. 1 includes an imaging optical system 12, an imaging element 13, a signal processing unit 14, a camera microcomputer 15, a first memory 16, a second memory 17, a media I / F 18, and a display. It has a unit 19 and an operation unit 20 that accepts various user operations. Here, the image sensor 13, the signal processing unit 14, the first memory 16, the second memory 17, the media I / F 18, the display unit 19, and the operation unit 20 are each connected to the camera microcomputer 15.

  The imaging element 13 is an imaging device that captures an image of a subject formed by the imaging optical system 12. The output of the image sensor 13 is connected to the signal processing unit 14. The imaging device 13 may be a progressive scanning solid-state imaging device 13 (for example, a CCD) or an XY addressing solid-state imaging device 13 (for example, a CMOS).

  In the shooting mode of the electronic camera 11, the imaging device 13 shoots (captures) a still image for recording accompanied by recording in a nonvolatile storage medium in response to an imaging start instruction from the operation unit 20. In addition, the imaging device 13 continuously captures images for observation (through images) at predetermined intervals even during standby for capturing. The through images acquired in time series are used for moving image display on the display unit 19 and various arithmetic processes by the camera microcomputer 15.

  The signal processing unit 14 performs analog signal processing (correlated double sampling, black level correction, etc.), A / D conversion processing, and digital signal processing (defective pixel correction, etc.) on the image signal input from the image sensor 13. ) In order. Image data output from the signal processing unit 14 is input to the camera microcomputer 15.

  The camera microcomputer 15 is a processor that comprehensively controls the operation of the electronic camera 11. The camera microcomputer 15 according to the first embodiment functions as an imaging control unit 21, an image processing unit 22, an image analysis unit 23, and a panorama synthesis unit 24 by executing a program.

  The imaging control unit 21 controls the charge accumulation time of the imaging element 13 and the signal readout timing (imaging timing).

  The image processing unit 22 performs image processing (gradation conversion processing, white balance adjustment processing, noise removal processing, contour enhancement processing, color conversion processing, etc.) on image data for one frame.

  The image analysis unit 23 performs image analysis in the panoramic shooting mode. The image analysis unit 23 according to the first embodiment performs a known face recognition process using a human face as a recognition target as a main subject. As an example, the image analysis unit 23 extracts feature points such as eyebrow, eye, nose, and lip end points from an image by a known feature point extraction process, and determines whether or not a face region is based on these feature points. judge. Alternatively, the image analysis unit 23 obtains a correlation coefficient between reference data (such as a face image) prepared in advance and an image to be determined, and determines that this is a face area when the correlation coefficient exceeds a certain threshold value. Also good.

  Further, the image analysis unit 23 in the first embodiment performs subject matching processing between two images that are continuous in the time axis direction in order to obtain an overlapping range of unit images in panoramic photography. Note that the above-described matching processing may be performed by, for example, performing feature point matching between two images using an edge or corner of the image as a feature point, or performing so-called block matching between two images. Also good.

  The panorama synthesizing unit 24 generates a panorama image by synthesizing a plurality of unit images acquired by panorama shooting in the panorama shooting mode.

  The first memory 16 temporarily stores image data in the pre-process and post-process of image processing. For example, the first memory 16 is an SDRAM that is a volatile storage medium. The second memory 17 is a non-volatile memory that stores programs executed by the camera microcomputer 15 and various data.

  The media I / F 18 has a connector for connecting a nonvolatile storage medium 25. Then, the media I / F 18 executes data writing / reading with respect to the storage medium 25 connected to the connector. The storage medium 25 is, for example, a hard disk or a memory card with a built-in semiconductor memory. In FIG. 1, a memory card is illustrated as an example of the storage medium 25.

  The display unit 19 is a display device that displays various images. For example, the display unit 19 displays a moving image (viewfinder display) of a through image in the shooting mode under the control of the camera microcomputer 15.

  Next, an example of operation in the panoramic shooting mode in the first embodiment will be described with reference to the flowchart of FIG. The processing in FIG. 2 is started when the electronic camera 11 is activated in the panoramic shooting mode. In each embodiment of the present specification, it is assumed that panoramic photography is performed by panning the electronic camera 11 in a horizontal direction with respect to a landscape including a person.

  Step # 101: The camera microcomputer 15 performs preliminary shooting of a panoramic shooting range using a through image prior to panoramic shooting, and obtains the maximum size of a human face that can be shot by panoramic shooting.

  For example, the camera microcomputer 15 in # 101 acquires a through image with the image sensor 13 in response to activation of the panoramic shooting mode. Then, the image analysis unit 23 performs face detection processing on each of the through images acquired in time series by preliminary shooting before the start of panoramic shooting (before a panoramic shooting start instruction described later). The image analysis unit 23 compares the size of the face in the pan direction (horizontal direction) each time a face is detected from the through image, and overwrites the first memory 16 with a larger face size value. Thereby, the maximum size can be recorded in the first memory 16.

  As another example, prior to panoramic shooting, the user may be caused to perform preliminary shooting of the same range as the range shot by panoramic shooting (pre-scan). In the preliminary shooting in this case, the camera microcomputer 15 instructs acquisition of a through image in a state where the user pans the electronic camera 11. In addition, the image analysis unit 23 extracts a human face from each through image acquired in the preliminary shooting. Then, the image analysis unit 23 may extract the face having the longest horizontal length from the faces extracted in the preliminary shooting.

  Step # 102: The imaging control unit 21 determines the imaging timing in panoramic shooting based on the maximum size (# 101) of the person's face. The imaging control unit 21 in the first embodiment determines the imaging timing so that the size of the overlapping range between unit images during panoramic shooting is equal to or greater than the maximum size of the face.

  For example, when the maximum size in the horizontal direction of the face is 200 pixels, the imaging control unit 21 in the first embodiment has an overlapping range of the previous unit image and the current imaging range from 200 pixels at the time of panoramic shooting. The next unit image is captured when the number of pixels becomes slightly larger. As shown in FIG. 4, in the example of the first embodiment, the overlapping range of unit images increases as the face size acquired in # 101 increases, and the unit image decreases as the face size acquired in # 101 decreases. The overlapping range of becomes smaller.

  Step # 103: The camera microcomputer 15 determines whether or not a panoramic shooting start instruction has been accepted. For example, in this specification, the camera microcomputer 15 determines that a panoramic image shooting instruction has been received when a release button (not shown) of the operation unit 20 is fully pressed by the user.

  If the above requirement is satisfied (YES side), the process proceeds to # 104. On the other hand, if the above requirement is not satisfied (NO side), the camera microcomputer 15 waits until there is an instruction to start panoramic photography.

  Step # 104: With the user panning the electronic camera 11, the camera microcomputer 15 performs panoramic shooting.

  During panoramic photography, the camera microcomputer 15 captures a through image. Further, the image analysis unit 23 executes matching processing using a through image, and continuously monitors changes in the image in the imaging range. Then, when the number of pixels in the overlapping range between the previous unit image and the current imaging range reaches a predetermined number of pixels that defines the imaging timing, the imaging control unit 21 captures the next unit image. As a result, the image sensor 13 sequentially captures the unit images so that the overlapping range between the unit images has a constant size larger than the face size.

  If the pan speed at # 104 is high, the imaging range may move greatly in a short time, and the image analysis unit 23 may not be able to perform matching. In such a case, since the above panorama shooting cannot be performed, the display unit 19 outputs a warning display indicating that “panorama shooting cannot be performed” under the control of the camera microcomputer 15.

  Step # 105: The camera microcomputer 15 determines whether or not a panoramic shooting end instruction has been received. For example, in this specification, the camera microcomputer 15 determines that the panorama shooting end instruction has been received when the user fully releases the release button.

  If the above requirement is satisfied (YES side), the process proceeds to # 106. On the other hand, if the above requirement is not satisfied (NO side), the camera microcomputer 15 returns to # 104 and repeats the above operation.

  Step # 106: The panorama synthesizing unit 24 connects a plurality of unit images (obtained in # 104) acquired in the panorama shooting in the pan direction to generate one panorama image.

  Here, when the two unit images are synthesized in # 106, the image analysis unit 23 performs face recognition processing on the overlapping range of each unit image and its neighboring range, and the human face is recognized from these overlapping ranges. It is determined whether it has been completed.

  When the face of a person cannot be recognized from any of the overlapping ranges of the two unit images (that is, when no face exists in the overlapping range), the panorama composition unit 24 uses the stitching in the overlapping range of one unit image. Most of the area except for the area is deleted and a known panorama composition process is performed.

  When a person's face can be recognized from the overlapping range of two unit images, the panorama synthesizing unit 24 deletes the overlapping range of one unit image and performs a known panorama synthesizing process. At this time, the panorama composition unit 24 leaves one image that satisfies a predetermined condition (for example, a state where the eyes are not closed, a state of smiling) out of the overlapping range of the unit images, The panorama composition may be performed by deleting most of the overlapping range of the other image (see FIG. 5A). Thereby, it is possible to obtain a panoramic image with a good appearance of a person's face. Note that the determination of the face state in the unit image may be executed by the image analysis unit 23 using a known algorithm for detecting eye meditation or smiling.

  If the person's face can be recognized only from the overlapping range of one unit image, the panorama composition unit 24 deletes most of the overlapping range of the unit image that cannot recognize the person's face and performs known panoramic composition. Processing is performed (see FIG. 5B). In this case, in the unit image in which the person's face cannot be recognized, there is a high possibility that inconvenience occurs, for example, the face is cut off at the edge of the image. Therefore, it is possible to obtain a panoramic image with good appearance of the person's face by prioritizing and using the unit image that can recognize the person's face.

  Then, the panorama composition unit 24 may generate the panorama image by repeating the above process for each combination of unit images. The panoramic image data generated in # 106 is recorded in the storage medium 25 under the control of the camera microcomputer 15. Above, description of the operation example of FIG. 2 is complete | finished.

  The electronic camera 11 according to the first embodiment determines the imaging timing so that the size of the overlapping range between unit images is larger than the maximum size of a face that can be captured when performing panoramic shooting. As a result, there is no case where the face is interrupted in both of the two unit images, so that the face shift in the panoramic image can be suppressed. In addition, since the electronic camera 11 in the first embodiment individually adjusts the imaging timing for each shooting according to the size of the face detected in the through image, the number of unit images acquired by panoramic shooting is more than necessary. The increase can be suppressed.

<Description of Second Embodiment>
FIG. 6 is a flowchart illustrating an operation example of the panoramic shooting mode in the second embodiment. FIG. 7 shows an outline of the imaging timing in the panoramic shooting mode of the second embodiment. This second embodiment is a modification of the above-described first embodiment. In the following embodiments, the configuration of the electronic camera 11 is the same as that of the first embodiment, and therefore, duplicate explanation is omitted.

  Step # 201: The camera microcomputer 15 determines whether a panoramic shooting start instruction has been received. If the above requirement is satisfied (YES side), the process proceeds to # 202. On the other hand, if the above requirement is not satisfied (NO side), the camera microcomputer 15 waits until there is an instruction to start panoramic photography.

  Step # 202: With the user panning the electronic camera 11, the camera microcomputer 15 starts panorama shooting.

  At the time of panoramic shooting in the second embodiment, first, the imaging control unit 21 acquires the first unit image simultaneously with the panoramic shooting start instruction. Thereafter, the camera microcomputer 15 captures a through image and continuously monitors changes in the image within the imaging range. Then, the image analysis unit 23 executes matching processing and face recognition processing using the through image.

  Step # 203: The image analysis unit 23 determines whether or not a person's face has been recognized during panorama shooting. If the above requirement is satisfied (YES side), the process proceeds to # 205. On the other hand, if the above requirement is not satisfied (NO side), the process proceeds to # 204.

  Step # 204: The image analysis unit 23 determines whether or not the number of pixels in the overlapping range between the previous unit image and the current imaging range has reached a predetermined number. If the above requirement is satisfied (YES side), the process proceeds to # 205. On the other hand, if the above requirement is not satisfied (NO side), the process proceeds to # 206.

  Note that the predetermined number of pixels in # 204 of the second embodiment can be determined regardless of the size of the face, and may be the minimum number of pixels as an overlapping range for performing panorama synthesis.

  Step # 205: The imaging control unit 21 drives the imaging element 13 to take a unit image.

  Step # 206: The camera microcomputer 15 determines whether or not a panoramic shooting end instruction has been received. If the above requirement is satisfied (YES side), the process proceeds to # 207. On the other hand, if the above requirement is not satisfied (NO side), the camera microcomputer 15 returns to # 203 and repeats the above operation.

  Step # 207: The panorama synthesizing unit 24 connects a plurality of unit images (obtained in # 202 and # 205) acquired by panoramic shooting in the pan direction to generate one panoramic image. In the process of # 207, face detection in the overlapping range is not performed, and the panorama composition unit 24 only needs to connect unit images. Above, description of the operation example of FIG. 6 is complete | finished.

  When the electronic camera 11 in the second embodiment detects a face during panoramic shooting (YES side of # 203) or when the overlapping range of unit images reaches a predetermined size (YES side of # 204), the unit image (# 205). For this reason, unit images with uninterrupted faces can be acquired with certainty, so that face misalignment in panoramic images can be suppressed. In addition, since the electronic camera 11 in the second embodiment changes the imaging timing when a face is detected in a through image, it is possible to suppress an increase in the number of unit images acquired by panoramic shooting more than necessary.

<Description of Third Embodiment>
FIG. 8 is a flowchart showing an operation example of the panorama shooting mode in the third embodiment. FIG. 9 shows an outline of the imaging timing in the panoramic shooting mode of the third embodiment.

  Note that the operations of # 301 and # 302 in FIG. 8 are the same as those of # 201 and # 202 of the second embodiment, so redundant description will be omitted.

  Step # 303: The image analysis unit 23 determines whether or not the number of pixels in the overlapping range between the previous unit image and the current imaging range has reached a predetermined number of pixels. If the above requirement is satisfied (YES side), the process proceeds to # 304. On the other hand, if the above requirement is not satisfied (NO side), the process proceeds to # 306. Note that the image analysis unit 23 in # 303 performs the above determination without targeting the unit image additionally captured in # 308 to be described later.

  Step # 304: The imaging control unit 21 drives the imaging element 13 to take a unit image.

  Step # 305: The camera microcomputer 15 determines whether or not a panoramic shooting end instruction has been received. If the above requirement is satisfied (YES side), the process proceeds to # 309. On the other hand, if the above requirement is not satisfied (NO side), the camera microcomputer 15 returns to # 303 and repeats the above operation.

  In the third embodiment, acquisition of unit images is repeated at a constant interval in which the overlapping range between images becomes a predetermined number of pixels by the loop of # 303 to # 305.

  Step # 306: The image analysis unit 23 determines whether or not a person's face has been recognized during panorama shooting. If the above requirement is satisfied (YES side), the process proceeds to # 307. On the other hand, if the above requirement is not satisfied (NO side), the process proceeds to # 305.

  Step # 307: The image analysis unit 23 determines whether or not the face size of the person recognized at # 306 is larger than a predetermined number of pixels (# 303). If the above requirement is satisfied (YES side), the process proceeds to # 308. On the other hand, if the above requirement is not satisfied (NO side), the process proceeds to # 305.

  Step # 308: The imaging control unit 21 drives the imaging element 13 to additionally take a unit image. Note that the # 308 unit image is captured when the face protrudes from the overlapping range of unit images acquired at regular intervals (the loop of # 303 to # 305). Thereafter, the process proceeds to # 305.

  Step # 309: The panorama synthesizing unit 24 connects a plurality of unit images (obtained in # 304 and # 308) acquired in the panorama shooting in the pan direction to generate one panorama image. In the process of # 309, if unit images overlap at the time of composition, the panorama composition unit 24 partially deletes the unit image obtained in # 304 and preferentially uses the unit image obtained in # 308. Generate a panoramic image. Above, description of the operation example of FIG. 8 is complete | finished.

  In the electronic camera 11 according to the third embodiment, when a face protrudes from an overlapping range of unit images acquired at a constant interval (loop of # 303 to # 305), the unit is set at an imaging timing different from the constant interval. Further images are taken (# 308). For this reason, unit images with uninterrupted faces can be acquired with certainty, so that face misalignment in panoramic images can be suppressed. In addition, since the electronic camera 11 in the third embodiment additionally captures a unit image only when the size of the face protrudes from the overlapping range, the number of unit images acquired in panoramic shooting increases more than necessary. Can be suppressed.

<Supplementary items of the embodiment>
(Supplement 1) In each of the above embodiments, the case where the recognition target is a human face has been described. However, the present invention is not limited to the case where the recognition target is a person's face, and can of course be applied to panoramic shooting using another subject such as a pet animal as the recognition target.

  (Supplement 2) Panorama shooting in the present invention is not limited to panning in the horizontal direction, and can naturally be applied to panoramic shooting by tilting the electronic camera in the vertical direction.

  (Supplement 3) In the above embodiment, the example in which each function of the imaging apparatus is realized by software by a program has been described. However, in the present invention, of course, the functions of the imaging control unit 21, the image processing unit 22, the image analysis unit 23, and the panorama synthesis unit 24 may be realized by hardware using an ASIC.

  From the above detailed description, features and advantages of the embodiments will become apparent. It is intended that the scope of the claims extend to the features and advantages of the embodiments as described above without departing from the spirit and scope of the right. Further, any person having ordinary knowledge in the technical field should be able to easily come up with any improvements and modifications, and there is no intention to limit the scope of the embodiments having the invention to those described above. It is also possible to use appropriate improvements and equivalents within the scope disclosed in.

DESCRIPTION OF SYMBOLS 11 ... Electronic camera, 12 ... Imaging optical system, 13 ... Imaging device, 14 ... Signal processing part, 15 ... Camera microcomputer, 16 ... 1st memory, 17 ... 2nd memory, 18 ... Media I / F, 19 ... Display part , 20 ... operation unit, 21 ... imaging control unit, 22 ... image processing unit, 23 ... image analysis unit, 24 ... panorama synthesis unit, 25 ... storage medium

Claims (7)

An imaging unit that captures an image of a subject;
A subject recognition unit that extracts a recognition target from an image acquired by the imaging unit;
An imaging control unit that captures a plurality of unit images at an imaging timing at which a part of the imaging range overlaps when panoramic imaging is performed by moving the imaging range of the imaging unit in a predetermined direction;
A panorama synthesizing unit that generates a single panoramic image by connecting a plurality of the unit images acquired in the panoramic shooting,
The imaging apparatus, wherein the imaging control unit determines an imaging timing in the panoramic shooting according to a size of the recognition target on the image. The imaging device according to claim 1,
The imaging apparatus, wherein the imaging control unit determines an imaging timing in the panorama shooting so that a size of an overlapping range between the unit images is equal to or larger than a size of the recognition target. In the imaging device according to claim 1 or 2,
The imaging apparatus, wherein the imaging control unit varies the imaging timing of the unit image according to the recognition target extraction result in the panoramic shooting. The imaging device according to claim 1,
The imaging control unit captures the unit images at regular intervals during the panoramic shooting, and determines that the recognition target protrudes from the overlapping range of the unit images, and an imaging timing different from the interval The image pickup apparatus further picks up the unit image. In the imaging device according to any one of claims 1 to 4,
The panorama synthesizing unit synthesizes and uses an image satisfying a predetermined condition among the unit images in the recognition target range of the panoramic image. In the imaging device according to any one of claims 1 to 5,
An imaging apparatus, further comprising: a warning unit that outputs a warning when the panoramic shooting cannot be performed because the moving speed of the imaging range is high. The imaging apparatus according to any one of claims 1 to 6,
The imaging unit pre-photographs at least a part of a range shot in the panoramic shooting before the panoramic shooting,
The imaging control unit determines an imaging timing in the panoramic shooting on the basis of a recognition target having the largest size among the recognition targets extracted from the through image acquired in the preliminary shooting. .

Images