JP2014011639A - Imaging device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To predict a photographed image acquirable in photographing in accordance with a photographing condition before photographing in advance and display it.SOLUTION: An imaging device according to the present invention sets a photographing condition for photographing by imaging means (S401), calculates subject information including the position and shape of a subject on the basis of an image captured by the imaging means (S403), creates a preview image obtained by predicting an image acquired in imaging in accordance with the photographing condition from the photographing condition and the subject information (S404), and displays the preview image.

Description

  The present invention relates to an imaging apparatus, and more particularly to an imaging apparatus capable of displaying an image in which the effects of strobe light emission and blurring are predicted before shooting.

  In general, a digital camera or a camera mounted on a mobile phone displays an image captured by an imaging unit on a display built in the main body, an electronic viewfinder, or an externally connected display before shooting. be able to. The preview display, which is a display before shooting, allows the user to determine shooting conditions and image composition at the time of shooting prior to shooting.

  However, the photographed image obtained by photographing may differ from the preview image displayed on the screen before photographing due to differences in photographing conditions such as the presence or absence of strobe light emission and the shutter speed. For this reason, even if shooting is performed while checking with a preview, an image expected by the user may not be obtained.

  For example, in the case of shooting using strobe light emission, a preview image is displayed without strobe light emission during preview. Therefore, what kind of images can be taken when the flash is fired, that is, how shadows are generated by the flash, or what changes in color and brightness are actually taken There is a problem that you do not know until you confirm.

  In addition, since the exposure time at the time of shooting and the exposure time of each frame at the time of preview do not match, the degree of the effects of blurring such as subject shake and camera shake may differ between the shot image and the preview image. is there. In particular, when shooting conditions are set that have a long exposure time at the time of shooting, when the preview is displayed at a frame rate that matches the exposure time for shooting, smooth movement cannot be displayed. It can be a jerky video.

  Therefore, in order to display smooth movement, it is common to set the frame rate at the time of preview high. However, with this setting, while previewing, the blurring becomes noticeable due to the high frame rate. On the other hand, the effect of blurring on shooting is not known until the actual shooting and display of the image. It will not be.

  Therefore, various proposals have been made in the past in order to match the preview image with the actually captured image.

  For example, using a twin-lens camera, the distance distribution of the subject is obtained from the principle of triangulation, and the light and dark distribution due to the amount of strobe light is predicted by changing the transmittance of the liquid crystal displayed in the finder according to the distance. (For example, refer to Patent Document 1).

  Further, a technique is disclosed in which means for detecting the position of the sun and the intensity of sunlight is provided on the upper surface of the camera body, and a shadow generation state is predicted according to the position of the sun and the intensity of sunlight (for example, , See Patent Document 2). In the case where the occurrence of a shadow is predicted, a method is shown in which a strobe is emitted toward a subject and used for suppressing the generation of a shadow.

JP 2001-83563 A JP-A-8-146484

  The technique disclosed in Patent Document 1 described above controls the brightness of the finder image according to the subject distance, and the position and shape of the subject or the color of the subject according to the strobe light color and light distribution characteristics. It does not predict the occurrence of changes or shadows. For this reason, it is possible to predict the luminance related to the portion where the strobe light hits, but there is a problem that it is impossible to predict the shadow and the color.

  Further, the technique disclosed in Patent Document 2 predicts the possibility of a shadow from the position of the sun and the intensity of sunlight, and what kind of shadow is generated in the captured image according to the position and shape of the subject. I can't predict what to do. From the above problems, even if the techniques disclosed in the above patent documents are used, it is assumed that the preview image and the actual captured image are different from each other. There is a problem that I do not know if is taken.

  In order to solve the above problems, an imaging apparatus according to the present invention includes a shooting condition setting unit that sets shooting conditions for shooting by an imaging unit, and a position and shape of a subject based on an image captured by the imaging unit. A preview image that creates a preview image that predicts an image that is acquired when the imaging unit captures an image according to the shooting condition from the subject information calculation unit that calculates subject information including the shooting condition and the subject information The image forming apparatus includes a creating unit and a display unit that displays the preview image.

  According to the present invention, it is possible to predict and display a photographed image that can be acquired when photographing according to photographing conditions before photographing.

It is a figure which shows the external appearance of the imaging device which concerns on an Example. It is a block diagram which shows the internal structure of the imaging device which concerns on an Example. It is a figure which shows the internal structure of the imaging part which concerns on an Example. 6 is a flowchart illustrating an operation procedure of the imaging apparatus according to the embodiment. 6 is a flowchart illustrating a processing procedure of subject information calculation processing according to the first embodiment. It is a figure which shows an example of the arrangement | positioning information which concerns on an Example. It is a figure which shows the relationship between a to-be-photographed object and an imaging part. 6 is a diagram illustrating an example of a multi-viewpoint image according to Embodiment 1. FIG. 6 is a flowchart illustrating a processing procedure of preview image creation processing according to the first embodiment. It is a figure which shows an example of the light distribution characteristic of a light source. It is a figure for demonstrating the concept of a ray tracing method. 6 is a flowchart illustrating a processing procedure of preview image rendering processing according to Embodiment 1; It is a figure for demonstrating an example of the effect of this invention. 12 is a flowchart illustrating a processing procedure of preview image creation processing according to the second embodiment. 10 is a flowchart illustrating a processing procedure of preview image rendering processing according to the second embodiment.

Example 1
Details of the imaging apparatus according to the embodiment of the present invention will be described below.

  First, FIG. 1 is a diagram illustrating an example of the appearance of the imaging apparatus according to the present embodiment. In FIG. 1, (a) represents the front surface of the imaging device, and (b) represents the back surface of the imaging device.

  As shown in FIG. 1A, nine imaging units 101 to 109 capable of acquiring a color image, a shooting button 110, and a light emitting unit 111 are provided on the front surface of the imaging device 100. Yes. The light emitting unit 111 includes light emitting elements 112 to 120 that can control light emission independently of each other.

  When the user presses the shooting button 110, the imaging units 101 to 109 receive light from the subject with a sensor (imaging device), and the received signal is A / D converted to simultaneously generate a plurality of color images (digital data). Can be acquired. A multi-viewpoint image group composed of a plurality of color images can be obtained by capturing the same subject from a plurality of viewpoint positions with such a multi-view imaging device.

  Although the number of imaging units is nine here, the number of imaging units is not limited to nine. That is, in the present invention, in order to acquire subject information related to a subject from a multi-viewpoint image group, it is only necessary to have a plurality of imaging units. In addition, when the subject information is acquired from another sensor or the like without acquiring from the multi-viewpoint image group, a monocular imaging device including one imaging unit may be used.

  In addition, although the example where an imaging part is arrange | positioned equally when providing two or more imaging parts is demonstrated, arrangement | positioning of an imaging part is arbitrary. For example, they may be arranged radially or linearly or randomly. Similarly, the light emitting portion 111 can have any number of light emitting elements and any arrangement in the present invention.

  Next, as illustrated in FIG. 1B, a display unit 121 including a liquid crystal display and an operation unit 122 for receiving various operations by the user are provided on the back surface of the imaging apparatus.

  The display unit 121 can display a shooting condition setting operation status or a GUI for confirming the setting, a preview image or video of a subject being shot, and an image after shooting.

  The operation unit 122 includes a mode switching button 123 and an operation button 124. When the user presses the mode switching button 123, the preview mode described later is switched. The operation button 124 is used by the user to perform a direction operation or the like in order to change or select an item, and is arranged in the form of a cross button, for example. In this figure, only two buttons are shown as an example for convenience of explanation, but any button form may be used. Further, although an example in which operations are performed using various buttons is shown in the present embodiment, a configuration in which the display unit 121 receives operations by a touch panel method instead of the buttons may be used. For example, various types of touch panels such as a pressure-sensitive type and an electromagnetic induction type can be used, and the coordinates touched by the user can be detected, and the user's instructions can be read based on the detected coordinates.

  FIG. 2 is a block diagram illustrating an internal configuration of the imaging apparatus 100.

  The CPU 201 is a central processing unit and controls the respective units connected via the bus 204 in an integrated manner.

  A RAM 202 is a RAM (Random Access Memory) that functions as a main memory, a work area, and the like of the CPU 201.

  The ROM 203 is a ROM (Read Only Memory) that stores a control program executed by the CPU 201.

  The bus 204 functions as a transfer path for various data. For example, digital data acquired by the imaging units 101 to 109 is sent to a predetermined processing unit via the bus 204.

  The display control unit 205 performs display control for displaying captured images and characters displayed on the display unit 121.

  The imaging control unit 206 controls the optical system based on an instruction from the CPU 201 such as focusing, opening / closing a shutter, and adjusting an aperture. In addition, the control parameter of the imaging unit is adjusted based on the arrangement information of the imaging device.

  The light emission control unit 207 selectively causes the light emitting elements 112-120 constituting the light emitting unit 111 to emit light based on an instruction from the CPU 201.

  The digital signal processing unit 208 performs various processes such as white balance processing, gamma processing, and noise reduction processing on the digital data received via the bus 204.

  The encoder unit 209 converts the digital data into a file format such as JPEG or MPEG.

  The external memory control unit 210 is an interface for connecting the imaging apparatus 100 to a PC or other media 213 (for example, a hard disk, a memory card, a CF card, an SD card, or a USB memory).

  The subject information calculation unit 211 performs calculations for obtaining the position, shape, texture, and the like of the subject. In this embodiment, a method of calculating from the multi-viewpoint image group acquired by the imaging units 101 to 109 or the multi-viewpoint image group output from the digital signal processing unit 208 is shown. Details of processing by the subject information calculation unit 211 will be described later.

  The image creating unit 212 creates an image that is assumed when the strobe is fired and photographed using CG. Details of processing by the image creation unit 212 will be described later.

  Although there are other components of the image pickup apparatus other than those described above, the description is omitted because it is not an essential component of the present invention.

  FIG. 3 is a diagram illustrating an internal configuration of the imaging units 101 to 109. The imaging units 101 to 109 include lenses 301 to 303, an aperture 304, a shutter 305, an optical low-pass filter 306, an iR cut filter 307, a color filter 308, a sensor 309, and an A / D conversion unit 310. The zoom lens 301, the focus lens 302, and the shake correction lens 303 function as lenses. The sensor 309 is a sensor such as a CMOS or CCD.

  When the light amount of the subject is detected by the sensor 309, the detected light amount is converted into a digital value by the A / D conversion unit 310, and is output as digital data to the bus 204.

  An imaging apparatus according to an embodiment of the present invention obtains subject information for specifying a subject before photographing, and creates a preview image by CG using the calculated subject information and photographing conditions set by a user. , To display. Hereinafter, the operation of the imaging apparatus having the above configuration will be described with reference to the flowchart of FIG.

  First, in step S401, the user uses the operation buttons 124 to set shooting conditions such as shutter speed, aperture, ISO sensitivity, and presence / absence of flash emission. In addition, when performing strobe light emission at the time of shooting, it is also possible to select a light emission pattern. Here, the light emission pattern indicates a combination of light emitting elements that emit light from among the plurality of light emitting elements 112 to 120. Note that the shooting conditions do not have to be set by the user, that is, may be set in advance.

  In step S402, a multi-viewpoint image group output from the digital signal processing unit 208 is acquired. Each of the multi-viewpoint images is an image picked up without a strobe light emission.

  In step S403, subject information is calculated from the multi-viewpoint image group acquired in step S402. This process is executed by the subject information calculation unit 211. Details of the operation of the subject information calculation unit 211 will be described later.

  In step S404, a preview image is created using the shooting conditions set in step S401 and the subject information calculated in step S403. This process is executed by the image creation unit 212. Details of the operation of the image creation unit 212 will be described later.

  In step S405, the preview image created in step S404 is displayed on the display unit 121 according to an instruction from the display control unit 205.

  In step S406, it is determined whether the user of the imaging apparatus has pressed the shooting button 110. When the photographing button 110 is pressed, the process proceeds to S407, and when it is not pressed, the process returns to S401. Until the user presses the shooting button 110, the setting change of the shooting conditions and the preview are repeated.

  In step S407, when the shooting button 110 is pressed, images are immediately captured by the imaging units 101 to 109 based on the shooting conditions set in step S401. The captured image is processed in the digital signal processing unit 208 and the encoder unit 209, and then stored in the PC or other media 213 through the external memory control unit 210.

  Details of each process will be described below.

(Subject information calculation process)
The subject information calculation process (S403) is performed in the subject information calculation unit 211. With this processing, the position, shape, and texture of the subject can be acquired from the multi-viewpoint image group output from the digital signal processing unit 208.

  Details of the subject information calculation process (S403) will be described below with reference to the flowchart of FIG. This process is performed during the preview before the user presses the shooting button 110.

  First, in step S501, image capturing unit information including arrangement information indicating a relative positional relationship between the image capturing units 101 to 109 and information on a focal length is acquired.

  Here, the arrangement information may be information indicating at least the relative positional relationship between the optical centers of all the imaging units. In this embodiment, it is assumed that the imaging unit information is recorded in advance in the ROM 203 and can be acquired by reading it. In the case of an imaging apparatus that can change the arrangement of individual imaging units, the arrangement of the imaging units may be changed based on a user instruction, and thereby the arrangement information may be changed.

  FIG. 6 specifically shows an example of arrangement information. In the arrangement information shown in FIG. 6, the imaging unit number is for specifying the imaging unit. For example, if the imaging unit number is 1, it corresponds to the imaging unit 101 shown in FIG. Similarly, the imaging numbers 2-9 correspond to the imaging units 102-109, respectively.

  In the arrangement information shown in FIG. 6, the optical center of the imaging unit 101 identified by the imaging number 1 is used as the origin. With the origin as a reference, the position of the optical center of the imaging units 102 to 109 can be expressed using two-dimensional coordinates of an X coordinate (horizontal direction) and a Y coordinate (vertical direction).

  Subsequently, in step S502, subject information indicating the position and shape of the subject is calculated from the above-described imaging unit information and the multi-viewpoint image group. Any method may be used to calculate the position and shape of the subject. Hereinafter, an example of the calculation method will be described with reference to FIGS. 7 and 8 with respect to a method for obtaining the distance to the subject using the two imaging units 101 and 102.

  In the imaging unit 101 and the imaging unit 102, the focal lengths of the lenses are the same. In addition, it is assumed that the two imaging units have the same Y coordinate and only the X coordinate is different.

  In FIG. 7, it is assumed that a subject 703 is a subject to be photographed. Further, it is assumed that the subject 703 is within the visual field range 701 of the imaging unit 101 and the visual field range 702 of the imaging unit 102.

  Under this condition, the imaging unit 101 acquires the image shown in FIG. Similarly, the imaging unit 102 acquires the image illustrated in FIG. Here, it is assumed that a position 704 that is one point on the surface of the subject 703 is shown in the coordinates (x1, 0) of the image by the imaging unit 101. Similarly, it is assumed that the image is captured at the coordinates (x2, 0) of the image by the imaging unit 102.

  In the following, in the two images, the same position of the subject appearing together is called a corresponding point. Here, assuming that the center of each image is the origin (0, 0), the distance D from the optical center of the imaging unit regarding the position 704 that is the corresponding point can be expressed by the following expression.

  In Equation 1, the pixel pitch of the image sensors of the two imaging units 101 and 102 is p, the distance between the optical centers of the imaging units 101 and 102 is B, and the focal length of the imaging units 101 and 102 is f.

  Based on Equation 1, the distance to one point in the subject can be calculated. Similarly, the distance from the optical center of the imaging unit can be obtained for other corresponding points of the subject. Furthermore, the shape of the subject can be obtained by changing the combination of the imaging units.

  In addition, when not only the X coordinate but also the Y coordinate are different or the direction of the optical axis is different, the rotation and parallel progression are obtained for each image so that the epipolar line is horizontal, and the two images are parallel. It is possible to handle the images as different images only in the X coordinate. Such parallelization of the image is not the main point of the present invention, and the description thereof will be omitted. As a method for obtaining corresponding points in two images, for example, a method of performing block matching in two images can be cited. Of course, the present invention is not limited to block matching, and any other method can be applied.

  Further, the texture of the subject can be acquired as the pixel value of the corresponding point used when obtaining the above-described distance. Note that the texture is expressed by three colors of RGB, and the signal value of each color is acquired.

(Preview image creation process)
Next, details of the preview image creation process (S404) shown in FIG. 4 will be described.

  The preview image creation process is executed by the image creation unit 212. Before the strobe light is emitted, an image assumed when the strobe light is emitted and photographed is generated by CG and output.

  Hereinafter, the processing procedure of the preview image creation processing S404 will be described with reference to the flowchart of FIG.

  First, in step S901, shooting conditions including information indicating the shutter speed, aperture, ISO sensitivity, and presence / absence of flash emission set by the user via the operation unit 122 are acquired.

  In step S902, it is determined whether or not the shooting condition acquired in step S901 includes a strobe emission instruction. If the flash is to be emitted, the process proceeds to step S903, and if not, the process proceeds to step S908.

  In step S903, the preview mode set by the user using the mode switching button 123 is acquired. The preview mode in the present embodiment is set to two modes: a strobe preview mode that displays the strobe light emission effect on the preview image, and a normal preview mode that does not display the strobe light effect similar to the conventional preview image.

  In step S904, it is determined whether the preview mode acquired in step S903 is the strobe preview mode. If the strobe preview mode is set (S904; YES), the process proceeds to step S905. On the other hand, in the normal preview mode (S904; NO), the process proceeds to step S908.

  In step S905, strobe characteristics indicating the light emission pattern and color of the strobe are acquired. Specifically, the strobe characteristic is light emission information indicating the light emission pattern set by the user using the operation button 124, the position and luminance distribution of the light emitting elements that emit light with the set light emission pattern, and the light distribution characteristic.

  Here, the luminance distribution of the light emitting element may be expressed using the three primary colors of RGB, or another color space may be used. It can also be expressed using spectral radiance. In the following, it is assumed that three colors of RGB are used.

  The light distribution characteristic is an intensity distribution in each direction of light emitted from the light source, and the range in which one light source illuminates the scene varies depending on the directivity of the light source. A specific example is shown in FIG.

  FIG. 10A shows an example of a light source that illuminates a wide range although the intensity in each direction is low. On the other hand, FIG. 10B is an example of a light source having high intensity in a specific direction.

  When the strobe light color is expressed by RGB three primary colors, the light distribution characteristics can be different for the three colors R, G, and B.

  In this embodiment, it is assumed that the light distribution characteristic of each light emitting element is recorded in the ROM 203 in advance. And the light distribution characteristic regarding the light emitting element which light-emits according to a light emission pattern is read, and it can add according to arrangement | positioning information, and can acquire the light distribution characteristic of the whole light emission part.

  Here, the data format of the light distribution characteristic may be a desired format as long as the correspondence between the light output angle from the light source and the light intensity can be obtained. For example, it may be a function of the light intensity with respect to the output angle, or may be in the form of a look-up table (hereinafter referred to as LUT) indicating the correspondence between the output angle and the light intensity.

  In step S906, subject information including the position, shape, and texture of the subject calculated by the subject information calculation unit 211 is acquired.

  In step S907, based on the shooting conditions acquired in step S901, the strobe characteristics acquired in step S905, and the subject information acquired in step S906, an image that is supposed to be shot with flash emission is rendered by CG. create. Details regarding the preview image rendering process will be described later.

  In step S908, a preview image is acquired when the strobe is not emitted or when the strobe preview is not performed even when the strobe is emitted. This preview image is 105 images arranged in the center among a plurality of imaging units. However, the preview image may be an image of another imaging unit, or may be an image obtained by performing arbitrary processing on the images of a plurality of imaging units.

  In step S909, the image obtained in step S907 or step S908 is output to the display control unit 205. Then, the display control unit 205 causes the display unit 121 to display the received image. Then, the preview image creation process S404 of the present embodiment ends.

  As described above, when the strobe preview mode is selected, a preview image predicting an image obtained when the strobe light is emitted can be created and displayed during the preview before the strobe light is emitted.

(Preview image rendering process)
The preview image rendering process S907 is a process for rendering an image obtained when the strobe emits light using the above-described shooting conditions, strobe characteristics, and subject information using CG. Various rendering methods using CG have been proposed so far, but any method may be used as long as it can reproduce the color of a subject and the generation of shadows by strobe light. An example of such a rendering technique is a ray tracing method. Hereinafter, the ray tracing method will be described with reference to FIG.

  Assume that a camera 1101, a screen 1102, subjects 1104 and 1105, and a light source 1106 are arranged as shown in FIG. The pixel 1103 is a pixel for determining a pixel value by ray tracing on the screen.

  In the ray tracing method, first, a ray is emitted from the camera 1101 toward the pixel 1103. The light beam is repeatedly reflected on the object and traced until it reaches the strobe light source. Then, the pixel value of the pixel 1103 on the screen is determined based on the attenuation of the light beam by the object colliding during tracking and the color of the light source.

  In the following, the procedure for calculating the luminance in the case where one light beam is emitted for one point on the screen will be described with reference to the flowchart of FIG. For example, 105 camera positions arranged in the center of the imaging units 101 to 109 are used as camera positions during rendering.

  First, in step S1201, the ray number is initialized. This is a number for grasping what number of light rays are processed when a plurality of light rays are emitted to each pixel on the screen. In initialization, 0 is substituted.

  In step S1202, the number of ray collisions is initialized. Specifically, 0 is substituted for the number of collisions.

  In step S1203, 1 is added to the ray number to update the value.

  In step S1204, the direction and intensity of the light beam are initialized. The initial direction of the light beam is a direction connecting pixels that determine pixel values on the screen from the camera position. The intensity of the light beam is represented by three colors of RGB, and the initial value is 1.

  In step S1205, the light beam is blown in the direction set to the light beam from the tracking start point, and a collision point with the subject or the light source is searched. The tracking start point is a collision point between a pixel on the screen or a light ray and a subject in the previous tracking. The search for the collision point may be performed by a desired method. For example, it is only necessary to sequentially perform collision determination with all objects in the scene to obtain a collision point closest to the tracking start point.

  In step S1206, it is determined whether the point where the light beam collides is a strobe as a light source or another subject. If it collides with the light source, the process proceeds to step S1212. If it collides with another subject, the process proceeds to step S1207.

  In step S1207, 1 is added to the variable held by the light beam and the number of collisions, and the value is updated.

  In step S1208, the direction of the light beam after the collision with the subject is determined. The light beam direction after the collision may be a random direction, and a random number is generated for each collision, and the direction corresponding to the random number is set.

  In step S1209, the reflectance of the subject that the light ray collides with is acquired. The reflectance of the subject is composed of a diffuse reflection component and a specular reflection component. The diffuse reflection component uses the texture of the subject obtained by the subject information calculation unit, the specular reflection component is the direction of the light ray before the collision, the direction of the light ray after the collision determined in step S1208, and the method of the subject at the collision point. A value corresponding to the line direction is calculated and used.

In this embodiment, the specular reflection component is calculated by the Phong model. When the direction of the light beam before the collision is d, the direction of the light beam after the collision is d ′, and the normal direction of the object is N, the specular reflectance R _s is expressed by the following equation.

Here, k _s represents a specular reflection coefficient, and n is a parameter representing the spread of specular reflection. Constant values are set in advance for k _s and n. However, in combination with the image recognition technology for the subject, a value corresponding to the recognition result may be set for each subject.

For example, when it is found by image recognition that the collision point is human skin, the parameter corresponding to human skin is used. It should be noted that other models can be used for calculating the specular reflectance. The reflectances R _R , R _G , and R _B of the subject at the collision point are as follows. Here, R _dR , R _dG , and R _dB represent diffuse reflectance, and are R, G, and B values of the texture of the subject.

In step S1210, the intensity of the light beam after the collision is calculated. The intensity of the light beam is obtained by multiplying the intensity of the light beam at that time by the reflectance of the subject calculated in step S1209. Assuming that the light intensity before the update of each color of RGB for the light beam number i is P _Ri , P _Gi , P _Bi , the light intensity after the update can be expressed by the following formulas: P _Ri ′, P _Gi ′, P _Bi ′.

  In step S1211, the number of collisions of the light beam is compared with a preset maximum number of collisions to determine the next processing step. If the maximum number of collisions is not reached, the process returns to step S1205. On the other hand, if the maximum number of collisions is exceeded, 0 is assigned to the light intensity, and the process proceeds to step S1212.

  Originally, it is desirable to trace until the light beam reaches the light source, but enormous calculation time is required to trace all the light beams until they reach the light source. Therefore, when the light beam does not reach the light source by the preset maximum number of collisions, the processing is terminated by assuming that the light beam does not reach the light source, thereby realizing high-speed processing.

  Therefore, the intensity of such a light beam is treated as zero. On the other hand, if the maximum number of collisions is not reached, the ray tracing process is repeated with the collision point with the object as the new ray tracing start point.

In step S1212, the luminance is calculated for the ray that has been traced. Specifically, the luminance is calculated by multiplying the intensity of the light beam, the luminance of the light source, and the light distribution characteristic of the light source. This is executed for each of the three colors RGB. The calculation of the brightness is expressed as follows. Here, I _Ri , I _Gi , and I _Bi represent the luminance of each color of RGB with respect to the light beam with the ray number i, L _R , L _G , and L _B are the luminance of the light source, S _R (θ), S _G (θ). , S _B (θ) represents the light distribution characteristics of the light source.

In the above equation, θ represents an incident angle when a light ray collides with a strobe as a light source.

  In step S1213, the preset number of rays to be emitted per pixel is compared with the ray number. If the ray number is less than the number of rays, the process returns to step S1202, and a new ray is emitted to the same pixel. On the other hand, if the light ray number is greater than or equal to the number of light rays, the process proceeds to step S1214.

In step S1214, the pixel value of the pixel is calculated from the average of the luminances of a plurality of light rays skipped with respect to the same pixel. First, the average brightness I _R , I _G , and I _B of each light ray is calculated as in the following equation.

Then, the luminance calculated by the above equation is added to the image obtained from the imaging unit 105 to determine the pixel value. If the preview images of the imaging unit 105 are I _R , I _G , and I _B , the luminances I _R ″, I _G ″, and I _B ″ after addition are as follows.

In the above equation, f (x) reproduces the saturation of the pixel value of the sensor and can be expressed by the following equation.

  The process related to the ray tracing method is executed according to the procedure as described above.

  Next, the effect of the present invention will be described with reference to FIG.

  Here, an example of a preview image in the case of performing flash photography on two types of subjects as shown in FIG. 13A is shown in FIG. By tracking the light multiple times, shadows can be reproduced if there is another object between the strobe light and the object. In the ray tracing method, a light ray easily reaches an object close to the camera, and a light ray does not easily reach a far object. Therefore, a subject close to the distance can be reproduced brightly, and an object that is far away and difficult to reach the strobe light can be reproduced darkly.

  In addition, although the example which determines the cancellation of the tracking process regarding the light ray before reaching the light source using the number of collisions has been shown, the determination may be performed using other variables. For example, the tracking process may be aborted when the light intensity falls below a certain threshold.

  As described above, by using the above-described imaging device, it is possible to predict the effect of the strobe light based on the shooting conditions set by the user and display it as a preview image before light emission. Thereby, it is possible to reduce a difference between a preview image at the time of preview and an image obtained by actual shooting. In addition, before the flash fires, the user can check the brightness change and shadows caused by the flash, start setting the flash output and direction, and other shooting conditions such as shutter speed and composition settings. Can be set in advance.

  For example, when the light emitting unit is composed of a plurality of light emitting elements as shown in this embodiment, the effect of the strobe when the light emission pattern is changed can be confirmed during the preview.

  In this embodiment, an example of using a multi-view camera capable of acquiring images from a plurality of viewpoints to acquire subject information has been described. However, a subject information acquisition method is a method of calculating image processing from multi-view images. It is not limited to. Even a monocular camera may be combined with various sensors such as a distance sensor.

(Example 2)
In the first embodiment, an imaging apparatus capable of displaying a preview image in which the effect of strobe light emission is predicted before shooting is shown. In the second embodiment, an imaging apparatus that predicts subject blur and camera shake that may occur at the shutter speed during shooting in a preview image will be described.

  The components of the imaging apparatus of the present embodiment are the same as those of the first embodiment, but the processing contents of the preview image creation process (S404) are different. Hereinafter, portions different from the first embodiment in each process will be described in detail.

(Preview image creation process)
In the preview image creation process (S404) of the present embodiment, an image in which subject blur or camera shake is predicted is created by CG and output during preview. The operation of the preview image creation process S404 will be described below with reference to the flowchart of FIG. In FIG. 14, only steps that are different from the flowchart of FIG. 9 described in the first embodiment will be described.

  In step S1401, the number of processing target frames is calculated. The number of frames to be processed is the number of frames obtained within the exposure time corresponding to the exposure time at the time of shooting during preview.

  For example, when the exposure time obtained from the shutter speed included in the shooting condition is 1 second and the exposure time during preview is 0.04 seconds (= 25 FPS), the number of frames to be processed is 25.

  In step S1402, information indicating the preview mode set by the user using the mode switching button 123 is acquired. There are two preview modes in the present embodiment: a blur preview mode that displays the effect of blur correction as a preview image, and a normal preview mode that does not display the effect of blur correction.

  In step S1403, if the information indicating the preview mode acquired in step S1402 is the blur preview mode, the process proceeds to step S906 (S1403; YES).

  On the other hand, in the normal preview mode, the process proceeds to step S908 (S1403; NO).

  In step S1404, the preview image predicted to be blurred is rendered by CG. Details regarding the preview image rendering process will be described later.

  Thus, the operation of the preview image creation process S404 of this embodiment is completed. When the blur preview mode is selected, a preview image in which blur is predicted is displayed during the preview.

(Preview image rendering process)
Next, the preview image rendering process according to the second embodiment will be described with reference to FIG.

  In the preview image rendering process (S1404), an average image obtained by averaging images related to each frame of the processing target frame group is created as a preview image.

  The processing target frame group is a group composed of past frames for the number of processing target frames obtained in S1401 with the current time as a reference.

  For example, when the exposure time at the time of shooting is 1 second and the exposure time at the time of preview is 0.04 seconds, the frame from the current time or reference time to the 25th frame obtained in the past is the processing target frame group. Become.

  First, in step S1501, a variable n for designating a frame is initialized. Specifically, 0 is assigned to the variable n. Here, if the frame number of the current time or the reference time is N (N is the number of frames to be processed), (N−n) is the frame number of the frame to be processed.

  In step S1502, a preview image of (N−n) frames is created by CG. Note that any method may be used as a method for creating a preview image, and the ray tracing method described in the first embodiment may be used.

  In step S1503, the frame number is updated. Specifically, the variable n is updated by adding 1 to the value.

  In step S1504, it is determined whether or not the processing has been performed on all the frames related to the processing target frame group. If the processing has not been completed for all the frames, the process returns to step S1502 to create a preview image for the frame to be processed.

  In step S1505, a preview image is created by averaging the preview images of the frames created in step S1502.

  The preview image rendering process ends according to the above processing procedure.

  This process reproduces camera shake and subject blur by creating an image that averages the number of images corresponding to the difference in exposure time when the exposure time during shooting differs from the exposure time during preview. A predicted preview image can be created.

  By using the imaging device described above, camera shake and subject blur that may occur according to the shutter speed set by the user are predicted and displayed as a preview image, so that the difference between the preview image and the actual captured image is displayed. Can be reduced. Accordingly, the user of the imaging apparatus can set shooting conditions such as a shutter speed while confirming the occurrence of blurring in the preview image.

(Other examples)
In the second embodiment, a preview image of each frame is created based on the already captured frames. However, by calculating subject information of subjects in a plurality of frames and estimating subject movement, a preview image at a time when an image is not actually acquired may be created, and an average image may be created. . Furthermore, for example, the subject position and shape in a future frame may be predicted from several past frames, and this may be used.

  In the second embodiment, the method of creating an image of each frame by CG has been described. However, when it is not necessary to generate a shadow by strobe light or to reproduce luminance, it is not always necessary to create by CG. In that case, you may utilize the frame corresponding to the image acquired by the imaging part 105 arrange | positioned in the center among imaging parts instead of CG.

Claims (5)

Shooting condition setting means for setting shooting conditions for shooting by the imaging means;
Subject information calculation means for calculating subject information including the position and shape of the subject based on the image captured by the imaging means;
A preview image creation means for creating a preview image that predicts an image acquired when the imaging means is imaged according to the shooting conditions from the shooting conditions and the subject information;
An image pickup apparatus comprising: a display unit that displays the preview image. Comprising a light emitting means comprising one or more light emitting elements,
When shooting using the light emitting means is included in the shooting conditions, the preview image creating means includes information on the arrangement information, light distribution characteristics, and luminance of the light emitting means in addition to the shooting conditions and the subject information. The imaging apparatus according to claim 1, wherein the preview image is created from the included light emission information. The shooting conditions include a shutter speed,
When the exposure time calculated from the shutter speed is longer than the exposure time at the time of preview, the preview image creation means acquires an image group captured by the imaging means within the exposure time calculated from the shutter speed. The imaging apparatus according to claim 1, wherein the preview image is created by averaging the image group.   The imaging according to any one of claims 1 to 3, wherein the subject information calculation unit obtains the subject information using a multi-viewpoint image group captured by the imaging unit from a plurality of viewpoints. apparatus.   The imaging device according to claim 1, wherein the subject information calculation unit obtains the position of the subject using a distance sensor.

Images