WO2019188933A1

WO2019188933A1 - Imaging device, imaging method, and camera system

Info

Publication number: WO2019188933A1
Application number: PCT/JP2019/012410
Authority: WO
Inventors: 祐樹杉原; 小林　潤; 一樹石田; 真彦宮田
Original assignee: 富士フイルム株式会社
Priority date: 2018-03-29
Filing date: 2019-03-25
Publication date: 2019-10-03
Also published as: JP7015906B2; JPWO2019188933A1

Abstract

The purpose of the present invention is to provide an imaging device, camera system and imaging method by which images exhibiting a long exposure effect can be obtained with excellent image quality. If it has been determined that blown-out highlights are occurring in an acquired image, this imaging device sets an imaging unit to a first mode for intermittently acquiring multiple images from the imaging element, and generates a composite image by acquiring multiple images by intermittent exposure; meanwhile, if it is determined that blown-out highlights are not occurring, the imaging device sets the imaging unit to a second mode for continuous exposure to acquire an image. In the first mode, by setting the exposure time of intermittent image acquisition and the number of exposures, it is possible to obtain a composite image exhibiting the effect of a long exposure time while preventing the occurrence of blown-out highlights. Further, in the second mode, because an image is acquired with continuous exposure, it is possible to obtain an image which exhibits an effect corresponding to the exposure time.

Description

Imaging apparatus, imaging method, and camera system

The present invention relates to an image pickup apparatus, an image pickup method, and a camera system, and more particularly to an image pickup apparatus, an image pickup method, and a camera system that obtain an effect of shooting by long exposure.

In photography, a so-called “panning (shooting where the main subject appears to be stationary and the background appears to flow)” or shooting that emphasizes the movement trajectory of the main subject may be performed by long exposure. As a method for obtaining such a long-time exposure effect, in addition to a photographing method for performing long-time continuous exposure, a plurality of images obtained by a plurality of short-time exposures are combined to obtain a long-time exposure effect. A photographing method for acquiring an image to be played is known. These photographing methods can be switched according to photographing conditions. For example, Patent Document 1 describes switching between a first mode in which continuous exposure is performed and a second mode in which a plurality of images are acquired and combined according to a shutter speed (exposure time). In the first mode, image stabilization control (camera shake correction) is performed by driving the shift lens, and in the second mode, normal image stabilization control is performed.

JP-A-2015-195497

In order to enhance the effects (the degree of background flow, the degree of movement of the subject, the smoothness of the flow and / or movement, etc.) due to the long exposure described above, continuous exposure for a long time is often effective. On the other hand, if continuous exposure is performed for a long time, there is a risk that the exposure of the captured image may be inappropriate (too bright) depending on conditions such as the brightness of the subject. With respect to such a problem, in the above-described Patent Document 1, since continuous exposure is performed in the first mode when the exposure time is long, it is difficult to make the exposure of the captured image appropriate.

As described above, the conventional technique does not set an appropriate shooting mode according to the shooting conditions, and an image having an effect of long exposure cannot be obtained with good image quality.

The present invention has been made in view of such circumstances, and an object of the present invention is to provide an imaging device, an imaging method, and a camera system that can obtain an image having a long-time exposure effect with good image quality.

In order to achieve the above-described object, an imaging apparatus according to the first aspect of the present invention includes an imaging unit that acquires an image of a subject using an imaging element on which an optical image of the subject is formed by a shooting lens, and an imaging unit that acquires the image. A determination unit that determines whether or not whiteout occurs in an image to be performed, a first mode in which a plurality of images are intermittently acquired from the image sensor, and images are continuously acquired from the image sensor A mode setting unit that sets one of the second mode and the first mode when it is determined that whiteout occurs, and when it is determined that whiteout does not occur, the first mode is set. A mode setting unit that sets the mode of 2, a detection unit that detects movement of the subject in the image, and a plurality of images acquired in the first mode are aligned according to the movement of the subject detected by the detection unit, Multiple aligned It comprises an image generator which synthesizes the images to generate a composite image.

When it is determined that whiteout occurs in an acquired image, the imaging device according to the first aspect sets the first mode in which a plurality of images are intermittently acquired from the imaging element to the imaging unit and is intermittent When a plurality of images are obtained by exposure, a composite image is generated. On the other hand, if it is determined that whiteout does not occur, a second mode in which continuous exposure is performed is set in the imaging unit, and the image is displayed. get. In the first mode, when the determination unit determines that “overexposure occurs”, the occurrence of overexposure is prevented by setting the exposure time and the number of exposures for intermittent image acquisition in the first mode. Can do. In the first mode, the image generation unit aligns the plurality of images according to the movement of the subject (for example, aligns the main subject by movement and / or rotation) and combines them, thereby effecting long exposure. It is possible to obtain a composite image that exhibits (background flow, subject trajectory, etc.). When generating the composite image, the image generation unit may perform other image processing such as blurring processing to enhance the effect of long exposure. On the other hand, when the determination unit determines that “the whiteout does not occur”, since the image is acquired by the continuous exposure in the second mode, an image having an effect corresponding to the exposure time can be obtained. As described above, according to the first aspect, it is possible to obtain the effect of long-time exposure with good image quality that does not cause whiteout.

In the first aspect and each of the following aspects, “out-of-white” means a state in which the output of the image sensor is saturated and the luminance difference of the subject cannot be expressed at all or almost in an image (originally a portion having a difference in luminance) However, it can be said that all have the maximum luminance or a luminance close to the maximum luminance). In addition, “intermittently acquiring a plurality of images” includes a mode in which an image sensor is intermittently exposed and a plurality of images corresponding to individual exposures are acquired, and an image sensor is continuously exposed. , (By an electronic shutter or the like) the image may be read intermittently from the image sensor, and a plurality of images corresponding to individual reading may be acquired.

Note that the imaging apparatus according to the first aspect can be applied to the camera body of a lens-integrated camera, the body part of a lens interchangeable camera, the body part of a surveillance camera, and the like, but is not limited to these examples.

The imaging device according to a second aspect further includes an exposure control unit that controls exposure with a preset exposure time in the first aspect, and the determination unit includes the first exposure time set by the exposure control unit. If it is greater than or equal to the threshold value, it is determined that whiteout occurs, and the mode setting unit is caused to set the first mode. The second mode prescribes a specific example of the determination of overexposure, and the determination unit has a long exposure time (initial exposure time) set in advance (specifically, the initial exposure time ≧ the first exposure time). In the case of the threshold value), it is determined that whiteout occurs, and the mode setting unit sets the first mode based on the determination result. The first threshold value may be set according to the type of subject and / or the effect (content, degree) to be obtained. Further, the first threshold value may be set according to the user's designation, or may be set by the imaging apparatus regardless of the user's designation.

In the imaging device according to the third aspect, in the first or second aspect, the determination unit makes a determination at the start of exposure. In the third mode, when “exposure is started”, the determination is made when the operation for shifting to the long-time exposure mode is performed. However, the present invention is not limited to these modes. However, the present invention is not limited to these modes.

The imaging device according to a fourth aspect is any one of the first to third aspects, wherein the imaging unit determines the number of images and / or the number of pixels acquired in the first mode according to the exposure time of the image. To set. The fourth aspect is for reducing the load on the imaging device when acquiring, processing, and synthesizing a plurality of images. The imaging unit has an exposure time (initial exposure time) set to obtain a desired effect. ) Is long and the load is high, the number of images to be taken and combined and / or the number of pixels can be reduced. In addition to these measures, it is also preferable to set the exposure time (multiple exposure time) of each image acquired intermittently according to the initial exposure time.

In any one of the first to fourth aspects, the imaging device according to the fifth aspect further includes an information presentation unit that presents information that prompts a change in the shooting direction according to the detection result, and the detection unit includes: The moving direction of the subject in the image is detected. According to the fifth aspect, the user can easily perform the photographing direction changing operation (pan and / or tilt) with reference to the presented information. Information can be presented by screen display, audio output, or the like.

In any one of the first to fifth aspects, the imaging device according to a sixth aspect further includes a photographing direction changing unit that automatically changes the photographing direction following the movement of the subject, and the mode setting unit includes: When the focal length of the photographing lens when obtaining the image is longer than the second threshold value, the image is obtained while changing the photographing direction by the photographing direction changing unit. By automatically changing the shooting direction by the shooting direction changing unit, it is possible to reduce the burden of pan / tilt operation during exposure for the user. Since the effect (the degree of background flow, etc.) due to the change in the shooting direction becomes higher when the focal length is long, the shooting direction changing unit is used in the sixth mode (focal length> second threshold). I was going to do the shooting. Thereby, in the sixth aspect, the effect of the pan / tilt operation during exposure can be obtained efficiently. Note that examples of the configuration of the shooting direction changing unit include a mode of changing the direction of the lens and / or the image sensor, and a mode of changing the shooting direction by driving an optical member such as a mirror. It is not limited to.

In any one of the first to sixth aspects, the imaging device according to the seventh aspect notifies the user which mode the mode setting unit has set between the first mode and the second mode. And a notification unit. According to the seventh aspect, the user can recognize the set mode. Notification to the user can be performed by screen display, light emission, vibration, voice output, or the like.

The imaging device according to the eighth aspect further includes a storage unit that stores the composite image generated by the image generation unit in any one of the first to seventh aspects. The composite image stored in the storage unit may be output by screen display, communication, printing, or the like.

In order to achieve the above-described object, an imaging method according to a ninth aspect of the present invention provides imaging in an imaging apparatus including an imaging unit that acquires an image of a subject with an imaging element on which an optical image of the subject is formed by a photographing lens. A method for determining whether or not whiteout occurs in an image acquired by an imaging unit; and a first mode for intermittently acquiring a plurality of images from an imaging device for the imaging unit; A mode setting step for setting one of a second mode in which images are continuously acquired from the image sensor, and when it is determined that whiteout occurs, the first mode is set and whiteout If it is determined that the image does not occur, a mode setting step for setting the second mode, a detection step for detecting movement of the subject in the image, and a subject for which a plurality of images acquired in the first mode are detected in the detection step Aligned in accordance with the movement, having, an image generating step of generating a synthesized image by synthesizing the plurality of images aligned. According to the ninth aspect, it is possible to obtain an image having a long-time exposure effect with good image quality as in the first aspect.

The imaging method according to the ninth aspect may further have the same configuration as the second to eighth aspects. In addition, a program that causes an imaging apparatus and / or camera system to execute the imaging method of these aspects, and a non-transitory recording medium that records a computer-readable code of such a program can also be cited as aspects of the present invention.

To achieve the above-described object, a camera system according to a tenth aspect of the present invention includes an imaging device according to any one of the first to eighth aspects, and a photographing lens. Since the camera system according to the tenth aspect includes the imaging device according to any one of the first to eighth aspects, an image that exhibits the effect of long exposure can be obtained with good image quality. The camera system according to the tenth aspect can be applied to a lens integrated camera, a lens interchangeable camera, a surveillance camera, and the like, but is not limited to these specific examples.

As described above, according to the imaging apparatus, imaging method, and camera system of the present invention, an image that exhibits the effect of long exposure can be obtained with good image quality.

FIG. 1 is a diagram illustrating a configuration of a camera system according to the first embodiment. FIG. 2 is a diagram illustrating a functional configuration of the image processing unit. FIG. 3 is a diagram illustrating how the shooting direction is changed. FIG. 4 is another diagram showing a state of changing the shooting direction. FIG. 5 is an overall flowchart of the imaging method. FIG. 6 is a table showing the relationship between shooting conditions and shooting modes. FIG. 7 is an individual flowchart of the imaging method. FIG. 8 is a diagram illustrating a display example of information that prompts the user to change the shooting direction. FIG. 9 is a diagram illustrating how a composite image is generated. FIG. 10 is another diagram showing how a composite image is generated. FIG. 11 is still another view showing a state of generating a composite image. FIG. 12 is another flowchart of the imaging method. FIG. 13 is still another individual flowchart of the imaging method. FIG. 14 is still another individual flowchart of the imaging method. FIG. 15 is a diagram illustrating an example of the effect of multiple exposure and long exposure.

Hereinafter, an embodiment for carrying out an imaging apparatus, an imaging method, and a camera system according to the present invention will be described in detail with reference to the accompanying drawings.

<First Embodiment>
<Overall configuration of imaging device>
FIG. 1 is a diagram illustrating a configuration of a camera system 10 (an imaging apparatus, a camera system) according to the first embodiment. The camera system 10 includes an interchangeable lens 100 (imaging lens, imaging unit) and an imaging apparatus main body 200 (imaging apparatus), and forms a subject image (optical image) on the imaging element 210 by a photography lens including a zoom lens 110 described later. Let The interchangeable lens 100 and the imaging apparatus main body 200 can be attached and detached via a mount (not shown).

<Configuration of interchangeable lens>
The interchangeable lens 100 includes a zoom lens 110 (zoom lens, photographic lens), a focus lens 120 (photographic lens), a diaphragm 130, a lens driving unit 140, and a variable apex angle prism 150. The lens driving unit 140 performs zoom (optical zoom) adjustment and focus adjustment by driving the zoom lens 110 and the focus lens 120 forward and backward according to a command from the image processing device 240 (exposure control unit 240A in FIG. 2). The zoom adjustment and the focus adjustment may be performed according to a zoom operation and a focus operation performed by the user (a zoom ring not shown, a rotation of the focus ring, etc.) in addition to being performed according to a command from the image processing apparatus 240. Good. Further, the lens driving unit 140 controls the diaphragm 130 in accordance with a command from the image processing device 240 to adjust the exposure. On the other hand, information such as the positions of the zoom lens 110 and the focus lens 120 and the degree of opening of the diaphragm 130 are input to the image processing device 240. The interchangeable lens 100 has an optical axis L1.

<Configuration of imaging device body>
The imaging apparatus main body 200 includes an imaging element 210 (imaging unit), an AFE 220 (AFE: Analog Front End, imaging unit), an A / D converter 230 (A / D: Analog to Digital, imaging unit), and an image processing device 240. Is provided. The image sensor 210 includes a light receiving surface on which a large number of light receiving elements are arranged in a matrix. Then, the subject light that has passed through the variable apex angle prism 150, the zoom lens 110, the focus lens 120, and the stop 130 is imaged on the light receiving surface of the image sensor 210 and is converted into an electric signal by each light receiver. An R (red), G (green), or B (blue) color filter is provided on the light receiving surface of the image sensor 210, and a color image of the subject can be acquired based on the signals of each color. As the imaging device 210, various photoelectric conversion devices such as a complementary metal-oxide semiconductor (CMOS) and a charge-coupled device (CCD) can be used. The AFE 220 performs noise removal and amplification of the analog image signal output from the image sensor 210, and the A / D converter 230 converts the captured analog image signal into a digital image signal having a gradation width.

<Configuration of image processing apparatus>
FIG. 2 is a diagram illustrating a functional configuration of the image processing apparatus 240 (image processing apparatus). The image processing apparatus 240 includes an exposure control unit 240A (exposure control unit), a determination unit 240B (determination unit), a mode setting unit 240C (mode setting unit), a detection unit 240D (detection unit), and an image generation unit 240E (image generation unit). ), An information presentation unit 240F (information presentation unit), an imaging direction changing unit 240G (imaging direction changing unit), and an informing unit 240H (informing unit), and based on the digital image signal input from the A / D converter 230. Thus, processing such as generation of a composite image is performed. Details of the processing by the image processing apparatus 240 will be described later.

The function of the image processing apparatus 240 can be realized by using various processors. The various processors include, for example, a CPU (Central Processing Unit) that is a general-purpose processor that executes various types of functions by executing software (programs). The above-mentioned various processors include programmable logic devices that are processors whose circuit configuration can be changed after manufacture, such as GPU (Graphics Processing Unit) and FPGA (Field Programmable Gate Array) that are specialized for image processing. (Programmable Logic Device: PLD) is also included. Further, the above-mentioned various processors include dedicated electric circuits that are processors having a circuit configuration designed exclusively for executing specific processing such as ASIC (Application Specific Specific Integrated Circuit).

The functions of each unit may be realized by a single processor, or may be realized by a plurality of processors of the same or different types (for example, a plurality of FPGAs, a combination of CPU and FPGA, or a combination of CPU and GPU). A plurality of functions may be realized by one processor. As an example of configuring a plurality of functions with one processor, first, as represented by a computer such as an image processing apparatus main body and a server, one processor is configured with a combination of one or more CPUs and software. There is a form in which this processor is realized as a plurality of functions. Second, as represented by a system-on-chip (SoC), there is a form of using a processor that realizes the functions of the entire system with a single integrated circuit (IC) chip. Thus, various functions are configured using one or more of the various processors described above as a hardware structure. Further, the hardware structure of these various processors is more specifically an electric circuit (circuitry) in which circuit elements such as semiconductor elements are combined.

When the processor or electrical circuit described above executes software (program), a processor (computer) readable code of the software to be executed is stored in a non-temporary recording medium such as a ROM (Read Only Memory). Refers to the software. The software stored in the non-temporary recording medium includes a program for executing image input, zoom processing, composition processing, and the like. The code may be recorded on a non-temporary recording medium such as various magneto-optical recording devices and semiconductor memories instead of the ROM. In processing using software, for example, RAM (Random Access Memory) is used as a temporary storage area, and for example, data stored in an EEPROM (Electronically Erasable Memory and Programmable Read Only Memory) (not shown) may be referred to. it can.

The image processing apparatus 240 includes a ROM 242 (ROM: Read Only Memory, non-temporary recording medium) in addition to the above-described units. The ROM 242 stores computer-readable codes of programs (including a program for executing the imaging method according to the present invention) necessary for image input, zoom processing, composition processing, and the like.

<Operation unit>
The operation unit 250 includes a release button (not shown), operation buttons (for example, a cross button, a Quick button, an OK button, etc.), a dial, a switch, and the like. Various operations such as setting contents and / or degree can be performed. The monitor 270 may be configured as a touch panel and used as the operation unit 250.

<Storage unit>
The storage unit 260 includes various magneto-optical recording media, non-temporary recording media such as a semiconductor memory, and a control circuit thereof, and stores captured images, synthesized images, and the like. As the recording medium, a type that can be attached to and detached from the imaging apparatus main body 200 can be used.

<Monitor and finder>
The monitor 270 (display device) is composed of, for example, a liquid crystal display panel, and can display a live view image, a captured image, a composite image, shooting assistance information, and the like. The monitor 270 can be disposed on the back side, the top side, or the like of the imaging apparatus main body 200. Similarly, the finder 280 is also composed of, for example, a liquid crystal display panel, a prism, a lens, and the like, and the user can visually recognize a live view image, a captured image, a composite image, shooting assistance information, and the like via an eyepiece unit (not shown). . As the finder 280, an “optical view finder (OVF)”, an “electronic view finder (EVF)”, or a combination thereof, a “hybrid view finder (HVF)” is shown. Can be used.

<Polarization movable part>
The variable apex angle prism 150 (photographing direction polarization unit) is configured by enclosing a liquid with a high refractive index between two transparent plate glasses arranged in the front and rear in the optical axis direction. Is attached to the side). In response to a command from the shooting direction changing unit 240G, the lens driving unit 140 extends and contracts the bellows-like expansion / contraction unit 152 provided on the side surface to change the inclination of the liquid, thereby changing the refraction angle of the subject light (ie, the shooting direction) Can be made. 3A shows a state in which the optical axis L1 has not changed, and FIG. 3B shows a state in which the optical axis has changed to the optical axis L2 in the downward direction due to the expansion / contraction of the expansion / contraction part. Yes. In addition, the direction of an optical axis can be changed to 2 axis | shafts by providing the location (for example, 3 places, 4 places, etc.) where the expansion-contraction part 152 is expanded-contracted. Note that an optical element such as the variable apex angle prism 150 may be provided not at the distal end portion of the interchangeable lens 100 but at the proximal end portion.

¡Changing the shooting direction can also be done by driving a mirror instead of a variable apex angle prism. For example, two mirrors 160 are arranged facing each other as shown in FIG. 4A, and the directions of these mirrors 160 are changed as shown in FIG. 4B (command from the imaging direction changing unit 240G). (The lens driving unit 140 is driven), the direction of the optical axis (that is, the photographing direction) can be changed (part (b) of FIG. 4 shows the optical axis L3 changed downward in the figure). ). The direction of the optical axis can be changed around the two axes by rotating the mirror 160 around the two axes.

Photographing by changing the orientation of the image sensor with a device such as a piezoelectric element instead of, or in addition to, providing the interchangeable lens 100 with members and / or mechanisms such as the variable apex angle prism 150 and the mirror 160 described above. The direction may be changed.

<Processing of imaging method>
Processing of the imaging method in the camera system 10 having the above-described configuration will be described. FIG. 5 is a flowchart illustrating the imaging method according to the first embodiment.

<Determining if whiteout occurs>
When the long exposure mode is started by an operation on a button, switch, etc. (not shown) of the operation unit 250, the determination unit 240B has a set exposure time (hereinafter referred to as “initial exposure time”) as a first threshold. It is determined whether the value is equal to or greater than the value (step S100: determination step). The exposure control unit 240A may set the initial exposure time according to the user's operation, or may set it without depending on the user's operation. The exposure control unit 240A can set the initial exposure time according to the content of the effect to be obtained (for example, the background and the subject appear to flow) and the degree thereof.

If the initial exposure time is greater than or equal to the first threshold value, the determination unit 240B determines that “whiteout occurs” and proceeds to step S110. In this case, as will be described in detail later, a first mode (a mode for shooting by multiple exposure) is set. On the other hand, when the initial exposure time is less than the first threshold value, the determination unit 240B determines that “no whiteout occurs” and proceeds to step S120. In this case, as will be described in detail later, a second mode (a mode in which shooting is performed by continuous exposure) is set. Note that “out-of-white” means “a state in which the output of the image sensor 210 is saturated and the luminance difference of the subject cannot be represented at all or almost in the image (originally the portion where the luminance is different is the highest luminance or the highest luminance). In a state where the luminance is close to).

The first threshold value can be about several tens of seconds (for example, 60 seconds), but is not limited to this value. In addition, since the exposure state of the image to be obtained may vary depending on the user, the first threshold value is changed according to conditions such as the content and / or degree of the effect to be obtained and the type of subject. Also good. By setting the first threshold value as described above, whiteout can be prevented and an image with good image quality can be obtained.

<Judgment timing for occurrence of overexposure>
The determination of whether or not whiteout occurs in step S100 (specifically, determination whether or not the exposure time is equal to or greater than the first threshold value) is made at the start of exposure. In the case of “Perform at the start of exposure”, for example, when judging when the operation for shifting to the long exposure mode is performed, when judging according to the operation for instructing the mode setting after the transition, the long exposure mode Judgment is made when AE conditions (AE: Automatic Exposure, automatic exposure control) and AF conditions (AF: Automatic Focus, automatic focus control) are determined in response to a shooting preparation instruction by operating a shutter button (not shown) after shifting to However, the present invention is not limited to these modes, but may include a case in which a determination is made before actual exposure is started after a shooting instruction is given by operating a shutter button (not shown).

<Judgment based on focal length>
In the first embodiment, the mode is set in consideration of the focal length of the interchangeable lens 100 in addition to determining whether or not whiteout occurs based on the exposure time. Specifically, when it is determined that “whiteout occurs” (YES in step S100), mode setting unit 240C determines whether or not the focal length of interchangeable lens 100 is equal to or less than the second threshold value. If the determination is affirmative (YES in step S110), the process proceeds to step S200 to set the first mode (1). If the determination in step S110 is negative, the process proceeds to step S300 to set the first mode (2).

Specifically, when the determination in step S110 is affirmative (focal length ≦ second threshold value), “first mode (1)” is set in which shooting is performed by multiple exposure and the shooting direction is not automatically changed. (Step S200: Mode setting step). On the other hand, if the determination in step S110 is negative (focal length> second threshold value), the amount of movement of the subject by driving the variable apex angle prism 150 is large and effective shooting is possible. The unit 240G drives the variable apex angle prism 150 via the lens driving unit 140, and sets “first mode (2)” in which shooting is performed with multiple exposure while changing the shooting direction (step S300: mode setting). Process).

The threshold for the focal length (the above-mentioned “second threshold”) is, for example, 35 mm or more and 50 mm when the size of the image sensor is converted to “35 mm full size” (so-called “35 mm conversion”). However, the value is not limited to this value.

Even when it is determined that “no whiteout occurs” (NO in step S100) and the second mode is set, the mode setting unit 240C determines whether or not the focal length of the interchangeable lens 100 is equal to or less than the second threshold value. Is determined (step S120). When the determination is affirmed, the process proceeds to step S400, and the mode setting unit 240C sets the second mode (1) for photographing by long-time exposure (continuous exposure). On the other hand, if the determination in step S110 is negative, the process proceeds to step S500, and the mode setting unit 240C causes the shooting direction changing unit 240G to drive the variable apex angle prism 150 via the lens driving unit 140 and change the shooting direction. A second mode (2) is set for performing photographing with long exposure (continuous exposure).

Fig. 6 shows a table summarizing the mode settings according to the exposure time and focal length threshold values.

<Processing in each mode>
Shooting and image processing are performed according to the mode set in steps S200, S300, S400, and S500 (in FIG. 5, the whole is collectively described as “step S600”). Hereinafter, specific processing in each mode in step S600 will be described.

<Processing in First Mode (1)>
FIG. 7 shows a process when the first mode (1) is set in step S200 of FIG. When the first mode (1) is set, the notification unit 240H notifies the user that the first mode (1) has been set by displaying on the monitor 270 and / or the viewfinder 280 (step S210: notification). Process). You may alert | report by the audio | voice output by the speaker which is not shown in figure.

In the first mode (1), it is determined in step S100 that “the exposure time (initial exposure time) is equal to or greater than the first threshold value, so that whiteout occurs”. Therefore, in the first mode (1), by performing photographing (intermittent image acquisition, multiple exposure) with an exposure time shorter than the “initial exposure time” (hereinafter referred to as “multiple exposure time”) a plurality of times, The effect of long exposure is obtained while preventing whiteout. For example, when it is determined that “whiteout occurs” at an initial exposure time of 10 seconds, it is conceivable that imaging with a multiple exposure time of 0.5 seconds is repeated 10 times within the initial exposure time. If the number of multiple exposures is increased, the same effect (smooth background flow, subject trajectory, etc.) as long-time exposure (continuous exposure) can be obtained. (Combining, image processing, composition, etc.) increases the load on the system. Therefore, the exposure control unit 240A (imaging unit) takes into consideration the load on the system in addition to the effect to be obtained (corresponding to the initial exposure time), the number of images acquired (number of frames), the multiple exposure time, and the number of pixels. Is set (step S220: exposure time setting step), and the image sensor 210 (image pickup unit) is controlled under the set conditions to perform exposure for the i-th frame (step S230: shooting step).

The detection unit 240D detects the subject and its movement from the captured image (step S240: detection step), and the information presentation unit 240F presents information that prompts the user to change the shooting direction according to the detection result (step S250: information presentation). Process). For example, as shown in FIG. 8, the information presentation unit 240 </ b> F can perform information presentation by displaying a symbol indicating a change in the photographing direction (pan and / or tilt direction) on the display area 1000 of the finder 280. . In the example of FIG. 8, the arrow 1020 facing left is displayed among the eight

arrows

1010 and 1020 in total (the arrow that is actually displayed is shown by a solid line, and the other is shown by a dotted line). Such display may be performed on the monitor 270 instead of the finder 280. At the time of display, the length of the arrow may be changed according to the change amount of the shooting direction. Further, instead of the display or in addition to the display, sound may be output by a speaker (not shown) (for example, “pan left”). By presenting such information, the user can easily change the shooting direction, and can easily shoot an image having the effect of long exposure. In step S240, the detection unit 240D can detect the moving direction and the moving amount of the subject by, for example, calculating the motion vector by comparing the positions of the main subjects between images with different shooting timings.

The exposure control unit 240A, the detection unit 240D, and the information presentation unit 240F perform the control from step S230 to S250 (control for acquiring a plurality of images intermittently from the image sensor 210; multiple exposure) for the number of shots N (step S260). Until YES, and the process proceeds to step S270.

FIG. 9 shows an example of an image (photographed image) obtained by each exposure (photographing) in step S230 described above. In FIG. 9, captured

images

1031, 1032, and 1033 are images that have different shooting ranges due to movement of

subjects

901, 902, and 903 (main subjects), and the image generation unit 240 E combines these captured

images

1031, 1032, and 1033. An image is generated (step S270: image generation step). An example of generating a composite image in step S270 will be described below.

<Synthetic Image Generation Example 1>
The image generation unit 240E moves, rotates, enlarges, or reduces each image to match the

subjects

901, 902, and 903, and synthesizes a plurality of aligned images to generate a temporary composite image 1100. As shown in FIG. 9, the temporary composite image 1100 is a panoramic composite image. Further, the image generation unit 240E cuts out a plurality of images with different cutout ranges from the temporary composite image 1100, such as the images 1101 to 1105 in FIG. By changing the clipping range, the effect of moving the subject can be given to the composite image. The image generation unit 240E performs image processing such as blurring, brightness and / or saturation change, enlargement, reduction, and deformation on the cut-out image. At this time, the brightness of the individual images may be adjusted so that the final synthesized image has the same brightness as that of the long-time exposure (continuous exposure). The content and degree of image processing may be set according to user designation, or may be set without user designation. The image generation unit 240E generates a final composite image (composite image) by combining the images that have undergone image processing. In the example of FIG. 10, images 1101 to 1105 cut out from the temporary composite image 1100 and these images are made translucent and superimposed to generate a final composite image 1200 (composite image). By these processes, it is possible to generate an image having a desired effect (the content of the effect such as the background flow, the locus of the subject, and the degree thereof). In addition, the image generation unit 240E stores the final composite image 1200 in the storage unit 260 (image storage process) and displays it on the monitor 270 (image display process). You may memorize | store and / or display a picked-up image, a temporary composite image, etc.

<Synthetic Image Generation Example 2>
In the generation example 2, the image generation unit 240E shifts the

subjects

901, 902, and 903 little by little when aligning the captured image. The amount to be shifted can be set according to the content and degree of the effect to be played (flow, movement, etc.). In addition, the image generation unit 240E performs image processing such as blurring, brightness and / or saturation change, enlargement, reduction, and deformation on each captured image. The content and degree of image processing may be set according to user designation, or may be set without user designation. The image generation unit 240E generates a final composite image (composite image) by combining the images that have undergone image processing. By these processes, for example, a final composite image 1300 in FIG. 11 or a final composite image 1301 can be obtained that emphasizes the effect of moving the subject from the right to the left of the image. The generated final composite image is stored in the storage unit 260 (image storage process) and displayed on the monitor 270 (image display process). The captured image may be stored and / or displayed.

<Processing in the first mode (2)>
FIG. 12 shows a process when the first mode (2) is set in step S300 of FIG. When the first mode (2) is set, the notification unit 240H notifies the user that the first mode (2) has been set as in the case of the first mode (1) (step S310: Notification process). In the same way as in the first mode (1), the exposure control unit 240A (imaging unit) considers the load on the system in addition to the effect to be obtained, and the number of images to be acquired (the number of frames), multiple exposure The time and the number of pixels are set according to the initial exposure time (step S320: exposure time setting step), and the image sensor 210 (image pickup unit) is controlled to perform exposure for the i-th frame (step S330: Shooting process).

The first mode (2) is common to the first mode (1) in that multiple exposure is performed (a plurality of images are intermittently acquired from the image sensor 210), but the focal length is as described above. This is a mode for a long case (focal length> second threshold value). Therefore, the shooting direction changing unit 240G drives the variable apex angle prism 150 via the lens driving unit 140 in accordance with the movement of the subject detected by the detection unit 240D, and automatically changes the shooting direction (step S350). : Shooting direction changing step) Shooting with multiple exposure (step S330) is performed. The exposure controller 240A repeats the multiple exposure shooting until the number of shots is N and the determination in step S360 is affirmative. In the first mode (2), effective shooting (shooting with a long exposure effect such as a moving amount of a subject) can be performed by driving the variable apex angle prism 150 by such shooting.

In the process of the first mode (2), the number of images to be acquired, the multiple exposure time, and the number of pixels are set (step S320: exposure time setting step), the subject and its movement are detected (step S340: detection step). ) Can be performed in the same manner as in the first mode (1) described above. Further, a process of generating, storing, and displaying a composite image (final image) from the captured image obtained by each exposure in step S330 (an image generation process in step S370, an image storage process in step S380, and an image display process). Can be performed in the same manner as in the first mode (1) described above. Through these processes, it is possible to obtain a composite image (final composite image) that exhibits desired effects (contents of the effect such as background flow, subject trajectory and the degree thereof) also in the first mode (2).

<Processing in Second Mode (1)>
FIG. 13 shows a process when the second mode (1) is set in step S400 of FIG. When the second mode (1) is set, the notification unit 240H notifies the user that the second mode (1) has been set, as in the first mode (step S410: notification step). . The exposure control unit 240A (imaging unit) controls the image sensor 210 (imaging unit) to start exposure (step S420: imaging process), and the detection unit 240D detects the subject and its moving direction at a specified frame rate. (Step S430: detection step). Then, the information presentation unit 240F presents information that prompts the user to change the shooting direction according to the detection result, as in the first mode (see step S440: information presentation step, see FIG. 8 and the like). Thus, the user can easily change the shooting direction, and can easily take an image having the effect of long exposure.

When the designated exposure time has elapsed and the exposure is completed (YES in step S450), the exposure control unit 240A stores the photographed image in the storage unit 260 (step S460) and displays it on the monitor 270 (step S470). In the second mode (1), since the exposure is performed for a long time (images are continuously acquired from the image sensor 210), an effect corresponding to the exposure time (initial exposure time) can be obtained. Note that the same processing (image cutout, processing, and composition) as described above with reference to FIG. 10 may be performed on the captured image to further enhance the effect of long exposure.

<Processing in Second Mode (2)>
FIG. 14 shows processing when the second mode (2) is set in step S500 of FIG. When the second mode (2) is set, the notification unit 240H notifies the user that the second mode (2) has been set as in the second mode (1) (step S510: notification). Step), the exposure control unit 240A (imaging unit) controls the image sensor 210 (imaging unit) to start exposure (step S520: imaging step). The second mode (2) is common to the second mode (1) in that continuous exposure is performed (images are continuously acquired from the image sensor 210), but the focal length is as described above. This is a mode for a long case (focal length> second threshold value). Therefore, the imaging direction changing unit 240G drives the variable apex angle prism 150 via the lens driving unit 140 in accordance with the movement of the subject detected by the detection unit 240D (step S530: detection process), and the imaging direction automatically. (Step S540: Shooting direction changing step). Shooting with long exposure is repeated until the designated exposure time (initial exposure time) has elapsed and the determination in step S550 is affirmed. In the second mode (2), it is possible to perform effective shooting with a large amount of movement of the subject by driving the variable apex angle prism 150 (shooting with a long-time exposure effect). When the designated exposure time elapses and exposure ends (YES in step S550), exposure control unit 240A stores the captured image in storage unit 260 (step S560) and displays it on monitor 270 (step S570). Processing similar to that described above with reference to FIG. 10 (image clipping, processing, and composition) may be performed on the captured image to further enhance the effect of long exposure.

<Effects of multiple exposure and long exposure>
FIG. 15 is a diagram showing an example of the effects of multiple exposure and long exposure. An image 2000 shows an example of an image by multiple exposure, and the locus of the subject (black portion in the figure) is discontinuously (stepwise) thinned. On the other hand, an image 2001 shows an example of an image by long-time exposure (continuous exposure), and the locus of the subject (gray portion in the figure) is continuously thinned. The discontinuous change in the case of multiple exposure as shown in the image 2000 becomes continuous when the number of multiple exposures (the number of frames) is increased. However, as described above with respect to steps S220 and S320, the effect to be obtained is obtained. In addition, it is preferable to set the number of multiple exposures, the multiple exposure time, and the number of pixels in consideration of the load on the system. Since the exposure time of the image 2001 is longer than the total exposure time (multiple exposure time) of the image 2000, the image 2001 is brighter. However, the brightness of each image or the synthesized image is obtained by image processing. By adjusting, the brightness of the image 2000 can be the same as that of the image 2001.

<Others>
Although the embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments, and various modifications can be made without departing from the spirit of the present invention. For example, the camera system 10 in the first embodiment can be realized by a digital camera, a smartphone, a tablet terminal, or the like.

DESCRIPTION OF SYMBOLS 10 Camera system 100 Interchangeable lens 110 Zoom lens 120 Focus lens 130 Diaphragm 140 Lens drive part 150 Variable apex angle prism 152 Expansion / contraction part 160 Mirror 200 Imaging device main body 210 Imaging element 220 AFE
230 A / D converter 240 Image processing device 240A Exposure control unit 240B Judgment unit 240C Mode setting unit 240D Detection unit 240E Image generation unit 240F Information presentation unit 240G Shooting direction change unit 240H Notification unit 242 ROM
250 Operation unit 260 Storage unit 270 Monitor 280 Viewfinder 901 Subject 902 Subject 903 Subject 1000 Display area 1010 Arrow 1020 Arrow 1031 Captured image 1032 Captured image 1033 Captured image 1100 Temporary composite image 1101 Image 1102 Image 1103 Image 1104 Image 1105 Image 1200 Final composite image 1300 Final composite image 1301 Final composite image 2000 Image 2001 Image L1 Optical axis L2 Optical axis L3 Optical axis N Number of shots S100 to S600 Each step of the imaging method

Claims

An imaging unit that acquires an image of the subject by an imaging element on which an optical image of the subject is formed by a photographing lens;
A determination unit that determines whether or not whiteout occurs in the image acquired by the imaging unit;
One of a first mode in which a plurality of images are intermittently acquired from the image sensor and a second mode in which the images are continuously acquired from the image sensor are set for the imaging unit. A mode setting unit that sets the first mode when it is determined that the whiteout occurs, and sets the second mode when it is determined that the whiteout does not occur When,
A detection unit for detecting movement of a subject in the image;
The plurality of images acquired in the first mode are aligned according to the movement of the subject detected by the detection unit, and the plurality of the aligned images are combined to generate a combined image. An image generator;
An imaging apparatus comprising:
An exposure control unit that controls exposure with a preset exposure time;
The determination unit determines that the whiteout occurs when the exposure time set by the exposure control unit is equal to or greater than a first threshold value, and causes the mode setting unit to set the first mode. The imaging device according to claim 1.
3. The imaging apparatus according to claim 1, wherein the determination unit performs the determination at the start of exposure.
The imaging apparatus according to any one of claims 1 to 3, wherein the imaging unit sets the number of images and / or the number of pixels acquired in the first mode according to an exposure time of the image.
Further comprising an information presentation unit for presenting information for prompting a change in the shooting direction according to the detection result;
The imaging device according to claim 1, wherein the detection unit detects a moving direction of the subject in the image.
A shooting direction changing unit that automatically changes the shooting direction following the movement of the subject;
The mode setting unit acquires the image while changing the shooting direction by the shooting direction changing unit when a focal length of the shooting lens when acquiring the image is longer than a second threshold. The imaging device according to any one of 1 to 5.
The imaging according to any one of claims 1 to 6, further comprising an informing unit that informs a user which mode of the first mode and the second mode is set by the mode setting unit. apparatus.
The imaging apparatus according to any one of claims 1 to 7, further comprising a storage unit that stores the composite image generated by the image generation unit.
An imaging method in an imaging apparatus including an imaging unit that acquires an image of the subject by an imaging element on which an optical image of the subject is formed by a photographing lens,
A determination step of determining whether or not whiteout occurs in the image acquired by the imaging unit;
One of a first mode in which a plurality of images are intermittently acquired from the image sensor and a second mode in which the images are continuously acquired from the image sensor are set for the imaging unit. A mode setting step in which the first mode is set when it is determined that the whiteout occurs, and the second mode is set when it is determined that the whiteout does not occur. When,
A detection step of detecting movement of the subject in the image;
The plurality of images acquired in the first mode are aligned according to the movement of the subject detected in the detection step, and the plurality of the aligned images are combined to generate a combined image. An image generation process;
An imaging method comprising:
A camera system comprising: the imaging device according to any one of claims 1 to 8; and the photographing lens.