JP2018067846A - Image processing apparatus, image processing method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a composite image with a finish desired by a user.SOLUTION: A RAM (103) holds a plurality of input images sequentially input from an imaging unit (105). An image processing unit (107) successively performs a combining process using the input images in the order of input to generate a composite image. A system control unit (101) regards a composite image completed by the image processing unit (107) as an output image by the timing when an end instruction is input from a user.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an image processing apparatus that processes and outputs an input image, an image processing method, and a program.

  There is an expression technique that makes it possible to acquire a trajectory image of a moving object such as a light of a running car or a light of fireworks by performing shooting with a longer exposure time than before. Hereinafter, shooting with a long exposure time is referred to as long-second shooting, and an image obtained by long-second shooting is referred to as a long-second image. However, in the case of this technique, the user (photographer) can determine how long exposure time can be taken to obtain a long-second image with a desired trajectory of the moving object before performing actual shooting. This requires some experience. Further, even an experienced user cannot always take a desired long-second image.

  Therefore, in recent years, by synthesizing a plurality of images obtained by continuously performing shooting with a short exposure time at a short time interval, an image corresponding to a long second image by long second shooting is generated. A technique for sequentially displaying the generation process on a live view screen of a camera has been proposed. In the following, shooting with a short exposure time is referred to as short-second shooting, images obtained by short-second shooting are referred to as short-second images, and multiple short-second images taken at short time intervals are synthesized. The generated image (an image corresponding to a long second image) is referred to as a pseudo long second image. By sequentially displaying the pseudo-long-second image generated in this way on the live view screen, the user can obtain a desired image (pseudo-long-second image) while confirming the finish of the trajectory image in real time. it can.

  Patent Document 1 discloses an electronic camera that generates a composite image by combining a plurality of frames of captured images captured by an imaging unit. Moreover, the electronic camera described in Patent Document 1 is configured to determine whether to stop imaging by the imaging unit based on an instruction from a user (photographer).

JP 2006-332934 A

  Here, the electronic camera described in Patent Document 1 stops imaging by the imaging unit when an instruction to end shooting is given from the user, and synthesizes the captured images obtained until the imaging is stopped. Thus, a composite image (that is, a pseudo long-second image) is generated. On the other hand, the user inputs an instruction to end shooting when an image with a desired finish is displayed while checking the composite image sequentially displayed on the monitor screen of the electronic camera. However, the combined result that is sequentially displayed on the monitor screen with respect to the captured image captured by the image capturing unit has a delay of at least the combined processing time. Therefore, the composition generated from the combination result displayed on the monitor screen when the user inputs the photographing end instruction and the captured image acquired until the photographing is stopped by the photographing end instruction from the user. It does not match the image (pseudo long-second image). That is, the composite image (pseudo long-second image) generated from the captured image becomes an image having a different finish from the desired pseudo-long-second image confirmed on the monitor screen by the user.

  Therefore, an object of the present invention is to obtain a composite image having a finish desired by a user.

  The present invention is an acquisition unit that acquires a plurality of input images input in order, a combining unit that sequentially generates a composite image by using the input images in the input order, and ends from the user Detection means for detecting timing at which an instruction is input, and an output image is a combined image in which the combining processing by the combining means has been completed until detection of timing at which an end instruction is input from a user It is characterized by that.

  According to the present invention, it is possible to obtain a composite image having a finish desired by a user.

It is a figure which shows schematic structure of the imaging device of this embodiment. It is a flowchart of the operation | movement at the time of pseudo | simulation long second mode of 1st Embodiment. It is a figure for demonstrating comparatively bright composition. It is a timing diagram of the process in 1st Embodiment. It is a flowchart of the operation | movement at the time of pseudo | simulation long second mode of 2nd Embodiment. It is a timing diagram of the process in 2nd Embodiment. It is a figure explaining the synthetic | combination process in 2nd Embodiment.

  Hereinafter, an embodiment of the present invention will be described in detail with reference to the accompanying drawings. The image processing apparatus of the present embodiment can be applied to digital cameras, digital video cameras, various mobile terminals such as smartphones and tablet terminals having camera functions, industrial cameras, in-vehicle cameras, medical cameras, and the like. In the present embodiment, an imaging apparatus such as a digital camera will be described as an application example of the image processing apparatus. In addition, the imaging apparatus according to the present embodiment sequentially generates images corresponding to images shot with a long exposure time from a plurality of images obtained by continuously shooting with a short exposure time at short time intervals. It has a function that can be displayed sequentially. In the present embodiment, as described above, shooting with a long exposure time is referred to as long-second shooting, and an image obtained by long-second shooting is referred to as a long-second image. It is also generated by combining short-time shooting for short exposure time, short-second images for short-time shooting, and multiple short-second images for continuous shooting at short time intervals. An image corresponding to the long second exposure image is referred to as a pseudo long second image.

<Schematic Configuration of Imaging Device of this Embodiment>
FIG. 1 is a block diagram showing a schematic configuration of an imaging apparatus 100 as an embodiment of an image processing apparatus of the present invention. In the imaging apparatus 100 illustrated in FIG. 1, each component (each block) of the imaging apparatus 100 is connected via a bus, for example.
The system control unit 101 is, for example, a CPU, and reads out a control program for each block included in the imaging apparatus 100 from a ROM 102 described later, develops it in a RAM 103 described later, and executes it to control the operation of each block of the imaging apparatus 100. . The ROM 102 is a rewritable nonvolatile memory, and stores parameters necessary for the operation of each block in addition to the control program for each block of the imaging apparatus 100. The RAM 103 is a rewritable volatile memory, and is used as a temporary storage area for data generated by the operation of each block of the imaging apparatus 100.

  The optical system 104 includes a lens group including a zoom lens and a focus lens, a diaphragm, a shutter, and the like. The optical system 104 forms an optical image of a subject or the like on the imaging surface of the imaging unit 105. The imaging unit 105 is an imaging element such as a CCD or a CMOS sensor, for example. The imaging unit 105 photoelectrically converts an optical image formed on the imaging surface by the optical system 104 and sequentially converts the obtained analog image signals into the A / D conversion unit 106. Output to. The A / D converter 106 converts the input analog image signal into digital image data, and outputs the obtained digital image data to the RAM 103. The RAM 103 temporarily stores the image data supplied from the A / D conversion unit 106.

  The image processing unit 107 performs various image processing such as white balance adjustment, color interpolation, gamma processing, and electronic zoom processing on the image data temporarily stored in the RAM 103. In the case of this embodiment, the image processing unit 107 also performs image processing for generating a pseudo long-second image. Detailed description of image processing such as pseudo long-second image generation will be described later. Note that each image processing in the image processing unit 107 may be realized by a CPU or the like executing a program.

  The display unit 108 is a display device such as a liquid crystal display (LCD). On the screen of the display unit 108, an image recorded in the RAM 103 or a recording unit 109, which will be described later, an operation user interface image for receiving an instruction from the user, and the like are displayed. The display unit 108 also displays a live view image captured by the imaging unit 105. Although details will be described later, in the present embodiment, a pseudo long-second image sequentially generated from a plurality of short-second images obtained by short-second shooting continuously performed at short time intervals is used as a display image, and is sequentially displayed on the display unit 108. Image display such as update display is also possible.

  The operation unit 110 includes, for example, a so-called touch sensor provided on the screen of the display unit 108, various buttons provided on the housing of the imaging apparatus 100, a shutter button that is mainly operated when taking an image, and the like. It is a user interface part. The shutter button is a button that can detect a two-step pressing operation of so-called half pressing and full pressing. When the shutter button is half-pressed, for example, focus or exposure is fixed, and when the shutter button is fully pressed, actual shooting is performed. The shutter button is also used in the pseudo long-second image photographing start instruction and photographing end instruction in the present embodiment. Note that the shutter button may be provided on a remote controller or the like that can communicate with the imaging device 100 by wire or wirelessly.

<First Embodiment>
Hereinafter, the processing mode of the first embodiment in the imaging apparatus 100 of FIG. 1 will be described. The imaging apparatus 100 according to the first embodiment can be operated in a processing mode in which pseudo long-second images sequentially generated from a plurality of short-second images obtained by short-time shooting continuously performed at short time intervals are sequentially updated and displayed. Yes. Hereinafter, this processing mode is referred to as a pseudo long second mode.

  FIG. 2 is a flowchart illustrating the operation of the imaging apparatus 100 according to the first embodiment in the pseudo long second mode. In the processing of each step shown in the flowchart of FIG. 2, the system control unit 101 controls each unit of the imaging apparatus 100 based on a program stored in the ROM 102, and controls input and output to the image processing unit 107. It is realized by. The same applies to other flowcharts described later. In the present embodiment, the imaging timing of the short second image in the imaging unit 105 and the display update timing of the image displayed on the display unit 108 are controlled independently. In the following, the flowchart of FIG. 2 will be described by dividing into processing at the time of taking a short second image consisting of step S201 and step S202, and processing at the time of image display and recording consisting of step S204 and subsequent steps.

First, the processing at the time of taking a short second image from step S201 to step S203 in FIG. 2 will be described.
The process of the flowchart in FIG. 2 starts when the imaging apparatus 100 according to the present embodiment shifts to the pseudo long-second mode, for example, by a mode instruction from a user (photographer) via the operation unit 110. After the transition to the pseudo long second mode, when a shooting start instruction is input from the user via the operation unit 110, the system control unit 101 sets shooting parameters such as the exposure time and shooting interval of the short second shooting. In step S201, the system control unit 101 controls the imaging unit 105 to capture an image according to the imaging parameter, that is, capture a short second image. In the present embodiment, it is assumed that an instruction to start shooting is made, for example, when the shutter button is fully pressed. After step S201, the system control unit 101 advances the process to step S202.

  In step S202, the system control unit 101 causes the memory (RAM 103) to sequentially store the short-second image data sequentially captured by the imaging unit 105 in step S201 as input image data. In parallel with the processing in step S202, the system control unit 101 determines whether or not a shooting end instruction from the user is detected from the operation unit 110 in step S203. In the present embodiment, the shooting end instruction is performed when the shutter button used for the above-described shooting start instruction is fully pressed again by the user. If the system control unit 101 determines in step S203 that the shooting end instruction has not been detected (No), the system control unit 101 returns the processing to step S201 and causes the imaging unit 105 to continue shooting the short-second image. That is, the processing from step S201 to step S203 is processing that is repeatedly performed until a photographing end instruction is detected. Therefore, in the image capturing apparatus 100, a short-second image is captured and stored in the memory until the user gives an instruction to end the capturing. On the other hand, if it is determined in step S203 that a shooting end instruction has been detected (Yes), the system control unit 101 advances the process to step S204.

Next, the flow of processing at the time of image display and recording after step S204 will be described.
The processing from step S204 to step S207 is processing performed by the image processing unit 107 under the control of the system control unit 101. In step S204, the image processing unit 107 acquires (reads from the memory) data of the input image (short-second image) stored in the memory (RAM 103). After step S204, the image processing unit 107 advances the processing to step S205.

  In step S205, the image processing unit 107 performs an image reduction process for reducing the short-second image read from the memory. In the present embodiment, since the number of pixels of the captured image (short-second image) of the imaging unit 105 is larger than the number of display pixels of the display unit 108, the image processing unit 107 performs image reduction processing for reducing the short-second image. After step S205, the image processing unit 107 advances the process to step S206.

  In step S206, the image processing unit 107 performs display alignment processing on the short-second image reduced in step S205. This display alignment process is a process that is performed assuming that the imaging apparatus 100 of the present embodiment performs handheld shooting by the user in the pseudo long second mode. Done to absorb. In addition, since the process etc. which detect the camera shake, and absorb the position shift by camera shake are well-known processes, the illustration and detailed description are abbreviate | omitted. After step S206, the image processing unit 107 advances the process to step S207.

  In step S207, the image processing unit 107 performs a process for synthesizing the reduced second image. In the present embodiment, the synthesis process is performed by a process called a comparatively bright synthesis process. The comparatively bright combining process is a process in which the luminance value of the pixel at the same coordinate in the reference image and the image to be combined is compared, and a combined image is generated by adopting a pixel with a higher luminance value (brighter). is there. Although details will be described later, in the case of the present embodiment, the image of the previous frame or the synthesized image in the time series order is set as the reference image, and the image of the next frame in the time series order is set as the synthesis target image, and these images are used. A composite image is generated by performing comparatively bright combination processing. It should be noted that, immediately after the start of shooting, when there is only a short second image of the first frame as the first image, the comparatively bright combination process is not performed in step S207. In the present embodiment, a pseudo long-second image is generated by sequentially performing such comparatively bright combination processing. Details of the comparatively bright combination process and the pseudo long-second image generated thereby will be described in FIG. 3 to be described later. After step S207, the system control unit 101 advances the process to step S208.

  In step S208, the system control unit 101 displays the pseudo long-second image generated in step S207 on the screen of the display unit 108. Note that, immediately after the start of shooting, if there is only a short second image of the first frame in the image, an image obtained by reducing the short second image of the first frame is displayed in step S208. Is called. Further, the system control unit 101 displays the number of seconds obtained by adding the exposure time when each of the short second images used for generating the pseudo long second image is captured together with the display of the pseudo long second image. Alternatively, it may be displayed on the screen of the display unit 108. For example, when the exposure time of each short-second image is 0.5 seconds and the number of frames of the short-second image used for generating the pseudo-long-second image is 10, the system control unit 101 sets the number of frames for 10 images. “5 seconds”, which is the number of seconds added to the exposure time, is displayed. This number of seconds represents how much the pseudo long-second image is an image corresponding to a long-second image obtained by long-second shooting with an exposure time. In this way, by displaying the above-mentioned number of seconds together with the pseudo long-second image, the user can grasp how much long-second exposure time the pseudo long-second image currently displayed corresponds to. It becomes like this. That is, the user can use the displayed number of seconds as reference information when determining the timing for instructing to end the shooting. After step S208, the system control unit 101 advances the process to step S209.

  The processing in step S209 and the next step S210 is performed by the image processing unit 107 under the control of the system control unit 101. In step S209, the image processing unit 107 performs registration processing of the recording image. The recording image in this case is a short-second image acquired by the imaging unit 105 and accumulated in the memory. The alignment process for the short second image may be the same process as the alignment process in step S206 described above, or may be a different process. In general, the registration process for the recording image is a process after photographing and a process for the recording image, and therefore, it is desirable that the process places more importance on the accuracy than the processing time.

  After step S209, when the process proceeds to step S210, the image processing unit 107 performs a process for synthesizing the short-second image acquired by the imaging unit 105 and stored in the memory. If only the short second image of the first first frame has been acquired immediately after shooting is started, the composition process is not performed in step S209. Further, the synthesis process at this time may be a comparatively bright synthesis process similar to the synthesis process in step S207 described above, or a process that can generate a pseudo long-second image by another synthesis process. As another example of the synthesis process, for example, a synthesis process in which pixels having the same coordinates in the reference image and the synthesis target image are subjected to addition averaging with a weight corresponding to the luminance of the pixel having a higher luminance. Can be mentioned. More specifically, in this synthesis process, a composite image is generated by multiplying pixels having the same coordinates in the reference image and the synthesis target image by a gain corresponding to the weight of the luminance and then averaging. According to this composition processing, for example, a composite image in which composition switching in a moving subject is not noticeable can be obtained. Then, the pseudo long-second image data generated by the synthesis processing in step S210 is later sent to the recording unit 109 as output image data under the control of the system control unit 101, and is recorded on the recording medium. Become. The data of the pseudo long-second image for recording may be output not only when recorded on the recording medium but also via a network or the like. After step S210, the system control unit 101 advances the process to step S211.

  In step S <b> 211, the system control unit 101 determines whether the user has detected a shooting end instruction from the operation unit 110. The shooting end instruction is issued when the user fully presses the shutter button of the operation unit 110, as in the case of step S203. If the system control unit 101 determines in step S211 that the photographing end instruction has not been detected (No), the system control unit 101 returns the process to step S204. As a result, the processing from step S204 described above is performed on the short-second image of the next frame. On the other hand, if it is determined in step S211 that a shooting end instruction has been detected (Yes), the system control unit 101 ends the process of the flowchart of FIG.

Hereinafter, the comparatively bright combination process will be described with a specific example.
In the following description, the display pseudo-second image and the short pseudo-second image for generating the recording pseudo-second image are not distinguished from each other, and are described as short-second images.

  As described above, the comparatively bright combination process is a process of generating a composite image by comparing the luminance of the reference image and the compositing target image and adopting the high luminance region as the compositing target image region. In this embodiment, when sequentially generating pseudo long-second images from a plurality of images in time series order, the previous image in the time series order is used as the reference image, and the next image in the time series order is the synthesis target image. A process of making them comparatively bright and composite is repeated in chronological order. In the case of the present embodiment, the first short-second image in the first frame in time series order is a reference image, and the next short-second image in the second frame is a synthesis target image. Further, the composite image generated by the comparatively bright composite processing of the first frame and the second frame is used as the reference image when the short second image of the next third frame becomes the composition target image. After that, the synthesized image generated in the previous comparative bright synthesis process in time series order is used as the reference image, and the short second image of the next frame in time series order is used as the synthesis target image, so that comparative bright synthesis is performed. Such processing is sequentially repeated in time series. Therefore, for example, when a pseudo long-second image is sequentially generated from a plurality of short-second images in time series in which a vehicle traveling with lights turned on at night is taken, the locus of a high-intensity part such as light light is An image is generated as if it was captured by pseudo-second exposure.

  3, in the imaging apparatus 100 of the present embodiment, comparatively bright combination is performed from, for example, three short-second images A, B, and C in chronological order in which an image of a vehicle traveling with lights turned on at night is taken. A state in which a pseudo long-second image is generated by the processing will be specifically described. In FIG. 3, it is assumed that the image 301 is the short-second image A, the image 302 is the short-second image B, and the image 303 is the short-second image C.

  In each of the short-time images 301, 302, and 303 in FIG. 3, automobiles 321, 322, and 323 running in the directions indicated by arrows in the drawing are shown as subjects (moving bodies). It is assumed that arrows t in the drawings of the respective short-second images 301, 302, and 303 represent the amount and direction of movement of the subject automobiles 321, 322, and 323 within the exposure time for short-second shooting. Note that the images of the cars 321, 322, and 323 are actually images that flow according to the amount of movement within the exposure time at the time of capturing the short-second images 301, 302, and 303, but are illustrated in FIG. 3. Is drawn as an image in which only the light of the light flows. The same applies to images 304 and 305 described later. In the example of FIG. 3, the light of the light that has flowed due to the movement of the automobiles 321, 322, and 323 is depicted as light tracks 311, 312, and 313, respectively.

  In the example of FIG. 3, the image processing unit 107 first generates a composite image 304 by performing comparatively bright composite processing 330 on the previous two short-second images 301 and the short-second image 302 in time series order. The composite image 304 generated by the comparative bright combination processing 330 includes a light trajectory 314 in which the light light trajectory 311 in the short-second image 301 and the light light trajectory 312 in the short-second image 302 are connected. It becomes an image having. The composite image 304 is a pseudo long second image synthesized based on the short second image 301 and the short second image 302. The arrow t in the figure of the composite image 304 is a combination of the short-second image 301 and the double-headed arrow t in the short-second image 302, and the automobile within the time equivalent to the long-second exposure in the pseudo-long-second image (304). The moving amount and moving direction of (321, 322) are shown.

  Further, the image processing unit 107 compares the composite image 304 generated from the short-second image 301 and the short-second image 302 and the short-second image 303 that is later in time series than the short-second images 301 and 302, by the comparatively bright composite processing 331. Thus, a composite image 305 is generated. The composite image 305 generated by the comparative bright combination processing 331 is an image having a light locus 315 in which the light locus 314 in the composite image 304 and the light locus 313 in the short-second image 303 are connected. The composite image 305 is a pseudo long-second image generated based on the composite image 304 and the short-second image 303. The arrow t in the figure of the composite image 305 is a combination of the arrow t in the composite image 304 and the double arrow t in the short-second image 303, which corresponds to the long-second exposure in the pseudo long-second image (305). The moving amount and moving direction of the automobile (321, 322, 323) within the time are shown.

Next, the imaging timing and display update timing described above will be described with reference to FIG.
In FIG. 4, the shooting instruction timing 403 represents the timing when the user issues a shooting start instruction and a shooting end instruction. Specifically, a vertical line 431 in the shooting instruction timing 403 indicates that the signal output from the operation unit 110 when the shutter button is fully pressed by the user is detected by the system control unit 101 as a shooting start instruction signal. Represents timing. A vertical line 432 in the shooting instruction timing 403 indicates a signal output from the operation unit 110 when the shutter button is fully pressed again by the user after shooting is started. The timing detected as a signal is shown. The system control unit 101 determines in step S203 or step S211 of FIG. 2 described above whether an imaging end instruction has been received.

  In FIG. 4, each vertical line 411 in the imaging timing 401 represents a timing at which a frame of a short second image is captured after the start of imaging. That is, the above-described imaging of the short second image in step S <b> 201 of FIG. 2 is performed at each timing represented by each vertical line 411 in the imaging timing 401. Further, the processing in step S202 and step S203 in FIG. 2 is performed between the time when the short-second image of one frame is captured and the time of the short-second image of the next frame is captured. As described above, the processing from step S201 to step S203 in FIG. 2 is repeated for each timing represented by each vertical line 411 of the imaging timing 401 until it is determined in step S203 that an imaging end instruction has been detected. As a result, each short second image is acquired in chronological order from the shooting start instruction to the shooting end instruction. In the example of FIG. 4, the frames of each short second image acquired in chronological order after the start of shooting are represented by numbers 1, 2, 3, 4,.

  Each vertical line 421 in the display update timing 402 in FIG. 4 represents the timing at which the display is updated on the display unit 108. That is, after the display of the first frame 1 immediately after the start of photographing is performed, the display update of each pseudo long second image for display generated in steps S204 to S208 in FIG. This is performed for each vertical line 421 within the timing 402. In addition, the numbers “1 to 2”, “1 to 3”, “1 to 4”, “1 to 5”,... It is generated based on the frame. Note that “1” in the display update timing 402 represents the first frame 1. For example, the pseudo long second images “1 to 2” represent images generated from the short second images of the frame 1 and the frame 2, and the pseudo long second images “1 to 3” are the pseudo long second images “1 to 2”. 2 "and the image generated from the short-second image of frame 3. The pseudo long second images “1 to 4” are generated from the pseudo long second images “1 to 3” and the short second image of the frame 4, and the pseudo long second images “1 to 5” are the pseudo long second images “1 to 4”. This represents an image generated from 4 ″ and the short-second image of frame 5. The same applies hereinafter. Further, as can be seen from FIG. 4, each pseudo long second image “1-2”, “1-3”, “1-4”,... Is the amount of processing time required for steps S204 to S208 in FIG. However, it is delayed from each frame 1, 2, 3, 4,. Although the illustration of the generation timing of the pseudo long-second image for recording in steps S209 and S210 in FIG. 2 is omitted, the generation timing of the pseudo long-second image for recording is based on the display update timing 402 in FIG. The timing is slightly delayed. A specific example of the generation timing of the pseudo long second image for recording will be described in detail with reference to FIG.

  As described above, in the example of FIG. 4, a shooting end instruction is input from the user at the timing of the vertical line 432 in the shooting instruction timing 403. Further, at the time when a shooting end instruction is input from the user, the shooting of the short-second image of the frame 11 is completed in the imaging unit 105 as shown in the example of the imaging timing 401. On the other hand, since there is a delay in the processing time of step S204 to step S208 described above, the pseudo long-second image “1-7” is displayed on the display unit 108 when the photographing end instruction is input from the user. It has become. Here, when the user confirms the finish of the pseudo long-second image by looking at the display on the display unit 108 and determines that the displayed pseudo-long-second image has a desired finish, an instruction to end shooting is given. Is considered to be entered. Therefore, in the example of FIG. 4, the pseudo long-second image of the finish desired by the user is displayed on the display unit 108 at the timing (vertical line 432) when the shooting end instruction is input at the shooting instruction timing 403. It is considered that the pseudo long second images are “1-7”.

  On the other hand, at the time when a shooting end instruction is input from the user, the imaging unit 105 has already finished capturing each short second image from frame 1 to frame 11. Therefore, at the time when a photographing end instruction is input from the user, short-second image data up to frame 11 (at least frames 8 to 11) is held in the RAM 103 (memory) in FIG. Here, if a recording pseudo long second image using each short second image up to the frame 11 remaining in the memory is generated, the recording pseudo long second image has the pseudo length desired by the user. The second image “1-7” is an image having a different finish. The pseudo long second image using each short second image up to the frame 11 will be described with reference to the example of the display update timing 402. The pseudo long second image ("1 to 11") generated at the timing of the vertical line 422 is used. ) In other words, the pseudo long-second images ("1 to 11") are different from the pseudo long-second images "1 to 7" desired by the user.

  Therefore, the imaging apparatus 100 according to the present embodiment can independently control the imaging timing 401, the display update timing 402, and the generation timing of the pseudo long second image for recording (not shown in FIG. 4). . That is, in the flowchart of FIG. 2, when it is determined that a shooting end instruction has been detected in steps S203 and S211 (Yes), the processing is ended, and processing after reading out the short second image from the memory in step S204 (step The process up to S210 is not performed. Therefore, the pseudo long-second image for recording when the photographing end instruction is input from the user is the pseudo long-second image “1 to 7” similarly to the display, and the short-second image of the frame 11 that is not intended by the user. The pseudo long second images ("1 to 11") up to are not generated.

  As described above, according to the imaging apparatus 100 of the present embodiment, the pseudo long second image displayed on the display unit 108 at the time when the user inputs the photographing end instruction, that is, the pseudo long second image of the finish intended by the user, It is generated as a pseudo long second image for recording. Therefore, according to the present embodiment, a pseudo long-second image having a finish desired by the user can be obtained.

  In the case of the imaging apparatus 100 according to the present embodiment, an instruction to end shooting is performed when the user fully presses the shutter button, for example. Here, when a physical force is applied to the image capturing apparatus 100, such as when the shutter button is fully pressed, vibration or the like may be transmitted to the image capturing apparatus 100 and the captured image may be blurred. In this case, if a pseudo long-second image is generated by synthesizing the captured images in which the blur occurs, the image quality may be deteriorated. On the other hand, in the case of the present embodiment, the short second image acquired by the imaging unit 105 when the user fully presses the shutter button to input the shooting end instruction is generated as a pseudo long second image for recording. Therefore, it is possible to obtain a pseudo long-second image with good image quality without blurring.

<Second Embodiment>
The first embodiment described above is an example of recording a pseudo long-second image in which the composition process is completed when a shooting end instruction is input from the user. Also in the following second embodiment, similarly to the first embodiment, a pseudo long-second image that has been combined at the time when a photographing end instruction is input is set as a finished image desired by the user. In the case of the second embodiment, the pseudo-long image data, which is already stored in the memory when the photographing end instruction is input but is not used to generate the pseudo-long-second image, is used. Image processing different from the composition processing at the time of generating the second image is performed. Specifically, in the second embodiment, a short second image that is stored in the memory but is not used to generate the pseudo long second image is used, and the pseudo long second image that has already been combined is processed. For example, processing for improving noise in a low luminance region (that is, a dark portion) is performed to improve image quality. Note that the schematic configuration of the imaging apparatus 100 according to the second embodiment is the same as that of FIG. 1 described above, and therefore illustration and description of each part are omitted.

  FIG. 5 shows a pseudo long-second image generated in the pseudo long-second mode in the second embodiment in the same manner as described above, and using a short-second image that was not used to generate the pseudo-long-second image. It is a flowchart which shows the flow of the process which improves the dark part noise of an image. In the flowchart of FIG. 5, each process from step S501 to step S511 is the same as each process from step S201 to step S211 in the flowchart of FIG. 2, and description thereof will be omitted. In the flowchart of FIG. 5 of the second embodiment, processes of step S512 and step S513 are added, and hereinafter, step S512 and step S513 will be described.

  If the system control unit 101 determines in step S511 in FIG. 5 that a shooting end instruction from the user has been detected via the operation unit 110 (Yes), the system control unit 101 proceeds to step S512 in FIG. The processing in step S512 is performed by the image processing unit 107 under the control of the system control unit 101.

  In step S512, the image processing unit 107 stores a short second image (a plurality of short images that are already stored in the memory (RAM 103) at the time when the photographing end instruction is input but are not used to generate a pseudo long second image. (Short second image) data is acquired from the memory. Then, the image processing unit 107 synthesizes the respective short-second images acquired from the memory by averaging them for each pixel. Hereinafter, the synthesized image generated by the addition average synthesis process is referred to as an addition average image. In addition, the addition average composition processing of each short second image is a process of performing addition average composition only for the low-luminance region (dark part) of each short second image, in addition to the example of performing addition average composition of each short second image as a whole. Also good. An addition average image obtained by performing addition average synthesis processing on a plurality of short-second images is an image in which noise components are smoothed by addition average and noise is improved. For example, a short-second image taken at night tends to be a noise-conspicuous image particularly in a dark portion that is a low-luminance region.However, when a plurality of short-second images are added and averaged and combined as in the present embodiment, An image with improved noise is obtained. After step S512, the image processing unit 107 advances the process to step S513.

  In step S513, the image processing unit 107 performs comparative dark combining processing using the pseudo long-second image generated in step S510 and the addition average image generated in step S512 as output image generation processing. The comparative dark combining process in the present embodiment is a combining process based on a result obtained by comparing the addition average image generated in step S512 and the pseudo long second image generated in step S510 with a predetermined luminance value. Specifically, for example, an area having a high luminance value equal to or higher than a predetermined luminance value employs a pseudo long second image, while an area having a luminance value lower than the predetermined luminance value employs an average image, and these pseudo long seconds are used. The image and the averaged image are combined. As a result, a region with a high luminance such as a light trajectory uses a pseudo long-second image region, and a region with a low luminance such as a dark portion in the background uses a region of an addition average image. An improved pseudo-long second image will be obtained.

  Further, in the synthesis process in step S513, the luminance value of the pixel at the same coordinate in the reference image and the synthesis target image is compared, and a synthesized image is generated so as to adopt a pixel with a lower luminance value (darker). Such processing may be used. In the case of this example, the addition average image generated in step S512 is used as a reference image, the pseudo long-second image generated in step S510 is used as a synthesis target image, and the synthesis process of these images is performed. As a result, the low-brightness area (dark part with low luminance value) of the pseudo long-second image generated in step S510 is replaced with the low-brightness area of the addition average image generated in step S512, and noise in the dark part is generated. An improved pseudo long second image is obtained.

Hereinafter, the processing in steps S512 and S513 in FIG. 5 will be described in more detail with reference to FIGS. 7 will be described before FIG.
An image 701 in FIG. 7 is a pseudo length for recording generated in steps S509 to S510 in FIG. 5 from three short-second images A, B, and C obtained by imaging a car that is running with lights on at night. Assume that it is a second image (pseudo long-second image A + B + C). The pseudo long second image 701 is assumed to be an image having a light locus 315 similar to that described for the composite image 305 (pseudo long second image) in FIG. 3 described above. In addition, the images 702 and 703 in FIG. 7 are stored in the memory at the time when the shooting end instruction is input, but are not used for generating the pseudo-long-second image (the short-second image D). And a short second image E). Further, it is assumed that the area 711 of the pseudo long second image 701, the area 712 of the short second image 702, and the area 713 of the short second image 703 are dark background areas. The pseudo long-second image 701 is an image obtained by performing comparatively bright combination processing on the short-second images A, B, and C each having a short exposure time. This is considered to be an area with a lot of dark part noise. Similarly, since the short-second images D and E are images having a short exposure time, the dark background regions 712 and 713 are considered to be regions having a low SN and a large amount of dark noise compared to the images with proper exposure, respectively. It is done.

  In the second embodiment, the short-second image 702 and the short-second image 703 are combined by the addition average combining processing 721 in step S512 of FIG. The addition average image 705 obtained by the addition average synthesis processing 721 is an image in which the dark noise in the dark background region 714 is improved as described above. In the second embodiment, the addition average image 705 and the pseudo long second image 701 are combined by the comparative dark combining process 722 in step S513 in FIG. An image 704 obtained by the comparative dark combination processing 722 becomes a pseudo long second image having a background region 715 in which the dark background region 711 of the pseudo long second image 701 is replaced by the dark background region 714 of the addition average image 705. . That is, the pseudo long second image 704 is an image with improved dark part noise. In the second embodiment, the pseudo long-second image 704 with improved dark part noise is an output image generated by the output image generation process, that is, a final pseudo long-second image for recording.

  In the present embodiment, the example in which the synthesis process of the pseudo long-second image 701 and the addition average image 705 is the comparative dark synthesis process 722 is described. However, the synthesis process is not performed by the comparative dark synthesis process 722 but by another method. It may be. For example, the image processing unit 107 uses the short-second images A, B, and C when the pseudo long-second image 701 is generated, and the short-second images 702 (D) and 703 (E) to obtain a moving object region such as an automobile. Is detected, and an optical cuff flow in the moving body region is obtained to determine a moving body. Then, the image processing unit 107 uses the area determined to be a moving object in the pseudo long second image 701 as it is, while the non-moving area (background area 711) is a non-moving area in the addition average image 705 (background area 711). A compositing process to be replaced by the background area 714) is performed. Also in this example, a pseudo long-second image with improved dark part noise can be obtained.

  In the above-described example, the process of replacing the background area 711 of the pseudo long second image 701 with the background area 714 of the addition average image 705 is a process of binaryly switching the background area 711 to the background area 714. The process of replacing the background area 711 with the background area 714 is not a binary switching process. For example, the same weighting is performed based on the luminance information of the pseudo long-second image 701, and a gain corresponding to the weight is multiplied. It is also possible to perform processing such that addition is performed after this.

  Next, the timing at which the processing of the flowchart of FIG. 5 described above is performed will be described with reference to FIG. FIG. 6 is a diagram expressed in the same manner as FIG. 4 described above. In FIG. 6, a shooting instruction timing 603 represents a timing when the user gives a shooting start instruction and a shooting end instruction by operating the shutter button. A vertical line 631 indicates a shooting start instruction, and a vertical line 632 indicates a shooting end instruction. Represents timing. In addition, each vertical line 611 in the imaging timing 601 represents the timing at which each frame of the short-second image is captured after the start of shooting, and the frames of each short-second image acquired in chronological order after the start of shooting are 1, 2, respectively. , 3,... Also, each vertical line 621 in the display update timing 602 represents a display update timing, “1” indicates the first frame, “1-2”, “1-3”,. Represents each frame used to generate the pseudo-long-second image. Further, each of the pseudo long-second images “1 to 2”, “1 to 3”,... Corresponds to the frames 1, 2, and 3 of the imaging timing 601 corresponding to the processing time required for steps S504 to S508 in FIG. , ... are delayed from each other.

  In FIG. 6, a recorded image generation timing 604 represents the timing at which a pseudo long second image for recording is generated in steps S509 to S510 in FIG. That is, after the first recording image is generated from the first short-frame image of the first frame 1 immediately after the start of photographing, the recording is sequentially performed at the timing of each vertical line 641 in the recording image generation timing 604. A pseudo long second image is generated. The numbers “1-2”, “1-3”,... In the recording image generation timing 604 indicate on which frame the pseudo long-second image for recording is generated. Represents. In the case of the present embodiment, as shown in FIG. 5, the pseudo long-second image for recording in steps S509 to S510 is generated and displayed in steps S504 to S508. It is done after processing. For this reason, as shown in FIG. 6, the recorded image generation timing 604 is slightly delayed from the display update timing 602.

  In the example of FIG. 6, a shooting end instruction is input from the user at the timing of the vertical line 632 in the shooting instruction timing 603, and the shooting of the frame 11 at the imaging timing 601 is completed at the time when the shooting end instruction is input. On the other hand, since there is a delay due to the processing time of steps S504 to S508 in FIG. 5 at the time of inputting the photographing end instruction, pseudo long-second images “1 to 7” are displayed on the display unit 108. That is, it is considered that the finished pseudo long-second image desired by the user is the pseudo long-second images “1 to 7” displayed on the display unit 108. Also in the imaging apparatus 100 of the second embodiment, as in the case of the first embodiment described above, the pseudo long-second image for recording is a pseudo long-second image at the time when the photographing end instruction is input, that is, a pseudo long-second image. Long-second images “1 to 7” are displayed. As described above, in the second embodiment as well, as in the first embodiment, the pseudo long-second image displayed when the user inputs the shooting end instruction, that is, the pseudo long-second image finished by the user is displayed. And generated as a pseudo-long-second image for recording. Accordingly, a pseudo long-second image having a finish desired by the user can be obtained.

  Further, in the case of the second embodiment, as described in step S512 and step S513 in FIG. 5, each short second image that is not used for generating the pseudo long second image at the time of inputting the photographing end instruction is added and averaged combined. The dark average synthesized image and the pseudo long-second image are subjected to comparative dark synthesis. In the example of FIG. 6, the short-second images of the frames 8, 9, 10, and 11 indicated by the frame section 652 are short-second images that are not used for generating the pseudo-long-second image at the time of inputting the photographing end instruction. is there. On the other hand, each short second image of each of the frames 1, 2, 3,..., 7 indicated by the frame section 651 is a short second image used for generating a pseudo long second image at the time of inputting the photographing end instruction. In the first embodiment described above, the short-second images of the frames 8, 9, 10, and 11 in the frame section 652 are not used. However, in the second embodiment, the short images of the frames 8, 9, 10, and 11 are used. An averaged image is generated by adding and averaging the second images. In the second embodiment, the pseudo long second image generated from the short second image of each frame in the frame section 651 is compared with the addition average image generated from the short second image of each frame in the frame section 652. Dark synthesis processing is performed. In the example of FIG. 6, the pseudo long-second image after the comparative dark combining process is generated at the timing of the vertical line 642 of the recording image generation timing 604.

  As described above, in the second embodiment, as in the first embodiment, it is possible to obtain a pseudo long-second image having a desired finish by the user, and also to obtain a pseudo long-second image having good image quality without blurring. Can be obtained. Further, according to the second embodiment, an addition average image obtained by adding and synthesizing each short second image that was not used for generation of the pseudo long second image when the photographing end instruction is input from the user, and the pseudo long second image And comparative dark synthesis processing. As a result, in the second embodiment, it is possible to obtain a pseudo long-second image with good image quality and improved noise in the dark part.

<Other embodiments>
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a recording medium, and one or more processors in the computer of the system or apparatus read and execute the program This process can be realized. In the present invention, each function of the above-described embodiment can be realized by cooperation of a circuit (for example, ASIC) and a program.

  The above-described embodiments are merely examples of implementation in carrying out the present invention, and the technical scope of the present invention should not be construed as being limited thereto. That is, the present invention can be implemented in various forms without departing from the technical idea or the main features thereof.

  101 system control unit, 102 ROM, 103 RAM, 104 optical system, 105 imaging unit, 106 A / D conversion unit, 107 image processing unit, 108 display unit, 109 recording unit, 110 operation unit

Claims (14)

Acquisition means for acquiring a plurality of input images input in order;
Using the input images in the input order, and sequentially performing a combining process to generate a combined image;
Detecting means for detecting timing when an end instruction is input from the user,
An image processing apparatus, wherein an output image is a combined image in which the combining process by the combining unit is completed before detection of a timing when an end instruction is input from a user. Acquisition means for acquiring a plurality of input images input in order;
Using the input images in the input order, and sequentially performing a combining process to generate a combined image;
Detecting means for detecting timing when an end instruction is input from a user;
Among a plurality of input images acquired by the acquisition unit, a plurality of input images that have not been subjected to the combining process by the combining unit until the timing when the end instruction is input from the user are sequentially added and averaged and added. An averaging means for generating an average image;
Output image generation means for generating an output image by combining the combined image in which the combining processing by the combining means has been completed by the timing when an end instruction is input from the user and the addition average image;
An image processing apparatus comprising:   The addition averaging means sequentially averages all or a plurality of low-luminance regions of a plurality of input images that have not been subjected to the composition processing by the composition means by the timing when an end instruction is input from the user. The image processing apparatus according to claim 2. The output image generation means includes
The luminance value of the composite image and the luminance value of the addition average image are compared with a predetermined luminance value,
The composite image and the addition average image are configured such that the region having a higher luminance value than the predetermined luminance value employs the composite image, and the region having a luminance value lower than the predetermined luminance value employs the addition average image. The image processing apparatus according to claim 2, wherein an image obtained by combining the images is used as the output image. A moving object determining means for determining an area of a moving object in the composite image and the addition average image;
The output image generation means sets an image obtained by combining an area that is not the moving body in the combined image with an area that is not the moving body in the addition average image as the output image. The image processing apparatus according to claim 2, wherein the image processing apparatus is an image processing apparatus.   The synthesizing unit sequentially synthesizes the high luminance regions of the plurality of input images in the input order in the order of the inputs, or multiplies each pixel by a gain based on the luminance of each pixel. 6. The composition processing for generating a locus of movement of a high-intensity region of a moving body in the plurality of input images in the input order by performing addition later is performed. The image processing apparatus according to any one of the above.   The synthesizing unit performs the synthesizing process using the synthesized image generated by the synthesizing process and the input image having the next input order with respect to the input image used to generate the synthesized image. The image processing apparatus according to claim 1, wherein the image processing apparatus is characterized in that:   The image processing apparatus according to claim 1, further comprising a display unit that updates and displays the composite image sequentially generated by the combining unit. The acquisition unit acquires a captured image as the input image,
9. The display unit according to claim 8, wherein the display unit updates and displays, together with the composite image, a time obtained by adding the exposure times when the plurality of captured images used to generate the composite image are captured. The image processing apparatus described. The acquisition means acquires and stores a captured image as the input image,
The said synthetic | combination means reads the input image memorize | stored in the said acquisition means in order of the said input, The said synthetic | combination process which requires predetermined | prescribed processing time is performed, The any one of Claim 1 thru | or 9 characterized by the above-mentioned. Image processing device.   The image processing apparatus according to claim 1, wherein the output image is recorded on a recording medium. An acquisition step of acquiring a plurality of input images sequentially input;
Using the input images in the input order, and sequentially performing a combining process to generate a combined image;
Detecting a timing at which an end instruction is input from the user, and
An image processing method for an image processing apparatus, wherein an output image is a combined image in which the combining process in the combining step is completed before detection of a timing at which an end instruction is input from a user. An acquisition step of acquiring a plurality of input images sequentially input;
Using the input images in the input order, and sequentially performing a combining process to generate a combined image;
A detection step of detecting a timing when an end instruction is input from the user;
Among a plurality of input images acquired in the acquisition step, a plurality of input images that have not been subjected to the combining process by the combining step until the timing when an end instruction is input from the user are sequentially added and averaged and added. An averaging process for generating an average image;
An output image generation step of generating an output image by combining the combined image in which the combining processing by the combining step is completed by the timing when an end instruction is input from the user and the addition average image;
An image processing method for an image processing apparatus, comprising:   The program for functioning a computer as each means of the image processing apparatus of any one of Claims 1 thru | or 11.

Images