JP2017184094A - Imaging control device, imaging control method, computer program and electronic equipment - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an imaging control device that can generate a natural HDR image even when a moving object is included in a subject.SOLUTION: An imaging control device is provided, the imaging control device comprising a control unit that generates a first image by performing first exposure on a pixel for first exposure time, and generates a second image by performing second exposure on a pixel for a second exposure time following the first image, and a synthesis unit that synthesizes a composite image synthesized by the first image and the second image, and a selected image selected from either of the first image or the second image. The control unit determines the first exposure time and the second exposure time such that the image being selected as the selected image switches on the synthesis unit based on an exposure ratio which is a ratio between the exposure time of the composite image and the exposure time of the selected image.SELECTED DRAWING: Figure 1

Description

  The present disclosure relates to an imaging control device, an imaging control method, a computer program, and an electronic device.

  Some digital cameras including a CCD image sensor or a CMOS image sensor have a function called HDR (High Dynamic Range) that widens the width of a dynamic range. As a technique for capturing an HDR image, there is a method of generating an HDR image using a plurality of images captured at different exposure times (see, for example, Patent Document 1).

JP 2011-4353 A

  When an HDR image is generated by combining an image captured by long exposure and an image captured by short exposure, a blank occurs between image captures depending on the exposure time ratio. When the moving object is included in the subject, the moving subject becomes unnatural when the HDR image is generated.

  Therefore, the present disclosure proposes a new and improved imaging control apparatus, imaging control method, computer program, and electronic device that can generate a natural HDR image even when a moving object is included in a subject. .

  According to the present disclosure, a first image is generated by exposing a pixel by first exposure with a first exposure time, and a pixel is exposed by second exposure with a second exposure time following the first image. A composition for synthesizing a control unit for generating a second image, a synthesized image obtained by synthesizing the first image and the second image, and a selected image selected from either the first image or the second image. And the control unit switches the image selected as the selected image by the combining unit based on an exposure ratio that is a ratio of an exposure time of the combined image and an exposure time of the selected image. An imaging control device for determining a first exposure time and the second exposure time is provided.

  According to the present disclosure, the first image is generated by exposing the pixel by the first exposure with the first exposure time, and the pixel is exposed by the second exposure by the second exposure time following the first image. Generating a second image, and synthesizing a synthesized image obtained by synthesizing the first image and the second image and a selected image selected from either the first image or the second image. And the first exposure time and the second exposure time are determined so that the image selected as the selected image is switched based on an exposure ratio that is a ratio of an exposure time of the composite image and an exposure time of the selected image. To provide an imaging control method.

  Further, according to the present disclosure, the computer generates a first image by exposing the pixel by the first exposure according to the first exposure time, and then the pixel is performed by the second exposure by the second exposure time following the first image. Generating a second image by exposure; a composite image obtained by combining the first image and the second image; and a selected image selected from either the first image or the second image. And the first exposure time and the second exposure so that the image selected as the selected image is switched based on an exposure ratio that is a ratio between the exposure time of the composite image and the exposure time of the selected image. A computer program is provided for determining and executing time.

  Moreover, according to this indication, an electronic device provided with the said imaging control apparatus is provided.

  As described above, according to the present disclosure, a novel and improved imaging control device, imaging control method, computer program, and computer program capable of generating a natural HDR image even when a moving object is included in a subject, Electronic equipment can be provided.

  Note that the above effects are not necessarily limited, and any of the effects shown in the present specification, or other effects that can be grasped from the present specification, together with or in place of the above effects. May be played.

It is explanatory drawing for demonstrating the imaging technique of the existing HDR image. It is explanatory drawing for demonstrating the imaging technique of the existing HDR image. 3 is an explanatory diagram illustrating a functional configuration example of an electronic device 10 according to an embodiment of the present disclosure. FIG. 2 is an explanatory diagram illustrating a configuration example of a sensor module 100 included in an imaging unit 11. FIG. It is explanatory drawing which shows the function structural example of the sensor module 100 which concerns on embodiment of this indication. 5 is a flowchart illustrating an operation example of the sensor module 100 according to an embodiment of the present disclosure. It is explanatory drawing which shows the example of selection of the short accumulation image by exposure ratio. It is explanatory drawing which shows the example which exposes two flame | frames in a very short time.

  Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In addition, in this specification and drawing, about the component which has the substantially same function structure, duplication description is abbreviate | omitted by attaching | subjecting the same code | symbol.

The description will be made in the following order.
1. Embodiment of the present disclosure 1.1. Outline 1.2. Functional configuration example of imaging apparatus 1.3. Configuration example of sensor module 1.4. Example of operation Summary

[1.1. Overview]
Before describing the embodiments of the present disclosure in detail, an outline of the embodiments of the present disclosure will be described first.

  As described above, some digital cameras including a CCD image sensor or a CMOS image sensor have a function called HDR that expands the width of a dynamic range. As a technique for capturing an HDR image, there is a method of generating an HDR image using a plurality of images captured at different exposure times.

  Hereinafter, an existing HDR image capturing technique will be described. 1 and 2 are explanatory diagrams for describing an existing HDR image capturing technique. 1 and 2 show an example in which an HDR image is generated by capturing two images with different or the same exposure time. Also, in FIGS. 1 and 2, by adding two frames, an image by long exposure (long accumulation image) is generated, and any one of the frames that are the basis of the long accumulation image is converted by an image by short exposure ( An example in the case of (short accumulation image) is shown.

  In the description of the following embodiment, when generating an HDR image, the longer exposure time among the exposure times set for each of the two images is referred to as the long exposure time, and the shorter exposure time is referred to as the short exposure time. This is called exposure time. In addition, one image obtained by exposure with a long exposure time is a long exposure image, one image obtained by exposure with a short exposure time is a short exposure image, and a pixel signal obtained by exposure with a long exposure time A pixel signal obtained by exposure with a long exposure signal and a short exposure time is also referred to as a short exposure signal.

  FIG. 1 exemplifies the relationship between the exposure ratio and the exposure timing when the short accumulation image is fixed to the second frame. Here, the exposure ratio represents the exposure time of the long accumulation image when the exposure time of the short accumulation image is 1. That is, the exposure ratio of 2 indicates that the exposure time of the long accumulation image: exposure time of the short accumulation image = 2: 1.

  For example, when an image sensor such as a CMOS image sensor is composed of pixels of H rows in the vertical direction, data can be read sequentially from the first row to the H row, but the last H row data has been read. Otherwise, reading from the first line of the next frame cannot be started.

  Therefore, if the output frame rate of the image from the image sensor is 30 fps (the time per frame is approximately 33.3 milliseconds) and the exposure ratio is approximately 5, the exposure time of the second frame is 6 .6 milliseconds can be used. When a bright subject and a dark subject are mixed, if the exposure ratio is set to 5 or more so that both subjects can be appropriately displayed in the captured image, the exposure time of the second frame is set to 6. It should be shorter than 6 milliseconds. However, the exposure time of the first frame cannot be longer than 26.7 milliseconds due to the above-described readout limitation. Accordingly, when the short accumulation image is fixed to the second frame, if the exposure ratio is set to 5 or more, blanks are generated in the first frame and the second frame. If a blank occurs in the first frame and the second frame, if the subject includes a moving object, the moving subject becomes unnatural when the HDR image is generated.

  FIG. 2 illustrates the relationship between the exposure ratio and the exposure timing when the short accumulation image is fixed to the first frame.

  When the short accumulation image is fixed to the first frame, when the output frame rate of the image from the image sensor is 30 fps, if the exposure ratio is set to about 1.25, the exposure time of the second frame is set to 6.6. It only has to be milliseconds. When a bright subject and a dark subject are mixed, if the exposure ratio is set to 1.25 or less so that both subjects can be appropriately displayed in the captured image, the exposure time of the second frame is reduced. It may be shorter than 6.6 milliseconds. However, the exposure time of the first frame cannot be longer than 26.7 milliseconds due to the above-described readout limitation. Accordingly, when the short accumulation image is fixed to the first frame, if the exposure ratio is set to 1.25 or less, blanks are generated in the first frame and the second frame. If a blank occurs in the first frame and the second frame, if the subject includes a moving object, the moving subject becomes unnatural when the HDR image is generated.

  Therefore, in view of the above-described content, the present disclosure has intensively studied a technique capable of generating an HDR image so that a subject does not become an unnatural image when generating an HDR image based on an exposure ratio. As a result, the present disclosure has devised a technique capable of generating an HDR image so that the subject does not become an unnatural image when generating the HDR image by the exposure ratio, as will be described below.

  The outline of the embodiment of the present disclosure has been described above.

[1.2. Functional configuration example]
Subsequently, a functional configuration example of the electronic device according to the embodiment of the present disclosure will be described. FIG. 3 is an explanatory diagram illustrating a functional configuration example of the electronic device 10 according to the embodiment of the present disclosure. Hereinafter, a functional configuration example of the electronic apparatus 10 according to the embodiment of the present disclosure will be described with reference to FIG.

  As illustrated in FIG. 3, the electronic device 10 according to the embodiment of the present disclosure includes an imaging unit 11, an image processing unit 12, a display unit 13, a control unit 14, a storage unit 15, and an operation unit 16. And comprising.

  The imaging unit 11 includes a lens, a sensor module, and the like, and accumulates electrons for a predetermined period according to an image formed on the light receiving surface of the sensor module through the lens. The imaging unit 11 performs predetermined signal processing on a signal corresponding to the accumulated electrons. Then, the imaging unit 11 outputs the signal after the signal processing to the image processing unit 12. The configuration of the sensor module included in the imaging unit 11 will be described in detail later.

  The imaging unit 11 includes, as the predetermined signal processing, camera shake correction processing using an electronic camera shake correction method, automatic white balance processing, automatic exposure processing, distortion correction processing, defect correction processing, noise reduction processing, high dynamic range synthesis processing, and the like. Signal processing may be performed.

  The image processing unit 12 includes, for example, an application processor (AP), and executes image processing using a signal output from the imaging unit 11. The image processing executed by the image processing unit 12 includes, for example, demosaic processing using a signal output from the imaging unit 11, display processing of the demosaiced image on the display unit 13, storage processing in the storage unit 15, and the like. There is.

  The display unit 13 is a display device configured by, for example, a liquid crystal display or an organic EL display. Display contents of the display unit 13 are controlled by the control unit 14. For example, the display unit 13 displays an image captured by the imaging unit 11 and subjected to image processing by the image processing unit 12 based on the control of the control unit 14.

  The control unit 14 includes, for example, a processor such as a CPU (Central Processing Unit), a ROM, a RAM, and the like, and controls the operation of each unit of the electronic device 10.

  The storage unit 15 is configured by a storage medium such as a flash memory or other nonvolatile memory. The storage unit 15 stores an image captured by the imaging unit 11 and subjected to image processing by the image processing unit 12. The image stored in the storage unit 15 can be displayed on the display unit 13 in accordance with the operation of the user of the electronic device 10.

  The operation unit 16 is a device for operating the electronic device 10 and includes, for example, buttons and a touch panel. When the operation unit 16 includes a touch panel, the touch panel is provided on the display surface of the display unit 13. When the user of the electronic device 10 wants to record the image captured by the imaging unit 11 in the electronic device 10, the user triggers a shutter trigger by operating a predetermined button of the operation unit 16. When the imaging unit 11 and the image processing unit 12 detect the occurrence of the shutter trigger, the imaging unit 11 and the image processing unit 12 execute a process for recording an image on the electronic device 10 according to the generation of the shutter trigger.

  The electronic device 10 shown in FIG. 3 is not limited to a specific device, and may take various forms such as a digital camera, a smartphone, a tablet portable terminal, a portable music player, and a game machine.

  Heretofore, the functional configuration example of the electronic device 10 according to the embodiment of the present disclosure has been described. Next, a configuration example of the image sensor included in the imaging unit 11 of the electronic device 10 according to the embodiment of the present disclosure will be described.

[1.3. Example of sensor module configuration]
FIG. 4 is an explanatory diagram illustrating a configuration example of the sensor module 100 included in the imaging unit 11. The sensor module 100 according to the embodiment of the present disclosure is an example of the image processing apparatus of the present disclosure, and is configured by stacking three substrates as illustrated in FIG. The sensor module 100 according to the embodiment of the present disclosure has a configuration in which a pixel substrate 110, a memory substrate 120, and a signal processing substrate 130 are stacked in this order.

  The pixel substrate 110 is a substrate having an image sensor composed of pixel regions in which unit pixels are formed in an array. Each unit pixel receives light from a subject, photoelectrically converts the incident light, accumulates charges, and outputs the charges as pixel signals at a predetermined timing. Pixel signals output from the pixel substrate 110 are stored in the memory substrate 120, and signal processing is performed in the signal processing substrate 130. The pixel substrate 110 includes an AD converter that converts an analog signal into a digital signal. That is, the pixel signal output from the pixel substrate 110 is a digital signal.

  The memory substrate 120 is a substrate having a memory such as a DRAM (Dynamic Random Access Memory) that temporarily stores pixel signals output from the pixel substrate 110. The memory substrate 120 has a capacity capable of temporarily storing pixel signals of a plurality of frames, for example, enough frames that can execute the camera shake correction process by the electronic camera shake correction method in the signal processing board 130. The pixel signal stored in the memory substrate 120 is read based on a read command from the signal processing substrate 130.

  The signal processing board 130 performs various signal processes on the pixel signals stored in the memory board 120. The signal processing executed by the signal processing board 130 is signal processing related to image quality with respect to the pixel signals stored in the memory board 120. For example, camera shake correction processing by an electronic camera shake correction method, automatic white balance processing, automatic exposure processing, distortion Signal processing such as correction processing, defect correction processing, noise reduction processing, and high dynamic range synthesis processing can be executed.

  FIG. 4 illustrates a configuration in which the sensor module 100 is stacked in the order of the pixel substrate 110, the memory substrate 120, and the signal processing substrate 130. However, the present disclosure is not limited to such an example. For example, the sensor module 100 may have a configuration in which a pixel substrate 110, a signal processing substrate 130, and a memory substrate 120 are stacked in this order.

  The configuration example of the sensor module 100 has been described above with reference to FIG. Subsequently, a functional configuration example of the sensor module 100 will be described.

  FIG. 5 is an explanatory diagram illustrating a functional configuration example of the sensor module 100 according to the embodiment of the present disclosure. Hereinafter, a functional configuration example of the sensor module 100 according to the embodiment of the present disclosure will be described with reference to FIG.

  The pixel substrate 110 includes an image sensor 111 having a pixel region in which unit pixels are formed in an array, and a control unit 112 that supplies a predetermined clock signal and timing signal to the image sensor 111. A pixel signal output from the image sensor 111 in response to a signal from the control unit 112 is once sent to the signal processing board 130 and then sent to the memory board 120.

  In the present embodiment, the control unit 112 supplies a timing signal to the image sensor 111 so that two exposure signals are output from the image sensor 111 so that no blank is generated between frames when an HDR image is captured. . Then, the control unit 112 supplies a timing signal to the image sensor 111 so that the exposure ratio is set when the HDR image is captured.

  The memory substrate 120 includes an image storage unit 121 configured by a DRAM (Dynamic Random Access Memory) or the like. The image storage unit 121 temporarily stores pixel signals output from the image sensor 111. The image storage unit 121 has a capacity capable of temporarily storing pixel signals of a plurality of frames. The pixel signal stored in the image storage unit 121 is read based on a read command from the signal processing board 130.

  The signal processing board 130 includes a pre-processing unit 131 and a post-processing unit 132.

  The preprocessing unit 131 performs signal processing on the pixel signal output from the image sensor 111. The preprocessing unit 131 stores the pixel signal after the signal processing in the image storage unit 121. The signal processing executed by the preprocessing unit 131 can include, for example, gain adjustment processing, clamping processing, pixel addition processing, and the like.

  The post-processing unit 132 performs signal processing on the pixel signal stored in the image storage unit 121. When the post-processing unit 132 performs signal processing on the pixel signal stored in the image storage unit 121, the post-processing unit 132 outputs the pixel signal after the signal processing to the image processing unit 12. The signal processing executed by the post-processing unit 132 can include, for example, automatic white balance processing, automatic exposure processing, distortion correction processing, defect correction processing, noise reduction processing, high dynamic range synthesis processing, and the like. Further, the post-processing unit 132 can execute a composition process of images captured in a plurality of frames.

  In the present embodiment, when the HDR image is generated, the post-processing unit 132 uses a frame obtained by combining the long exposure image and the short exposure image output from the image sensor 111 as the long accumulation image. Then, the post-processing unit 132 determines whether to use the image based on the exposure signal of the first frame or the image based on the exposure signal of the second frame, as the short accumulation image, according to the exposure ratio. Accordingly, the post-processing unit 132 can function as an example of the control unit and the synthesis unit of the present disclosure.

  The sensor module 100 according to the embodiment of the present disclosure has such a configuration, and can generate an HDR image so that the subject does not become an unnatural image when generating the HDR image based on the exposure ratio.

  The functional configuration example of the sensor module 100 according to the embodiment of the present disclosure has been described above.

[1.4. Example of operation]
Subsequently, an operation example of the sensor module 100 according to the embodiment of the present disclosure will be described. FIG. 6 is a flowchart illustrating an operation example of the sensor module 100 according to the embodiment of the present disclosure. FIG. 6 is a flowchart of the sensor module 100 according to the embodiment of the present disclosure when generating an HDR image. Hereinafter, an operation example of the sensor module 100 according to the embodiment of the present disclosure according to the embodiment of the present disclosure will be described using FIG. 6.

  When generating the HDR image, the sensor module 100 according to the embodiment of the present disclosure determines whether the exposure ratio between the long accumulation image and the short accumulation image that is the basis of the HDR image is greater than 2 (step S101). ). For example, the control unit 112 executes the determination process in step S101.

  As a result of the determination in step S101, when the exposure ratio between the long accumulation image and the short accumulation image that is the basis of the HDR image is larger than 2 (step S101, Yes), the sensor module 100 then proceeds to the exposure signal of the first frame. The exposure times of the first frame and the second frame are determined so as to be a short accumulation image (step S102). The determination process in step S102 is executed by, for example, the control unit 112.

  On the other hand, as a result of the determination in step S101, when the exposure ratio between the long accumulation image and the short accumulation image that is the basis of the HDR image is 2 or less (No in step S101), the sensor module 100 then performs the second frame. The exposure times of the first frame and the second frame are determined so that the exposure signal is a short accumulation image (step S103). The determination process in step S103 is executed by, for example, the control unit 112.

  After determining the exposure time of the first frame and the second frame, the sensor module 100 subsequently exposes the image sensor 111 with the exposure time determined in step S102 or step S103, and receives the exposure signal of the first frame from the image sensor 111. Read (step S104). The process in step S <b> 104 is performed, for example, when the control unit 112 supplies a signal for reading an exposure signal to the image sensor 111.

  When the exposure signal of the first frame is read from the image sensor 111, the sensor module 100 then temporarily stores the read exposure signal of the first frame in the image storage unit 121 (step S105). When the exposure ratio is larger than 2, the exposure signal of the first frame becomes a short accumulation image.

  When the read exposure signal of the first frame is temporarily stored in the image storage unit 121, the sensor module 100 subsequently exposes the image sensor 111 with the exposure time determined in step S102 or step S103, and the image sensor 111 transmits the second frame. The exposure signal is read out (step S106). The process of step S <b> 106 is performed, for example, when the control unit 112 supplies a signal for reading an exposure signal to the image sensor 111.

  When the exposure signal of the second frame is read from the image sensor 111, the sensor module 100 then temporarily stores the read exposure signal of the second frame in the image storage unit 121 (step S107). When the exposure ratio is 2 or less, the exposure signal of the second frame is a short accumulation image.

  When the readout exposure signal of the second frame is temporarily stored in the image storage unit 121, the sensor module 100 subsequently acquires the exposure signal of the first frame and the exposure signal of the second frame stored in the image storage unit 121. Then, an addition signal is generated by adding the obtained exposure signal of the first frame and the exposure signal of the second frame (step S108). The post-processing unit 132 executes the addition signal generation processing in step S105, for example. When the sensor module 100 generates the addition signal, the sensor module 100 temporarily stores the generated addition signal in the image storage unit 121.

  When the addition signal is generated, the sensor module 100 then generates an HDR image by combining the long accumulation image and the short accumulation image with the addition signal generated in step S108 as a long accumulation image (step S109). For example, the post-processing unit 132 executes the HDR image generation processing in step S109.

  By executing such an operation, the sensor module 100 according to the embodiment of the present disclosure can generate an HDR image so that the subject does not become an unnatural image when generating the HDR image based on the exposure ratio.

  Here, in step S106 of the above-described series of processing, it will be described that the processing is divided depending on whether the exposure ratio between the long accumulation image and the short accumulation image is larger than 2.

  FIG. 7 is an explanatory diagram illustrating a selection example of the short accumulation image based on the exposure ratio when the HDR image is generated by the sensor module 100 according to the embodiment of the present disclosure. In the example shown in FIG. 7, exposure is performed so that there is no free time between the first frame and the second frame.

  When the exposure ratio is 2 or less, the sensor module 100 selects the exposure signal of the second frame as the short accumulation image. On the other hand, when the exposure ratio exceeds 2, the sensor module 100 selects the exposure signal of the first frame as the short accumulation image.

  As described above, the sensor module 100 switches the frame to be selected as the short accumulation image between the case where the exposure ratio is 2 or less and the case where it exceeds 2, so that the first frame and the second frame are generated when the long accumulation image is generated. It is possible to prevent blanks from occurring in the frame. That is, the sensor module 100 according to the embodiment of the present disclosure generates an HDR image according to an exposure ratio by switching a frame to be selected as a short accumulation image depending on whether the exposure ratio is 2 or less and exceeds 2. Sometimes an HDR image can be generated so that the subject does not become an unnatural image.

  In the example described above, the sensor module 100 switches the frame to be selected for the short accumulation image depending on whether the exposure ratio is 2 or less and exceeds 2. However, the sensor module 100 switches the frame to be selected for the short accumulation image. The exposure ratio is not limited to such an example.

  Further, in the above-described example, the sensor module 100 switches the frame to be selected as the short accumulation image depending on whether the exposure ratio is 2 or less and exceeds 2, but the exposure ratio is around 2 depending on the brightness of the subject. In such a situation, the frame selected for the short accumulation image is frequently switched. When the frame to be selected for the short image is frequently switched, the direction of blurring of the subject is frequently switched when the generated HDR image includes a moving subject.

  Therefore, in order to avoid frequent switching of the frame selected for the short accumulation image, the sensor module 100 has hysteresis in switching of the frame selected for the short accumulation image in the vicinity of the exposure ratio at which the frame selected for the short accumulation image is switched. May be provided. For example, when the exposure ratio changes in the direction in which the exposure ratio increases, the sensor module 100 determines to use the exposure signal of the first frame as the short accumulation image when the exposure ratio becomes 2.1 times instead of 2 times, for example. However, when the exposure ratio changes in a decreasing direction, the exposure signal of the second frame may be determined to be a short accumulation image when the exposure ratio becomes 1.9 times instead of doubling.

  As described above, the sensor module 100 frequently switches the frame to be selected for the short accumulation image by providing hysteresis in the switching of the frame to be selected for the short accumulation image in the vicinity of the exposure ratio at which the frame to be selected for the short accumulation image is switched. You can avoid that.

  Depending on the condition of the exposure time, the subject may be blown out if it is not exposed in a very short time. Depending on the conditions of the exposure time, a blank may occur between the first frame and the second frame. For example, the read time of the first frame and the exposure time of the second frame overlap. FIG. 8 is an explanatory diagram showing an example in which exposure of two frames is performed in an extremely short time. For example, when the exposure ratio is high, the sensor module 100 sets the first frame as a short accumulation image. However, if a blank occurs between the first frame and the second frame as shown in FIG. If a moving subject is included in the image used as the subject, the subject becomes unnatural.

  Therefore, when a blank occurs between the first frame and the second frame depending on the exposure time condition, the sensor module 100 does not combine the long accumulation image and the short accumulation image, but only the short accumulation image. It may be output. The sensor module 100 does not combine the long accumulation image and the short accumulation image depending on the condition of the exposure time, so that the subject does not become unnatural even when capturing a moving subject. Such an image can be generated.

<2. Summary>
As described above, according to the embodiment of the present disclosure, when generating an HDR image with an expanded dynamic range by combining a plurality of images, a frame to be used as a short accumulation image is selected according to the exposure ratio. A sensor module 100 is provided.

  The sensor module 100 according to the embodiment of the present disclosure generates a first image by exposing pixels to the first exposure based on the first exposure time, and then performs the second exposure based on the second exposure time following the first image. A second image is generated by exposing the pixel. When the sensor module 100 according to the embodiment of the present disclosure synthesizes a synthesized image obtained by synthesizing the first image and the second image and a selection image selected from either the first image or the second image. The image to be selected as the selected image is switched at the time point when the relationship between the first exposure time and the second exposure time satisfies a predetermined condition.

  The sensor module 100 according to the embodiment of the present disclosure has such a configuration, and when generating an HDR image, by selecting a frame to be used as a short accumulation image according to an exposure ratio, a moving subject Even when an image is picked up, an HDR image can be generated so that the subject does not become unnatural.

  Each step in the processing executed by each device in the present specification does not necessarily have to be processed in time series in the order described as a sequence diagram or flowchart. For example, each step in the processing executed by each device may be processed in an order different from the order described as the flowchart, or may be processed in parallel.

  In addition, it is possible to create a computer program for causing hardware such as a CPU, ROM, and RAM incorporated in each device to exhibit functions equivalent to the configuration of each device described above. A storage medium storing the computer program can also be provided. In addition, by configuring each functional block shown in the functional block diagram with hardware or a hardware circuit, a series of processing can be realized with hardware or a hardware circuit.

  The preferred embodiments of the present disclosure have been described in detail above with reference to the accompanying drawings, but the technical scope of the present disclosure is not limited to such examples. It is obvious that a person having ordinary knowledge in the technical field of the present disclosure can come up with various changes or modifications within the scope of the technical idea described in the claims. Of course, it is understood that it belongs to the technical scope of the present disclosure.

  Further, the effects described in the present specification are merely illustrative or exemplary and are not limited. That is, the technology according to the present disclosure can exhibit other effects that are apparent to those skilled in the art from the description of the present specification in addition to or instead of the above effects.

The following configurations also belong to the technical scope of the present disclosure.
(1)
A first image is generated by exposing pixels to the first exposure with the first exposure time, and a second image is generated by exposing pixels to the second exposure with the second exposure time following the first image. A control unit;
A synthesis unit that synthesizes a synthesized image obtained by synthesizing the first image and the second image, and a selection image selected from either the first image or the second image;
With
The control unit includes the first exposure time and the first exposure time so that an image selected as the selected image by the combining unit is switched based on an exposure ratio that is a ratio of an exposure time of the combined image and an exposure time of the selected image. An imaging control apparatus that determines a second exposure time.
(2)
The imaging control apparatus according to (1), wherein the control unit causes the pixel to be exposed under a condition that no empty time is provided between the first exposure and the second exposure.
(3)
An output unit that outputs at least one of the image synthesized by the synthesis unit or the selected image;
The imaging control apparatus according to (1) or (2), wherein the output unit outputs only the selected image when the lead time of the first exposure and the second exposure time overlap in time.
(4)
The imaging control apparatus according to claim 1, wherein the first exposure time is equal to or shorter than the second exposure time.
The imaging control device according to any one of (1) to (3).
(5)
When the exposure ratio is larger than 2, the first image is selected as the selected image, and when the exposure ratio is smaller than 2, the second image is selected as the selected image. The imaging control device described.
(6)
The control unit determines the first exposure time and the second exposure time so that hysteresis occurs when the image selected as the selection image by the combining unit is switched. Any one of (1) to (5) An imaging control device according to claim 1.
(7)
It is configured by stacking three semiconductor substrates consisting of a first semiconductor substrate, a second semiconductor substrate, and a third semiconductor substrate,
The first semiconductor substrate includes at least the pixel.
A storage unit for storing the first image and the second image is formed on the second semiconductor substrate,
The image processing apparatus according to any one of (1) to (6), wherein at least the control unit is formed on the third semiconductor substrate.
(8)
The image processing apparatus according to (7), wherein the second semiconductor substrate is provided between the first semiconductor substrate and the third semiconductor substrate.
(9)
The image processing apparatus according to (7), wherein the third semiconductor substrate is provided between the first semiconductor substrate and the second semiconductor substrate.
(10)
A first image is generated by exposing pixels to the first exposure with the first exposure time, and a second image is generated by exposing pixels to the second exposure with the second exposure time following the first image. And
Synthesizing a synthesized image obtained by synthesizing the first image and the second image, and a selection image selected from either the first image or the second image;
Determining the first exposure time and the second exposure time based on an exposure ratio which is a ratio of an exposure time of the composite image and an exposure time of the selected image so that an image selected as the selected image is switched; When,
An imaging control method.
(11)
On the computer,
A first image is generated by exposing pixels to the first exposure with the first exposure time, and a second image is generated by exposing pixels to the second exposure with the second exposure time following the first image. And
Synthesizing a synthesized image obtained by synthesizing the first image and the second image, and a selection image selected from either the first image or the second image;
Determining the first exposure time and the second exposure time based on an exposure ratio which is a ratio of an exposure time of the composite image and an exposure time of the selected image so that an image selected as the selected image is switched; When,
A computer program that executes
(12)
An electronic apparatus comprising the imaging control device according to any one of (1) to (10).

10 Electronic device 100 Sensor module

Claims (12)

A first image is generated by exposing pixels to the first exposure with the first exposure time, and a second image is generated by exposing pixels to the second exposure with the second exposure time following the first image. A control unit;
A synthesis unit that synthesizes a synthesized image obtained by synthesizing the first image and the second image, and a selection image selected from either the first image or the second image;
With
The control unit includes the first exposure time and the first exposure time so that an image selected as the selected image by the combining unit is switched based on an exposure ratio that is a ratio of an exposure time of the combined image and an exposure time of the selected image. An imaging control apparatus that determines a second exposure time.   The imaging control apparatus according to claim 1, wherein the control unit causes the pixel to be exposed under a condition that no empty time is provided between the first exposure and the second exposure. An output unit that outputs at least one of the image synthesized by the synthesis unit or the selected image;
The imaging control apparatus according to claim 1, wherein the output unit outputs only the selected image when the lead time of the first exposure and the second exposure time overlap in time.   The imaging control apparatus according to claim 1, wherein the first exposure time is equal to or shorter than the second exposure time.   5. The first image is selected as the selected image when the exposure ratio is greater than 2, and the second image is selected as the selected image when the exposure ratio is less than 2. Imaging control device.   The imaging control apparatus according to claim 1, wherein the control unit determines the first exposure time and the second exposure time so that hysteresis occurs when an image selected as the selection image by the combining unit is switched. It is configured by stacking three semiconductor substrates consisting of a first semiconductor substrate, a second semiconductor substrate, and a third semiconductor substrate,
The first semiconductor substrate includes at least the pixel.
A storage unit for storing the first image and the second image is formed on the second semiconductor substrate,
The image processing apparatus according to claim 1, wherein at least the control unit is formed on the third semiconductor substrate.   The image processing apparatus according to claim 7, wherein the second semiconductor substrate is provided between the first semiconductor substrate and the third semiconductor substrate.   The image processing apparatus according to claim 7, wherein the third semiconductor substrate is provided between the first semiconductor substrate and the second semiconductor substrate. A first image is generated by exposing pixels to the first exposure with the first exposure time, and a second image is generated by exposing pixels to the second exposure with the second exposure time following the first image. And
Synthesizing a synthesized image obtained by synthesizing the first image and the second image, and a selection image selected from either the first image or the second image;
Determining the first exposure time and the second exposure time based on an exposure ratio which is a ratio of an exposure time of the composite image and an exposure time of the selected image so that an image selected as the selected image is switched; When,
An imaging control method. On the computer,
A first image is generated by exposing pixels to the first exposure with the first exposure time, and a second image is generated by exposing pixels to the second exposure with the second exposure time following the first image. And
Synthesizing a synthesized image obtained by synthesizing the first image and the second image, and a selection image selected from either the first image or the second image;
Determining the first exposure time and the second exposure time based on an exposure ratio which is a ratio of an exposure time of the composite image and an exposure time of the selected image so that an image selected as the selected image is switched; When,
A computer program that executes   An electronic apparatus comprising the imaging control device according to claim 1.

Images