TWI803424B - Dynamic image generating method and dynamic image sensor thereof - Google Patents

Abstract

The present invention relates to a dynamic image generating method and a dynamic image sensor thereof. The dynamic image sensor includes a first exposure pixel, a second exposure pixel and an image processing module. The dynamic image sensor applies the dynamic image generating method, which generates the short-exposure image signal to be tested by exposing the short-exposure time to be tested through the first exposure pixel, and the second exposure pixel exposes the to-be-measured long-exposure time to generate the to-be-measured long-exposure image information. The image processing module confirms whether the short-exposure image signal to be measured and the long-exposure image signal to be measured are higher than the lower limit of pinching or lower than the upper limit of pinching. If so, the optimal short exposure time and the optimal long exposure time are generated; otherwise, increase the short exposure time to be measured by the short exposure fixed value or decrease the long exposure time to be measured by the long exposure fixed value, and repeat the automatic exposure time method until the optimal short exposure time and the optimal long exposure time are obtained.

Description

Dynamic image generation method and dynamic image sensor thereof

The present invention relates to a dynamic image generation method, in particular to a dynamic image generation method with high detection rate and high dynamic range and a dynamic image sensor thereof.

In recent years, the demand for the self-driving car industry has become increasingly vigorous. For self-driving cars, image sensors for detecting real-time road conditions are essential components. Currently, the dynamic vision sensor (Dynamic Vision Sensor, DVS) is the mainstream image sensor used to detect real-time road conditions. The reason is that the dynamic vision sensor records images in units of events. Information. This dynamic event-based sensor brings machine autonomy closer to reality, making it suitable for vision-based high-speed applications in the field of autonomous vehicles.

However, during driving on the road, occasional situations where the illumination level changes drastically within a short period of time may cause a partial (or entire) temporary over-exposure (or under-exposure) of the image sensor. During this short period of time, the image recognition algorithm of the self-driving car cannot make correct object detection in the face of insufficient detail images, thus increasing the risk of accidents.

Therefore, how to optimize the image sensor so that the image sensor can quickly adjust the exposure level at a very high frame rate to produce the best exposure time, and through tone mapping of the same frame of photos And exposure fusion to generate high dynamic range images, so that the image details can have sufficient clarity and recognition under various illumination environments, which is one of the problems that developers need to solve urgently.

Therefore, the inventor of this case came up with the present invention after observing the above-mentioned deficiency.

The object of the present invention is to provide a dynamic image sensor, by exposing the first exposure pixels for a short exposure time, while the second exposure pixels are exposed for a long exposure time, and generating high dynamic range image information through tone mapping and exposure fusion. In this way, the present invention utilizes the design of long and short exposure arrays to expose and fuse image information that is exposed at different times and with different exposure times within the same frame time, achieving the effects of enhancing the clarity and brightness of moving images to improve The accuracy of the identification algorithm for identification of dynamic images reduces the risk of accidents.

Another object of the present invention is to provide a dynamic image generation method, which uses the image processing module to increase the short exposure time to be measured by a short exposure fixed value to form a new short exposure time to be measured, or reduce the long exposure time to be measured The long exposure setting value forms a new long exposure time to be measured, so as to produce the best short exposure time and the best long exposure time. In this way, the present invention exchanges the spatial resolution for additional exposure information, which is beneficial to converge to the optimal exposure time in a short time, so that the exposure of high dynamic range image information is good. In addition, through the calculation method of simultaneously pinching the upper and lower ranges, the automatic exposure time method of the present invention can realize a dynamic calculation method to generate the optimal exposure time to adapt to various shooting environments, and at the same time reduce the calculation time for generating the optimal exposure time. The efficiency and applicability of the present invention are greatly improved.

Another object of the present invention is to provide a dynamic image generation method, which processes the short-exposure image signal and the long-exposure image signal through binarization, and multiplies the binarized short-exposure image signal by the short-exposure weight. The transformed long-exposure image signal is multiplied by the long-exposure weight, wherein the long-exposure weight is greater than the short-exposure weight. This solves the problem of unclear dynamic images in low-brightness environments, making the pixels with high brightness have higher values and those with low brightness lower, which increases the contrast of object outlines in the image and further improves the clarity of dynamic images.

In order to achieve the above object, the present invention provides a method for generating a dynamic image with a high frame rate and a high dynamic range, which is applied in an environment where a dynamic image sensor receives a dynamic image in a dynamic range, and the dynamic image sensor includes a first Exposure pixels, a second exposure pixel and an image processing module, the first exposure pixel and the second exposure pixel are coupled to the image processing module, the dynamic image generation method includes the following steps: a simultaneous exposure step, the The first exposure pixel is exposed for a short exposure time to generate a short exposure image signal, and the second exposure pixel is exposed for a long exposure time to generate a long exposure image signal; a tone mapping step, the image processing module generates the short exposure image signal And the long-exposure image signal executes a tone mapping (tone mapping) algorithm to generate a short-exposure image information and a long-exposure image information; an exposure fusion step, the image processing module uses the short-exposure image information and the long-exposure image information Exposure fusion is performed on the image information to generate a high dynamic range image information; and an output step, the image processing module outputs the high dynamic range image information.

Preferably, according to the dynamic image generation method of the present invention, the tone mapping algorithm is selected from one of the Gamma curve algorithm and the Academy Color Encoding System curve algorithm .

Preferably, the dynamic image generation method according to the present invention, wherein, before the simultaneous exposure step, the dynamic image generation method further includes an automatic exposure time method, and the automatic exposure time method includes the following steps: a testing step, the first An exposure pixel is exposed for a short exposure time to be measured, and the second exposure pixel is exposed for a long exposure time to be measured, so as to generate a short exposure image signal to be measured and a long exposure image signal to be measured; a determination step, the image processing The module confirms whether the signal of the short-exposure image to be tested is higher than a pinch lower limit, and confirms whether the signal of the long-exposure image to be tested is lower than a pinch upper limit; Time increases a short exposure fixed value to produce the new short exposure time to be measured, or reduces the long exposure time to be measured by a long exposure fixed value to generate a new long exposure time to be measured; an optimal exposure time generation step , the image processing module generates an optimal short exposure time and an optimal long exposure time according to the short exposure time to be measured and the long exposure time to be measured; wherein, if the determination step determines that the signal of the short exposure image to be measured is high When the pinching lower limit is reached, and the long-exposure image signal to be measured is lower than the pinching upper limit, the optimal exposure time generating step is performed after the determination step is completed, and in the simultaneous exposure step, the first exposure pixel exposes the optimal exposure time. The optimal short exposure time, while the second exposure pixel is exposed to the optimal long exposure time; on the contrary, the adjustment step is executed after the determination step and the testing step is repeatedly executed.

Preferably, according to the dynamic image generation method of the present invention, wherein, a short exposure time upper limit and a long exposure time lower limit are stored in the dynamic image sensor, and the short exposure time upper limit is the The maximum value of the short exposure time is measured, and the lower limit value of the long exposure time is the minimum value of the long exposure time to be measured.

Preferably, in the dynamic image generation method according to the present invention, if the short exposure time to be measured is the upper limit of the short exposure time, and the signal of the short exposure image to be measured is still lower than the lower limit of pinching in the determining step , the automatic exposure time method further includes a superposition step, the superposition step is for the image processing module to superimpose the short-exposure image signal to be tested to generate the short-exposure image signal.

Preferably, according to the dynamic image generation method of the present invention, the dynamic image sensor further includes a brightness sensing module, which is coupled to the image processing module, and the brightness sensing module is used for Sensing the brightness of the environment where the dynamic image sensor is located, the exposure time adjustment method further includes an upper and lower limit adjustment step, the image processing module adjusts the pinching lower limit according to the brightness sensed by the brightness sensing module and The pinching upper limit, when the brightness sensed by the brightness sensing module is lower than a low brightness value, then correspondingly reduce the pinching lower limit, when the brightness sensed by the brightness sensing module is higher than a high brightness value, then The pinch upper limit is raised correspondingly.

Preferably, according to the dynamic image generation method of the present invention, the tone mapping step includes: a critical value generation step, the image processing module averages the pixel values of the short-exposure image signal to generate a short Exposure brightness threshold, and average the pixel values of the long-exposure image signal to generate a long-exposure brightness threshold; a high dynamic range synthesis step, the image processing module according to the short-exposure brightness threshold, the The pixel values in the short-exposure image signal that are higher than the short-exposure brightness threshold are defined as 1, and the pixel values in the short-exposure image signal that are lower than or equal to the short-exposure brightness threshold are defined as 0, and the image The processing module defines the pixel values in the long-exposure image signal higher than the long-exposure brightness threshold as 1 according to the long-exposure brightness threshold, and defines the pixel values in the long-exposure image signal as being lower than or equal to the long-exposure brightness The pixel values of the critical value are defined as 0; a weighting calculation step, the image processing module multiplies the binarized short-exposure image signal by a short-exposure weight, and the image processing module multiplies the binarized short-exposure image signal The long-exposure image signal is multiplied by a long-exposure weight, wherein the long-exposure weight is greater than the short-exposure weight.

Moreover, in order to achieve the above object, the present invention further provides a dynamic image sensor with a high frame rate and high dynamic range based on the above dynamic image generation method, which is applied in an environment for receiving dynamic images with a dynamic range. The dynamic image sensor includes: a sensing array including a plurality of first exposure pixels and a plurality of second exposure pixels, the first exposure pixels have a short exposure time, and the second exposure pixels have a Long exposure time; an image processing module, which is coupled to the sensing array; wherein, the first exposure pixel is exposed to the short exposure time to generate a short exposure image signal, while the second exposure pixel is exposed to the long exposure time generating a long exposure image signal, the image processing module performs a tone mapping algorithm on the short exposure image signal and the long exposure image signal to generate a short exposure image information and a long exposure image information, and the image processing module Exposure fusion is performed on the short-exposure image information and the long-exposure image information to generate a high dynamic range image information.

Preferably, in the dynamic image sensor according to the present invention, the first exposure pixels and the second exposure pixels are arranged alternately, and the number of the first exposure pixels is the number of the second exposure pixels One of 1 times, 2 times and 3 times.

To sum up, the dynamic image generation method provided by the present invention mainly utilizes the dynamic image sensor of the present invention to expose the first exposure pixel for a short exposure time, while the second exposure pixel is exposed for a long exposure time, and through tone mapping and exposure Fusion produces high dynamic range image information. In this way, the present invention utilizes the design of long and short exposure arrays to expose and fuse image information that is exposed at different times and with different exposure times within the same frame time, thereby achieving the effects of enhancing the clarity and brightness of moving images, and greatly improving The accuracy of the identification algorithm for identification of dynamic images reduces the risk of accidents. In addition, by the image processing module, the short exposure time to be measured is increased by a short exposure fixed value to form a new short exposure time to be measured, or the long exposure time to be measured is reduced by a long exposure fixed value to form a new long exposure time to be measured. Produces the best short exposure times as well as the best long exposure times. In this way, the present invention exchanges the spatial resolution for additional exposure information, which is beneficial to converge to the optimal exposure time in a short time, so that the exposure of high dynamic range image information is good.

In order to enable those skilled in the art to understand the purpose, features and effects of the present invention, the present invention will be described in detail below by means of the following specific embodiments and accompanying drawings.

The inventive concept will now be explained more fully hereinafter with reference to the accompanying drawings, in which exemplary embodiments of the inventive concept are shown. Advantages and features of the inventive concept and methods for achieving it will be apparent below by referring to the exemplary embodiments described in more detail with reference to the accompanying drawings. It should be noted, however, that the inventive concept is not limited to the following exemplary embodiments, but can be implemented in various forms. Therefore, the exemplary embodiments are provided only to disclose the inventive concept and to make one skilled in the art understand the category of the inventive concept. In the drawings, the exemplary embodiments of the inventive concepts are not limited to the specific examples provided herein and are exaggerated for clarity.

The terminology used herein is for describing particular embodiments only and is not intended to be limiting of the invention. As used herein, the terms "a", "an" and "the" in the singular are intended to include the plural forms as well, unless the context clearly dictates otherwise. As used herein, the term "and/or" includes any and all combinations of one or more of the associated listed items. It will be understood that when an element is referred to as being "connected" or "coupled" to another element, it can be directly connected or coupled to the other element or intervening elements may be present.

Similarly, it will be understood that when an element such as a layer, region or substrate is referred to as being "on" another element, it can be directly on the other element or intervening elements may be present. In contrast, the term "directly" means that there are no intervening elements. It should be further understood that when the words "comprising" and "comprising" are used herein, it indicates the existence of stated features, integers, steps, operations, elements, and/or components, but does not exclude one or more other features. , integers, steps, operations, elements, components, and/or the presence or addition of groups thereof.

Furthermore, exemplary embodiments in the detailed description will be explained by means of cross-sectional views that are idealized exemplary views of the inventive concept. Accordingly, the shapes of the exemplary figures may be modified according to manufacturing techniques and/or allowable errors. Accordingly, exemplary embodiments of the inventive concepts are not limited to the specific shapes shown in the exemplary figures, but may include other shapes that may be produced according to manufacturing processes. Regions illustrated in the drawings have general characteristics and are used to illustrate specific shapes of elements. Accordingly, this should not be seen as limiting the scope of the inventive concept.

It should also be understood that although the terms “first”, “second”, “third” etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish various elements. Thus, a first element in some embodiments could be termed a second element in other embodiments without departing from the teachings of the present invention. Exemplary embodiments of aspects of the inventive concept illustrated and illustrated herein include their complementary counterparts. Throughout this specification, the same reference number or the same designator designates the same element.

Additionally, exemplary embodiments are described herein with reference to cross-sectional and/or plan views, which are idealized exemplary illustrations. Accordingly, deviations from the illustrated shapes as a result, for example, of manufacturing techniques and/or tolerances are to be expected. Thus, example embodiments should not be construed as limited to the shapes of regions illustrated herein but are to include deviations in shapes that result, for example, from manufacturing. Thus, the regions shown in the figures are schematic and their shapes are not intended to illustrate the actual shape of a region of a device and are not intended to limit the scope of example embodiments.

Please refer to FIG. 1 , which is a schematic diagram of a dynamic image sensor according to the present invention. As shown in FIG. 1 , a dynamic image sensor 100 according to the present invention includes: a sensing array 11 and an image processing module 12 .

Specifically, the dynamic image sensor 100 according to the present invention may be a complementary metal oxide semiconductor (Complementary Metal-Oxide Semiconductor, CMOS) image sensor, and may be selected from a back-illuminated CMOS image sensor or a front-illuminated CMOS image sensor. One of the type CMOS image sensors, but the present invention is not limited thereto.

Specifically, as shown in FIG. 1, the sensing array 11 according to the present invention includes a plurality of first exposure pixels 111 and a plurality of second exposure pixels 112, the first exposure pixels 111 have a short exposure time, and the second exposure pixels 111 have a short exposure time. The exposure pixels 112 have a long exposure time. In some embodiments, each of the first exposure pixels 111 and the second exposure pixels 112 are CMOS imaging pixels. More specifically, in some embodiments, the short exposure time may be 256 times different from the long exposure time, and the short exposure time is at least one shutter speed of the shutter of the motion image sensor 100, however The present invention is not limited thereto.

Specifically, in some embodiments, the dynamic image sensor 100 according to the present invention may have one of a rolling shutter (Rolling Shutter) mechanism and a global shutter (Global Shutter) mechanism, wherein the rolling shutter mechanism is used At this time, due to the time difference in the exposure of the rolling shutter mechanism, when the dynamic image sensor 100 shoots a dynamic image, the exposure time of the upper and lower half of the image is different, which may cause the upper half of the image to appear first, but the lower end of the image has not yet The time gap that appears causes the image to be distorted and deformed. In a preferred embodiment of the present invention, since the present invention is mainly aimed at generating dynamic images in the dynamic range, in order to avoid the jello effect (Jello Effect) of dynamic images, a global shutter mechanism is adopted, that is, on the sensor array 11 All the first exposure pixels 111 and the second exposure pixels 112 are simultaneously exposed to obtain image signals or image charges, but the present invention is not limited thereto.

It should be further explained that since the present invention relates to tone mapping of images exposed at different times and with different exposure times, the tone mapping will be described below first. Tone mapping refers to the conversion of lighting results from high dynamic range image information (High Dynamic Range, HDR) to low dynamic range image information (Low Dynamic Range, LDR) that can be displayed normally on a display device. In the scene, the high dynamic range image information whose brightness (brightness) maximum and minimum values differ within 106 orders of magnitude is processed by the tone mapping algorithm and mapped to a low dynamic range (such as 8-bit Yuan) image information, while still retaining the color details and shading changes in the original shooting scene. Since the tone mapping algorithm is familiar to those skilled in the art, it will not be repeated here.

It is worth mentioning that since the present invention relates to image exposure fusion with different exposure times and different exposure times, the exposure fusion will be described below first. The exposure fusion method involved in this paper is based on the first exposure pixels 111 and the second exposure pixels 112 , and the exposure fusion process generally includes four steps of preprocessing, transformation, synthesis and inverse transformation. Preprocessing generally refers to preparation operations before exposure fusion, which will not be repeated here. The main methods of transformation in the exposure fusion process generally include Principal Component Analysis (PCA) method, HIS (Intensity-Hue-Saturation) transformation method, wavelet transformation method, multi-resolution method such as Laplacian pyramid fusion algorithm, etc. one or a combination thereof. The synthesis in the exposure fusion process refers to the comprehensive processing of the transformation results of the image signals generated by the first exposure pixel 111 and the second exposure pixel 112. There are many processing methods for the comprehensive processing, such as: selection method, weighted average method and optimization Among them, the selection method can be to select the corresponding transformation coefficients from the original image sequence to form a new set of transformation coefficients, and the weighted average method can determine the weight of the transformation coefficients corresponding to the image signal according to specific rules, and through The weighted average operation produces new coefficients. The inverse transformation in the exposure fusion process refers to the corresponding inverse operation of the new transformation coefficients of the integrated image signal, for example, the radiance map is reversed from the existing image to generate the final fused image .

Specifically, as shown in FIG. 1, the image processing module 12 according to the present invention is coupled to the sensing array 11. The image processing module 12 is mainly used to execute an algorithm. In some embodiments, the algorithm The legal system includes a tone mapping algorithm, and the tone mapping algorithm can be selected from one of the gamma curve algorithm and the professional color coding system curve algorithm. More specifically, the image processing module 12 may include one or a combination of a server, a personal computer, and an Application Specific Integrated Circuit (ASIC), but the present invention is not limited thereto.

In order to further understand the structural characteristics of the present invention, the use of technical means and the expected effect, the use of the present invention will be described hereby. It is believed that the present invention can be understood more deeply and specifically. Please refer to Figure 2. 2 is a block diagram illustrating the steps of implementing the dynamic image generation method of the present invention. The method for generating dynamic images with high frame rate and high dynamic range according to the present invention comprises the following steps:

In the simultaneous exposure step S11 , the first exposure pixel 111 is exposed for a short exposure time to generate a short exposure image signal, while the second exposure pixel 112 is exposed for a long exposure time to generate a long exposure image signal, and then the tone mapping step S12 is performed.

In the tone mapping step S12, the image processing module 12 executes the tone mapping algorithm on the short-exposure image signal and the long-exposure image signal to generate short-exposure image information and long-exposure image information, and then executes the exposure fusion step S13.

In the exposure fusion step S13 , the image processing module 12 performs exposure fusion on the short-exposure image information and the long-exposure image information to generate high dynamic range image information, and then executes the output step S14 .

In the outputting step S14, the image processing module 12 outputs the high dynamic range image information.

In order to further understand the structural features, technical means and expected effects of the present invention, the actual implementation process of the first embodiment of the present invention will be described, and it is believed that the present invention can be understood more deeply and specifically as follows. Said:

Please refer to FIG. 3 , together with FIG. 1 and FIG. 2 , FIG. 3 is a flow chart illustrating the actual execution process of the dynamic image generation method according to the present invention. The actual execution process of the dynamic image sensor 100 according to the present invention is described as follows: First, the simultaneous exposure step S11 is performed, the first exposure pixel 111 is exposed for a short exposure time 21 to generate a short exposure image signal 23, and at the same time the second exposure pixel 112 is exposed for a long exposure Time 22 to generate a long-exposure image signal 24; then, execute the tone mapping step S12, the image processing module 12 executes the tone-mapping algorithm on the short-exposure image signal 23 and the long-exposure image signal 24 to generate short-exposure image information 25 and long-exposure image information 26; afterward, execute the exposure fusion step S13, the image processing module 12 performs exposure fusion on the short exposure image information 25 and the long exposure image information 26, and generate high dynamic range image information 27; finally, execute the output step S14, the image processing module Group 12 outputs high dynamic range image information 27 .

Therefore, it can be seen from the above description that the dynamic image generation method according to the present invention uses the first exposure pixel 111 to expose for a short exposure time 21, while the second exposure pixel 112 exposes for a long exposure time 22, and generates high Dynamic Range Image Information 27 . In this way, the present invention utilizes the design of the long and short exposure arrays to perform exposure and fusion of image information within one frame, thereby achieving effects such as enhancing the definition and brightness of moving images.

Hereinafter, the first embodiment of the dynamic image sensor 100 of the present invention will be described with reference to the drawings, so that those skilled in the art of the present invention can understand possible changes more clearly. Components denoted by the same reference numerals as above are substantially the same as those described above with reference to FIG. 1 . The same components, features, and advantages as those of the dynamic image sensor 100 will not be repeated.

Please refer to FIG. 4 , which is a schematic diagram of a dynamic image sensor according to a first embodiment of the present invention. As shown in FIG. 4 , the dynamic image sensor 100A according to the first embodiment of the present invention includes: a sensing array 11A and an image processing module 12A.

Specifically, as shown in FIG. 4 , in this embodiment, the sensing array 11A according to the present invention includes a plurality of first exposure pixels 111A and a plurality of second exposure pixels 112A, the first exposure pixels 111A and the second exposure pixels The pixels 112A are arranged in a staggered manner, and the number of the first exposure pixels 111A and the second exposure pixels 112A are equal, wherein, the first exposure pixels 111A have a short exposure time, and the second exposure pixels 112A have a long exposure time, and the long exposure time can be short exposure One of 2 times, 4 times and 8 times of the time, but the present invention is not limited thereto. It can be understood that the staggered arrangement of the first exposure pixels 111A and the second exposure pixels 112A can ensure that each block of the motion image generated by the motion image sensor 100A of the present invention has the same long exposure time and short exposure times. In addition, the number of the first exposure pixels 111A and the second exposure pixels 112A in the sensing array 11A of the present invention, as well as the relationship between the long exposure time and the short exposure time can be determined according to the environment in which the motion image sensor 100A is used. Modification, for example, when the dynamic image sensor 100A is used in a brighter environment, the number of the first exposure pixels 111A with a short exposure time may be less than the second exposure pixels 112A with a long exposure time, and The long exposure time may have a higher multiple than the short exposure time to ensure that the dynamic image generated by the dynamic image sensor 100A has better clarity and brightness, but the invention is not limited thereto.

It should be further explained that, in this embodiment, the tone mapping algorithm of the image processing module 12A is selected from the Gamma curve (Gamma curve) algorithm and the professional color coding system curve (Academy Color Encoding System curve) S algorithm One of them, but the present invention is not limited thereto.

It is worth mentioning that, in this embodiment, the image processing module 12A can store a pinch upper limit and a pinch lower limit, wherein the pinch upper limit can represent the threshold for judging overexposure, and pinch The lower limit value may represent a threshold for judging too dark, and the lower limit of video signal and pinch may be, for example, a DN (Digital Number) value or a value converted therefrom. For example, when the image processing module 12A increases the exposure time to the maximum, but the image signal to be tested is still lower than the lower limit of pinching, the image processing module 12A can determine that the current environment is too dark. , the image processing module 12A can perform image overlay through the image signal to be tested to generate a brighter and more detailed image signal, but the present invention is not limited thereto.

It is worth mentioning that, in this embodiment, the image processing module 12A can store a long exposure setting value and a short exposure setting value, and the user can also adjust the long exposure setting value according to the brightness of the current environment through the image processing module 12A. value and short exposure setting. For example, when the environment is brighter, the short exposure setting value can have a larger value and the long exposure setting value can have a smaller value, so as to generate the optimal exposure time faster and effectively save the dynamic image sensor The number of operations and the time of image capture performed by the device 100A to generate the optimal exposure time can effectively save the power of the electronic device.

Please refer to FIG. 5 . FIG. 5 is a block diagram illustrating the steps of the method for generating a dynamic image according to the first embodiment of the present invention. The present invention is based on the dynamic image sensor 100A of the first embodiment, and further provides a dynamic image generation method for implementing the dynamic image sensor 100 of the first embodiment, which includes the following steps:

In the test step S21, the first exposure pixel 111A is exposed for a short exposure time to be tested, and at the same time, the second exposure pixel 112A is exposed for a long exposure time to be measured, so as to generate a short exposure image signal to be tested and a long exposure image signal to be tested, and then execute the determination step S22 .

In the determination step S22 , the image processing module 12A confirms whether the short-exposure image signal to be tested is higher than a lower limit of pinching, and confirms whether the signal of the long-exposure image to be tested is lower than an upper limit of pinching.

In the adjustment step S23, the image processing module 12A increases the short exposure time to be measured by a short exposure constant value, or decreases the long exposure time to be measured by a long exposure constant value to form a new short exposure time to be measured and a new Long exposure time to be measured.

In the optimal exposure time generating step S24 , the image processing module 12A generates an optimal short exposure time and an optimal long exposure time according to the short exposure time to be measured and the long exposure time to be measured.

Simultaneous exposure step S25 , the first exposure pixel 111A is exposed for an optimal short exposure time to generate a short exposure image signal, while the second exposure pixel 112A is exposed for an optimal long exposure time to generate a long exposure image signal, and then the tone mapping step S26 is performed.

In the tone mapping step S26, the image processing module 12A executes the tone mapping algorithm on the short-exposure image signal and the long-exposure image signal to generate short-exposure image information and long-exposure image information, and then executes the exposure fusion step S27.

In the exposure fusion step S27 , the image processing module 12A performs exposure fusion on the short-exposure image information and the long-exposure image information to generate high dynamic range image information, and then executes the output step S28 .

In the output step S28, the image processing module 12A outputs the high dynamic range image information.

In order to further understand the structural features, technical means and expected effects of the present invention, the actual implementation process of the first embodiment of the present invention will be described. It is believed that the present invention can be understood more deeply and specifically as follows. :

Please refer to FIG. 6 , together with FIG. 4 and FIG. 5 , FIG. 6 is a flow chart illustrating the actual execution process of the dynamic image generation method according to the first embodiment of the present invention. The actual execution process of the dynamic image sensor 100A according to the first embodiment of the present invention is described as follows: First, the test step S21 is executed, the first exposure pixel 111A is exposed to the short exposure time 31A to be tested, and the second exposure pixel 112A is exposed to the long exposure time to be measured The exposure time 32A is used to generate the short-exposure image signal 33A to be tested and the long-exposure image signal 34A to be tested; then, the determination step S22 is performed, and the image processing module 12A confirms whether the short-exposure image signal 33A to be tested is higher than the pinching lower limit 35A, And confirm whether the long-exposure image signal 34A to be measured is lower than the pinch upper limit 36A; wherein, if the determination step S22 determines that the short-exposure image signal 33A to be tested is lower than the pinch lower limit 35A, or the long-exposure image signal 34A to be measured is higher than the pinch lower limit If the upper limit 36A is squeezed, the adjustment step S23 is performed, and the image processing module 12A increases the short exposure time 31A to be measured by the short exposure fixed value 37A to form a new short exposure time to be measured 31A, or decreases the long exposure time to be measured 32A by a long Exposure fixed value 38A, to form a new long exposure time 32A to be measured, and repeat the test step S21; 34A is lower than the pinching upper limit 36A, then execute the optimal exposure time generating step S24, the image processing module 12A generates the optimal short exposure time 39A and the optimal long exposure time according to the short exposure time 31A to be measured and the long exposure time 32A to be measured 40A; after that, the simultaneous exposure step S3725 is executed, the first exposure pixel 111 is exposed to the optimal short exposure time 39A to generate a short exposure image signal 23A, and the second exposure pixel 112 is exposed to the optimal long exposure time 40A to generate a long exposure image signal 24A; then , execute the tone mapping step S3826, the image processing module 12A executes the tone mapping algorithm on the short-exposure image signal 23A and the long-exposure image signal 24A to generate the short-exposure image information 25A and the long-exposure image information 26A; after that, execute the exposure fusion step S3927 , the image processing module 12A performs exposure fusion of the short-exposure image information 25A and the long-exposure image information 26A to generate high dynamic range image information 27A; finally, the output step S4028 is executed, and the image processing module 12A outputs the high dynamic range image information 27A.

For further explanation, please refer to FIG. 7 , which is a schematic diagram illustrating the tone mapping algorithm according to the first embodiment of the present invention. As shown in FIG. 7 , together with FIG. 5 and FIG. 6 , FIG. 7 is a schematic diagram of a single pixel of the short-exposure image signal 23A and the long-exposure image signal 24A generated after adjustment from the testing step S21 to the optimal exposure time generating step S24. Before executing the tone mapping, in this embodiment, the pixel value 51A of the short-exposure image signal 23 and the long-exposure image signal 24A is partially overlapped (shown as two digits in FIG. 7 ), and compared with the long-exposure image signal 24A. Higher weights (Figure 7 is an exemplary representation of 4 times the weight, that is, the operation of shifting two bits to the left in binary representation) are superimposed, so that the higher the pixel value 51A of the long-exposure image signal 24A, the higher the pixel value 51A of the short-exposure image signal 23A. The lower the pixel value 51A is, the effect of high dynamic range image information can be achieved. As shown in FIG. 7 , the superimposed short-exposure image signal 23 and long-exposure image signal 24A are then subjected to tone mapping algorithms, such as global tone mapping functions such as Gamma curve and ACES curve, or for example : Reinhard tone mapping and other methods that consider local pixel characteristics, and map high dynamic range image information back to low dynamic range image information. In this embodiment, the tone mapping algorithm is simply linear transformation, but the invention is not limited thereto.

It is worth mentioning that the dynamic image sensor according to the present invention can store a short exposure time upper limit value (not shown in the figure) and a long exposure time lower limit value (not shown in the figure), wherein the short exposure time upper limit value is the maximum value of the short exposure time 31A to be measured, and the lower limit of the long exposure time is the minimum value of the long exposure time 32A to be measured. The upper limit of the exposure time represents the upper limit of the short exposure time 31A of the first exposure pixel 111A on the hardware, and the lower limit of the long exposure time represents the lower limit of the second exposure pixel 112A on the hardware of the long exposure time 32A. limit. In this way, according to the dynamic image generation method of the first embodiment of the present invention, through the optimal short exposure time 39A and the optimal long exposure time 40A, the distribution of the short exposure image signal 23A and the long exposure image signal 24A of each frame is realized. The probability is as close as possible, and through the upper limit value of the short exposure time and the lower limit value of the long exposure time, it is ensured that the distance between the distribution center (for example: average or median) of the short exposure image signal 23A and the long exposure image signal 24A is not Too close, it can be understood that when the values of the optimal short exposure time 39A and the optimal long exposure time 40A are closer, the technical effect of the present invention will be reduced, and it is not conducive to the subsequent exposure fusion calculation.

Therefore, it can be seen from the above description that according to the dynamic image generation method and its dynamic image sensor 100A according to the first embodiment of the present invention, the short exposure time 31A to be measured can be increased by the short exposure constant value 37A by the image processing module 12A Up to the upper limit of the short exposure time, or reduce the long exposure time 32A to be measured from the long exposure fixed value 38A to the lower limit of the long exposure time to generate the best short exposure time 39A and the best long exposure time 40A. In this way, the present invention exchanges the spatial resolution for additional exposure information, which is beneficial to converge to the optimal exposure time in a short time, so that the high dynamic range image information 27A is exposed well. In addition, through the computing method of pinching, the automatic exposure time method of the present invention can realize a dynamic calculation method to generate the optimal exposure time to adapt to various shooting environments, and at the same time reduce the computing time for generating the optimal exposure time, greatly improving the cost. Efficiency and applicability of the invention.

Other examples of the dynamic image sensor 100 are provided below, so that those skilled in the art of the present invention can understand possible variations more clearly. Components indicated by the same reference numerals as in the above embodiment are substantially the same as those described above with reference to FIGS. 1 to 7 . The same components, features, and advantages as those of the dynamic image sensor 100 will not be repeated.

Please refer to FIG. 8 , which is a schematic diagram of a dynamic image sensor according to a second embodiment of the present invention. As shown in FIG. 8 , the dynamic image sensor 100B according to the present invention includes: a sensing array 11B, an image processing module 12B, and a brightness sensing module 13B.

Specifically, the dynamic image sensor 100B according to the second embodiment of the present invention further includes a brightness sensing module 13B, and the brightness sensing module 13B according to the second embodiment of the present invention is coupled to The image processing module 12B and the brightness sensing module 13B are used to sense the brightness of the environment where the dynamic image sensor 100B is located, so that the image processing module 12B can adjust the folder according to the brightness sensed by the brightness sensing module 13B The lower limit of squeeze and the upper limit of squeeze, when the brightness sensed by the brightness sensing module 13B is low, the lower limit of squeeze is correspondingly reduced, and when the brightness sensed by the brightness sensing module is high, it is correspondingly increased The pinching upper limit, however the invention is not limited thereto.

Please refer to FIG. 9 and FIG. 10. FIG. 9 is a block diagram illustrating the steps of the method for generating a dynamic image according to the second embodiment of the present invention; FIG. 10 is a flow chart illustrating the actual execution process of the superimposing steps according to the second embodiment of the present invention picture. According to the present invention, based on the dynamic image sensor 100B of the second embodiment, a method for generating a dynamic image for executing the dynamic image sensor 100B of the second embodiment is further provided, which includes the following steps:

In the upper and lower limit adjustment step S31, the image processing module 12B adjusts the pinching lower limit 35B and the pinching upper limit 36B according to the brightness sensed by the brightness sensing module 13B, when the brightness sensed by the brightness sensing module is lower than the low brightness value ( Not shown in the figure), then correspondingly lower the pinching lower limit 35B, when the brightness sensed by the brightness sensing module is higher than the high brightness value (not shown in the figure), then correspondingly increase the pinching upper limit 36B, and then execute the test steps S32.

In the test step S32, the first exposure pixel 111B exposes the short exposure time 31B to be tested, and the second exposure pixel 112A exposes the long exposure time 32B to generate the short exposure image signal 33B to be tested and the long exposure image signal 34B to be tested, and then Execute the decision step S33.

In the determination step S33 , the image processing module 12B confirms whether the short-exposure image signal 33B to be tested is higher than the pinch lower limit 35B, and whether the long-exposure image signal 32B to be tested is lower than the pinch upper limit 36B.

In the adjustment step S34, the image processing module 12B increases the short exposure time 31B to be measured by a short exposure fixed value 37B, or decreases the long exposure time 32B to be measured by a long exposure fixed value 38B to form a new short exposure time to be measured 31B And the new long exposure time 32B to be tested.

In the optimal exposure time generating step S35 , the image processing module 12B generates an optimal short exposure time 39B and an optimal long exposure time 40B according to the short exposure time to be measured 31B and the long exposure time to be measured 32B.

In the superposition step S36, the image processing module 12B superimposes the generated short-exposure image signal 33B or the generated short-exposure image signal 33B and the long-exposure image signal 32B to generate a short-exposure image signal 23B, and simultaneously perform the simultaneous exposure step S37.

At the same time in the exposure step S37 , the second exposure pixel 112B is exposed for an optimal long exposure time to generate a long exposure image signal, and then the tone mapping step S38 is executed.

Tone mapping step S38, after executing the simultaneous exposure step S37, the image processing module 12B executes the tone mapping algorithm on the short-exposure image signal 23B and the long-exposure image signal 24B to generate short-exposure image information and long-exposure image information; After the optimal exposure time generation step S35, the first exposure pixel 111A is exposed for the optimal short exposure time to generate a short exposure image signal 23B, and the second exposure pixel 112A is exposed for the optimal long exposure time to generate a long exposure image signal, and then exposure fusion is performed. Step S39.

In the exposure fusion step S39 , the image processing module 12B performs exposure fusion on the short-exposure image information and the long-exposure image information to generate high dynamic range image information, and then executes the output step S14 ′.

In the output step S40, the image processing module 12B outputs the high dynamic range image information.

It should be further explained that the low brightness value according to the present invention represents an environment under low illumination, which may easily cause underexposure of the image, and the low brightness value may be 1 lux. In addition, the high luminance value according to the present invention represents being in a high luminance or high illuminance environment, which may easily cause image overexposure. The high luminance value may be 1000 lux, but the present invention is not limited thereto.

表示影像訊號的電子數，

表示散粒噪聲的電子數，＜

＞表示對對角括號中的

於各個時間的值取平均值。請參閱下方公式(2)所示，由於散粒噪聲(shot noise)符合泊松分布(Poisson distribution)，由於泊松分布在大量粒子數時趨向於常態分布，在大量粒子存在時訊號中的散粒雜訊會呈現常態分布，散粒雜訊的標準差此時等於平均粒子數的平方根，亦即變異數(variance)等於其平均值，使得公式2之等號成立。請參閱下方公式(3)所示，根據信噪比(Signal-to-noise ratio, SNR)之公式並代入公式(1)以及公式(2)，使得公式(3)之等式成立。可以理解的是，根據公式(1)-(3)所示，訊噪比正比於影像訊號的電子數，因此，本發明透過將已產生的待測短曝影像訊號33B或已產生的待測短曝影像訊號33B與待測長曝影像訊號34B進行疊加產生短曝影像訊號23B的方式，可以實現降低信噪比之功效，然而本發明不限於此。

….(1)

….(2)

….(3) Specifically, please refer to FIG. 10. Compared with the first embodiment, the dynamic image generation method according to the second embodiment of the present invention further includes a superposition step S36. If the short exposure time 31B to be measured is the When the upper limit of the short exposure time is determined, and the short exposure image signal 33B to be tested is still lower than the pinch lower limit 35B in step S33, according to the second embodiment of the present invention, the superposition step S36 is executed to pass through the image processing module 12B Superimpose the generated short-exposure image signal 33B or the generated short-exposure image signal 31B and the long-exposure image signal 32B to generate a short-exposure image signal 23B, so as to enhance the clarity and brightness of the moving image, etc. effect. In addition, please refer to the following formula 1. When the shot noise caused by photoelectrons in the image signal accounts for the main component of the noise, it represents the number n _photon of electrons causing shot noise and the number N _signal of electrons in the image signal. The relationship is shown in formula (1), where,

Represents the number of electrons in the video signal,

represents the number of electrons of shot noise, <

> Indicates the pair of diagonal brackets

The values at each time were averaged. Please refer to the formula (2) below, since the shot noise conforms to the Poisson distribution (Poisson distribution), since the Poisson distribution tends to the normal distribution when there are a large number of particles, the scattered noise in the signal exists when there are a large number of particles The particle noise will show a normal distribution, and the standard deviation of the shot noise is equal to the square root of the average number of particles at this time, that is, the variance is equal to its average value, so that the equality of the formula 2 is established. Please refer to formula (3) below, according to the formula of signal-to-noise ratio (SNR) and substituting into formula (1) and formula (2), the equation of formula (3) is established. It can be understood that, according to formulas (1)-(3), the signal-to-noise ratio is proportional to the number of electrons in the image signal. The method of superimposing the short-exposure image signal 33B and the long-exposure image signal 34B to generate the short-exposure image signal 23B can realize the effect of reducing the signal-to-noise ratio, but the present invention is not limited thereto.

….(1)

….(2)

 ….(3)

Please refer to FIG. 11 and FIG. 12. FIG. 11 is a block diagram illustrating the tone mapping step according to the second embodiment of the present invention; FIG. 12 is a flowchart illustrating the actual execution process of the tone mapping step according to the second embodiment of the present invention . According to the second embodiment of the present invention, the dynamic image sensor 100 actually executes the steps of tone mapping as follows: First, execute the critical value generation step S381, the image processing module 12B takes the pixel values of the short-exposure image signal 23B The average value is used as the short-exposure brightness critical value 41B, and the average value of these pixel values of the long-exposure image signal 24B is used as the long-exposure brightness critical value 42B, which is used as the benchmark S381 of the high dynamic range synthesis step S382; after that, the high dynamic range is executed. In the range composition step S382, the image processing module 12B defines the pixel values higher than the short-exposure brightness threshold 41B in the short-exposure image signal 23B as 1 according to the short-exposure brightness threshold 41B, and sets the short-exposure image signal 23B as low The pixel values equal to or equal to the short-exposure brightness threshold value 41B are defined as 0, and the image processing module 12B converts the pixel values higher than the long-exposure brightness threshold value 42B in the long-exposure image signal 24B according to the long-exposure brightness threshold value 42B is defined as 1, and those pixel values in the long-exposure image signal 24B that are lower than or equal to the long-exposure brightness critical value 42B are defined as 0; finally, the weighted calculation step S383 is executed, and the image processing module 12B binarizes the The short-exposure image signal 23B is multiplied by the short-exposure weight 43B, and the image processing module multiplies the binarized long-exposure image signal 24B by the long-exposure weight 44B to generate a new short-exposure image signal 23B' and a long-exposure image signal 24B', wherein the long exposure weight 44B is greater than the short exposure weight 43B.

It can be understood that, in this embodiment, the short-exposure brightness threshold 41B and the long-exposure brightness threshold 42B are generated by averaging the pixel values of the short-exposure image signal 23B and the long-exposure image signal 24B, The short-exposure brightness critical value 41B and the long-exposure brightness critical value 42B are used as the benchmark for the subsequent high dynamic range synthesis step S382, so that the present invention can use the actually generated pixel value (that is, the brightness of the environment) as the basis for binarization. The benchmark is to increase the sensitivity of the image processing module 12B of the present invention to the brightness of the dynamic image, so as to generate a dynamic image with better bright details, but the present invention is not limited thereto. For example, the short-exposure brightness critical value 41B and the long-exposure brightness critical value 42B of the present invention can also be stored in the image processing module 12B in advance by means of statistical calculations, so as to meet most usage environments, thereby reducing the cost of the image processing module 12B. computing, while effectively saving the power of electronic devices, users can choose which method is more appropriate according to their needs.

Thus, the dynamic image sensor according to the second embodiment of the present invention senses the brightness in the environment through the brightness sensing module 13B, so that the image processing module 12B can sense the brightness according to the brightness sensing module 13B. Brightness adjustment pinch lower limit and pinch upper limit greatly reduce the calculation time of image processing module 12B to generate the optimal short exposure time and optimal long exposure time, and can realize the dynamic image that is most suitable for the current environment according to the environment, and has wide applicability . In addition, according to the tone mapping method of the second embodiment of the present invention, the short-exposure image signal 23B and the long-exposure image signal 24B are processed through binarization, and the binarized short-exposure image signal is multiplied by the short-exposure weight 43B, multiply the binarized long-exposure image signal 24B by the long-exposure weight 44B, wherein the long-exposure weight 44B is greater than the short-exposure weight 43B. In this way, the problem of unclear dynamic images in low-brightness environments is solved, so that the higher the value of the high pixel and the lower the value of the low pixel in the original pixel, the contrast of the outline of the object in the image is increased, and the clarity of the dynamic image is further improved.

Please refer to FIG. 13 , which is a schematic diagram of a dynamic image sensor according to a third embodiment of the present invention. As shown in FIG. 13 , the dynamic image sensor 100C according to the present invention includes: a sensing array 11C and an image processing module 12C. Compared with the first embodiment, the third embodiment of the present invention is mainly different in that the number of the first exposure pixels 111C of the sensing array 11C is 1/3 times the number of the second exposure pixels 112C, but not limited thereto . Relevant descriptions of this embodiment may refer to the foregoing, and details are not repeated here.

Please refer to FIG. 14 , which is a schematic diagram of a dynamic image sensor according to a fourth embodiment of the present invention. As shown in FIG. 14 , the dynamic image sensor 100D according to the present invention includes: a sensing array 11D and an image processing module 12D. Compared with the first embodiment, the fourth embodiment of the present invention mainly differs in that the number of the first exposure pixels 111D of the sensing array 11D is three times the number of the second exposure pixels 112D, but not limited thereto. Relevant descriptions of this embodiment may refer to the foregoing, and details are not repeated here.

It can be understood that the number of the first exposure pixels 111D and the second exposure pixels 112D can be adjusted according to the needs of the user, without greatly affecting the steps of the method for generating a dynamic image provided by the present invention, and the present invention belongs to the technical field Those with ordinary knowledge can make various changes and adjustments based on the above examples, which will not be listed here.

Please refer to FIG. 15 , which is a schematic diagram of a dynamic image sensor according to a fifth embodiment of the present invention. As shown in FIG. 15 , the dynamic image sensor 100E according to the present invention includes: a sensing array 11E and an image processing module 12E. Compared with the first embodiment, the fifth embodiment of the present invention mainly differs in that the sensing array 11E according to the fourth embodiment of the present invention may include first exposure pixels 111E, long exposure images 112E and medium exposure pixels 113E, wherein , the medium exposure pixel 113E has a medium exposure time, the length of the medium exposure time is between the short exposure time and the long exposure time, and the ratio of the number of the first exposure pixel 111E, the second exposure pixel 112E and the medium exposure pixel 113E 1:1:2, but not limited to this. Relevant descriptions of this embodiment may refer to the foregoing, and details are not repeated here. In this way, the present invention further provides a dynamic image sensor with three different simultaneous exposures and different exposure times, so that it can further process image information that is not simultaneously exposed and with different exposure times in the same frame time for different environments. Exposure fusion achieves effects such as enhancing the definition and brightness of dynamic images, and further improves the applicability of the dynamic image sensor of the present invention.

Please refer to FIG. 16 , which is a schematic diagram of a dynamic image sensor according to a sixth embodiment of the present invention. As shown in FIG. 16 , the dynamic image sensor 100F according to the present invention includes: a sensing array 11F and an image processing module 12F. Compared with the first embodiment, the sixth embodiment of the present invention is mainly different in that the sensing array 11E according to the sixth embodiment of the present invention may include first exposure pixels 111F, long exposure images 112F, and very short exposure pixels 114F And the extremely long exposure pixel 115F, wherein, the extremely short exposure pixel 114F has an extremely short exposure time, the extremely long exposure pixel 115F has an extremely long exposure time, and the length of the extremely short exposure time is smaller than the short exposure time, the extremely long exposure time is longer than the long exposure time, and the first exposure pixels 111F, the long exposure image 112F, the extremely short exposure pixels 114F and the extremely long exposure pixels 115F have the same number, but not limited thereto. Relevant descriptions of this embodiment may refer to the foregoing, and details are not repeated here. In this way, the present invention further provides a dynamic image sensor with four different simultaneous exposures and different exposure times, so that it can be further aimed at the environment with extreme brightness and will not be exposed at the same time and with different exposure times within the same frame time. The image information is subjected to exposure fusion to achieve effects such as enhancing the definition and brightness of the dynamic image, and further improving the applicability of the dynamic image sensor of the present invention.

Thus, the present invention has the following implementation effects and technical effects:

Firstly, based on the dynamic image sensor of the present invention, combined with the dynamic image generation method provided by the present invention, the first exposure pixel is exposed for a short exposure time, while the second exposure pixel is exposed for a long exposure time, and through Tone mapping and exposure fusion generate high dynamic range image information. In this way, the present invention utilizes the design of long and short exposure arrays to expose and fuse image information that is exposed at different times and with different exposure times within the same frame time, thereby achieving the effects of enhancing the clarity and brightness of moving images, and greatly improving The accuracy of the identification algorithm for identification of dynamic images reduces the risk of accidents.

Second, based on the dynamic image sensor of the present invention, combined with the dynamic image generation method provided by the present invention, the short exposure time to be measured is increased by the short exposure value to form a new short exposure time to be measured by the image processing module. Exposure time, or reduce the long exposure time to be measured by the long exposure fixed value to form a new long exposure time to be measured, so as to generate the best short exposure time and the best long exposure time. In this way, the present invention exchanges the spatial resolution for additional exposure information, which is beneficial to converge to the optimal exposure time in a short time, so that the exposure of high dynamic range image information is good. In addition, through the computing method of pinching, the automatic exposure time method of the present invention can realize a dynamic calculation method to generate the optimal exposure time to adapt to various shooting environments, and at the same time reduce the computing time for generating the optimal exposure time, greatly improving the cost. Efficiency and applicability of the invention.

Third, the tone mapping of the dynamic image generation method according to the present invention is different from the tone mapping algorithm of the prior art. The short-exposure image signal and the long-exposure image signal are processed through binarization, and the binarized short-exposure image The signal is multiplied by the short-exposure weight, and the binarized long-exposure image signal is multiplied by the long-exposure weight, wherein the long-exposure weight is greater than the short-exposure weight. This solves the problem of unclear dynamic images in low-brightness environments, making the pixels with high brightness have higher values and those with low brightness lower, which increases the contrast of object outlines in the image and further improves the clarity of dynamic images.

The above is to illustrate the implementation of the present invention through specific specific examples. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification.

The above descriptions are only preferred embodiments of the present invention and are not intended to limit the scope of the present invention; all other equivalent changes or modifications that do not deviate from the spirit disclosed in the present invention shall be included in the scope of the following patents Inside.

100, 100A, 100B, 100C, 100D, 100E, 100F: Dynamic image sensor 11, 11A, 11B, 11C, 11D, 11E, 11F: sensing array 111, 111A, 111B, 111C, 111D, 111E, 111F: first exposure pixels 112, 112A, 112B, 112C, 112D, 112E, 112F: second exposure pixels 113E: Medium exposure pixels 114F: extremely short exposure pixels 115F: extremely long exposure pixels 12, 12A, 12B, 12C, 12D, 12E, 12F: image processing module 21: Short exposure time 22: Long exposure time 23, 23A, 23B, 23B': short exposure image signal 24, 24A, 24B, 24B': long exposure image signal 25, 25A: short exposure image information 26, 26A: Long exposure image information 27, 27A: High dynamic range image information 31A, 31B: short exposure time to be tested 32A, 32B: long exposure time to be tested 33A, 33B: short-exposure video signal to be tested 34A, 34B: Long exposure video signal to be tested 35A, 35B: pinching lower limit 36A, 36B: pinching upper limit 37A, 37B: short exposure fixed value 38A, 38B: long exposure fixed value 39A, 39B: best short exposure time 40A, 40B: best long exposure time 41B: short exposure brightness threshold 42B: Long exposure brightness threshold 43B: Short exposure weight 44B: Long exposure weight 51A: Pixel value S11, S25, S37: Simultaneous exposure steps S12, S26, S38: tone mapping steps S13, S27, S39: exposure fusion steps S14, S28, S40: output steps S21, S32: test steps S22, S33: Judgment step S23, S34: adjustment steps S24, S35: optimal exposure time generation steps S31: Upper and lower limit adjustment steps S36: Overlay steps S381: Critical value generation step S382: High dynamic range synthesis steps S383: weighted calculation steps

1 is a schematic diagram of a dynamic image sensor according to the present invention; FIG. 2 is a block diagram illustrating the steps of the method for generating a dynamic image of the present invention; Fig. 3 is a flow chart illustrating the steps of the actual execution process of the dynamic image generation method according to the present invention; 4 is a schematic diagram of a dynamic image sensor according to a first embodiment of the present invention; 5 is a block diagram illustrating the steps of the method for generating a dynamic image according to the first embodiment of the present invention; 6 is a flow chart illustrating the steps of the actual execution process of the dynamic image generation method according to the first embodiment of the present invention; 7 is a schematic diagram illustrating a tone mapping algorithm according to a first embodiment of the present invention; 8 is a schematic diagram of a dynamic image sensor according to a second embodiment of the present invention; FIG. 9 is a block diagram illustrating the steps of the method for generating a dynamic image according to the second embodiment of the present invention; FIG. 10 is a flow chart illustrating the actual execution process of the superposition step according to the second embodiment of the present invention; 11 is a block diagram illustrating a tone mapping step according to a second embodiment of the present invention; 12 is a flow chart illustrating the actual execution of the tone mapping step according to the second embodiment of the present invention; 13 is a schematic diagram of a dynamic image sensor according to a third embodiment of the present invention; 14 is a schematic diagram of a dynamic image sensor according to a fourth embodiment of the present invention; 15 is a schematic diagram of a dynamic image sensor according to a fifth embodiment of the present invention; and FIG. 16 is a schematic diagram of a dynamic image sensor according to a sixth embodiment of the present invention.

100:Motion image sensor

11:Sensing array

111: First exposure pixel

112: second exposure pixel

12: Image processing module

Claims (8)

A dynamic image generation method with high frame rate and high dynamic range, which is applied in an environment where a dynamic image sensor receives a dynamic image in a dynamic range, and the dynamic image sensor includes a first exposure pixel and a second exposure pixel Pixels and an image processing module, the first exposure pixel and the second exposure pixel are coupled to the image processing module, the dynamic image generation method includes the following steps: a simultaneous exposure step, the first exposure pixel is exposed for a short The exposure time generates a short-exposure image signal, and the second exposure pixel is exposed for a long exposure time to generate a long-exposure image signal; a tone mapping step, the image processing module executes the short-exposure image signal and the long-exposure image signal A tone mapping (tone mapping) algorithm to generate a short exposure image information and a long exposure image information; an exposure fusion step, the image processing module performs exposure fusion (exposure) on the short exposure image information and the long exposure image information fusion), generating a high dynamic range image information; and an output step, the image processing module outputs the high dynamic range image information, wherein, before the simultaneous exposure step, the dynamic image generation method further includes an automatic exposure time method , the automatic exposure time method includes the following steps: a testing step, the first exposure pixel is exposed for a short exposure time to be measured, and the second exposure pixel is exposed for a long exposure time to be measured, so as to generate a short exposure image signal to be measured and a long-exposure image signal to be tested; a determination step, the image processing module confirms whether the short-exposure image signal to be tested is higher than a pinch lower limit, and confirms whether the long-exposure image signal to be tested is lower than a pinch upper limit ; an adjustment step, the image processing module increases the short exposure time to be measured by a short exposure constant value to generate a new short exposure time to be measured, or decreases the long exposure time to be measured by a long exposure constant value to generate The new long exposure time to be measured; and an optimal exposure time generation step, the image processing module generates an optimal short exposure time and an optimal long exposure time according to the short exposure time to be measured and the long exposure time to be measured time, and wherein, if the judging step determines that the signal of the short-exposure image to be tested is higher than the lower limit of pinch, and the signal of the long-exposure image to be tested is lower than the upper limit of pinch, the optimal exposure time is executed after the judging step generation step, and in the simultaneous exposure step, the first exposure pixel is exposed for the optimal short exposure time, and at the same time the The second exposure pixel is exposed for the optimal long exposure time; on the contrary, the adjusting step is executed after the determining step and the testing step is repeatedly executed. The dynamic image generation method as claimed in claim 1, wherein the tone mapping algorithm is selected from one of a Gamma curve algorithm and an Academy Color Encoding System curve algorithm. The automatic exposure time method as described in claim item 1, wherein, a short exposure time upper limit and a long exposure time lower limit are stored in the dynamic image sensor, and the short exposure time upper limit is the The maximum value of the short exposure time, the lower limit of the long exposure time is the minimum value of the long exposure time to be measured. The automatic exposure time method as described in Claim 3, wherein, if the short exposure time to be measured is the upper limit of the short exposure time, and the signal of the short exposure image to be measured is still lower than the lower limit of pinching in the determining step, Then the automatic exposure time method further includes a superposition step, the superposition step is that the image processing module combines the generated short-exposure image signal to be measured or the generated short-exposure image signal to be measured with the long-exposure image signal to be measured The short-exposure image signal is superimposed to generate the short-exposure image signal. The automatic exposure time method as described in claim 1, wherein the dynamic image sensor further includes a brightness sensing module, which is coupled to the image processing module, and the brightness sensing module is used for sensing Measure the brightness of the environment where the dynamic image sensor is located. The exposure time adjustment method further includes an upper and lower limit adjustment step. The image processing module adjusts the pinching lower limit and the brightness according to the brightness sensed by the brightness sensing module. The upper limit of pinching, when the brightness sensed by the brightness sensing module is lower than a low brightness value, the lower limit of pinching is correspondingly reduced, and when the brightness sensed by the brightness sensing module is higher than a high brightness value, the corresponding Raise the upper pinch limit. The dynamic image generation method as described in claim 1, wherein the tone mapping step further includes: a critical value generation step, the image processing module averages the pixel values of the short-exposure image signal to generate a short Exposure brightness threshold, and average the pixel values of the long-exposure image signal to generate a long-exposure brightness threshold; a high dynamic range synthesis step, the image processing module according to the short-exposure brightness threshold, the In the short-exposure image signal, the pixel values higher than the short-exposure brightness threshold are defined as 1, and the short-exposure image signal The pixel values in the signal that are lower than or equal to the short-exposure brightness threshold are defined as 0, and the image processing module determines the pixel values in the long-exposure image signal that are higher than the long-exposure brightness threshold according to the long-exposure brightness threshold define the pixel values as 1, and define the pixel values lower than or equal to the long-exposure brightness threshold in the long-exposure image signal as 0; and a weighting calculation step, the image processing module binarizes The short-exposure image signal is multiplied by a short-exposure weight, and the image processing module multiplies the binarized long-exposure image signal by a long-exposure weight, wherein the long-exposure weight is greater than the short-exposure weight. A dynamic image sensor with high frame rate and high dynamic range, which is applied in the environment of receiving dynamic images with dynamic range, the dynamic image sensor includes: a sensing array, which includes a plurality of first exposures Pixels and a plurality of second exposure pixels, the first exposure pixels have a short exposure time, the second exposure pixels have a long exposure time; and an image processing module, which is coupled to the sensing array; Wherein, the dynamic image sensor performs a simultaneous exposure step: the first exposure pixel is exposed to the short exposure time to generate a short exposure image signal, and the second exposure pixel is exposed to the long exposure time to generate a long exposure image signal, wherein , the dynamic image sensor performs a tone mapping step: the image processing module executes a tone mapping algorithm on the short-exposure image signal and the long-exposure image signal to generate short-exposure image information and long-exposure image information, Wherein, the dynamic image sensor performs an exposure fusion step: the image processing module performs exposure fusion on the short-exposure image information and the long-exposure image information to generate a high dynamic range image information, wherein, in the simultaneous exposure step Before, the dynamic image sensor further implements an automatic exposure time method, and the automatic exposure time method includes the following steps: a test step, the first exposure pixel is exposed for a short exposure time to be tested, and the second exposure pixel is exposed for a short exposure time at the same time The long exposure time to be measured is used to generate a short-exposure image signal to be measured and a long-exposure image signal to be measured; a determination step, the image processing module confirms whether the short-exposure image signal to be tested is higher than a pinch lower limit, and Confirming whether the signal of the long-exposure image to be tested is lower than a pinch upper limit; an adjustment step, the image processing module increases the short exposure time to be measured by a short exposure constant value to generate a new short exposure time to be measured, or decreases the long exposure time to be measured by a long exposure constant value to generate a new The long exposure time to be measured; and an optimal exposure time generation step, the image processing module generates an optimal short exposure time and an optimal long exposure time according to the short exposure time to be measured and the long exposure time to be measured , and wherein, if the judging step judges that the signal of the short-exposure image to be tested is higher than the lower limit of pinching, and the signal of the long-exposure image to be tested is lower than the upper limit of pinching, the optimal exposure time is executed after the judging step to generate step, and in the simultaneous exposure step, the first exposure pixel is exposed for the optimal short exposure time, while the second exposure pixel is exposed for the optimal long exposure time; otherwise, the adjustment step is executed after the determination step and repeated The test steps. The dynamic image sensor according to claim 7, wherein the first exposure pixels and the second exposure pixels are arranged alternately, and the number of the first exposure pixels is less than the number of the second exposure pixels One of 1x, 2x and 3x.

Images