WO2018220993A1

WO2018220993A1 - Signal processing device, signal processing method and computer program

Info

Publication number: WO2018220993A1
Application number: PCT/JP2018/014210
Authority: WO
Inventors: 毅都築
Original assignee: ソニーセミコンダクタソリューションズ株式会社
Priority date: 2017-05-29
Filing date: 2018-04-03
Publication date: 2018-12-06
Also published as: JP2018201158A

Abstract

[Problem] To provide a signal processing device which is capable of shortening processing time without degrading dynamic range performance or flexibly adjusting a tradeoff between image quality and processing time upon receiving an HDR image. [Solution] Provided is a signal processing device provided with: a synthesis processing unit which synthesizes, at least (N-1) times, an image of N frames which have exposure times that are different from each other; and a motion adaptation processing unit which performs a motion adaptation process in one-time synthesis that uses two of the group of the images in the synthesis processing unit, wherein the number of motion adaptation processes in the motion adaption processing unit is equal to or less than (N-2), when the synthesis processing unit performs the synthesis (N-1) times.

Description

Signal processing apparatus, signal processing method, and computer program

The present disclosure relates to a signal processing device, a signal processing method, and a computer program.

In recent years, as a technique for capturing an HDR (High Dynamic Range) image, an imaging technique for combining image signals captured by a plurality of types of exposure has been used.

In recent years, smartphones are the most popular platform for performing composition processing, and in this case, processing is performed by software on an AP (application processor). In an environment where the computing performance of the AP is not sufficient, it is required to obtain a wide dynamic range with a reasonable processing time while maintaining a constant image quality with respect to movement, and the purpose is to shorten the time for obtaining an HDR image. Techniques are disclosed (see Patent Documents 1 and 2).

JP 2013-152334 A Japanese Patent Laying-Open No. 2015-186062

However, with the existing technology, the processing time for obtaining the HDR image is shortened, but the image quality is deteriorated, or the deterioration of the image quality of the HDR image is suppressed, but it is difficult to shorten the processing time. Further, in the existing technology, when obtaining an HDR image, the trade-off between image quality and processing time is not adjusted flexibly.

Therefore, in the present disclosure, when obtaining an HDR image, the dynamic range performance is not degraded, the processing time is shortened, and the trade-off between the image quality and the processing time can be adjusted flexibly. A signal processing apparatus, a signal processing method, and a computer program are proposed.

According to the present disclosure, a composition processing unit that performs composition processing at least N−1 times on images of N frames (N is an integer of 3 or more) with different exposure times, and the image processing unit A motion adaptation processing unit that performs motion adaptation processing at the time of one synthesis using two of the groups, and the motion adaptation when the synthesis processing unit performs the N-1 synthesis processings A signal processing device is provided in which the number of motion adaptation processes in the processing unit is set to N-2 or less.

Further, according to the present disclosure, the processor performs at least N−1 synthesis processing on images of N frames (N is an integer of 3 or more) having different exposure times, and Performing motion adaptation processing at the time of one synthesis using two of them, and setting the number of motion adaptation processing at the time of performing the N-1 synthesis processing to be N-2 times or less, A signal processing method is provided.

Further, according to the present disclosure, the computer performs at least N−1 synthesis processing on images of N frames (N is an integer of 3 or more) having different exposure times, And performing the motion adaptation process at the time of one synthesis using two of them, and the number of the motion adaptation processes at the time of performing the N-1 synthesis process is N-2 times or less. A computer program is provided.

As described above, according to the present disclosure, when obtaining an HDR image, it is possible to shorten the processing time without reducing the performance of the dynamic range, and to flexibly adjust the trade-off between the image quality and the processing time. In addition, a new and improved signal processing apparatus, signal processing method, and computer program can be provided.

Note that the above effects are not necessarily limited, and any of the effects shown in the present specification, or other effects that can be grasped from the present specification, together with or in place of the above effects. May be played.

It is explanatory drawing which shows the mode of the production | generation of the HDR image which the signal processing apparatus in this embodiment presupposes. It is explanatory drawing which shows the example of the image sensor which has arrange | positioned different exposure pixels in two dimensions. It is explanatory drawing which shows the function structural example of the signal processing apparatus which concerns on the same embodiment. It is explanatory drawing explaining the production | generation process of the HDR image by the signal processing apparatus which concerns on the same embodiment. It is a flowchart which shows the operation example of the signal processing apparatus which concerns on the same embodiment. FIG. 6 is an explanatory diagram illustrating a specific example of the operation of the signal processing device according to the embodiment. FIG. 6 is an explanatory diagram illustrating a specific example of the operation of the signal processing device according to the embodiment. It is a block diagram which shows the schematic structural example of the vehicle control system which is an example of the mobile body control system with which the technique which concerns on this indication can be applied. It is a figure which shows the example of the installation position of the imaging part 12031. FIG.

Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In addition, in this specification and drawing, about the component which has the substantially same function structure, duplication description is abbreviate | omitted by attaching | subjecting the same code | symbol.

The description will be made in the following order.
1. Embodiment of the present disclosure 1.1. Background 1.2. Configuration example 1.3. Example of operation 2. Application examples to mobile objects Summary

<1. Embodiment of the present disclosure>
[1.1. History]
First, before the embodiments of the present disclosure are described in detail, the background that led to the embodiments of the present disclosure will be described.

As described above, in recent years, as a technique for capturing an HDR (High Dynamic Range) image, an imaging technique for combining image signals captured by a plurality of types of exposure has been used. If the size of the image sensor is small, the dynamic range becomes small, and so-called whiteout or blackout tends to occur in the captured image. An image in which whiteout or blackout has occurred is different from how it is seen with the naked eye. Therefore, the HDR technology for expanding the dynamic range is useful as a technology for expanding the dynamic range of a small size image sensor.

The HDR image is obtained by combining a plurality of images with different exposures. When generating an HDR image, it is necessary to consider the case where a moving object is included in the subject. As a response to motion, the purpose is to reduce the blur of the moving object that occurs in images with long exposure times (long accumulation images), or to reduce the displacement and double image of moving objects if the frame composition type In some cases, a motion adaptation process is performed in which a motion region is detected to increase a blend ratio of an image having a short exposure time (short accumulation image), or a motion vector is detected to align an object. In recent years, there is a tendency to increase the number of exposures to obtain a wider dynamic range.

In recent years, smartphones are the most popular platform for performing composition processing, and in this case, processing is performed by software on an AP (application processor). However, the computing performance of APs for smartphones is not yet sufficient, and the computing performance of APs mounted on models of middle class and below is relatively low. In such an environment, if complicated calculation processing is performed after increasing the number of exposures to obtain an HDR image, a long processing time is required. The user waits for a long time until the HDR composite image is output after shooting with the HDR function enabled. In an environment where the calculation performance is not sufficient, it is not easy to obtain a wide dynamic range in a reasonable processing time while maintaining a constant image quality with respect to movement.

In an environment where the computing performance of the AP is not sufficient, it is required to obtain a wide dynamic range with a reasonable processing time while maintaining a constant image quality with respect to movement, and the purpose is to shorten the time for obtaining an HDR image. Technology is disclosed.

Therefore, in view of the above-mentioned points, the present disclosure person can shorten the processing time without reducing the performance of the dynamic range and flexibly adjust the trade-off between the image quality and the processing time when obtaining the HDR image. We have intensively studied the technology that can be used. As a result, as described below, the present disclosure, when obtaining an HDR image, shortens the processing time without reducing the performance of the dynamic range, or flexibly adjusts the trade-off between image quality and processing time. It came to devise the technology that can do.

As above, the background to the embodiment of the present disclosure has been described. Subsequently, an embodiment of the present disclosure will be described in detail.

[1.2. Configuration example]
In the present embodiment, a signal processing apparatus is assumed that inputs a plurality of image frames with different exposures and generates an HDR image using the plurality of image frames. FIG. 1 is an explanatory diagram showing a state of generating an HDR image that is assumed by the signal processing apparatus according to the present embodiment. FIG. 1 shows a state where an HDR image is generated using four image frames. The four image frames are designated as long accumulation, middle accumulation, short accumulation, and ultrashort accumulation in order of long exposure time. For example, an image with long accumulation has an exposure time of 1/30 seconds, an image with intermediate accumulation has an exposure time of 1/120 seconds, an image with short accumulation has an exposure time of 1/480 seconds, and an image with ultra short accumulation has The exposure time is 1/1920 seconds. Of course, the exposure time of each image is not limited to this example. Further, FIG. 1 shows a state in which an HDR image is generated using four images having different exposure times. However, the number of images that are the basis of the HDR image is not limited to four, and two images are generated. That is all you need. Further, FIG. 1 shows a state in which the exposure time is long, but the HDR image may be combined from the short exposure time when generating the HDR image.

Since the dynamic range of the image obtained by synthesizing the four images is expanded, gradation compression processing is performed in order to fit the normal dynamic range. After the gradation compression process, the camera signal processing system is used even when HDR is not performed. Camera signal processing includes white balance, demosaic, color matrix, edge enhancement, noise reduction, and the like.

In addition, as a method for obtaining a plurality of images with different exposures, as described above, images may be taken in time sequential manner while changing the exposure time, or different exposure pixels are arranged two-dimensionally in the image sensor, and once. Each exposure plane may be obtained by shooting. Alternatively, these methods may be combined. FIG. 2 is an explanatory diagram showing an example of an image sensor in which differently exposed pixels are two-dimensionally arranged. The image sensor 10 shown in FIG. 2 has a configuration in which long accumulation pixels 11, middle accumulation pixels 12, short accumulation pixels 13, and ultrashort accumulation pixels 14 are arranged in a matrix.

Further, as a method of obtaining a plurality of images with different exposures, a method of changing the sensitivity without changing the shutter time may be used. If the shutter time is shortened, it is required to deal with flicker. Therefore, the correspondence to flicker can be omitted by changing the sensitivity without changing the shutter time.

Next, a functional configuration example of the signal processing device according to the embodiment of the present disclosure will be described. FIG. 3 is an explanatory diagram illustrating a functional configuration example of the signal processing device 100 according to the embodiment of the present disclosure. FIG. 3 shows a functional configuration example of the signal processing device 100 that executes processing for generating an HDR image. Hereinafter, a functional configuration example of the signal processing device 100 according to the embodiment of the present disclosure will be described with reference to FIG.

As shown in FIG. 3, the signal processing apparatus 100 according to the present embodiment includes a saturation detection unit 102, a motion detection unit 104, a black crush detection unit 106, a blend rate determination unit 108, a synthesis unit 110, It is comprised including.

The saturation detection unit 102 detects the degree of saturation of an image having a longer exposure time among two images having different exposure times input to the signal processing apparatus 100. The degree of image saturation is information that is defined as multi-valued with respect to a pixel value that is 0 below a certain threshold and exceeds the threshold. The saturation detection unit 102 sends information on the detected degree of saturation to the blend rate determination unit 108. Information on the degree of saturation is used by the blend rate determination unit 108 to determine the blend rate (synthesis ratio) of two images.

The motion detection unit 104 detects a motion between images using two images input to the signal processing device 100. As a process for detecting motion, for example, the motion detection unit 104 normalizes brightness by applying an exposure ratio gain to a short storage frame in advance, and then calculates a difference between pixel values of the long storage frame and the short storage frame. take. The motion detection unit 104 determines that there is motion between images if the difference value is greater than a predetermined threshold value. Further, the motion detection unit 104 may calculate an average value in a certain range and take the difference as a process of detecting the motion. In addition, the motion detection unit 104 may detect a change by paying attention to a frequency or a gradient in order to detect a motion more accurately as a process for detecting the motion. In addition, the motion detection unit 104 may detect a corresponding change in position between a plurality of frames, or may detect a motion by a method of detecting a movement vector and using the value. The motion detection unit 104 sends information on the area detected as motion to the blend rate determination unit 108. The blend rate determination unit 108 increases the blend rate of an image with a shorter exposure time for the purpose of reducing blur and misalignment in an area detected as motion.

The black crushing detection unit 106 detects the degree of black crushing of an image having a shorter exposure time out of two images having different exposure times. The degree of black crushing in an image is information that is defined as multivalued for pixel values that are 0 or more below a certain threshold. The black crushing detection unit 106 sends information on the detected degree of black crushing to the blend rate determination unit 108. Information on the degree of black crushing is used in the blend rate determination unit 108 to determine the blend rate of two images.

The blend rate determination unit 108 determines the blend rate of two images input to the signal processing apparatus 100. In determining the blend ratio of two images, the blend rate determination unit 108 determines the degree of saturation of the image detected by the saturation detection unit 102, the level of black collapse of the image detected by the black collapse detection unit 106, and the motion detection unit 104. The information on the presence / absence of the motion detected by is used. For example, if the degree of saturation of the image detected by the saturation detection unit 102 is equal to or greater than a predetermined threshold, the blend rate determination unit 108 increases the blend rate of the image with the shorter exposure time. For example, if the degree of black crushing of the image detected by the black crushing detection unit 106 is equal to or greater than a predetermined threshold, the blend rate determination unit 108 increases the blend rate of the image with the longer exposure time. Further, for example, in the region detected by the motion detection unit 104 as a motion, the blend rate determination unit 108 increases the blend rate of an image with a shorter exposure time for the purpose of reducing blur and misalignment. The blend rate determination unit 108 determines a final blend rate from these elements, and sends information on the determined blend rate to the synthesis unit 110.

The synthesizing unit 110 performs a process of synthesizing two images input to the signal processing device 100 based on the blend rate determined by the blend rate determining unit 108. The image synthesized by the synthesis unit 110 is further synthesized with another image that is an HDR image generation target.

The signal processing apparatus 100 generates an HDR image by performing a series of processes on all HDR image generation targets.

In the HDR image generation processing in the signal processing apparatus 100, although it depends on the algorithm, the amount of calculation of the motion adaptation processing including motion detection of the subject between the images is relatively large. If motion adaptation processing is not performed at all, the amount of calculation is small and the synthesis processing is completed in a relatively short time. Instead, blurring of the moving object is noticeable, and artifacts in which multiple contours are generated can be seen.

In the present embodiment, focusing on this trade-off, an object is to flexibly control the relationship between the image quality of the HDR image and the processing time required for generating the HDR image.

FIG. 4 is an explanatory diagram illustrating HDR image generation processing by the signal processing apparatus 100 according to the present embodiment. FIG. 4 shows an example in which an HDR image is generated from four images having different exposure times. In FIG. 4, the shutter time of each frame is 1/30 seconds, 1/120 seconds, 1/480 seconds, 1/1920 seconds from the long storage side, and the exposure ratio interval increases by 4 times. A 64 times case is shown. It is desirable that the exposure ratio interval is about 16 times at the maximum. In addition, the shutter time of each frame can change according to the subject and the environment (brightness, etc.) at the time of imaging.

Normally, when an HDR image is generated from four images, the motion adaptation process is executed a total of three times. In the example shown in FIG. Relieves noticeable blur that occurs in the long storage frame with the longest shutter time, but does not replace it with the short storage side. Only long accumulation images show severe blur, and in many cases the amount of blur is within the allowable range except for long accumulation images, so if you perform motion adaptation processing only once on the long accumulation side, a composite image with sufficient quality Can be generated. Even when motion detection is performed and the replacement to the short accumulating side is advanced, the S / N of the moving body on the short accumulating side decreases as the shutter time becomes short. It may be unusable in terms of image quality. From such a viewpoint, the signal processing apparatus 100 according to the present embodiment performs the motion adaptation process only once on the long accumulation side. By performing the motion adaptation process only once on the long storage side, it is possible to reduce the amount of calculation required for the motion adaptation process while eliminating adverse effects such as a fatal blur of the moving object and a significant S / N reduction.

In FIG. 4, it has been described that the motion adaptation process is performed only once, but the number of times of the motion adaptation process is not limited to this example. A feature of the signal processing apparatus 100 according to the present embodiment is that the number of motion adaptive processes is fixed to a number smaller than the number of times originally necessary.

Even if the number of motion adaptation processes is reduced, fixing the number of times may not be sufficient for generating an HDR image with a certain level of quality. The amount of blur included in each frame is determined by many factors. If the moving speed of the moving object is high, the blur amount naturally increases. If the distance between the camera and the moving subject is long or the angle of view of the lens is wide, the moving speed is relatively slow and the blur amount is small. Further, the allowable limit for the blur amount is subjective, and varies depending on the evaluator (for example, the photographer) of the image. In addition, the actual shutter time of each frame depends on whether the scene to be photographed is bright or dark, how many times the exposure ratio between frames is set, how wide the overall exposure ratio, that is, the dynamic range, AE (Auto It depends on various factors such as the method and policy of exposure (auto exposure). Furthermore, as described above, the S / N of the moving object decreases as the shutter time becomes shorter, so this is a trade-off between reducing blur or securing S / N of moving object. The balance with the S / N securing of the moving object is also subjective and again varies depending on the evaluator. Considering the above factors, the appropriate number of motion adaptation processes is not easily determined, and it can be said that it is better to control flexibly according to the situation.

Therefore, the signal processing apparatus 100 according to the present embodiment determines whether or not the shutter time of the image that is the basis for generating the HDR image is equal to or greater than a predetermined threshold value. If the shutter time is equal to or greater than a predetermined threshold value, the signal processing apparatus 100 performs motion adaptation processing to perform image synthesis processing because blur of the moving object may be conspicuous in the image. In the present embodiment, the composition process is a process for determining the composition ratio of two images in the blend ratio determination unit 108 or a composition of images based on the composition ratio of two images by the composition unit 110. Any one of processing shall be said. Further, in the present embodiment, when generating an HDR image, description will be made assuming that images are synthesized in order from an image with a short exposure time, but the present disclosure is not limited to such an example, and an image with a long exposure time is ordered. Images may be combined.

[1.3. Example of operation]
FIG. 5 is a flowchart illustrating an operation example of the signal processing device 100 according to the embodiment of the present disclosure. FIG. 5 shows an operation example of the signal processing apparatus 100 when performing HDR image generation processing. In the flowchart shown in FIG. 5, N is the number of combined frames, _TN is the shutter time of the Nth frame, and TH _Shut is a threshold value of a shutter time that is defined in advance in consideration of blur and S / N. Here, it is assumed that the smaller N is the long accumulation side and the larger N is the short accumulation side. Hereinafter, an operation example of the signal processing apparatus 100 according to the embodiment of the present disclosure will be described with reference to FIG.

The signal processing apparatus 100, first initializes the value of N to 1 (step S101), followed by, _{T N} is determined whether a more _{TH Shut} (step S102). The determination in step S102 is executed by, for example, the motion detection unit 104.

As a result of the determination in step S102, if _TN is equal to or higher than TH _Shut (step S102, Yes), the signal processing apparatus 100 uses the two input images (Nth and N + 1th frames) to perform motion adaptation processing. Is performed (step S103). The process of step S103 is performed by, for example, the motion detection unit 104.

On the other hand, as a result of the determination in step S102, if _TN is not equal to or greater than TH _Shut (step S102, No), the signal processing apparatus 100 skips the motion adaptation process in step S103.

Subsequently, the signal processing apparatus 100 performs a process of combining the two input images (step S104). The compositing unit 110 executes the image compositing process. As described above, the signal processing apparatus 100 uses information on the degree of saturation, information on the degree of black crushing, information on movement between the two images, and the like when combining two images.

Subsequently, the signal processing apparatus 100 determines whether the value of N is equal to a value that is 1 less than the number of frames to be synthesized (step S105). If the value of N is equal to one less than the number of frames to be combined (step S105, Yes), the signal processing apparatus 100 ends the series of processes assuming that the processing has been completed for all the frames to be combined. On the other hand, if the value of N is not equal to a value that is one less than the number of frames to be combined (step S105, No), the signal processing apparatus 100 increments the value of N by 1 (step S106), and the process of step S102 is performed. Return.

FIG. 6 is an explanatory diagram illustrating a specific example of the operation of the signal processing device 100 according to the embodiment of the present disclosure. In FIG. 6, it is assumed that TH _Shut is 1/60 second.

Since the shutter time (1/30 second) of the long accumulation frame is longer than the threshold value TH _Shut , the risk of occurrence of blur in the long accumulation frame is great. Then, assuming that the S / N of the intermediate storage frame is sufficient, the signal processing device 100 executes a motion adaptation process between the long storage frame and the intermediate storage frame.

Next, the medium蓄frame shutter time (1/120 sec) is shorter than the threshold value _{TH Shut,} blur risk of middle蓄frame is small. Further, assuming that the S / N of the intermediate storage frame is insufficient, the signal processing device 100 does not execute the motion adaptation process between the intermediate storage frame and the short storage frame.

Furthermore, the signal processing apparatus 100 does not execute the motion adaptation process between the short accumulation frame and the ultra short accumulation frame in which the shutter time is shortened. In this way, the signal processing apparatus 100 according to the present embodiment can reduce the number of motion adaptation processes while considering blur and S / N. In the example shown in FIG. 6, the place where the motion adaptation process is executed three times is reduced to one if the threshold TH _Shut is not taken into consideration. The signal processing apparatus 100 according to the present embodiment defines a shutter time threshold TH _Shut as an element that reflects the image quality of the combined HDR image, such as blur and S / N, and the finally set shutter time. The number of executions of the motion adaptation process is adjusted from the above relationship.

The signal processing apparatus 100 according to the present embodiment may execute the HDR synthesis process after obtaining the number of executions of the motion adaptation process in advance. Formulas (1) and (2) show the calculation formula for the number of executions.

Assuming that the processing time of the entire HDR process is T _SUM , the sum of the time T _HDR required for one HDR synthesis and the time T _MOVE required for one motion adaptation process is multiplied by the number of synthesized frames N, and T _SUM is obtained. (See Equation (1)). From this relationship, the number M of motion adaptation processes when it is desired to keep the entire processing time _TSUM below a certain value is expressed by Equation (2). The signal processing apparatus 100 according to the present embodiment can improve the image quality of the moving object within a range that does not exceed a certain processing time by performing the motion adaptation process for the number M obtained by Expression (2).

The signal processing apparatus 100 can use both a method for determining whether or not the motion adaptation process can be performed based on a threshold value of the shutter time and a method for obtaining the number of executions of the motion adaptation process in advance. In this case, the signal processing apparatus 100 may use the smaller number of the upper limit of the number of motion adaptation processes considering blur and S / N and the number of times M obtained as described above.

The signal processing apparatus 100 according to the present embodiment may determine whether or not the motion adaptation process can be performed based on information on the amount of motion between images. FIG. 7 is an explanatory diagram illustrating a specific example of the operation of the signal processing device 100 according to the embodiment of the present disclosure.

The signal processing apparatus 100 first performs motion detection on the long accumulation frame and the middle accumulation frame, and detects the magnitude of motion 12 between the images. If the detected movement is large, the moving speed of the moving object is fast with respect to the shutter time, which means that there is a high risk of occurrence of blur and double image. It may be assumed that the magnitude of the movement and the shutter time are basically proportional. Here, the signal processing apparatus 100 can also calculate the movement amounts Motion 23 and Motion 34 on the short accumulation side from the Motion 12.

Then, after defining the amount of motion TH _{Motion in} which blur and double images are not conspicuous, the signal processing device 100 performs motion adaptation processing only between frames whose motion amount is equal to or greater than TH _Motion. . In the example illustrated in FIG. 7, the motion adaptation process is executed between the long accumulation frame and the intermediate accumulation frame, and the motion adaptation process is not performed between the other frames.

As described above, by determining whether or not the motion adaptation process can be performed based on the magnitude of the motion amount, the signal processing apparatus 100 does not perform the motion adaptation process if the motion amount between frames is zero or sufficiently small. There is no degradation in image quality, and motion adaptation processing for subsequent frames can be automatically turned off. That is, the signal processing apparatus 100 according to the embodiment of the present disclosure has an effect that the processing time can be shortened according to the amount of motion in the shooting scene.

The signal processing apparatus 100 uses both a method for determining whether or not the motion adaptive process can be performed based on the shutter time threshold value, and a method for determining whether or not the motion adaptive process can be performed based on information on the amount of motion between images. It is also possible. In this case, the signal processing apparatus 100 has an upper limit of the number of times of motion adaptation processing considering blur and S / N and an upper limit number of times of motion adaptation processing considering information on the amount of motion between images. The smaller number may be used.

Further, the signal processing apparatus 100 can use both a method for determining whether or not the motion adaptive process can be executed based on information on the amount of motion between images, and a method for obtaining the number of executions of the motion adaptive process in advance. In this case, the signal processing apparatus 100 may use the upper limit of the number of times of the motion adaptation processing considering information on the amount of motion between images and the smaller number of times M obtained as described above.

In addition, the signal processing device 100 determines whether or not to execute the motion adaptation process based on a threshold of the shutter time, determines whether or not to execute the motion adaptation process based on information on the amount of motion between images, It is also possible to use a method for determining whether or not to execute the motion adaptation process based on the information on the amount of motion in between. In this case, the signal processing apparatus 100 has an upper limit of the number of times of motion adaptation processing considering blur and S / N and an upper limit number of times of motion adaptation processing considering information on the amount of motion between images. Of the times M obtained as described above, the smaller number may be used.

When determining whether to execute the motion adaptation process based on the amount of motion, the amount of motion may be determined for the entire image of each frame, but the main subject often exists in the center of the image. Therefore, the signal processing apparatus 100 may determine whether or not to execute the motion adaptation process based on a motion amount in a predetermined range of a part, for example, the central part, instead of the entire image.

<2. Application example to mobile objects>
The technology according to the present disclosure (present technology) can be applied to various products. For example, the technology according to the present disclosure is realized as a device that is mounted on any type of mobile body such as an automobile, an electric vehicle, a hybrid electric vehicle, a motorcycle, a bicycle, personal mobility, an airplane, a drone, a ship, and a robot. May be.

FIG. 8 is a block diagram illustrating a schematic configuration example of a vehicle control system that is an example of a mobile control system to which the technology according to the present disclosure can be applied.

The vehicle control system 12000 includes a plurality of electronic control units connected via a communication network 12001. In the example shown in FIG. 8, the vehicle control system 12000 includes a drive system control unit 12010, a body system control unit 12020, a vehicle exterior information detection unit 12030, a vehicle interior information detection unit 12040, and an integrated control unit 12050. As a functional configuration of the integrated control unit 12050, a microcomputer 12051, an audio image output unit 12052, and an in-vehicle network I / F (interface) 12053 are illustrated.

The drive system control unit 12010 controls the operation of the device related to the drive system of the vehicle according to various programs. For example, the drive system control unit 12010 includes a driving force generator for generating a driving force of a vehicle such as an internal combustion engine or a driving motor, a driving force transmission mechanism for transmitting the driving force to wheels, and a steering angle of the vehicle. It functions as a control device such as a steering mechanism that adjusts and a braking device that generates a braking force of the vehicle.

The body system control unit 12020 controls the operation of various devices mounted on the vehicle body according to various programs. For example, the body system control unit 12020 functions as a keyless entry system, a smart key system, a power window device, or a control device for various lamps such as a headlamp, a back lamp, a brake lamp, a blinker, or a fog lamp. In this case, the body control unit 12020 can be input with radio waves transmitted from a portable device that substitutes for a key or signals from various switches. The body system control unit 12020 receives input of these radio waves or signals, and controls a door lock device, a power window device, a lamp, and the like of the vehicle.

The vehicle outside information detection unit 12030 detects information outside the vehicle on which the vehicle control system 12000 is mounted. For example, the imaging unit 12031 is connected to the vehicle exterior information detection unit 12030. The vehicle exterior information detection unit 12030 causes the imaging unit 12031 to capture an image outside the vehicle and receives the captured image. The vehicle outside information detection unit 12030 may perform an object detection process or a distance detection process such as a person, a car, an obstacle, a sign, or a character on a road surface based on the received image.

The imaging unit 12031 is an optical sensor that receives light and outputs an electrical signal corresponding to the amount of received light. The imaging unit 12031 can output an electrical signal as an image, or can output it as distance measurement information. Further, the light received by the imaging unit 12031 may be visible light or invisible light such as infrared rays.

The vehicle interior information detection unit 12040 detects vehicle interior information. For example, a driver state detection unit 12041 that detects a driver's state is connected to the in-vehicle information detection unit 12040. The driver state detection unit 12041 includes, for example, a camera that images the driver, and the vehicle interior information detection unit 12040 determines the degree of fatigue or concentration of the driver based on the detection information input from the driver state detection unit 12041. It may be calculated or it may be determined whether the driver is asleep.

The microcomputer 12051 calculates a control target value of the driving force generator, the steering mechanism, or the braking device based on the information inside / outside the vehicle acquired by the vehicle outside information detection unit 12030 or the vehicle interior information detection unit 12040, and the drive system control unit A control command can be output to 12010. For example, the microcomputer 12051 realizes an ADAS (Advanced Driver Assistance System) function including vehicle collision avoidance or impact mitigation, following traveling based on inter-vehicle distance, vehicle speed maintaining traveling, vehicle collision warning, or vehicle lane departure warning, etc. It is possible to perform cooperative control for the purpose.

Further, the microcomputer 12051 controls the driving force generator, the steering mechanism, the braking device, and the like based on the information around the vehicle acquired by the vehicle exterior information detection unit 12030 or the vehicle interior information detection unit 12040. It is possible to perform cooperative control for the purpose of automatic driving that autonomously travels without depending on the operation.

Further, the microcomputer 12051 can output a control command to the body system control unit 12020 based on information outside the vehicle acquired by the vehicle outside information detection unit 12030. For example, the microcomputer 12051 controls the headlamp according to the position of the preceding vehicle or the oncoming vehicle detected by the outside information detection unit 12030, and performs cooperative control for the purpose of anti-glare, such as switching from a high beam to a low beam. It can be carried out.

The sound image output unit 12052 transmits an output signal of at least one of sound and image to an output device capable of visually or audibly notifying information to a vehicle occupant or the outside of the vehicle. In the example of FIG. 8, an audio speaker 12061, a display unit 12062, and an instrument panel 12063 are illustrated as output devices. The display unit 12062 may include at least one of an on-board display and a head-up display, for example.

FIG. 9 is a diagram illustrating an example of an installation position of the imaging unit 12031.

In FIG. 9, the vehicle 12100 includes

imaging units

12101, 12102, 12103, 12104, and 12105 as the imaging unit 12031.

The

imaging units

12101, 12102, 12103, 12104, and 12105 are provided, for example, at positions such as a front nose, a side mirror, a rear bumper, a back door, and an upper part of a windshield in the vehicle interior of the vehicle 12100. The imaging unit 12101 provided in the front nose and the imaging unit 12105 provided in the upper part of the windshield in the vehicle interior mainly acquire an image in front of the vehicle 12100. The

imaging units

12102 and 12103 provided in the side mirror mainly acquire an image of the side of the vehicle 12100. The imaging unit 12104 provided in the rear bumper or the back door mainly acquires an image behind the vehicle 12100. The forward images acquired by the

imaging units

12101 and 12105 are mainly used for detecting a preceding vehicle or a pedestrian, an obstacle, a traffic light, a traffic sign, a lane, or the like.

FIG. 9 shows an example of the shooting range of the imaging units 12101 to 12104. The imaging range 12111 indicates the imaging range of the imaging unit 12101 provided in the front nose, the imaging ranges 12112 and 12113 indicate the imaging ranges of the

imaging units

12102 and 12103 provided in the side mirrors, respectively, and the imaging range 12114 The imaging range of the imaging part 12104 provided in the rear bumper or the back door is shown. For example, by superimposing the image data captured by the imaging units 12101 to 12104, an overhead image when the vehicle 12100 is viewed from above is obtained.

At least one of the imaging units 12101 to 12104 may have a function of acquiring distance information. For example, at least one of the imaging units 12101 to 12104 may be a stereo camera including a plurality of imaging elements, or may be an imaging element having pixels for phase difference detection.

For example, the microcomputer 12051, based on the distance information obtained from the imaging units 12101 to 12104, the distance to each three-dimensional object in the imaging range 12111 to 12114 and the temporal change in this distance (relative speed with respect to the vehicle 12100). In particular, it is possible to extract, as a preceding vehicle, a three-dimensional object that travels at a predetermined speed (for example, 0 km / h or more) in the same direction as the vehicle 12100, particularly the closest three-dimensional object on the traveling path of the vehicle 12100. it can. Further, the microcomputer 12051 can set an inter-vehicle distance to be secured in advance before the preceding vehicle, and can perform automatic brake control (including follow-up stop control), automatic acceleration control (including follow-up start control), and the like. Thus, cooperative control for the purpose of autonomous driving or the like autonomously traveling without depending on the operation of the driver can be performed.

For example, the microcomputer 12051 converts the three-dimensional object data related to the three-dimensional object to other three-dimensional objects such as a two-wheeled vehicle, a normal vehicle, a large vehicle, a pedestrian, and a utility pole based on the distance information obtained from the imaging units 12101 to 12104. It can be classified and extracted and used for automatic avoidance of obstacles. For example, the microcomputer 12051 identifies obstacles around the vehicle 12100 as obstacles that are visible to the driver of the vehicle 12100 and obstacles that are difficult to see. The microcomputer 12051 determines the collision risk indicating the risk of collision with each obstacle, and when the collision risk is equal to or higher than the set value and there is a possibility of collision, the microcomputer 12051 is connected via the audio speaker 12061 or the display unit 12062. By outputting an alarm to the driver and performing forced deceleration or avoidance steering via the drive system control unit 12010, driving assistance for collision avoidance can be performed.

At least one of the imaging units 12101 to 12104 may be an infrared camera that detects infrared rays. For example, the microcomputer 12051 can recognize a pedestrian by determining whether a pedestrian is present in the captured images of the imaging units 12101 to 12104. Such pedestrian recognition is, for example, whether or not the user is a pedestrian by performing a pattern matching process on a sequence of feature points indicating the outline of an object and a procedure for extracting feature points in the captured images of the imaging units 12101 to 12104 as infrared cameras. It is carried out by the procedure for determining. When the microcomputer 12051 determines that there is a pedestrian in the captured images of the imaging units 12101 to 12104 and recognizes the pedestrian, the audio image output unit 12052 has a rectangular contour line for emphasizing the recognized pedestrian. The display unit 12062 is controlled so as to be superimposed and displayed. Moreover, the audio | voice image output part 12052 may control the display part 12062 so that the icon etc. which show a pedestrian may be displayed on a desired position.

Heretofore, an example of a vehicle control system to which the technology according to the present disclosure can be applied has been described. The technology according to the present disclosure can be applied to the microcomputer 12051 among the configurations described above. Specifically, the signal processing apparatus 100 can be applied to a microcomputer 12051. By applying the technology according to the present disclosure to the microcomputer 12051, when obtaining an HDR image, the performance of the dynamic range is not reduced, the processing time is shortened, and the trade-off between the image quality and the processing time is flexibly adjusted. It becomes possible to do.

<3. Summary>
As described above, according to the embodiment of the present disclosure, the number of motion detections is set to a smaller number than the number of times of execution, thereby avoiding harmful effects such as fatal blur and significant S / N reduction of moving objects in HDR images. However, the signal processing device 100 that can reduce the amount of calculation when generating the HDR image is provided.

Further, according to the embodiment of the present disclosure, it is possible to control the number of times of motion adaptation processing from the finally determined shutter time of each frame, and shorten the processing time while maintaining the image quality of the HDR image. A signal processing device 100 is provided.

Further, according to the embodiment of the present disclosure, it is possible to control the number of executions of the motion adaptation process within a range in which the total synthesis process time does not exceed a predetermined value, The signal processing apparatus 100 is provided in which the image quality of the HDR image including the image quality is automatically adjusted.

Further, according to the embodiment of the present disclosure, it is possible to control the number of executions of the motion adaptation process from the result of motion detection, and to shorten the processing time while maintaining the image quality of the HDR image including the moving object. A signal processing apparatus 100 that can be used is provided.

Each step in the processing executed by each device in this specification does not necessarily have to be processed in chronological order in the order described as a sequence diagram or flowchart. For example, each step in the processing executed by each device may be processed in an order different from the order described as the flowchart, or may be processed in parallel.

In addition, it is possible to create a computer program for causing hardware such as CPU, ROM, and RAM incorporated in each device to exhibit functions equivalent to the configuration of each device described above. A storage medium storing the computer program can also be provided. Moreover, a series of processes can also be realized by hardware by configuring each functional block shown in the functional block diagram with hardware.

The preferred embodiments of the present disclosure have been described in detail above with reference to the accompanying drawings, but the technical scope of the present disclosure is not limited to such examples. It is obvious that a person having ordinary knowledge in the technical field of the present disclosure can come up with various changes or modifications within the scope of the technical idea described in the claims. Of course, it is understood that it belongs to the technical scope of the present disclosure.

In addition, the effects described in this specification are merely illustrative or illustrative, and are not limited. That is, the technology according to the present disclosure can exhibit other effects that are apparent to those skilled in the art from the description of the present specification in addition to or instead of the above effects.

The following configurations also belong to the technical scope of the present disclosure.
(1)
A composition processing unit that performs composition processing at least N-1 times on images of N frames (N is an integer of 3 or more) having different exposure times;
A motion adaptation processing unit for performing motion adaptation processing at the time of one synthesis using two of the groups of images in the synthesis processing unit;
With
The signal processing device, wherein the number of motion adaptation processes in the motion adaptation processing unit when the synthesis processing unit performs the N-1 synthesis processes is N-2 times or less.
(2)
The signal processing according to (1), wherein the motion adaptation processing unit performs the motion adaptation processing only on an image whose exposure time of the image that is a target of the synthesis processing in the synthesis processing unit is a predetermined threshold value or more. apparatus.
(3)
The signal processing apparatus according to (1), wherein the motion adaptive processing unit performs the motion adaptive processing at the maximum number of times that the synthesis processing execution time in the synthesis processing unit does not exceed a predetermined threshold.
(4)
A motion detection processing unit that performs motion detection processing on the two images synthesized by the synthesis processing unit;
The motion adaptation processing unit determines whether to perform motion adaptation processing on the two images according to the magnitude of motion between the two images detected by the motion detection processing. The signal processing device according to any one of 3).
(5)
The signal processing apparatus according to any one of (1) to (4), wherein the synthesis processing unit performs a process of generating a synthesis ratio of the two images to be synthesized as the synthesis process.
(6)
The signal processing according to any one of (1) to (4), wherein the synthesis processing unit performs synthesis of the two images based on a synthesis ratio of the two images to be synthesized as the synthesis processing. apparatus.
(7)
The signal processing apparatus according to any one of (1) to (6), wherein the composition processing unit performs composition processing at least N−1 times in order from the image with the long exposure time.
(8)
Processor
Performing composition processing at least N−1 times on images of N frames (N is an integer of 3 or more) having different exposure times;
Performing a motion adaptation process in one synthesis using two of the group of images;
Including
A signal processing method in which the number of motion adaptation processes when performing N-1 synthesis processes is N-2 or less.
(9)
On the computer,
Performing composition processing at least N−1 times on images of N frames (N is an integer of 3 or more) having different exposure times;
Performing a motion adaptation process in one synthesis using two of the group of images;
And execute
A computer program in which the number of motion adaptation processes when performing the N-1 synthesis processes is N-2 or less.

10: Image sensor 11: Long accumulation pixel 12: Medium accumulation pixel 13: Short accumulation pixel 14: Ultra short accumulation pixel 100: Signal processing device

Claims

A composition processing unit that performs composition processing at least N-1 times on images of N frames (N is an integer of 3 or more) having different exposure times;
A motion adaptation processing unit for performing motion adaptation processing at the time of one synthesis using two of the groups of images in the synthesis processing unit;
With
The signal processing device, wherein the number of motion adaptation processes in the motion adaptation processing unit when the synthesis processing unit performs the N-1 synthesis processes is N-2 times or less.
The signal processing apparatus according to claim 1, wherein the motion adaptation processing unit performs motion adaptation processing only on an image whose exposure time of the image that is a target of the synthesis processing in the synthesis processing unit is a predetermined threshold value or more. .
The signal processing apparatus according to claim 1, wherein the motion adaptive processing unit performs the motion adaptive processing at the maximum number of times that the synthesis processing execution time in the synthesis processing unit does not exceed a predetermined threshold.
A motion detection processing unit that performs motion detection processing on the two images synthesized by the synthesis processing unit;
The motion adaptation processing unit according to claim 1, wherein the motion adaptation processing unit determines whether to perform motion adaptation processing on the two images according to a magnitude of motion between the two images detected by the motion detection processing. Signal processing device.
The signal processing apparatus according to claim 1, wherein the synthesis processing unit performs a process of generating a synthesis ratio of the two images to be synthesized as the synthesis process.
The signal processing apparatus according to claim 1, wherein the synthesis processing unit performs synthesis of the two images based on a synthesis ratio of the two images to be synthesized as the synthesis processing.
The signal processing apparatus according to claim 1, wherein the synthesis processing unit performs synthesis processing at least N-1 times in order from the image having the long exposure time.
Processor
Performing composition processing at least N−1 times on images of N frames (N is an integer of 3 or more) having different exposure times;
Performing a motion adaptation process in one synthesis using two of the group of images;
Including
A signal processing method in which the number of motion adaptation processes when performing N-1 synthesis processes is N-2 or less.
On the computer,
Performing composition processing at least N−1 times on images of N frames (N is an integer of 3 or more) having different exposure times;
Performing a motion adaptation process in one synthesis using two of the group of images;
And execute
A computer program in which the number of motion adaptation processes when performing the N-1 synthesis processes is N-2 or less.