WO2022219874A1

WO2022219874A1 - Signal processing device and method, and program

Info

Publication number: WO2022219874A1
Application number: PCT/JP2022/002779
Authority: WO
Inventors: 駿阿久津
Original assignee: ソニーセミコンダクタソリューションズ株式会社
Priority date: 2021-04-15
Filing date: 2022-01-26
Publication date: 2022-10-20
Also published as: JP2022163882A

Abstract

The present technology relates to a signal processing device and method and a program that make it possible to suppress the generation of artifacts. The signal processing device comprises: a first filter processing unit that generates a first low frequency luminance signal by filtering a first image signal using a first filter that extracts low frequency components; a second filter processing unit that generates a second low frequency luminance signal having the same phase as the first low frequency luminance signal by filtering a second image signal having a different phase from the first image signal using a second filter that is different from the first filter, extracts low frequency components, and corrects phase differences; and a movement determination unit that determines movement on the basis of the first low frequency luminance signal and the second low frequency luminance signal. The present technology can be applied to a signal processing device.

Description

SIGNAL PROCESSING APPARATUS AND METHOD, AND PROGRAM

The present technology relates to a signal processing device, method, and program, and more particularly to a signal processing device, method, and program capable of suppressing the occurrence of artifacts.

Conventionally, an image sensor having a sub-pixel structure in which large pixels and small pixels with different sizes (sensitivities) are provided in one unit pixel is known (see, for example, Patent Document 1).

By using an image sensor with such a sub-pixel structure, an image with a wider dynamic range can be obtained by HDR (High Dynamic Range) synthesis of an image captured with large pixels and an image captured with small pixels. be able to.

However, if, for example, the exposure time is set to differ between large pixels and small pixels, there will be differences in imaging results for moving subjects and subject areas such as LEDs (Light Emitting Diodes) between large pixels and small pixels. Since such a difference causes moving object artifacts during HDR synthesis, it is common to perform motion determination and perform correction according to the determination result.

For example, Patent Literature 2 discloses detecting the difference between a plurality of images captured with different exposure times and combining the plurality of images with a composition coefficient according to the difference.

WO2016/147885 JP 2018-19387 A

By the way, in the motion determination described above, the presence or absence of motion is determined based on the difference between two images having different sensitivities, that is, the magnitude of the difference between the two image signals.

Such motion determination is performed on the premise that if the subject is stationary, the image signals (pixel values) will match if the sensitivity difference between the image signals with different sensitivities is corrected.

However, in an image sensor with a sub-pixel structure, the arrangement positions of large pixels and small pixels are different in the first place. The premise does not apply (does not hold).

Therefore, even if motion is judged based on two image signals obtained by an image sensor with a sub-pixel structure, and HDR synthesis is performed according to the judgment result, erroneous judgment and correction of motion judgment will occur, and HDR Artifacts will occur in the image generated by synthesis.

This technology has been developed in view of this situation, and is intended to suppress the occurrence of artifacts.

A signal processing device according to one aspect of the present technology performs filtering on a first image signal using a first filter that extracts a low-frequency component, thereby generating a first low-frequency luminance signal. and a second filter for extracting a low-frequency component and correcting a phase difference, unlike the first filter, for a second image signal having a phase different from that of the first image signal. A second filter processing unit that generates a second low-frequency luminance signal having the same phase as the first low-frequency luminance signal by performing filtering with the first low-frequency luminance signal and the second low-frequency luminance signal and a motion determination unit that performs motion determination based on the low-frequency luminance signal.

A signal processing method or program according to one aspect of the present technology generates a first low-frequency luminance signal by filtering a first image signal using a first filter that extracts low-frequency components, A second image signal having a phase different from that of the first image signal is filtered by a second filter that extracts low-frequency components and corrects a phase difference, unlike the first filter. a step of generating a second low-frequency luminance signal having the same phase as the first low-frequency luminance signal and performing motion determination based on the first low-frequency luminance signal and the second low-frequency luminance signal; including.

In one aspect of the present technology, a first low-frequency luminance signal is generated by filtering a first image signal using a first filter that extracts low-frequency components, and the first image is Unlike the first filter, a second image signal having a different phase from the signal is filtered by a second filter that extracts low-frequency components and corrects the phase difference. A second low-frequency luminance signal having the same phase as the low-frequency luminance signal is generated, and motion determination is performed based on the first low-frequency luminance signal and the second low-frequency luminance signal.

It is a figure explaining the image sensor of a sub-pixel structure. It is a figure explaining the phase shift of a large pixel image and a small pixel image. It is a figure explaining motion determination. It is a figure which shows the structural example of a signal processing apparatus. FIG. 4 is a diagram showing an example of a large-pixel luminance generation filter and a small-pixel luminance generation filter; 10 is a flowchart for explaining image compositing processing; It is a figure explaining the effect of this technique. FIG. 10 is a diagram showing another example of a small-pixel luminance generation filter; It is a figure which shows the structural example of a computer. 1 is a block diagram showing an example of a schematic configuration of a vehicle control system; FIG. FIG. 4 is an explanatory diagram showing an example of installation positions of an outside information detection unit and an imaging unit;

Embodiments to which the present technology is applied will be described below with reference to the drawings.

<First Embodiment>
<About this technology>
The present technology is intended to suppress the occurrence of artifacts by using a low-frequency luminance signal for motion determination and performing phase difference correction only on the signal used for motion determination.

In the following, an example of performing HDR synthesis based on two image signals obtained by an image sensor with a sub-pixel structure will be described. In such a case, the image sensor is provided with unit pixels arranged, for example, as shown in FIG.

For example, in one unit pixel PX1, a large pixel SP1 and a small pixel SP2 having different sensitivities are provided adjacent to each other.

That is, the large pixel SP1 and the small pixel SP2 are formed so that the area (size) of the light receiving surface of the large pixel SP1 is larger than that of the small pixel SP2, and the large pixel SP1 is larger than the small pixel SP2. also has high sensitivity. Also, here, the small pixel SP2 is arranged on the upper right side of the large pixel SP1 in the figure.

In an image sensor with a sub-pixel structure, multiple unit pixels with the same configuration as the unit pixel PX1 are arranged in a matrix.

Furthermore, in this example, each unit pixel is provided with a color filter of one of R (red), G (green), and B (blue). unit pixels are arranged in a checkered pattern.

That is, in a pixel row including a unit pixel PX1 and composed of a plurality of unit pixels arranged in the horizontal direction in the drawing, a unit pixel having a B color filter and a unit pixel having a G color filter are arranged in the horizontal direction in the drawing. are arranged alternately.

In addition, in the pixel row provided vertically adjacent to the pixel row including the unit pixel PX1, the unit pixel having the R color filter and the unit pixel having the G color filter are arranged in the figure. They are arranged alternately in the middle and the horizontal direction.

For example, focusing on a unit pixel PX1 having a G color filter, the unit pixel adjacent in the vertical direction in the figure of the unit pixel PX1 has an R color filter, and the unit pixel PX1 is adjacent in the horizontal direction in the figure. Each unit pixel has a B color filter.

It should be noted that each unit pixel of the image sensor may have any color filter, and each unit pixel may have a configuration in which no color filter is provided.

Further, hereinafter, an image signal of an image picked up by large pixels provided in an image sensor, that is, an image signal composed of pixel signals read out from each large pixel by performing an exposure operation (imaging processing) on the image sensor is converted into a large image signal. Also referred to as a pixel image signal, an image based on a large pixel image signal is also referred to as a large pixel image.

Similarly, hereinafter, an image signal of an image picked up by small pixels provided in an image sensor, that is, an image signal composed of pixel signals read out from each small pixel after an exposure operation is performed on the image sensor, is referred to as a small pixel image signal. An image based on a small pixel image signal is also called a small pixel image.

Now, let's consider capturing a large-pixel image and a small-pixel image with an image sensor having unit pixels arranged as shown in Fig. 1, and performing HDR synthesis.

In this case, motion determination is performed based on the large pixel image (large pixel image signal), small pixel image (small pixel image signal), and HDR synthesis gain, and a predetermined synthesis coefficient obtained from the result of the motion determination is used. Assume that HDR synthesis of a large pixel image and a small pixel image is performed.

Note that the HDR synthesis gain is a coefficient for matching the brightness of the large pixel image and the small pixel image, and is calculated in advance based on, for example, the sensitivity difference and the exposure time difference between the large pixel image and the small pixel image. Also, the synthesis coefficient is a coefficient that indicates the synthesis ratio (mixing ratio) of the large pixel image and the small pixel image at the time of HDR synthesis.

For example, when the subject shown in FIG. 2 is captured by an image sensor, the large pixel image and the small pixel image are basically the same image.

In other words, in both the large-pixel image and the small-pixel image, subjects such as people in the room, the window of the room, and the building outside the window are shown. In this example, for example, in the region R11, the window portion of an outdoor building exists as a subject.

However, the phases of the large pixel image and the small pixel image are different, that is, the positions of the corresponding pixels are different, so there is a deviation in the imaging result.

Specifically, for example, the pixel corresponding to the large pixel SP1 in FIG. 1 in the large pixel image is the pixel of interest. At this time, in the small pixel image, the pixel corresponding to the pixel of interest, that is, the pixel on the small pixel image that has the same positional relationship as the position of the pixel of interest in the large pixel image (hereinafter also referred to as the corresponding pixel) is the small pixel SP2. becomes a pixel corresponding to .

In this way, the target pixel and the corresponding pixel have the same positional relationship, but as shown in FIG. 1, the large pixel SP1 and the small pixel SP2 have different arrangement positions, that is, there is a deviation in arrangement position. Therefore, even if the subject is stationary, different subjects (different parts of the subject) are captured in the pixel of interest and the corresponding pixel.

Also, in motion determination, the presence or absence of motion is determined based on the magnitude of the difference between the large-pixel image and the small-pixel image, which have different sensitivities.

For example, it is assumed that the signal levels (pixel values) shown in FIG. 3 can be obtained with respect to the brightness of the subject with large pixels and small pixels.

In FIG. 3, the horizontal axis indicates the brightness of the subject, and the vertical axis indicates the pixel signal value (pixel value), that is, the signal level when the subject is captured.

The polygonal line L11 indicates the signal level (pixel signal value) obtained in the large pixel SP1 for each brightness, and the straight line L12 indicates the signal level (pixel signal value) obtained in the small pixel SP2 for each brightness. pixel signal value).

Now, suppose that the pixel signal of the large pixel SP1 and the pixel signal of the small pixel SP2 have the same phase, that is, the center position of the large pixel SP1 and the center position of the small pixel SP2 are at the same position.

Further, as a result of imaging with the subject stationary, for example, as indicated by an arrow Q11, a signal level P11 is obtained as the pixel signal value of the small pixel SP2, and a signal level P11 is obtained as the pixel signal value of the large pixel SP1. Suppose that P12 is obtained.

In such a case, the value obtained by multiplying the signal level P11 by the HDR synthesis gain matches the signal level P12.

Therefore, for example, if the difference between the signal level P11 multiplied by the HDR synthesis gain and the signal level P12 is compared with a predetermined threshold value, noise level, or the like to determine motion, a determination result indicating that there is no motion can be obtained.

On the other hand, as a result of capturing an image while the subject is moving, a signal level P21 is obtained as the pixel signal value of the small pixel SP2 as indicated by an arrow Q12, and a signal level P22 is obtained as the pixel signal value of the large pixel SP1. is obtained.

In such a case, the value obtained by multiplying the signal level P21 by the HDR synthesis gain matches the signal level P23 at the same brightness on the polygonal line L11, but the actual signal level of the large pixel SP1 Does not match P22.

That is, even if the signal level P21 is multiplied by the HDR synthesis gain, the resulting value does not match the signal level P22. A difference occurs.

This is because the subject to be imaged is a moving subject, and if the exposure time, etc., differs between the large-pixel image and the small-pixel image, the brightness of the subject changes due to changes in the subject captured between the large-pixel SP1 and small-pixel SP2 pixels. Because it changes.

Therefore, for example, if the difference between the multiplication result of the HDR synthesis gain for the signal level P21 and the signal level P22 is compared with a predetermined threshold value, noise level, etc., and motion determination is performed, a determination result indicating that there is motion can be obtained.

However, as described above, the image signal of the large pixel image and the image signal of the small pixel image, that is, the pixel signal of the large pixel and the pixel signal of the small pixel are out of phase.

Therefore, if the subject is stationary, the pixel signal obtained by multiplying the pixel signal of the small pixel by the HDR synthesis gain will match the pixel signal of the large pixel corresponding to that small pixel. Therefore, the premise of motion determination that pixel signals of small pixels and large pixels match does not apply.

Therefore, when motion determination is performed based on the pixel signals of the large pixels and the pixel signals of the small pixels that are actually obtained, the phase difference between the pixel signals of the large pixels and the small pixels can be used even for a stationary object. The signal difference may become large, resulting in an erroneous determination.

Also, if HDR synthesis is performed using such motion determination results, there is a risk that erroneous correction of motion may occur. Such erroneous determination and erroneous correction cause artifacts in an image generated by HDR synthesis (hereinafter also referred to as a synthesized image).

In order to suppress erroneous determination and erroneous correction due to such a phase difference, for example, phase difference correction is performed on the image signal of the small pixel image, and the pixel signal of the large pixel and the pixel signal of the small pixel are signals of the same phase. It is also conceivable to perform motion determination and HDR synthesis as

However, when performing phase difference correction, it is difficult to correct (shift) the phase of the high-frequency signal without destroying the pattern of the image, that is, without causing distortion or the like.

In addition, when phase difference correction is performed on an image, artifacts such as false colors and rattling occur, causing adverse effects such as loss of sharpness of the image. and so on, the cost becomes high. In particular, when phase difference correction is performed, even if noise reduction or the like is performed, only a synthesized image with a lower sharpness than when motion determination is performed without phase difference correction may be obtained.

Therefore, in the present technology, by using a low-frequency luminance signal for motion determination and performing phase difference correction only on the signal used for motion determination, it is possible to suppress the occurrence of erroneous determination and erroneous correction. I made it As a result, it is possible to suppress the occurrence of artifacts and obtain a composite image of higher quality.

In particular, in this technology, a low-frequency luminance signal is used to determine motion, and such a low-frequency signal can be phase-corrected (shifted) more easily than a high-frequency signal. Therefore, according to the present technology, occurrence of erroneous determination and erroneous correction can be suppressed without causing an increase in cost such as an increase in circuit scale.

In addition, since this technology uses signals that have not been phase-difference-corrected to generate a composite image, it is less likely that adverse effects such as loss of image sharpness due to the occurrence of artifacts will occur.

<Configuration example of signal processing device>
FIG. 4 is a diagram illustrating a configuration example of an embodiment of a signal processing device to which the present technology is applied.

The signal processing device 11 shown in FIG. 4 is composed of, for example, an in-vehicle camera device.

The signal processing device 11 has an image sensor 21 , a signal generation section 22 , a motion determination section 23 and an HDR synthesis section 24 .

The image sensor 21 is an image sensor having a sub-pixel structure, and functions as an imaging unit that generates an image signal of a large-pixel image and an image signal of a small-pixel image by performing imaging.

For example, the image sensor 21 is provided with a plurality of large pixels and small pixels having different sensitivities and phases (arrangement positions) from each other in the arrangement shown in FIG.

The image sensor 21 captures an image of a subject by photoelectrically converting incident light, and supplies image signals of the resulting large-pixel image and small-pixel image to the signal generating unit 22 and the HDR synthesizing unit 24 . . That is, an image signal of a large pixel image is generated by imaging with a plurality of large pixels forming the image sensor 21, and an image signal of a small pixel image is generated by imaging by a plurality of small pixels forming the image sensor 21. FIG. The image signal of the large pixel image and the image signal of the small pixel image are signals with phases different from each other.

Although an example in which the image sensor 21 is provided in the signal processing device 11 will be described here, the image sensor 21 may be provided outside the signal processing device 11 .

The signal generation unit 22 generates a low-frequency luminance signal (low-frequency luminance signal) for motion determination based on the image signal of the large pixel image and the image signal of the small pixel image supplied from the image sensor 21, Supplied to the motion determination unit 23 .

The signal generating section 22 has a large pixel filtering section 31 and a small pixel filtering section 32 .

The large-pixel filter processing unit 31 performs filtering (filtering) on the image signal of the large-pixel image using a large-pixel luminance generation filter to convert the large-pixel low-frequency signal, which is the low-frequency luminance signal of the large-pixel image. Generate a luminance signal.

For example, the large-pixel luminance generation filter is a low-pass filter that extracts low-frequency components, more specifically, luminance components below a predetermined frequency (low-frequency luminance components) from the image signal of the large-pixel image.

The small pixel filter processing unit 32 performs filtering (filtering) on the image signal of the small pixel image using a small pixel luminance generation filter, thereby reducing the low-frequency luminance of the phase difference corrected small pixel image. A small pixel low frequency luminance signal is generated. The small-pixel low-frequency luminance signal thus obtained is a signal having the same phase as the large-pixel low-frequency luminance signal.

Here, the luminance generation filter for small pixels is a filter different from the luminance generation filter for large pixels. For example, the small-pixel luminance generation filter functions as a low-pass filter that extracts low-frequency components (low-frequency luminance components) from the image signal of the small-pixel image. It is considered as a filter or the like that also realizes phase difference correction such as

The signal generation unit 22 converts the large pixel low-frequency luminance signal obtained by the large pixel filtering unit 31 and the small pixel low-frequency luminance signal obtained by the small pixel filtering unit 32 into low-frequency signals for motion determination. It is supplied to the motion determination unit 23 as a luminance signal.

The motion determination unit 23 performs motion determination based on the large-pixel low-frequency luminance signal and the small-pixel low-frequency luminance signal supplied from the signal generation unit 22, and supplies the determination result to the HDR synthesis unit 24.

The HDR synthesizing unit 24 HDR synthesizes the image signal of the large pixel image and the image signal of the small pixel image supplied from the image sensor 21 based on the determination result of the motion determination supplied from the motion determining unit 23 . The synthesized image obtained as a result is output to the subsequent stage.

<Generation of low-frequency luminance signal>
Here, examples of the large-pixel luminance generation filter and the small-pixel luminance generation filter will be described.

For example, when the pixel arrangement of the image sensor 21 is the arrangement shown in FIG. 1, the large pixel luminance generation filter and the small pixel luminance generation filter are the filters shown in FIG.

In FIG. 5, the portion indicated by arrow Q21 shows an example of the luminance generation filter for large pixels, and the portion indicated by arrow Q22 shows an example of the luminance generation filter for small pixels.

Here, the large-pixel luminance generation filter is a filter with a size of 3×3, and the numerical values written in each square are the coefficients of the filter that are multiplied by the pixel values of the pixels corresponding to those squares. represents. Note that the size of the filter here means the number of taps of the filter, that is, the number of pixels (size) of the pixel area to be filtered.

As a specific example, a case will be described in which filtering is performed using the large pixel luminance generation filter shown in FIG.

In this case, the values of the pixel signals (pixel values) of the large pixels adjacent to the upper left, right above, and upper right of the large pixel SP1 in FIG. 1 are multiplied by the coefficient values 1, 2, and 1. be.

Further, in FIG. 1, for each pixel signal value (pixel value) of the large pixel adjacent to the left side of the large pixel SP1, the large pixel SP1, and the large pixel adjacent to the right side of the large pixel SP1 in FIG. , are multiplied by the coefficient values 2, 4, and 2.

Similarly, in FIG. 1, the pixel signal values (pixel values) of the large pixels adjacent to the lower left, just below, and lower right of the large pixel SP1 are multiplied by

coefficient values

1, 2, and 1. be.

Furthermore, the sum of a total of 9 pixel values multiplied by the coefficients in this way (the sum of the pixel values after coefficient multiplication) is obtained, and the sum is divided by 16, which is the sum of the 9 coefficients, The result of the division is the value of the pixel signal of one pixel obtained by filtering, that is, the value of the large pixel low-frequency luminance signal for one pixel.

In this case, the phase of the newly generated large pixel low-frequency luminance signal, that is, the position of the pixel is the center position of the large pixel SP1.

It should be noted that when applying the luminance generation filter for large pixels, the coefficient indicated by the arrow Q21 is used regardless of the color of the large pixel color filter in the center of the target 3×3 pixel region. In other words, pixel signals of large pixels having R, G, and B color filters are used to generate one (one pixel) large pixel low-frequency luminance signal.

On the other hand, as indicated by an arrow Q22, the small pixel luminance generation filter is also a 3×3 size filter like the large pixel luminance generation filter. Here, the numerical values written in each rectangle represent the coefficients of the filters by which the pixel values of the pixels corresponding to those rectangles are multiplied.

The small-pixel luminance generation filter is a 3×3 filter, and the coefficients of the pixels forming the uppermost pixel row and the rightmost pixel column in the target pixel region are All are 0. Therefore, the small-pixel luminance generation filter is substantially a 2×2 size filter.

As a specific example, a case will be described in which filtering is performed using the small pixel luminance generation filter shown in FIG.

In this case, the value of the pixel signal (pixel value) of each of the small pixels adjacent to the left of the small pixel SP2 in FIG.

Also, in FIG. 1, the value of the pixel signal (pixel value) of the small pixels positioned to the left and directly below the small pixel SP2 is multiplied by 1, which is the value of the coefficient.

Furthermore, the sum of the four pixel values multiplied by the coefficients in this way is obtained, the sum is divided by 4 which is the sum of the four coefficients, and the result of the division is the one obtained by filtering. It is the value of the pixel signal of the pixel, that is, the value of the small pixel low-frequency luminance signal for one pixel.

In this case, the phase of the newly generated small-pixel low-frequency luminance signal, that is, the position of the pixel is the center position of the four small-pixels used to generate the small-pixel low-frequency luminance signal. It becomes the center position of the pixel SP1. Therefore, it can be seen that the generated small pixel low-frequency luminance signal has the same phase as the corresponding large pixel low-frequency luminance signal, and phase difference correction is realized by the small pixel luminance generation filter.

Note that when applying the small pixel luminance generation filter, the arrow The factor shown in Q22 is used.

As described above, the large-pixel luminance generation filter and the small-pixel luminance generation filter are filters having at least different coefficients.

<Description of image synthesis processing>
Next, the operation of the signal processing device 11 will be described. That is, the image synthesizing process by the signal processing device 11 will be described below with reference to the flowchart of FIG.

In step S11, the large pixels and small pixels of the image sensor 21 receive and photoelectrically convert the incident light to capture large pixel images and small pixel images.

For example, large pixels and small pixels are imaged so that the length of the exposure time and the timing of the start or end of exposure are different.

The image sensor 21 converts the image signal of the large pixel image obtained by imaging with each large pixel and the image signal of the small pixel image obtained by imaging with each small pixel into a signal generation unit 22 and an HDR synthesis unit. 24.

In step S<b>12 , the large pixel filter processing unit 31 performs filtering on the image signal of the large pixel image supplied from the image sensor 21 using the large pixel luminance generation filter, thereby reducing the large pixel intensity for each of a plurality of pixel positions. generating a frequency luminance signal.

In step S<b>13 , the small pixel filter processing unit 32 filters the image signal of the small pixel image supplied from the image sensor 21 using the small pixel luminance generation filter, thereby reducing the small pixel intensity for each of a plurality of pixel positions. generating a frequency luminance signal.

In the small pixel filter processing unit 32, a low-frequency luminance signal is extracted from the image signal and phase difference correction (phase correction) is also performed. A phase luminance signal is obtained.

The signal generation unit 22 supplies the large-pixel low-frequency luminance signal and the small-pixel low-frequency luminance signal obtained by the above processing to the motion determination unit 23 .

Although the processing of steps S12 and S13 is executed before demosaic processing is performed on the large-pixel image and the small-pixel image, it may be executed after the demosaicing processing.

In step S14, the motion determination unit 23 performs motion determination based on the large-pixel low-frequency luminance signal and the small-pixel low-frequency luminance signal supplied from the signal generation unit 22, and supplies the determination result to the HDR synthesis unit 24.

For example, the motion determination unit 23 calculates the difference between the value obtained by multiplying the small pixel low frequency luminance signal by the HDR synthesis gain and the large pixel low frequency luminance signal for each pixel position, and calculates the difference obtained, The motion amount of the subject is obtained by comparing the noise level and a predetermined motion determination threshold. As a result of the motion determination, the amount of motion of the subject for each pixel position is obtained.

In step S15, the HDR synthesizing unit 24 HDR synthesizes the image signal of the large pixel image and the image signal of the small pixel image supplied from the image sensor 21 based on the determination result of the motion determination supplied from the motion determining unit 23. By doing so, an image signal of a synthesized image is generated.

For example, the HDR synthesizing unit 24 generates a synthesizing coefficient for each pixel position based on the image signal of the large pixel image, adjusts the synthesizing coefficient by performing motion compensation according to the determination result of the motion determination, and finalizes the Get the composite coefficient. Then, the HDR synthesizing unit 24 synthesizes (mixes) the value obtained by multiplying the image signal of the small pixel image by the HDR synthesis gain and the image signal of the large pixel image using the synthesizing coefficient to obtain the image signal of the synthesized image. get

When the image signal of the synthesized image obtained in this way is output to the subsequent stage by the HDR synthesizing unit 24, the image synthesizing process ends.

As described above, the signal processing device 11 generates a large-pixel low-frequency luminance signal and a small-pixel low-frequency luminance signal of the same phase, performs motion determination, synthesizes the large-pixel image and the small-pixel image, and combines them. Generate an image. By performing motion determination using low-frequency luminance signals of the same phase in this way, it is possible to suppress the occurrence of erroneous determination and erroneous correction. As a result, it is possible to suppress the occurrence of artifacts and obtain a composite image of higher quality.

For example, by performing phase difference correction, the motion determination unit 23 can improve the determination accuracy of motion determination as shown in FIG. Note that FIG. 7 indicates that the brighter the part of the subject (image), the greater the motion amount obtained in the motion determination.

In FIG. 7, in the portion indicated by the arrow Q31, a low-frequency luminance signal is extracted from the large pixel image and the small pixel image obtained by imaging the subject shown in FIG. The result of motion determination is shown.

For example, in the region R11, a window of a building exists as a subject, and in the judgment result indicated by the arrow Q31, the window of the building has a particularly fine pattern, that is, a portion containing high-frequency components. It can be seen that an erroneous determination has occurred.

On the other hand, in FIG. 7, in the portion indicated by arrow Q32, the signal generator 22 generates a low-frequency luminance signal from the large-pixel image and the small-pixel image obtained by imaging the subject shown in FIG. The determination result when extracting and correcting the phase difference and performing the motion determination by the motion determination unit 23 is shown. That is, the portion indicated by the arrow Q32 shows the result of motion determination obtained by the processing performed by the signal generation unit 22 and the motion determination unit 23 of the signal processing device 11 .

In this example, it can be seen that the occurrence of erroneous determinations is suppressed compared to the example indicated by the arrow Q31 even in a portion containing high-frequency components such as the region R11.

As described above, in the present technology, phase difference correction is performed and motion determination is performed using a low-frequency luminance signal. Thus, it is possible to suppress the occurrence of erroneous determination at the time of motion determination.

Moreover, by using the determination result obtained in this way, the large pixel image and the small pixel image obtained by the image sensor 21, which are not low-frequency luminance signals, are used as they are to perform HDR synthesis. Artifacts such as jerkyness can also be made less likely to cause adverse effects such as loss of image sharpness. As a result, a composite image of higher quality and wider dynamic range can be obtained.

<Modification>
Note that FIG. 5 illustrates an example in which the large-pixel luminance generation filter and the small-pixel luminance generation filter are filters of size 3×3, but these filters may be of any size, such as 5×5. may be a filter of

Also, the large-pixel luminance generation filter and the small-pixel luminance generation filter may be filters of the same size, or may be filters of different sizes.

Furthermore, the small pixel luminance generation filter may be, for example, the filter shown in FIG.

In FIG. 8, each rectangle represents a pixel in the pixel area to which the small pixel luminance generation filter is applied, and the numerical value in each rectangle indicates the coefficient of the filter by which the pixel value of the pixel corresponding to the rectangle is multiplied. represent.

In this example, the small pixel luminance generation filter is a 3×3 size filter, and the pixel value of the pixel at the center position of the target pixel area and and the pixel values of the adjacent pixels below are multiplied by 63 as a factor.

Further, the pixel values of the pixels adjacent to the upper left, upper, right, and lower right in the drawing are multiplied by 1 as coefficients to the pixel at the center position of the pixel region, and the pixel at the center position of the pixel region is multiplied by 1 as a coefficient. On the other hand, in the drawing, the pixel value of the pixel adjacent to the upper right is multiplied by 0 as a coefficient.

Then, the sum of the pixel values of each pixel multiplied by these coefficients is divided by 256, and the result of the division is the value of the pixel signal of one pixel obtained by filtering, that is, the small pixel low-frequency luminance signal for one pixel. is assumed to be the value of By doing so, extraction of the low-frequency luminance signal and correction of the phase difference can also be realized.

In addition, an example of performing motion determination and HDR synthesis on image signals of a large-pixel image and a small-pixel image captured by a single image sensor 21 having a sub-pixel structure has been described above. However, the two images to be subjected to motion determination and HDR synthesis are not limited to this, and may be any images as long as they are images of the same subject and of different phases.

For example, two images with different phases may be captured using two cameras, motion determination and HDR synthesis may be performed based on the two images, and a synthesized image may be generated.

<Computer configuration example>
By the way, the series of processes described above can be executed by hardware or by software. When executing a series of processes by software, a program that constitutes the software is installed in the computer. Here, the computer includes, for example, a computer built into dedicated hardware and a general-purpose personal computer capable of executing various functions by installing various programs.

FIG. 9 is a block diagram showing a hardware configuration example of a computer that executes the series of processes described above by a program.

In the computer, a CPU (Central Processing Unit) 501, a ROM (Read Only Memory) 502, and a RAM (Random Access Memory) 503 are interconnected by a bus 504.

An input/output interface 505 is further connected to the bus 504 . An input unit 506 , an output unit 507 , a recording unit 508 , a communication unit 509 and a drive 510 are connected to the input/output interface 505 .

The input unit 506 consists of a keyboard, mouse, microphone, imaging device, and the like. The output unit 507 includes a display, a speaker, and the like. A recording unit 508 is composed of a hard disk, a nonvolatile memory, or the like. A communication unit 509 includes a network interface and the like. A drive 510 drives a removable recording medium 511 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory.

In the computer configured as described above, for example, the CPU 501 loads the program recorded in the recording unit 508 into the RAM 503 via the input/output interface 505 and the bus 504 and executes the above-described series of programs. is processed.

The program executed by the computer (CPU 501) can be provided by being recorded on a removable recording medium 511 such as package media, for example. Also, the program can be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

In the computer, the program can be installed in the recording unit 508 via the input/output interface 505 by loading the removable recording medium 511 into the drive 510 . Also, the program can be received by the communication unit 509 and installed in the recording unit 508 via a wired or wireless transmission medium. In addition, the program can be installed in the ROM 502 or the recording unit 508 in advance.

The program executed by the computer may be a program that is processed in chronological order according to the order described in this specification, or may be executed in parallel or at a necessary timing such as when a call is made. It may be a program in which processing is performed.

Also, the signal processing device 11 described above can be used in various cases for sensing light such as visible light, infrared light, ultraviolet light, and X-rays, for example, as follows.

・Devices that capture images for viewing purposes, such as digital cameras and mobile devices with camera functions. Devices used for transportation, such as in-vehicle sensors that capture images behind, around, and inside the vehicle, surveillance cameras that monitor running vehicles and roads, and ranging sensors that measure the distance between vehicles. Devices used in home appliances such as TVs, refrigerators, air conditioners, etc., to capture images and operate devices according to gestures ・Endoscopes, devices that perform angiography by receiving infrared light, etc. equipment used for medical and healthcare purposes ・Equipment used for security purposes, such as surveillance cameras for crime prevention and cameras for personal authentication ・Skin measuring instruments that photograph the skin and images of the scalp Equipment used for beauty, such as microscopes used for beauty ・Equipment used for sports, such as action cameras and wearable cameras for use in sports ・Cameras, etc. for monitoring the condition of fields and crops , agricultural equipment

<Example of application to a moving object>
In this way, the technology (the present technology) according to the present disclosure can be applied to various products. For example, the technology according to the present disclosure can be realized as a device mounted on any type of moving body such as automobiles, electric vehicles, hybrid electric vehicles, motorcycles, bicycles, personal mobility, airplanes, drones, ships, and robots. may

FIG. 10 is a block diagram showing a schematic configuration example of a vehicle control system, which is an example of a mobile control system to which the technology according to the present disclosure can be applied.

A vehicle control system 12000 includes a plurality of electronic control units connected via a communication network 12001. In the example shown in FIG. 10, the vehicle control system 12000 includes a drive system control unit 12010, a body system control unit 12020, an exterior information detection unit 12030, an interior information detection unit 12040, and an integrated control unit 12050. Also, as the functional configuration of the integrated control unit 12050, a microcomputer 12051, an audio/image output unit 12052, and an in-vehicle network I/F (interface) 12053 are illustrated.

The drive system control unit 12010 controls the operation of devices related to the drive system of the vehicle according to various programs. For example, the driving system control unit 12010 includes a driving force generator for generating driving force of the vehicle such as an internal combustion engine or a driving motor, a driving force transmission mechanism for transmitting the driving force to the wheels, and a steering angle of the vehicle. It functions as a control device such as a steering mechanism to adjust and a brake device to generate braking force of the vehicle.

The body system control unit 12020 controls the operation of various devices equipped on the vehicle body according to various programs. For example, the body system control unit 12020 functions as a keyless entry system, a smart key system, a power window device, or a control device for various lamps such as headlamps, back lamps, brake lamps, winkers or fog lamps. In this case, the body system control unit 12020 can receive radio waves transmitted from a portable device that substitutes for a key or signals from various switches. The body system control unit 12020 receives the input of these radio waves or signals and controls the door lock device, power window device, lamps, etc. of the vehicle.

The vehicle exterior information detection unit 12030 detects information outside the vehicle in which the vehicle control system 12000 is installed. For example, the vehicle exterior information detection unit 12030 is connected with an imaging section 12031 . The vehicle exterior information detection unit 12030 causes the imaging unit 12031 to capture an image of the exterior of the vehicle, and receives the captured image. The vehicle exterior information detection unit 12030 may perform object detection processing or distance detection processing such as people, vehicles, obstacles, signs, or characters on the road surface based on the received image.

The imaging unit 12031 is an optical sensor that receives light and outputs an electrical signal according to the amount of received light. The imaging unit 12031 can output the electric signal as an image, and can also output it as distance measurement information. Also, the light received by the imaging unit 12031 may be visible light or non-visible light such as infrared rays.

The in-vehicle information detection unit 12040 detects in-vehicle information. The in-vehicle information detection unit 12040 is connected to, for example, a driver state detection section 12041 that detects the state of the driver. The driver state detection unit 12041 includes, for example, a camera that captures an image of the driver, and the in-vehicle information detection unit 12040 detects the degree of fatigue or concentration of the driver based on the detection information input from the driver state detection unit 12041. It may be calculated, or it may be determined whether the driver is dozing off.

The microcomputer 12051 calculates control target values for the driving force generator, the steering mechanism, or the braking device based on the information inside and outside the vehicle acquired by the vehicle exterior information detection unit 12030 or the vehicle interior information detection unit 12040, and controls the drive system control unit. A control command can be output to 12010 . For example, the microcomputer 12051 realizes the functions of ADAS (Advanced Driver Assistance System) including collision avoidance or shock mitigation, follow-up driving based on inter-vehicle distance, vehicle speed maintenance driving, vehicle collision warning, or vehicle lane deviation warning. Cooperative control can be performed for the purpose of

In addition, the microcomputer 12051 controls the driving force generator, the steering mechanism, the braking device, etc. based on the information about the vehicle surroundings acquired by the vehicle exterior information detection unit 12030 or the vehicle interior information detection unit 12040, so that the driver's Cooperative control can be performed for the purpose of autonomous driving, etc., in which vehicles autonomously travel without depending on operation.

Also, the microcomputer 12051 can output a control command to the body system control unit 12020 based on the information outside the vehicle acquired by the information detection unit 12030 outside the vehicle. For example, the microcomputer 12051 controls the headlamps according to the position of the preceding vehicle or the oncoming vehicle detected by the vehicle exterior information detection unit 12030, and performs cooperative control aimed at anti-glare such as switching from high beam to low beam. It can be carried out.

The audio/image output unit 12052 transmits at least one of audio and/or image output signals to an output device capable of visually or audibly notifying the passengers of the vehicle or the outside of the vehicle. In the example of FIG. 10, an audio speaker 12061, a display unit 12062, and an instrument panel 12063 are illustrated as output devices. The display unit 12062 may include at least one of an on-board display and a head-up display, for example.

FIG. 11 is a diagram showing an example of the installation position of the imaging unit 12031. FIG.

In FIG. 11, the vehicle 12100 has

imaging units

12101, 12102, 12103, 12104, and 12105 as the imaging unit 12031.

The

imaging units

12101, 12102, 12103, 12104, and 12105 are provided at positions such as the front nose of the vehicle 12100, the side mirrors, the rear bumper, the back door, and the upper part of the windshield in the vehicle interior, for example. An image pickup unit 12101 provided in the front nose and an image pickup unit 12105 provided above the windshield in the passenger compartment mainly acquire images in front of the vehicle 12100 .

Imaging units

12102 and 12103 provided in the side mirrors mainly acquire side images of the vehicle 12100 . An imaging unit 12104 provided in the rear bumper or back door mainly acquires an image behind the vehicle 12100 . Forward images acquired by the

imaging units

12101 and 12105 are mainly used for detecting preceding vehicles, pedestrians, obstacles, traffic lights, traffic signs, lanes, and the like.

Note that FIG. 11 shows an example of the imaging range of the imaging units 12101 to 12104. In FIG. The imaging range 12111 indicates the imaging range of the imaging unit 12101 provided in the front nose, the imaging ranges 12112 and 12113 indicate the imaging ranges of the

imaging units

12102 and 12103 provided in the side mirrors, respectively, and the imaging range 12114 The imaging range of an imaging unit 12104 provided on the rear bumper or back door is shown. For example, by superimposing the image data captured by the imaging units 12101 to 12104, a bird's-eye view image of the vehicle 12100 viewed from above can be obtained.

At least one of the imaging units 12101 to 12104 may have a function of acquiring distance information. For example, at least one of the imaging units 12101 to 12104 may be a stereo camera composed of a plurality of imaging elements, or may be an imaging element having pixels for phase difference detection.

For example, based on the distance information obtained from the imaging units 12101 to 12104, the microcomputer 12051 determines the distance to each three-dimensional object within the imaging ranges 12111 to 12114 and changes in this distance over time (relative velocity with respect to the vehicle 12100). , it is possible to extract, as the preceding vehicle, the closest three-dimensional object on the course of the vehicle 12100, which runs at a predetermined speed (for example, 0 km/h or more) in substantially the same direction as the vehicle 12100. can. Furthermore, the microcomputer 12051 can set the inter-vehicle distance to be secured in advance in front of the preceding vehicle, and perform automatic brake control (including following stop control) and automatic acceleration control (including following start control). In this way, cooperative control can be performed for the purpose of automatic driving in which the vehicle runs autonomously without relying on the operation of the driver.

For example, based on the distance information obtained from the imaging units 12101 to 12104, the microcomputer 12051 converts three-dimensional object data related to three-dimensional objects to other three-dimensional objects such as motorcycles, ordinary vehicles, large vehicles, pedestrians, and utility poles. It can be classified and extracted and used for automatic avoidance of obstacles. For example, the microcomputer 12051 distinguishes obstacles around the vehicle 12100 into those that are visible to the driver of the vehicle 12100 and those that are difficult to see. Then, the microcomputer 12051 judges the collision risk indicating the degree of danger of collision with each obstacle, and when the collision risk is equal to or higher than the set value and there is a possibility of collision, an audio speaker 12061 and a display unit 12062 are displayed. By outputting an alarm to the driver via the drive system control unit 12010 and performing forced deceleration and avoidance steering via the drive system control unit 12010, driving support for collision avoidance can be performed.

At least one of the imaging units 12101 to 12104 may be an infrared camera that detects infrared rays. For example, the microcomputer 12051 can recognize a pedestrian by determining whether or not the pedestrian exists in the captured images of the imaging units 12101 to 12104 . Such recognition of a pedestrian is performed by, for example, a procedure for extracting feature points in images captured by the imaging units 12101 to 12104 as infrared cameras, and performing pattern matching processing on a series of feature points indicating the outline of an object to determine whether or not the pedestrian is a pedestrian. This is done by a procedure that determines When the microcomputer 12051 determines that a pedestrian exists in the captured images of the imaging units 12101 to 12104 and recognizes the pedestrian, the audio image output unit 12052 outputs a rectangular outline for emphasis to the recognized pedestrian. is superimposed on the display unit 12062 . Also, the audio/image output unit 12052 may control the display unit 12062 to display an icon or the like indicating a pedestrian at a desired position.

An example of a vehicle control system to which the technology according to the present disclosure can be applied has been described above. The technology according to the present disclosure can be applied to the imaging unit 12031, the vehicle exterior information detection unit 12030, and the like among the configurations described above. Specifically, for example, the signal processing device 11 shown in FIG. 4 can be used as the imaging unit 12031 and the vehicle exterior information detection unit 12030, suppressing the occurrence of erroneous determination during motion determination, and achieving higher quality and dynamic range. A wide composite image can be obtained.

It should be noted that the embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible without departing from the gist of the present technology.

For example, this technology can take the configuration of cloud computing in which one function is shared by multiple devices via a network and processed jointly.

In addition, each step described in the flowchart above can be executed by a single device, or can be shared by a plurality of devices.

Furthermore, when one step includes multiple processes, the multiple processes included in the one step can be executed by one device or shared by multiple devices.

Furthermore, this technology can also be configured as follows.

(1)
a first filtering unit that generates a first low-frequency luminance signal by filtering the first image signal with a first filter that extracts low-frequency components;
A second image signal having a phase different from that of the first image signal is filtered by a second filter that extracts low-frequency components and corrects a phase difference, unlike the first filter. a second filter processing unit that generates a second low-frequency luminance signal having the same phase as the first low-frequency luminance signal;
A signal processing device comprising: a motion determination unit that performs motion determination based on the first low-frequency luminance signal and the second low-frequency luminance signal.
(2)
The signal processing device according to (1), further comprising a synthesizing unit that synthesizes the first image signal and the second image signal based on the motion determination result.
(3)
The signal processing device according to (1) or (2), further comprising an imaging unit that generates the first image signal and the second image signal by performing imaging.
(4)
The imaging unit includes a plurality of first pixels for obtaining the first image signal, and a plurality of second pixels for obtaining the second image signal, the sensitivity of which is different from that of the first pixels. The signal processing device according to (3).
(5)
(4) The signal processing device according to (4), wherein the first pixel is larger than the second pixel.
(6)
The signal processing device according to any one of (1) to (5), wherein the second filter has the same size as the first filter.
(7)
The signal processing device according to any one of (1) to (5), wherein the second filter has a size different from that of the first filter.
(8)
A signal processing device
generating a first low-frequency luminance signal by filtering the first image signal with a first filter that extracts low-frequency components;
A second image signal having a phase different from that of the first image signal is subjected to filtering by a second filter that extracts a low-frequency component and corrects a phase difference, unlike the first filter. generating a second low-frequency luminance signal having the same phase as the first low-frequency luminance signal,
A signal processing method that performs motion determination based on the first low-frequency luminance signal and the second low-frequency luminance signal.
(9)
generating a first low-frequency luminance signal by filtering the first image signal with a first filter that extracts low-frequency components;
A second image signal having a phase different from that of the first image signal is subjected to filtering by a second filter that extracts a low-frequency component and corrects a phase difference, unlike the first filter. generating a second low-frequency luminance signal having the same phase as the first low-frequency luminance signal,
A program that causes a computer to execute processing including a step of motion determination based on the first low-frequency luminance signal and the second low-frequency luminance signal.

11 Signal processing unit, 21 Image sensor, 22 Signal generation unit, 23 Motion determination unit, 24 HDR synthesis unit, 31 Large pixel filter processing unit, 32 Small pixel filter processing unit

Claims

a first filtering unit that generates a first low-frequency luminance signal by filtering the first image signal with a first filter that extracts low-frequency components;
A second image signal having a phase different from that of the first image signal is filtered by a second filter that extracts low-frequency components and corrects a phase difference, unlike the first filter. a second filter processing unit that generates a second low-frequency luminance signal having the same phase as the first low-frequency luminance signal;
A signal processing device comprising: a motion determination unit that performs motion determination based on the first low-frequency luminance signal and the second low-frequency luminance signal.
The signal processing device according to claim 1, further comprising a synthesizing unit that synthesizes the first image signal and the second image signal based on the motion determination result.
The signal processing device according to claim 1, further comprising an imaging unit that generates the first image signal and the second image signal by performing imaging.
The imaging unit includes a plurality of first pixels for obtaining the first image signal, and a plurality of second pixels for obtaining the second image signal, the sensitivity of which is different from that of the first pixels. The signal processing device according to claim 3, comprising:
The signal processing device according to claim 4, wherein the first pixel is larger than the second pixel.
The signal processing device according to claim 1, wherein the second filter has the same size as the first filter.
The signal processing device according to claim 1, wherein the second filter has a size different from that of the first filter.
A signal processing device
generating a first low-frequency luminance signal by filtering the first image signal with a first filter that extracts low-frequency components;
A second image signal having a phase different from that of the first image signal is filtered by a second filter that extracts low-frequency components and corrects a phase difference, unlike the first filter. generating a second low-frequency luminance signal having the same phase as the first low-frequency luminance signal,
A signal processing method that performs motion determination based on the first low-frequency luminance signal and the second low-frequency luminance signal.
generating a first low-frequency luminance signal by filtering the first image signal with a first filter that extracts low-frequency components;
A second image signal having a phase different from that of the first image signal is filtered by a second filter that extracts low-frequency components and corrects a phase difference, unlike the first filter. generating a second low-frequency luminance signal having the same phase as the first low-frequency luminance signal,
A program that causes a computer to execute a process including a step of motion determination based on the first low-frequency luminance signal and the second low-frequency luminance signal.