WO2022075198A1

WO2022075198A1 - Imaging control device and imaging device

Info

Publication number: WO2022075198A1
Application number: PCT/JP2021/036286
Authority: WO
Inventors: 晋宝玉
Original assignee: ソニーセミコンダクタソリューションズ株式会社
Priority date: 2020-10-09
Filing date: 2021-09-30
Publication date: 2022-04-14
Also published as: JP2022063000A

Abstract

The present disclosure quickly sets the storage period of luminance difference signals. An imaging control device according to the present disclosure has a storage period adjustment unit. The storage period adjustment unit provided to this imaging control device adjusts, on the basis of at least one luminance difference frame generated by storing a time-series frame configured from the luminance signal of a subject and a luminance difference signal representing a luminance change amount of the subject each time the frame is generated, a luminance difference signal storage period for storing a luminance difference signal for the next luminance difference frame that is generated.

Description

Imaging control device and imaging device

The present disclosure relates to an image pickup control device and an image pickup device.

An image processing device that detects the movement of a subject using an image signal generated by an image sensor or the like is used. For example, a device has been proposed that detects the movement of a subject by using a luminance difference signal, which is an image signal generated by EVS (Event-based Vision Sensor) among image pickup devices (see, for example, Patent Document 1).

Japanese Patent Application Laid-Open No. 2015-028780

However, in the above-mentioned conventional technique, it is difficult to set the accumulation period of the brightness difference signal at high speed, so that there is a problem that it is difficult to detect the movement of the subject moving at high speed.

Therefore, in the present disclosure, we propose an image pickup control device and an image pickup device that set the accumulation period of the luminance difference signal at high speed.

The image pickup control device according to the present disclosure has a storage period adjusting unit. The accumulation period adjusting unit stores a time-series frame composed of the luminance signal of the subject and a luminance difference signal which is a signal of the amount of change in the luminance of the subject, and the luminance difference frame generated by accumulating each time the frame is generated. The luminance difference signal accumulation period, which is the period for accumulating the luminance difference signal when the next luminance difference frame is generated, is adjusted based on at least one of the above.

It is a figure which shows the structural example of the image pickup apparatus which concerns on 1st Embodiment of this disclosure. It is a figure which shows the structural example of the image pickup device which concerns on embodiment of this disclosure. It is a figure which shows an example of the luminance difference frame and the frame which concerns on embodiment of this disclosure. It is a figure which shows an example of the luminance difference frame and the frame which concerns on embodiment of this disclosure. It is a figure which shows an example of the luminance difference frame and the frame which concerns on embodiment of this disclosure. It is a figure which shows an example of the relationship between the motion speed and the spatial frequency which concerns on embodiment of this disclosure. It is a figure which shows an example of the adjustment of the luminance difference signal accumulation period which concerns on embodiment of this disclosure. It is a figure which shows an example of the adjustment of the luminance difference signal accumulation period which concerns on embodiment of this disclosure. It is a figure which shows an example of the frame period which concerns on 1st Embodiment of this disclosure. It is a figure which shows an example of the adjustment process of the luminance difference signal accumulation period which concerns on 1st Embodiment of this disclosure. It is a figure which shows the structural example of the image pickup apparatus which concerns on 2nd Embodiment of this disclosure. It is a figure which shows an example of the frame period which concerns on the 2nd Embodiment of this disclosure. It is a figure which shows the structural example of the luminance difference frame which concerns on 3rd Embodiment of this disclosure. It is a figure which shows an example of the adjustment process of the luminance difference signal accumulation period which concerns on 3rd Embodiment of this disclosure. It is a figure which shows an example of the frame period which concerns on 4th Embodiment of this disclosure. It is a figure which shows an example of the frame period which concerns on 4th Embodiment of this disclosure. It is a figure which shows an example of compression of the luminance difference frame which concerns on 5th Embodiment of this disclosure. It is a figure which shows an example of compression of the luminance difference frame which concerns on 5th Embodiment of this disclosure. It is a figure which shows the structural example of the image pickup apparatus which concerns on the modification of this disclosure. It is a figure which shows the structural example of the image pickup apparatus which concerns on the modification of this disclosure. It is a figure which shows the structural example of the image pickup apparatus which concerns on the modification of this disclosure. It is a figure which shows the structural example of the image pickup apparatus which concerns on the modification of this disclosure. It is a figure which shows the structural example of the image pickup apparatus which concerns on the modification of this disclosure. It is a figure which shows the structural example of the pixel of the image sensor which concerns on this disclosure. It is a figure which shows the structural example of the pixel of the image sensor which concerns on this disclosure. It is a figure which shows the structural example of the pixel of the image sensor which concerns on this disclosure.

An image processing device that detects the movement of a subject using an image signal generated by an image sensor or the like is used. For example, a device that detects the movement of a subject by using a luminance difference signal, which is an image signal generated by EVS (Event-based Vision Sensor) among image pickup devices, has been proposed. Here, the luminance difference signal is a signal corresponding to a change in luminance in the image of the subject. The EVS is an image pickup device that detects this luminance difference signal as a time-asynchronous event. In this device, an event is captured and the event occurrence time is sequentially recorded to generate an occurrence time map. This device analyzes the pattern of events on this occurrence time map and extracts the velocity component.

However, the above-mentioned conventional technique has a problem that it is difficult to detect the movement of a subject moving at high speed. Since events are generated asynchronously in time, it is necessary to set a predetermined period and accumulate events to generate an image. If the period of this accumulation is insufficient, it becomes difficult to analyze the pattern. On the other hand, if the accumulation period is excessive, the image is blurred and the motion detection error increases. In the above prior art, it is detected whether or not the accumulation is sufficient after the extraction of the velocity component. If the storage period is not sufficient, the storage will be performed again, and there is a problem that it cannot cope with the high-speed movement of the subject.

Therefore, it is expected to realize a technology that can overcome the above-mentioned problems and set the accumulation period of the luminance difference signal at high speed.

Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. The explanation will be given in the following order. In each of the following embodiments, the same parts are designated by the same reference numerals, so that overlapping description will be omitted.
1. 1. First embodiment 2. Second embodiment 3. Third embodiment 4. Fourth Embodiment 5. Fifth Embodiment 6. Modification example 7. Pixel configuration example

(1. First Embodiment)
[Configuration of image pickup device]
FIG. 1 is a diagram showing a configuration example of an image pickup apparatus according to the first embodiment of the present disclosure. The figure is a block diagram showing a configuration example of the image pickup apparatus 1. The image pickup device 1 is a device that generates image data of the subject and detects the movement of the subject. The image pickup device 1 includes

image pickup elements

10 and 20 and an image pickup control device 30.

The image pickup element 10 is an element that generates an image of a subject. The image pickup device 10 is provided with a pixel array unit (pixel array unit 11 described later) in which a plurality of pixels that generate a luminance signal according to the light from the subject are arranged in a two-dimensional matrix. A frame, which is a still image for one screen, is formed by the luminance signals of all the pixels arranged in this pixel array. The image sensor 10 can generate time-series frames constituting the moving image. The image sensor 10 can generate and output a frame at a rate of, for example, 120 frames / second (FPS). A photoelectric conversion unit that performs photoelectric conversion of the incident light is arranged in the pixel, and the photoelectric conversion of the incident light irradiated to the photoelectric conversion unit is performed during a predetermined exposure period. The charge generated by this photoelectric conversion is accumulated during the exposure period. This accumulated charge is converted into an image signal after the lapse of the exposure period and output from the pixel. The details of the configuration of the image pickup device 10 will be described later. The image sensor 10 is an example of the first image sensor described in the claims.

The image sensor 20 is an element that generates and outputs a luminance difference signal. EVS can be used for the image sensor 20. As described above, the luminance difference signal is a signal corresponding to a change in the luminance signal. Similar to the image pickup element 10, the image pickup element 20 includes a pixel array unit in which a plurality of pixels are arranged in a two-dimensional grid pattern. The pixel of the image pickup device 20 outputs a signal when the amount of incident light changes from a predetermined threshold value. Further, this signal can be configured to include a signal indicating either an increase or a decrease in the amount of incident light. Each pixel in the pixel array section asynchronously generates a signal. Based on these asynchronous signals, the image pickup device 20 generates a luminance difference signal including the xy coordinates in the pixel array portion of the pixel that outputs the signal and the signal generation time, and sequentially outputs the signal. For convenience, it is assumed that the pixels of the image sensor 20 output the luminance difference signal.

When the subject moves, the luminance difference signal is output from the pixel corresponding to the contour of the subject. This is because the brightness of the contour portion changes with the movement of the subject. The movement of the subject can be detected by detecting the change in the position of the contour portion where the luminance difference signal is output. The details of the configuration of the image pickup device 20 will be described later. The image sensor 20 is an example of the second image sensor described in the claims.

The image sensor 10 and the image sensor 20 each output a frame and a luminance difference signal corresponding to the same subject.

The image pickup control device 30 controls the

image pickup elements

10 and 20. The image pickup control device 30 controls the image pickup element 10 to generate time-series frames. Further, the image pickup control device 30 controls the image pickup element 20 to asynchronously generate the luminance difference signal. The generated luminance difference signal is accumulated in the image pickup control device 30, and a luminance difference frame, which is a frame composed of the luminance difference signal, is generated. The generation of the luminance difference frame can be performed in synchronization with the generation of the frame. Hereinafter, the period for generating a frame is referred to as a frame generation period. The image pickup control device 30 accumulates the luminance difference signal for each frame generation period to generate the luminance difference frame. Hereinafter, the period for accumulating the luminance difference signal is referred to as a luminance difference signal accumulating period.

The period for generating the frame and the luminance difference frame in the image pickup control device 30 is referred to as a frame period. The frame period includes a frame generation period and a luminance difference signal storage period. The image pickup control device 30 can generate a time-series frame and a luminance difference frame by repeating the frame period. Further, the image pickup control device 30 adjusts the luminance difference signal accumulation period in the next frame period. The adjustment of the luminance difference signal accumulation period in this next frame period can be performed based on the already generated luminance difference frame.

Further, the image pickup control device 30 can further detect the movement of the subject. The movement of the subject can be detected based on the luminance difference frame. The detected movement of the subject is output from the image pickup control device 30 as motion data.

The image pickup control device 30 in the figure includes a control unit 340, a storage period adjustment unit 310, a luminance difference frame generation unit 320, and a processing unit 330.

The control unit 340 controls the

image pickup devices

10 and 20. The control unit 340 controls the image sensor 10 to output time-series frames. At this time, the control unit 340 sets the frame generation period. Specifically, the control unit 340 can set the exposure period when generating the image signal as the frame generation period. Further, the control unit 340 controls the image pickup device 20 to output a luminance difference signal.

The luminance difference frame generation unit 320 generates a luminance difference frame. The luminance difference frame can be generated by the luminance difference frame generation unit 320, for example, by accumulating the luminance difference signal output from the image pickup device 20 in a memory or the like. Specifically, the luminance difference signal can be accumulated by recording the xy coordinates of the pixel that generated the luminance difference signal and the generation time in the memory each time the luminance difference signal is output. When a plurality of luminance difference signals having the same xy coordinates are accumulated, the occurrence time is overwritten. The luminance difference signal is accumulated based on the luminance difference signal accumulation period output from the accumulation period adjusting unit 310 described later.

The storage period adjusting unit 310 adjusts the luminance difference signal storage period. The accumulation period adjusting unit 310 adjusts the accumulation period when forming the next luminance difference frame based on the luminance difference frame output from the luminance difference frame generation unit 320, and adjusts the accumulation period when forming the next luminance difference frame with respect to the luminance difference frame generation unit 320. Output. The accumulation period adjusting unit 310 in the figure adjusts the luminance difference signal accumulating period based on the luminance difference frame and the frame. Further, the accumulation period adjusting unit 310 in the figure further adjusts the exposure period and outputs the exposure period to the control unit 340. The details of adjusting the luminance difference signal storage period in the storage period adjusting unit 310 will be described later.

The processing unit 330 processes the luminance difference frame generated by the luminance difference frame generation unit 320. The processing unit 330 in the figure describes an example in which a process of detecting the movement of the subject is performed based on the luminance difference frame. The detected motion is output to an external device as motion data. Motion detection can be performed by known methods, such as estimating optical flow.

In the above-mentioned optical flow, the luminance difference signal, that is, the time when the event was last detected is held in each pixel of the region of interest, and continuous events originating from the region of interest are detected. An event in which the detection time increases monotonically can be assumed to be a continuous event accompanying the movement of the subject.

In the image pickup apparatus 1 of the figure, the luminance difference signal generated asynchronously is divided into frames in a predetermined accumulation period to form a luminance difference frame. Further, the image pickup apparatus 1 can detect continuous events by comparing a plurality of luminance difference frames. The storage period needs to be set to a period in which an amount of luminance difference signals that can detect the contour of the subject is stored. On the other hand, when the accumulation period is long and an excessive luminance difference signal is accumulated, the outline of the subject is blurred. In this case, it becomes difficult to detect the movement of the subject. In addition, the processing load such as power consumption increases due to the operation of accumulating the excessive luminance difference signal. In addition, a lot of noise will be accumulated. It is also necessary to adjust the accumulation period depending on the speed of movement of the subject. This is because the faster the movement, the higher the frequency of occurrence of the luminance difference signal. It is also necessary to adjust the accumulation period depending on the brightness of the subject. This is because when the brightness of the subject is low due to low illuminance or the like, the detection frequency of the luminance difference signal decreases.

The accumulation period adjustment unit 310 adjusts the luminance difference signal accumulation period set in the previous frame period based on the luminance difference frame output by the luminance difference frame generation unit 320. Further, the accumulation period adjusting unit 310 outputs the luminance difference signal accumulating period for application to the generation of the next luminance difference frame to the luminance difference frame generation unit 320. That is, the accumulation period adjusting unit 310 performs feedback control based on the luminance difference frame. As will be described later, the storage period adjusting unit 310 can also adjust the luminance difference signal storage period based on the frame output by the image sensor 10. As described above, the accumulation period adjusting unit 310 adjusts the luminance difference signal accumulating period based on at least one of the frame and the luminance frame.

The configuration of the image pickup apparatus 1 is not limited to this example. For example, the image pickup control device 30 can be configured not to have the processing unit 330. In this case, the processing unit 330 can be arranged outside the image pickup control device 30.

[Structure of image sensor]
FIG. 2 is a diagram showing a configuration example of the image pickup device according to the embodiment of the present disclosure. The figure is a block diagram showing a configuration example of the image pickup device 10. The image pickup device 10 includes a pixel array unit 11, a vertical drive unit 12, and a column signal processing unit 13.

The pixel array unit 11 is configured by arranging the pixels 100 in a two-dimensional matrix. Pixels 100 for one screen are arranged in the pixel array unit 11. Further,

signal lines

18 and 19 are arranged in an XY matrix in the pixel array unit 11 and are wired to the pixel 100. The signal line 18 is a signal line arranged for each line of the pixel 100 of the pixel array unit 11 and transmitting a control signal to the pixel 100. The signal line 19 is a signal line that is arranged in each row of the pixels 100 of the pixel array unit 11 and transmits an image signal generated by the pixels 100.

The vertical drive unit 12 generates a control signal for the pixel 100. The vertical drive unit 12 generates a control signal for the pixel 100 and outputs it for each row of the pixel array unit 11.

The column signal processing unit 13 processes the image signal generated by the pixel 100. An image signal is input to the column signal processing unit 13 for each column of the pixel array unit 11 and processed. The processing of the image signal in the column signal processing unit 13 corresponds to, for example, an analog digital conversion process for converting an analog image signal generated by the pixel 100 into a digital image signal.

The frame generation in the image sensor 10 can be performed as follows. First, the vertical drive unit 12 outputs a control signal to reset all the pixels 100 of the pixel array unit 11 and start exposure. An image signal is generated for each pixel 100 based on the electric charge accumulated in the photoelectric conversion unit after the elapse of a predetermined exposure period. After that, based on the control of the vertical drive unit 12, the generated image signals are sequentially output for each row of the pixel array unit 11. This output image signal is converted into a digital image signal by the column signal processing unit 13 and output. The image pickup device 10 can form and output a frame by generating and outputting an image signal for one screen.

The image sensor 20 can also have the same configuration as the image sensor 10 in the figure. In the case of the image pickup device 20, the pixel 100 generates a signal based on the change in the amount of charge generated by the photoelectric conversion of the photoelectric conversion unit. The vertical drive unit 12 scans the pixels 100 for each row of the pixel array unit 11. When a signal is generated in the pixel 100, it is output via the signal line 19 during this scan. The column signal processing unit 13 converts the signal of the pixel 100 for each row into a luminance difference signal and sequentially outputs it to the image pickup control device 30.

[Brightness difference frame and frame]
3A to 3C are diagrams showing an example of a luminance difference frame and a frame according to the embodiment of the present disclosure. The figure is a diagram illustrating a luminance difference frame and a frame in a moving subject. FIG. 3A shows a moving subject. It is assumed that the subject 400 having a circular shape moving to the right in the figure is assumed, and the subject 400 at time t0 moves to the position of 400'at time t1. In such a case, a luminance difference signal is generated along the contour of the subject 400.

FIG. 3B shows a set of luminance difference signals in three-dimensional coordinates based on the xy coordinates parallel to the paper surface and the time t of the axis perpendicular to the paper surface. The black dots in the figure represent the luminance difference signals. When the subject 400 moves in the positive x-axis direction during the period from t0 to t1, the luminance of the overwritten shape while the annular contour 403 of the subject 400 moves from the position 401 of t0 to the position 402 of t1. A difference frame is formed.

FIG. 3C shows a frame 410 and a luminance difference frame 411 when the subject 400 moves. The rectangle in FIG. 3C represents the above-mentioned region of interest. The frame 410 has a shape in which the moving circular subject 400 is overwritten. On the other hand, the luminance difference frame 411 has a shape in which the contours of the moving annular subject 400 are overwritten. The frame 410 and the luminance difference frame 411 having such a shape have different spatial frequencies. Here, the spatial frequency is a frequency component of an electric signal when the shading of the image corresponds to the electric signal. The finer the shade period of the image, the higher the spatial frequency. Further, the steeper the change in the shade of the outline of the image, the higher the frequency component is included. Comparing the spatial frequency of the image along the aa'line of the frame 410 of FIG. 3C with the spatial frequency of the image along the bb'line of the brightness difference frame 411, the frame 410 composed of continuous regions. The spatial frequency of is low. On the other hand, the spatial frequency of the luminance difference frame 411 composed of the intermittent regions becomes high.

[Spatial frequency]
FIG. 4 is a diagram showing an example of the relationship between the motion speed and the spatial frequency according to the embodiment of the present disclosure. The figure is a diagram showing the relationship between the brightness and movement speed of a subject and the spatial frequency of a frame and a luminance difference frame. In the figure, the upper side of the column of spatial frequency describes the outline of the frame and the luminance difference frame, and the lower side describes the relative spatial frequency. Further, the accumulation period in the figure represents the adjustment of the luminance difference signal accumulation period. The spatial frequency can be expressed by the number of line pairs (lp / mm) per unit width. In this case, the spatial frequency of 0 to 2.5 lp / mm can correspond to a low frequency, 2.5 to 3.0 lp / mm corresponds to a medium frequency, and 3.0 lp / mm or more corresponds to a high frequency. The spatial frequency is the spatial frequency of the image captured by the image pickup element 10 or the like, and is different from the original spatial frequency of the subject. Further, the spatial frequency changes depending on the pixel size of the image pickup device 10 or the like and the configuration of the optical system for forming the subject on the image pickup device 10 or the like. Therefore, it is necessary to make corrections based on the pixel size and the like.

When the subject in the figure is in high illuminance, the frame has a low spatial frequency when the moving speed of the subject is high. This is because the image of the subject in the frame occupies a large area. On the other hand, when the moving speed of the subject is low, the spatial frequency of the frame becomes relatively high. This is because the area of the image of the subject in the frame becomes narrow.

When the subject has a relatively high illuminance, the luminance difference frame has a high spatial frequency regardless of the moving speed of the subject. This is because the luminance difference frame is composed of intermittent regions.

When the subject in the figure has a relatively low illuminance, the frame has a low spatial frequency when the moving speed of the subject is high, and a high spatial frequency when the moving speed of the subject is low. The spatial frequency is the same as when the illuminance is relatively high.

When the subject has low illuminance, the luminance difference frame has a medium spatial frequency when the moving speed of the subject is high. That is, the spatial frequency is lower than when the subject has high brightness and moves at high speed. This is because the outline of the image of the luminance difference frame is blurred because the luminance is low. On the other hand, when the moving speed is low, the spatial frequency becomes relatively high. This is because the area of the image of the subject in the frame is narrower than that in the case where the moving speed is high.

The storage period adjusting unit 310 (see FIG. 1) can adjust the luminance difference signal storage period by utilizing the relationship between the speed of the subject and the spatial frequency. For example, when the spatial frequency of the frame is low frequency (2.5 lp / mm or less) and the spatial frequency of the brightness difference frame is medium to high frequency (2.5 lp / mm or more), the storage period adjusting unit 310 may use the subject. It can be judged that the movement speed of is high. In this case, the storage period adjusting unit 310 can make adjustments to reduce the amount of information by shortening the luminance difference signal storage period. On the other hand, when the spatial frequency of the frame and the spatial frequency of the luminance difference frame are high frequencies (3.0 lp / mm or more), the accumulation period adjusting unit 310 can determine that the moving speed of the subject is low. In this case, the storage period adjusting unit 310 can make adjustments to lengthen the luminance difference signal storage period.

The accumulation period adjusting unit 310 can determine the moving speed of the subject based on, for example, the moving distance in the frame period. Specifically, the storage period adjusting unit 310 sets the speed at which the moving distance exceeds 12 pixels (pixels) at a frame rate of 60 FPS (frame period: 1 / 60s) to a high speed, and sets the moving distance to 12 pixels or less. The speed can be slowed down. Further, the storage period adjusting unit 310 assumes that the frame period 1 / 60s corresponding to the frame rate 60 FPS is set as the threshold value as the luminance difference signal storage period, 1/60s or less is a short storage period, and 1 / 60s or more is a long storage period. can do.

Further, the accumulation period adjustment unit 310 can also adjust the luminance difference signal accumulation period in consideration of the brightness (illuminance) of the subject. The accumulation period adjusting unit 310 can detect the brightness of the subject from the statistical value of the luminance signals constituting the frame, for example, the average value of the luminance signals in the region of interest. When the brightness of the subject is expressed by the illuminance, the accumulation period adjusting unit 310 can, for example, set the illuminance to be low when the illuminance is less than 10 lpx and to be high when the illuminance exceeds 100 lpx. The analysis based on the statistical values of the luminance signals constituting the frame corresponds to the analysis of the spatial region. In this way, the accumulation period adjusting unit 310 can also combine the analysis of the spatial domain with the analysis of the spatial frequency domain.

When the brightness of the subject exceeds 100 lp (high illuminance) and the moving speed of the subject becomes high due to the analysis of the spatial frequency of the frame and the luminance difference frame, the accumulation period adjusting unit 310 accumulates the luminance difference signal. The period can be adjusted to 1 / 60s or less. Further, when the moving speed of the subject becomes slow, the accumulation period adjusting unit 310 can adjust the luminance difference signal accumulating period to a value in the vicinity of 1 / 60s.

On the other hand, when the brightness of the subject is less than 10 lp (low illuminance) and the moving speed of the subject becomes high due to the analysis of the spatial frequency of the frame and the luminance difference frame, the accumulation period adjusting unit 310 performs the luminance difference. The signal storage period can be adjusted to a value near 1 / 60s. Further, when the moving speed of the subject becomes slow, the accumulation period adjusting unit 310 can adjust the luminance difference signal accumulating period to 1 / 60s or more.

For the spatial frequency of the frame and the brightness difference frame, perform fast Fourier transform, discrete Fourier transform, discrete cosine transform, etc. in the horizontal and vertical directions of the image in the region of interest, and acquire the two-dimensional frequency component as frequency information. Thereby, it can be extracted. Further, the accumulation period adjusting unit 310 can also use a method of extracting information corresponding to frequency information by executing a convolution operation using a matrix operator such as a Laplacian matrix. Further, the accumulation period adjusting unit 310 may perform a Fourier transform on the luminance signal constituting the frame to extract a two-dimensional frequency component and extract a cross-correlation with the frequency component of the luminance difference frame. Further, the accumulation period adjusting unit 310 may calculate the luminance difference signal accumulation period for each region by using the heuristic weighting of the luminance signal statistical value and the frequency component of the luminance difference signal and the neural network. can.

The accumulation period adjustment unit 310 can also adjust the luminance difference signal accumulation period based on the statistical value of the luminance signal in the frame. For example, the accumulation period adjusting unit 310 can calculate an average value of the luminance signal as a statistical value of the luminance signal, and can adjust the luminance difference signal accumulation period based on this average value. The storage period adjusting unit 310 can adjust the luminance difference signal storage period to a longer time, for example, when the average value of the luminance signals is relatively low. Further, the accumulation period adjusting unit 310 can also adjust the exposure period of the frame based on the statistical value of the luminance signal in the frame. For example, the accumulation period adjusting unit 310 can calculate an average value of the luminance signal as a statistical value of the luminance signal and adjust the exposure period based on this average value.

[Adjustment of luminance difference signal storage period]
5A and 5B are diagrams showing an example of adjusting the luminance difference signal storage period according to the embodiment of the present disclosure. The figure shows an example of adjusting the luminance difference signal accumulation period with reference to the frame generation period. In the figure, black circles represent luminance difference signals that are accumulated to form a luminance difference frame, and white circles represent luminance difference signals that are discarded without being accumulated.

FIG. 5A shows an example in which the luminance difference signal accumulation period is adjusted by dividing the frame generation period into two or more subframe periods and selecting the subframe period for accumulating the luminance difference signal from the subframe periods. It is a representation. In FIG. 5A, the subframe period is represented by "SF". FIG. 5A shows an example of dividing the frame generation period into eight from SF1 to SF8. Further, FIG. 5A shows an example in which the luminance difference signal accumulation period is adjusted by selecting every four subframe periods.

FIG. 5B shows an example in which a certain period from the start of the frame generation period is set as the luminance difference signal accumulation period, and the luminance difference signal accumulation period is adjusted by increasing or decreasing this period.

In any of the examples of FIGS. 5A and 5B, the accumulation period adjusting unit 310 can control the amount of the luminance difference signal included in the luminance difference frame and delete the unnecessary luminance difference signal. This corresponds to substantial denoising.

[Relationship between frame period and accumulation period]
FIG. 6 is a diagram showing an example of a frame period according to the first embodiment of the present disclosure. The figure is a diagram showing the relationship between the frame period, the exposure period, and the luminance difference signal storage period. In the figure, consecutive frame periods are identified by n-1, n and n + 1. “Exposure” in the figure represents an exposure period when forming a frame in the image sensor 10. In the figure, the period required for the output (transfer) of the frame in the image sensor 10 is ignored. In this case, the exposure period is almost equal to the frame generation period.

“Accumulation” in the figure represents a period during which the luminance difference signal is accumulated in the luminance difference frame generation unit 320. The period for accumulating the luminance difference signal is the period for generating the luminance difference frame. The dotted rectangle represents the period for adjusting the exposure period and the accumulation period of the luminance difference signal based on the frame and the luminance difference frame. In the frame period (n), the exposure period and the luminance difference signal storage period are adjusted based on the frame and the luminance difference frame generated in the frame period (n-1). Next, the frame and the luminance difference frame in the frame period (n) are formed by using the adjusted exposure period and the luminance difference signal accumulation period.

In this way, the image pickup apparatus 1 can generate a time-series frame and a luminance difference frame while adjusting the exposure period and the luminance difference signal accumulation period.

[Adjustment processing of luminance difference signal accumulation period]
FIG. 7 is a diagram showing an example of the luminance difference signal storage period adjustment process according to the first embodiment of the present disclosure. The figure is a flowchart showing an example of the adjustment process of the luminance difference signal accumulating period according to the first embodiment of the present disclosure.

First, the storage period adjusting unit 310 outputs the initial values of the exposure period and the luminance difference signal storage period (step S101). The initial value of the exposure period and the initial value of the luminance difference signal storage period are output to the control unit 340 and the luminance difference frame generation unit 320, respectively.

Next, a frame and a luminance difference frame are generated (step S102). Specifically, the control unit 340 controls the image pickup element 10 to generate a frame based on the output exposure period, and the brightness difference frame generation unit 320 generates a brightness difference frame based on the output brightness difference signal storage period. To generate. The accumulation period adjusting unit 310 acquires the generated frame and the luminance difference frame (step S103).

Next, the accumulation period adjusting unit 310 calculates the statistical values of the spatial frequency and the luminance signal from the frame and the luminance difference frame (step S104).

Next, the luminance difference frame generation unit 320 adjusts the exposure period and the luminance difference signal accumulation period based on the calculated spatial frequency and the statistical value of the luminance signal (step S105).

Next, the luminance difference frame generation unit 320 outputs the adjusted exposure period and the luminance difference signal storage period (step S106), and shifts to the process of step S102. By the above procedure, the luminance difference signal accumulation period can be adjusted.

As described above, the image pickup control device 30 of the first embodiment of the present disclosure adjusts the luminance difference signal accumulation period in the next frame period based on the luminance difference frame generated immediately before. Thereby, the accumulation period according to the moving speed of the subject and the like can be set before the processing in the processing unit 330. Even when the subject moves at high speed, an appropriate storage period can be set.

(2. Second embodiment)
The image pickup control device 30 of the first embodiment described above has output all the generated frames and the luminance difference frames. On the other hand, the image pickup control device 30 of the second embodiment of the present disclosure is different from the above-mentioned first embodiment in that the frame and the luminance difference frame are thinned out.

[Configuration of image pickup device]
FIG. 8 is a diagram showing a configuration example of the image pickup apparatus according to the second embodiment of the present disclosure. Similar to FIG. 1, the figure is a block diagram showing a configuration example of the image pickup apparatus 1. The image pickup apparatus 1 of FIG. 8 is different from the image pickup apparatus 1 of FIG. 1 in that it further includes a frame output unit 40.

The frame output unit 40 outputs the frame generated by the image pickup device 10 to the outside of the image pickup device 1. The frame output unit 40 outputs a frame selected based on the control signal output from the storage period adjustment unit 310.

The luminance difference frame generation unit 320 in the figure outputs the generated luminance difference frame to the accumulation period adjustment unit 310, and processes the luminance difference frame selected based on the control signal output from the accumulation period adjustment unit 310. Output to unit 330.

The accumulation period adjusting unit 310 in the figure further controls the thinning of the frame and the luminance difference frame. This thinning is to exclude a part of the time series frame and the luminance difference frame and output. For example, the accumulation period adjusting unit 310 can thin out frames and luminance difference frames at every other frame period in a continuous frame period. The accumulation period adjusting unit 310 in the figure determines whether or not it is possible to reduce the luminance difference frame when adjusting the luminance difference signal accumulating period. For example, the accumulation period adjusting unit 310 thins out the luminance difference frames when the movement of the subject can be detected even when the luminance difference frames are reduced. This thinning corresponds to the adjustment (shortening) of the luminance difference signal accumulation period for a plurality of frame periods. The accumulation period adjusting unit 310 in the figure further determines whether or not the frame can be reduced.

The accumulation period adjusting unit 310 in the figure outputs a control signal to the frame output unit 40 and the luminance difference frame generation unit 320, selects the frame to be thinned out and the luminance difference frame, and performs thinning.

[Relationship between frame period and accumulation period]
FIG. 9 is a diagram showing an example of a frame period according to the second embodiment of the present disclosure. FIG. 6 is a diagram showing the relationship between the frame period, the exposure period, and the luminance difference signal storage period, as in FIG. It differs from FIG. 6 in that the frame and the luminance difference frame are thinned out. In the figure, the exposure period and the luminance difference signal accumulation period from the frame period (n-1) to the frame period (n + 2) are shown. Further, at the end of the frame period in the figure, a frame and a luminance difference frame are output. The arrows in the figure represent the output of this frame or the like. The figure shows an example of thinning out frames and luminance difference frames every one frame period. Thinning is performed in the frame period (n) and the frame period (n + 2) in the figure, and the output of the frame and the luminance difference frame is stopped. The frame and the luminance difference frame are generated in all the frame periods, and are used by the accumulation period adjusting unit 310 for adjusting the luminance difference signal accumulation period.

Since the configuration of the image pickup apparatus 1 other than this is the same as the configuration of the image pickup apparatus 1 in the first embodiment of the present disclosure, the description thereof will be omitted.

As described above, the image pickup control device 30 of the second embodiment of the present disclosure can reduce unnecessary luminance difference frames by thinning out the luminance difference frames.

(3. Third embodiment)
The image pickup control device 30 of the first embodiment described above adjusts the luminance difference signal accumulation period for the luminance difference frame. On the other hand, the image pickup control device 30 of the third embodiment of the present disclosure is different from the above-mentioned first embodiment in that the luminance difference signal storage period is adjusted for each region arranged in the luminance difference frame. ..

[Structure of image sensor]
FIG. 10 is a diagram showing a configuration example of a luminance difference frame according to the third embodiment of the present disclosure. The figure is a diagram showing a configuration example of the luminance difference frame 200. The luminance difference frame 200 in the figure is divided into a plurality of regions 210. The dotted rectangle in the figure represents the area 210. The luminance difference frame 200 in the figure shows an example of being divided into 6 × 5 regions 210. The region 210 can be composed of a plurality of luminance difference signals. Further, the region 210 can also be configured by one luminance difference signal.

Similar to the luminance difference frame 200 in the figure, the region 210 is also arranged in the frame generated by the image pickup element 10. Specifically, the image pickup device 10 includes a pixel array unit 11 divided into regions 210, and a different exposure period can be set for each region 210. By forming the frame using the image sensor 10 having such a configuration, the frame can be divided into a plurality of regions 210. Further, the control unit 340 of the third embodiment of the present disclosure sets an exposure period for each region 210 with respect to the image pickup device 10.

The storage period adjusting unit 310 in the third embodiment of the present disclosure adjusts the luminance difference signal storage period for each region 210 of the luminance difference frame 200, and sets the luminance difference signal storage period for each region 210 as the luminance difference frame generation unit. Output to 320. Further, the luminance difference frame generation unit 320 in the third embodiment of the present disclosure accumulates the luminance difference signal for each region 210 based on the luminance difference signal accumulation period for each region 210, and generates the luminance difference frame 200. .. Further, the accumulation period adjusting unit 310 adjusts the exposure period of the frame for each area 210 and outputs the exposure period to the control unit 340. As a result, the accumulation period adjustment unit 310 can adjust the luminance difference signal accumulation period according to the movement of the subject in each area 210. The storage period adjusting unit 310 can set an optimum luminance difference signal storage period for each subject even when a stationary subject such as a background and a moving subject are simultaneously imaged.

[Adjustment processing of luminance difference signal accumulation period]
FIG. 11 is a diagram showing an example of the luminance difference signal storage period adjustment process according to the third embodiment of the present disclosure. FIG. 7 is a flowchart showing an example of the adjustment process of the luminance difference signal accumulation period, as in FIG. 7. It differs from the process of FIG. 7 in that the luminance difference signal accumulation period is adjusted for each region 210.

First, the storage period adjusting unit 310 outputs the initial values of the exposure period and the luminance difference signal storage period (step S121). Next, a frame is generated in the image sensor 10 (step S122). Next, the luminance difference frame generation unit 320 generates a luminance difference frame for each region 210 (step S123). Next, the accumulation period adjusting unit 310 acquires the generated frame and the luminance difference frame (step S124).

Next, the storage period adjustment unit 310 determines whether or not the luminance difference signal storage period has been adjusted in all the regions 210 (step S125). As a result, when the luminance difference signal storage period has not been adjusted in all the regions 210 (steps S125, No), the storage period adjusting unit 310 has not completed the adjustment of the luminance difference signal storage period 210. Is selected (step S126). Next, the accumulation period adjusting unit 310 calculates the spatial frequency and the luminance signal statistics in the selected region 210 (step S127). Next, the luminance difference frame generation unit 320 adjusts the exposure period and the luminance difference signal accumulation period based on the calculated spatial frequency and the luminance signal statistical value (step S128).

Next, the luminance difference frame generation unit 320 outputs the adjusted exposure period and the luminance difference signal storage period (step S129), and proceeds to the process of step S125.

In step S125, when the adjustment of the luminance difference signal accumulation period is completed in all the regions 210 (steps S125, Yes), the process proceeds to step S122. By the above procedure, the luminance difference signal accumulation period can be adjusted.

When a different luminance difference signal storage period is applied to each region 210, the luminance difference frame 200 is also generated for each region 210 in ascending order of the luminance difference signal storage period. In this case, the luminance difference frame generation unit 320 can output the luminance difference frame to the processing unit 330 for each region 210. Further, the processing unit 330 can process the luminance difference frame for each area 210. For example, the processing unit 330 can set different parameters for each area 210 and process them individually.

As described above, the image pickup control device 30 of the third embodiment of the present disclosure adjusts the luminance difference signal accumulation period for each region 210 to generate the luminance difference frame. As a result, different luminance difference signal accumulation periods can be set and adjusted for the subject included in each region 210. When photographing a plurality of subjects having different movements, the optimum luminance difference signal storage period can be applied to each subject.

(4. Fourth Embodiment)
The image pickup control device 30 of the first embodiment described above generates a frame and a luminance difference frame after performing an adjustment process of the luminance difference signal storage period. On the other hand, the image pickup control device 30 of the fourth embodiment of the present disclosure performs the above-mentioned first embodiment in parallel with the adjustment process of the luminance difference signal storage period and the generation of the frame and the luminance difference frame. Different from the form.

[Relationship between frame period and accumulation period]
12A and 12B are diagrams showing an example of the frame period according to the fourth embodiment of the present disclosure. FIG. 6 is a diagram showing the relationship between the frame period, the exposure period, and the luminance difference signal storage period, as in FIG. It differs from the frame period of FIG. 6 in that the adjustment of the luminance difference signal accumulation period and the generation of the frame are performed in parallel.

FIG. 12A is a diagram showing an example in which the same exposure period is applied in all frame periods without adjusting the exposure period. In FIG. 12A, the luminance difference signal storage period adjustment process is performed in parallel with the exposure period. After that, the luminance difference signal is accumulated based on the adjusted luminance difference signal accumulation period, that is, the luminance difference frame is generated. When the period for adjusting the luminance difference signal storage period and generating the luminance difference frame is shorter than the exposure period (frame generation period), the frame period can be made equal to the frame generation period. It is possible to increase the frame rate.

FIG. 12B is a diagram showing an example in the case of adjusting the exposure period. In FIG. 12B, the exposure period and the accumulation period of the luminance difference signal and the adjustment process of the exposure period and the luminance difference signal accumulating period are performed in parallel. As shown in FIG. 12B, the exposure period and the luminance difference signal storage period are adjusted in the frame period (n) based on the frame generated in the frame period (n-1) and the luminance difference frame. Based on the exposure period adjusted to this frame period (n) and the luminance difference signal accumulation period, the exposure and the luminance difference signal are accumulated in the frame period (n + 1). As described above, in FIG. 12B, a delay (latency) for two frame periods will occur. In FIG. 12B as well, the frame period can be made equal to the frame generation period, as in FIG. 12A. Such a process of adjusting the luminance difference signal accumulation period and generating a frame in parallel is called a pipeline process.

As described above, the image pickup control device 30 of the fourth embodiment of the present disclosure can increase the frame rate by adjusting the luminance difference signal accumulation period and generating the frame in parallel.

(5. Fifth Embodiment)
The image pickup control device 30 of the first embodiment described above outputs the luminance difference frame to the processing unit 330. On the other hand, the image pickup control device 30 of the fifth embodiment of the present disclosure is different from the above-mentioned first embodiment in that it outputs a compressed luminance difference frame.

[Compression of luminance difference frame]
13A and 13B are diagrams showing an example of compression of the luminance difference frame according to the fifth embodiment of the present disclosure. FIG. 13A is a diagram showing an example in the case where the luminance difference frame is not compressed. In the figure, the upward arrow represents the luminance difference signal output from the image sensor 20. As shown in FIG. 13A, the luminance difference signal is output asynchronously from the image pickup device 20. An example is described in which a relatively large number of luminance difference signals are output during the frame period (n). In FIG. 13A, the first luminance difference signal is output at time t1 and the last luminance difference signal is output at time t2. The luminance difference signals output during the period from time t1 to t2 are accumulated to generate a luminance difference frame. As described above, the luminance difference frame records the xy coordinates and the generation time of the luminance difference signal. Further, an example is described in which the output luminance difference signal is small in the frame period (n + 1).

FIG. 13B is a diagram showing an example in the case of compressing the luminance difference frame. In the frame period (n) of FIG. 13B, the accumulation of the luminance difference signals records the time t1 when the first luminance difference signal is output, the time t2 when the last luminance difference signal is output, and the number of luminance difference signals. Do it by. That is, as shown in the frame period (n) of FIG. 13B, it is assumed that the luminance difference signals are output at equal intervals between the times t1 and t2. As a result, the amount of information held in the luminance difference frame can be reduced. The period from time t1 to t2 corresponds to the adjusted luminance difference signal storage period. Further, as shown in the frame period (n + 1) of FIG. 13B, when the luminance difference signal is small, the luminance difference signal is reduced. For example, when the number of luminance difference signals output during the luminance difference signal storage period is equal to or less than a predetermined threshold value, it is treated as if the luminance difference signal was not output. Even in this case, the amount of information held in the luminance difference frame can be reduced. Such compression of the luminance difference frame can be performed by the luminance difference frame generation unit 320. The luminance difference frame generation unit 320 may perform compression based on at least one of the spatial frequency of the luminance difference signal, the spatial frequency of the luminance signal, and the statistical value of the luminance signal generated by the storage period adjusting unit 310. can.

When the subject suddenly moves, the luminance difference signal accumulation period may not be adjusted in time, and a luminance difference frame in which the luminance difference signal is excessively accumulated may be generated. Further, when the adjustment of the luminance difference signal accumulation period and the frame generation described in FIGS. 12A and 12B are performed in parallel, the luminance difference in which the luminance difference signal is excessively accumulated due to the delay due to the pipeline processing is performed. Frames may be generated. Even in such a case, by compressing the luminance difference frame, it is possible to prevent the capacity of the luminance difference frame from increasing.

As described above, the image pickup control device 30 according to the fifth embodiment of the present disclosure can prevent the capacity of the luminance difference frame from increasing by further compressing the luminance difference frame.

(6. Modification example)
Although the image pickup control device 30 of the first embodiment described above uses the image pickup element 10 and the image pickup element 20, other configurations can be adopted.

[Configuration of image pickup device]
14A to 14E are diagrams showing a configuration example of the image pickup apparatus according to the modified example of the present disclosure. The figure is a simplified diagram showing a configuration example of the image pickup apparatus 1. FIG. 14A is a diagram showing an example in which a camera 50 and a camera 60 are provided in place of the image sensor 10 and the image sensor 20 in FIG. 1, respectively. In this case, the camera 50 generates a frame, and the camera 60 outputs a luminance difference signal.

FIG. 14B is a diagram showing an example in which an image pickup device 70 is provided instead of the image pickup element 10 and the image pickup element 20. The image pickup device 70 is an image pickup device in which pixels that generate a luminance signal and a luminance difference signal are arranged.

FIG. 14C is a diagram showing an example in which an illuminance meter 80 is provided instead of the image pickup element 10. As the illuminance meter 80, for example, an illuminance meter 80 that measures the illuminance for each region of the subject and outputs it as a luminance signal can be used.

FIG. 14D is a diagram showing an example in which the image pickup device 20 is omitted. In FIG. 14D, the storage period adjusting unit 310 adjusts the exposure period based on the frame generated by the image pickup device 10.

FIG. 14E is a diagram showing an example in which the image pickup device 10 is omitted. In FIG. 14E, the accumulation period adjusting unit 310 adjusts the luminance difference signal accumulating period based on the luminance difference frame generated by the image sensor 20.

The configuration of the image pickup apparatus 1 is not limited to this example. For example, a configuration including a camera 50 and an image pickup device 20 can be adopted.

Further, it is also possible to adopt a configuration in which an image pickup element that generates a frame is further added to the image pickup device 1 of FIG. In this case, the control unit 340 can be applied to an image pickup device to which an exposure time shorter than that of the image pickup device 10 is added. The storage period adjusting unit 310 can compare the spatial frequencies of the frames generated by the image pickup device 10 and the added image pickup device, and adjust the luminance difference signal storage period based on the comparison result.

(7. Pixel configuration example)
The configuration of the pixel 100 in the image pickup device 10 and the image pickup device 20 of the above-mentioned image pickup device 1 will be described.

[Pixel composition]
15A to 15C are diagrams showing a configuration example of pixels of the image pickup device according to the present disclosure. FIG. 15A is a diagram showing a configuration example of the pixel 100 in the image sensor 10. The pixel 100 of FIG. 15A includes a photoelectric conversion unit 110, a charge holding unit 120, and an image signal generation unit 130, and generates a luminance signal. The photoelectric conversion unit 110 performs photoelectric conversion of incident light. For example, a photodiode can be used for the photoelectric conversion unit 110. The photoelectric conversion unit 110 performs photoelectric conversion of incident light during the exposure period to generate and store electric charges. The charge holding unit 120 holds the electric charge generated and accumulated by the photoelectric conversion unit 110. After the end of the exposure period, the charge accumulated in the photoelectric conversion unit 110 is transferred to the charge holding unit 120 by a charge transfer unit (not shown) and held. The image signal generation unit 130 generates an image signal (luminance signal) based on the electric charge held by the electric charge holding unit 120.

FIG. 15B is a diagram showing a configuration example of the pixel 100 in the image sensor 20. The pixel 100 of FIG. 15B includes a photoelectric conversion unit 150, a current-voltage conversion unit 160, a buffer 170, a subtractor 180, and a quantizer 190, and generates a luminance difference signal. The photoelectric conversion unit 150 performs photoelectric conversion of incident light in the same manner as the photoelectric conversion unit 110. Unlike the photoelectric conversion unit 110, the electric charge generated by the photoelectric conversion unit 150 is not accumulated and is output to the outside in the form of a photocurrent. The current-voltage conversion unit 160 converts the photocurrent output from the photoelectric conversion unit 150 into a voltage signal. Further, the current-voltage conversion unit 160 logarithmically compresses the signal voltage in the process of conversion. The buffer 170 amplifies the signal converted by the current-voltage conversion unit 160 and separates it from the subtractor 180 in the next stage.

The subtractor 180 performs subtraction processing on the signal output by the buffer 170, and outputs a signal having a voltage corresponding to the amount of change in the signal. The quantizer 190 quantizes the signal output by the subtractor 180. In this quantization, for example, a predetermined threshold value is compared with the signal output by the subtractor 180, and when the signal output by the subtractor 180 exceeds the threshold value, a 1-bit digital signal is output. Can be quantized by. The quantizer 190 can perform quantization by comparing with a threshold value for changes in both directions of rising and falling of the signal output by the subtractor 180. In this case, the quantizer 190 can output a 2-bit digital signal.

FIG. 15C is a diagram showing a configuration example of the pixel 100 in the image pickup device 70 described in FIG. 14B. The pixel 100 of FIG. 15C includes a photoelectric conversion unit 110, a charge holding unit 120, an image signal generation unit 130, a current / voltage conversion unit 160, a buffer 170, a subtractor 180, and a quantizer 190. The pixel 100 of FIG. 15C can output a luminance signal and a luminance difference signal. The photoelectric conversion unit 110 of FIG. 15C is commonly connected to the charge holding unit 120 and the current-voltage conversion unit 160. Since the configuration of the pixel 100 other than this is the same as that in FIGS. 15A and 15B, the description thereof will be omitted.

Note that the configuration of the second embodiment of the present disclosure can be applied to other embodiments. Specifically, the frame output unit 40, the accumulation period adjustment unit 310, and the luminance difference frame generation unit 320 of FIG. 8 can be applied to the third to fifth embodiments of the present disclosure.

The configuration of the third embodiment of the present disclosure can be applied to other embodiments. Specifically, the adjustment of the luminance difference signal accumulation period for each region 210 in FIG. 11 can be applied to the second, fourth, and fifth embodiments of the present disclosure.

The configuration of the fourth embodiment of the present disclosure can be applied to other embodiments. Specifically, the configuration in which the adjustment of the luminance difference signal storage period and the generation of the frame in FIGS. 12A and 12B are performed in parallel can be applied to the second, third and fifth embodiments of the present disclosure. can.

The configuration of the fifth embodiment of the present disclosure can be applied to other embodiments. Specifically, the compression of the luminance difference frame of FIG. 13B can be applied to the second to fourth embodiments of the present disclosure.

(effect)
The image pickup control device 30 has a storage period adjusting unit 310. The storage period adjusting unit 310 stores a time-series frame composed of the luminance signal of the subject and a luminance difference signal which is a signal of the amount of change in the luminance of the subject, and the luminance difference frame generated by accumulating each time the frame is generated. The luminance difference signal accumulation period, which is the period for accumulating the luminance difference signal when the next luminance difference frame is generated, is adjusted based on at least one of the above. Thereby, the image pickup control device 30 can apply the adjusted luminance difference signal accumulation period to the generation of the next luminance difference frame when generating the time-series luminance difference frame.

Further, the storage period adjusting unit 310 may adjust the luminance difference signal storage period based on the spatial frequency of the luminance difference signal and the spatial frequency of the luminance signal. This makes it possible to make adjustments based on the spatial frequency of the luminance difference signal and the spatial frequency of the luminance signal.

Further, the accumulation period adjustment unit 310 may adjust the luminance difference signal accumulation period based on the statistical value of the luminance signal. This makes it possible to make adjustments based on the statistical values of the luminance signal.

Further, the accumulation period adjusting unit 310 may adjust the luminance difference signal accumulating period by using the average of the luminance signals as the statistical value. This makes it possible to make adjustments based on the average of the luminance signals.

Further, the storage period adjusting unit 310 may store the luminance difference signal in a period synchronized with the frame generation period, which is the frame generation period. As a result, the luminance difference frame can be generated in synchronization with the frame generation period.

Further, the accumulation period adjusting unit 310 divides the frame generation period into a plurality of subframe periods, selects the divided subframe period, and accumulates the luminance difference signal in the selected subframe period. The luminance difference signal storage period may be adjusted accordingly. This makes it possible to adjust the luminance difference signal storage period by selecting the subframe period.

Further, the accumulation period adjusting unit 310 adjusts the luminance difference signal accumulation period by starting the accumulation of the luminance difference signal during the frame generation period and adjusting the period until the luminance difference signal accumulation is stopped. You may. This makes it possible to adjust the luminance difference signal accumulation period by changing the period from the start to the stop of the luminance difference signal accumulation.

Further, the accumulation period adjusting unit 310 may further adjust the frame generation period, which is the period for generating the frame. Thereby, the luminance difference signal accumulation period and the frame generation period can be adjusted.

Further, the storage period adjusting unit 310 may adjust the exposure period when generating the luminance signal as the frame generation period. This makes it possible to adjust the frame generation period by adjusting the exposure period.

Further, the accumulation period adjusting unit 310 may adjust the luminance difference signal accumulating period for each of a plurality of regions formed in the luminance difference frame. This makes it possible to apply different luminance difference signal storage periods for each region.

Further, the luminance difference frame generation unit 320 that generates the luminance difference frame based on the adjusted luminance difference signal storage period may be further provided. This makes it possible to generate a luminance difference frame based on the adjusted luminance difference signal storage period.

Further, the luminance difference frame generation unit 320 may further thin out the generated luminance difference frames. This makes it possible to reduce unnecessary luminance difference frames.

Further, the luminance difference frame generation unit 320 may further compress the generated luminance difference frame. As a result, the amount of data in the luminance difference frame can be reduced.

Further, the brightness difference frame generation unit 320 may perform the compression based on at least one of the spatial frequency of the brightness difference signal, the spatial frequency of the brightness signal, and the statistical value of the brightness signal. This makes it possible to perform compression based on at least one of the spatial frequency of the luminance difference signal and the luminance signal and the statistical value of the luminance signal.

Further, it may further have a processing unit 330 that performs processing based on the generated luminance difference frame. This makes it possible to perform processing based on the luminance difference frame in which the luminance difference signal storage period is adjusted.

Further, the processing unit 330 may perform the above-mentioned movement detection processing of the subject. This makes it possible to detect the movement of the subject based on the luminance difference frame in which the luminance difference signal storage period is adjusted.

Further, the image pickup device 1 has a first image pickup element (image pickup element 10), a second image pickup element (image pickup element 20), and a storage period adjusting unit 310. The first image sensor (image sensor 10) outputs a luminance signal of the subject. The second image sensor (image sensor 20) outputs a luminance difference signal, which is a signal of the amount of change in the luminance of the subject. The storage period adjusting unit 310 generates a time-series frame composed of the luminance signal output by the first image pickup element and a luminance difference signal output by the second image pickup element for each generation of the frame. Based on at least one of the luminance difference frames formed by accumulating, the luminance difference signal accumulation period, which is the period for accumulating the luminance difference signal when the next luminance difference frame is generated, is adjusted. The image pickup apparatus 1 can apply the adjusted luminance difference signal storage period to the generation of the next luminance difference frame when generating the time-series luminance difference frame.

It should be noted that the effects described in the present specification are merely examples and are not limited, and other effects may be obtained.

The present technology can also have the following configurations.
(1)
Based on at least one of the time-series frames composed of the luminance signal of the subject and the luminance difference signal generated by accumulating the luminance difference signal which is a signal of the amount of change in the luminance of the subject every time the frame is generated. An image pickup control device having a storage period adjusting unit for adjusting a luminance difference signal storage period, which is a period for accumulating the luminance difference signal when generating the next luminance difference frame.
(2)
The imaging control device according to (1), wherein the storage period adjusting unit adjusts the luminance difference signal storage period based on the spatial frequency of the luminance difference signal and the spatial frequency of the luminance signal.
(3)
The imaging control device according to (1) or (2), wherein the accumulation period adjusting unit adjusts the luminance difference signal accumulating period based on the statistical value of the luminance signal.
(4)
The imaging control device according to (3), wherein the accumulation period adjusting unit adjusts the luminance difference signal accumulating period by using the average of the luminance signals as the statistical value.
(5)
The imaging control device according to any one of (1) to (4), wherein the storage period adjusting unit stores the luminance difference signal in a period synchronized with the frame generation period, which is the frame generation period.
(6)
The accumulation period adjusting unit divides the frame generation period into a plurality of subframe periods, selects the divided subframe period, and accumulates the luminance difference signal in the selected subframe period. The image pickup control device according to (5) above, which adjusts the luminance difference signal storage period.
(7)
The accumulation period adjusting unit adjusts the luminance difference signal accumulation period by adjusting the period until the luminance difference signal accumulation is started and the luminance difference signal accumulation is stopped during the frame generation period. The image pickup control device according to 5).
(8)
The imaging control device according to any one of (1) to (7) above, wherein the accumulation period adjusting unit further adjusts a frame generation period, which is a period for generating the frame.
(9)
The image pickup control device according to (8) above, wherein the accumulation period adjusting unit adjusts the exposure period at the time of generating the luminance signal as the frame generation period.
(10)
The imaging control device according to any one of (1) to (9), wherein the accumulation period adjusting unit adjusts the luminance difference signal accumulating period for each of a plurality of regions formed in the luminance difference frame.
(11)
The imaging control device according to any one of (1) to (10), further comprising a luminance difference frame generation unit that generates the luminance difference frame based on the adjusted luminance difference signal storage period.
(12)
The image pickup control device according to (11) above, wherein the luminance difference frame generation unit further thins out the generated luminance difference frames.
(13)
The image pickup control device according to (11) above, wherein the luminance difference frame generation unit further compresses the generated luminance difference frame.
(14)
The image pickup control according to (13), wherein the luminance difference frame generation unit performs the compression based on at least one of the spatial frequency of the luminance difference signal, the spatial frequency of the luminance signal, and the statistical value of the luminance signal. Device.
(15)
The imaging control device according to any one of (11) to (14), further comprising a processing unit that performs processing based on the generated luminance difference frame.
(16)
The imaging control device according to (15) above, wherein the processing unit performs a motion detection process of the subject.
(17)
The first image sensor that outputs the luminance signal of the subject and
A second image sensor that outputs a luminance difference signal, which is a signal of the amount of change in the luminance of the subject,
Luminance formed by accumulating a time-series frame composed of the luminance signal output by the first image pickup element and the luminance difference signal output by the second luminance signal for each generation of the frame. An image pickup apparatus having a storage period adjusting unit for adjusting a luminance difference signal storage period, which is a period for accumulating the luminance difference signal when generating the next luminance difference frame based on at least one of the luminance frames.
(18)
The imaging device according to (17), wherein the storage period adjusting unit adjusts the luminance difference signal storage period based on the spatial frequency of the luminance signal and the spatial frequency of the luminance signal.
(19)
The image pickup apparatus according to (17) or (18), wherein the accumulation period adjusting unit adjusts the luminance difference signal accumulating period based on the statistical value of the luminance signal.
(20)
The image pickup apparatus according to (19), wherein the accumulation period adjusting unit adjusts the luminance difference signal accumulating period by using the average of the luminance signals as the statistical value.
(21)
The imaging device according to any one of (17) to (20), wherein the storage period adjusting unit stores the luminance difference signal in a period synchronized with the frame generation period, which is the frame generation period.
(22)
The accumulation period adjusting unit divides the frame generation period into a plurality of subframe periods, selects the divided subframe period, and accumulates the luminance difference signal in the selected subframe period. The image pickup apparatus according to (21) above, which adjusts the luminance difference signal storage period.
(23)
The accumulation period adjusting unit adjusts the luminance difference signal accumulation period by adjusting the period until the luminance difference signal accumulation is started and the luminance difference signal accumulation is stopped during the frame generation period. 21) The imaging device according to.
(24)
The image pickup apparatus according to any one of (17) to (23), wherein the accumulation period adjusting unit further adjusts a frame generation period, which is a period for generating the frame.
(25)
The image pickup apparatus according to (24), wherein the storage period adjusting unit adjusts the exposure period when generating the luminance signal as the frame generation period.
(26)
The image pickup apparatus according to any one of (17) to (25), wherein the accumulation period adjusting unit adjusts the luminance difference signal accumulating period for each of a plurality of regions formed in the luminance difference frame.
(27)
The image pickup apparatus according to any one of (17) to (26), further comprising a luminance difference frame generation unit that generates the luminance difference frame based on the adjusted luminance difference signal storage period.
(28)
The image pickup apparatus according to (27), wherein the luminance difference frame generation unit further thins out the generated luminance difference frames.
(29)
The image pickup apparatus according to (27), wherein the luminance difference frame generation unit further compresses the generated luminance difference frame.
(30)
The image pickup apparatus according to (29), wherein the luminance difference frame generation unit performs the compression based on at least one of the spatial frequency of the luminance difference signal, the spatial frequency of the luminance signal, and the statistical value of the luminance signal. ..
(31)
The image pickup apparatus according to any one of (27) to (30), further comprising a processing unit that performs processing based on the generated luminance difference frame.
(32)
The image pickup apparatus according to (31), wherein the processing unit performs a motion detection process of the subject.
(33)
Based on at least one of the time-series frames composed of the luminance signal of the subject and the luminance difference signal generated by accumulating the luminance difference signal which is a signal of the amount of change in the luminance of the subject every time the frame is generated. , An imaging control method comprising an accumulation period adjusting procedure for adjusting a luminance difference signal accumulation period, which is a period for accumulating the luminance difference signal when generating the next luminance difference frame.

1

Imaging device

10, 20, 70 Imaging element 11 Pixel array unit 30 Imaging control device 40 Frame output unit 50, 60 Camera 80 Illuminance meter 100 pixels 110, 150

Photoelectric conversion unit

200, 411 Luminance difference frame 210 Area 310 Storage period adjustment unit 320 Luminance difference frame generation unit 330 Processing unit 340 Control unit 400 Subject 410 Frame

Claims

Based on at least one of the time-series frames composed of the luminance signal of the subject and the luminance difference signal generated by accumulating the luminance difference signal which is a signal of the amount of change in the luminance of the subject every time the frame is generated. An image pickup control device having a storage period adjusting unit for adjusting a luminance difference signal storage period, which is a period for accumulating the luminance difference signal when generating the next luminance difference frame.
The imaging control device according to claim 1, wherein the storage period adjusting unit adjusts the luminance difference signal storage period based on the spatial frequency of the luminance signal and the spatial frequency of the luminance signal.
The imaging control device according to claim 1, wherein the accumulation period adjusting unit adjusts the luminance difference signal accumulating period based on the statistical value of the luminance signal.
The imaging control device according to claim 3, wherein the accumulation period adjusting unit adjusts the luminance difference signal accumulating period using the average of the luminance signals as the statistical value.
The imaging control device according to claim 1, wherein the storage period adjusting unit stores the luminance difference signal in a period synchronized with the frame generation period, which is the frame generation period.
The accumulation period adjusting unit divides the frame generation period into a plurality of subframe periods, selects the divided subframe period, and accumulates the luminance difference signal in the selected subframe period. The imaging control device according to claim 5, wherein the luminance difference signal storage period is adjusted.
The claim that the accumulation period adjusting unit adjusts the luminance difference signal accumulation period by starting the accumulation of the luminance difference signal in the frame generation period and adjusting the period until the luminance difference signal accumulation is stopped. 5. The imaging control device according to 5.
The imaging control device according to claim 1, wherein the accumulation period adjusting unit further adjusts a frame generation period, which is a period for generating the frame.
The imaging control device according to claim 8, wherein the storage period adjusting unit adjusts the exposure period when generating the luminance signal as the frame generation period.
The imaging control device according to claim 1, wherein the accumulation period adjusting unit adjusts the luminance difference signal accumulating period for each of a plurality of regions formed in the luminance difference frame.
The imaging control device according to claim 1, further comprising a luminance difference frame generation unit that generates the luminance difference frame based on the adjusted luminance difference signal storage period.
The imaging control device according to claim 11, wherein the luminance difference frame generation unit further thins out the generated luminance difference frames.
The imaging control device according to claim 11, wherein the luminance difference frame generation unit further compresses the generated luminance difference frame.
The image pickup control device according to claim 13, wherein the luminance difference frame generation unit performs the compression based on at least one of the spatial frequency of the luminance difference signal, the spatial frequency of the luminance signal, and the statistical value of the luminance signal. ..
The imaging control device according to claim 11, further comprising a processing unit that performs processing based on the generated luminance difference frame.
The imaging control device according to claim 15, wherein the processing unit performs processing for detecting the movement of the subject.
The first image sensor that outputs the luminance signal of the subject and
A second image sensor that outputs a luminance difference signal, which is a signal of the amount of change in the luminance of the subject,
Luminance formed by accumulating a time-series frame composed of the luminance signal output by the first image pickup element and the luminance difference signal output by the second luminance signal for each generation of the frame. An image pickup apparatus having a storage period adjusting unit for adjusting a luminance difference signal storage period, which is a period for accumulating the luminance difference signal when generating the next luminance difference frame based on at least one of the luminance frames.