JP2023130522A - Information processing device, information processing method, and program - Google Patents

Abstract

To achieve more appropriate functions in an information processing technique using a digital image.SOLUTION: In an imaging apparatus 1, an imaging unit 10 transmits data indicating the change of the state of a pixel corresponding to an imaging element on the basis of the change of the output of the imaging element included in an imaging sensor for imaging a subject. A data receiving part 21 asynchronously receives the data indicating the change of the state of the pixel transmitted by the imaging unit 10 for each pixel. A storage part 20B stores the data indicating the change of the state of the pixel, time information on the data indicating the change of the state of the pixel, and the address of the pixel in a preset storage format. A data storage control part 22 allows the storage part 20B to store the data indicating the change of the state of the pixel received by the data receiving part 21 as storage format data.SELECTED DRAWING: Figure 1

Description

The present invention relates to an information processing device, an information processing method, and a program.

In recent years, digital images have come to be utilized in a wide range of fields such as automatic driving of automobiles and AI (Artificial Intelligence).
As a digital image, a still image or a moving image, which is a color image of the real world, is generally used.
Note that Patent Document 1 discloses a technique for recognizing objects such as vehicles using digital images.

JP2018-097766A

However, when building an application using digital images, conventionally used digital images (such as still images or moving images captured as color images of the real world) may not be suitable for the intended function.
For example, when trying to detect an obstacle in automatic driving of a car, etc., the closer the object is to the camera, the faster the object moves, which increases the possibility that blurring will occur in the captured digital image. Therefore, the closer an object is to the camera and the more attention it receives, the more difficult it is to accurately detect the object, which may lead to a situation where the obstacle cannot be detected properly.
In addition, conventionally used digital images are output after processing to generate one frame of image based on the data output from the image sensor, so in applications that require high-speed processing, sufficient processing is required. It may be difficult to maintain speed.
That is, with conventional information processing technology using digital images, there are cases where appropriate functions cannot be realized.

An object of the present invention is to enable more appropriate functions to be realized in information processing technology using digital images.

In order to solve the above problems, an imaging device according to an embodiment of the present invention includes:
Imaging means for transmitting data representing a change in the state of a pixel corresponding to the image sensor based on a change in the output of an image sensor included in an image sensor that images a subject;
data receiving means for asynchronously receiving data representing a change in the state of the pixel transmitted by the imaging means for each pixel;
a storage means in which data representing a change in the state of the pixel, time information regarding the data representing a change in the state of the pixel, and an address of the pixel are stored in a preset storage format;
storage control means for storing data representing a change in the state of the pixel received by the data receiving means in the storage means as data in the storage format;
It is characterized by having the following.

According to the present invention, it is possible to realize more appropriate functions in information processing technology using digital images.

FIG. 1 is a schematic diagram showing the overall configuration of an imaging device 1 according to the present embodiment. 2 is a schematic diagram showing the configuration of a storage area of an output data storage section 24. FIG. 2 is a schematic diagram showing a reference range of the amount of received data per unit time (the amount of output data of the imaging device 12) with respect to the amount of movement of the imaging unit 10. FIG. 3 is a flowchart illustrating the flow of data reception processing executed by the information processing unit 20. FIG. 3 is a flowchart illustrating the flow of threshold adjustment processing executed by the information processing unit 20. FIG. 3 is a flowchart illustrating the flow of data storage processing executed by the information processing unit 20. FIG. 3 is a flowchart illustrating the flow of application execution processing executed by the information processing unit 20. FIG. FIG. 2 is a schematic diagram showing a configuration example of an image sensor in which an image sensor is provided with a filter that transmits light of a specific wavelength. FIG. 3 is a schematic diagram showing an example of a state in which pixel data of storage format 1 is stored in a storage area having one-dimensional continuous addresses. 1 is a schematic diagram showing the overall configuration of an imaging device 1 in a specific application example 1. FIG. 2 is a block diagram showing a functional configuration of an information processing unit 20. FIG. FIG. 2 is a schematic diagram illustrating the concept of interpolating the shape of an object using pixel data including data representing the state of pixels captured by the imaging unit 10. FIG. FIG. 2 is a schematic diagram showing an example of a route on which automatic driving simulation is performed. 3 is a flowchart illustrating the flow of environment construction processing executed by the imaging device 1. FIG. 3 is a flowchart illustrating the flow of simulation processing executed by the imaging device 1. FIG. 3 is a flowchart illustrating the flow of analysis processing executed by the imaging device 1. FIG.

Embodiments of the present invention will be described below with reference to the drawings.
[First embodiment]
[composition]
FIG. 1 is a schematic diagram showing the overall configuration of an imaging device 1 according to this embodiment.
The imaging device 1 detects the state of each pixel (here, a change in brightness value) in an imaging unit 10 having an imaging sensor in which imaging elements are arranged, and the information processing unit 20 stores data ( (hereinafter referred to as "pixel data" as appropriate) is received and stored in a storage area in a predetermined storage format. The information processing unit 20 also reads out the data (pixel data) of the detection results of the imaging unit 10 stored in the storage area, and uses it in various applications, as well as the imaging parameters ( (threshold value, etc. for detecting the state of each pixel).
Therefore, in the imaging device 1, it is possible to detect a change in the subject in the angle of view for each pixel, and to output a pixel in which a change has occurred as pixel data indicating that a change has occurred.
This makes it possible for the imaging device 1 to detect changes in the subject in the angle of view at high speed with a small amount of data.
Furthermore, the imaging device 1 can read out pixel data stored in a predetermined storage format and use it for various applications.
Therefore, according to the imaging device 1, it is possible to realize more appropriate functions in information processing technology using digital images.
The configuration of the imaging device 1 will be specifically described below.

As shown in FIG. 1, the imaging device 1 includes an imaging unit 10, an information processing unit 20, and a sensor unit 30, and the information processing unit 20 connects the imaging unit 10 and the sensor unit 30 to a wired or wireless communication network. connected by. Note that it is also possible to configure two or all of the imaging unit 10, the information processing unit 20, and the sensor unit 30 as one unit.

The imaging unit 10 includes a lens 11 and an imaging device 12.
The lens 11 focuses the light rays from the subject within the angle of view and forms an image of the subject on the image sensor of the imaging device 12 .
The imaging device 12 includes an imaging sensor in which a plurality of imaging elements are arranged, and a processing circuit that processes output signals of the imaging sensor. The imaging device 12 outputs a signal representing the state of each part of the subject image formed by the lens 11 for each pixel corresponding to the imaging element. That is, pixel data including data representing the state of each pixel is output from the imaging device 12 based on the signal representing the state of each part of the subject image. In this embodiment, a change in the brightness value of each pixel is used as an example of a parameter representing the state of each pixel. That is, when the amount of change in charge accumulated in each image sensor reaches a set threshold value, the imaging device 12 sets "+1" representing an increase or a decrease as data (polarity) representing the state of each pixel. Outputs "-1" representing the value. Specifically, when the increase in the brightness detected for each pixel exceeds a set threshold Th1, the imaging device 12 outputs data (+1) representing the increase, and the brightness detected for each pixel is increased. When the decrease is equal to or greater than the set threshold Th1, data (-1) representing the decrease is output. Note that the threshold Th1 used to determine whether the brightness increases or decreases is set by the information processing unit 20.
With the above configuration, the imaging device 12 asynchronously outputs pixel data including data (+1 or -1) representing the state of the pixel for each pixel.

The information processing unit 20 is configured by an information processing device such as a PC (Personal Computer), a microcomputer, or a smartphone, and includes a processor 20A and a storage section 20B.
The processor 20A is configured by an arithmetic processing device such as a CPU (Central Processing Unit). When the processor 20A executes a program for information processing, the processor 20A implements a data receiving section 21, a data storage control section 22, and a processing section 23 as functional configurations.

The storage unit 20B is constituted by a storage device such as a hard disk or a semiconductor memory, and an output data storage unit 24 is formed in a part of its storage area.
The output data storage section 24 stores pixel data output from the imaging unit 10.
FIG. 2 is a schematic diagram showing the configuration of the storage area of the output data storage section 24. As shown in FIG.
As shown in FIG. 2, a two-dimensional memory (frame memory) having an address corresponding to an address of an image captured by the imaging unit 10 is formed in the output data storage section 24. Specifically, the output data storage unit 24 includes a frame memory (frame memory for increase 24a) used when the data representing the state of each pixel is increasing, and a frame memory used when the data representing the state of each pixel is decreasing. A frame memory (frame memory for reduction 24b) used when the data is in progress is formed independently.

The pixel data output from the imaging unit 10 is stored in the increase frame memory 24a and the decrease frame memory 24b by the data storage control section 22, as described later.
Note that the storage contents of the output data storage unit 24 (increase frame memory 24a and decrease frame memory 24b) are reset at predetermined time intervals (for example, every 10 [ms]). However, in order to make it possible to refer to past pixel data even after resetting, the backup frame memory 24a for increasing and the frame memory 24b for decreasing may be formed in separate storage areas.

The data receiving unit 21 asynchronously receives pixel data output from the imaging device 12 of the imaging unit 10, and temporarily stores the received pixel data in a buffer for received data. Furthermore, the data receiving section 21 transfers the pixel data stored in the received data buffer to the data storage control section at a predetermined timing that defines the internal operation of the information processing unit 20 (for example, every 1 [ms]). Output to 22. Specifically, upon receiving pixel data from the imaging device 12, the data receiving unit 21 stores the received time information, the pixel address, and data (+1 or -1) representing the state of each pixel in the received data buffer. to be memorized. Then, at a predetermined timing, the data receiving unit 21 associates the time information stored in the received data buffer, the address of the pixel, and the data (+1 or -1) representing the state of each pixel, and sends the data It is output to the storage control unit 22.

Furthermore, in the present embodiment, the data receiving unit 21 changes the threshold Th1 used to determine whether the brightness in the imaging device 12 increases or decreases based on the amount of received data per unit time and the amount of movement of the imaging unit 10. .
FIG. 3 is a schematic diagram showing a reference range of the amount of received data per unit time (the amount of output data of the imaging device 12) with respect to the amount of movement of the imaging unit 10.
As shown in FIG. 3, as the amount of movement (rotational motion and translational motion) of the imaging unit 10 increases, the reference value (broken line in FIG. 3) of the amount of received data per unit time increases. Further, the reference range of the amount of data received per unit time is set within a certain amount of data from the reference value.
According to FIG. 3, an appropriate range (reference range) for the amount of received data per unit time is determined according to the amount of movement of the imaging unit 10, and if it deviates from the reference range, the brightness in the imaging device 12 increases. It is considered that the threshold value Th1 used for determining the decrease is an inappropriate value (too much or too small).

That is, when the amount of received data per unit time exceeds the reference range, it is considered appropriate to increase the threshold Th1 used to determine whether the brightness in the imaging device 12 increases or decreases. Therefore, the data receiving unit 21 outputs an instruction to increase the threshold Th1 to the imaging device 12. Furthermore, when the amount of received data per unit time is below the reference range, it is considered appropriate to decrease the threshold Th1 used to determine whether the brightness in the imaging device 12 increases or decreases. Therefore, the data receiving unit 21 outputs an instruction to decrease the threshold Th1 to the imaging device 12.
Note that the appropriate range (reference range) for the amount of data received per unit time is determined by calibrating the image capturing unit 10 by capturing images of various subjects in a state where there is no movement, a state where the movement is small, a state where the movement is large, etc. You can set specific values by doing this.

Further, even when an instruction to change the threshold Th1 used for determining an increase or decrease in brightness in the imaging device 12 is input from the processing unit 23, the data receiving unit 21 changes the threshold Th1 according to the content of the change instruction. An instruction to increase or decrease is output to the imaging device 12.
Note that the data receiving unit 21 may focus on or zoom in on a specific area within the angle of view based on the distribution of received data output from the imaging device 12 (address bias, etc.). . For example, when detecting an object using a surveillance camera, data representing the state of pixels is likely to be output in a concentrated manner in the area where an intruder is photographed. If the address is unevenly distributed, the data receiving section 21 (or other functional block of the information processing unit 20) can output an instruction to the imaging unit 10 to zoom in to the vicinity of the address.

The data storage control section 22 stores the pixel data (time information, pixel address, data representing the state of each pixel (+1 or -1)) input from the data receiving section 21 in the output data storage section 24.
In the pixel data input from the data receiving unit 21, when the data representing the state of the pixel is “+1” representing an increase, the data storage control unit 22 stores the address of the pixel of the pixel data in the increase frame memory 24a. Store time information in the address corresponding to . Further, in the pixel data input from the data receiving unit 21, when the data representing the pixel state is “-1” representing a decrease, the data storage control unit 22 stores the pixel data in the frame memory for decrease 24b. Time information is stored in an address corresponding to the pixel address. In this embodiment, when pixel data is output multiple times for the same pixel, the time information of the pixel is overwritten in the increase frame memory 24a or the decrease frame memory 24b.

As a result, the time information is sequentially written into the frame memory for increase 24a at the address of the pixel whose luminance value has increased over the threshold Th1, and the frame memory for decrease 24b is written with the time information of the pixel whose luminance value has increased over the threshold Th1. The time information is sequentially written to the address of the pixel whose brightness value has decreased by the threshold value Th1 or more.
Note that the data storage control unit 22 may acquire the value of the threshold Th1 used for determination when the pixel data is output, and store the value of the threshold Th1 at the address of the pixel in addition to the time information. good.

The processing unit 23 reads out the pixel data stored in the output data storage unit 24 and executes various processes depending on the application. The processing unit 23 can execute various applications using the pixel data output by the imaging unit 10, such as an application for environment recognition in automatic driving, and an application for object detection in a surveillance camera. Applications, as well as related machine learning and data processing, can be executed.

Further, the processing unit 23 can appropriately output an instruction to change the threshold value Th1 used for determining an increase or decrease in brightness in the imaging device 12 to the data receiving unit 21 in response to a request from an application being executed or the like.
Note that the processing section 23 can read out the pixel data stored in the output data storage section 24, perform various processing, and use the pixel data.
For example, for the entire pixel data stored in the increase frame memory 24a and the decrease frame memory 24b, the brightness increases and decreases in the imaging device 12 by the number of changes in the brightness value of each pixel within a predetermined time. By integrating the threshold value Th1 used for the determination with a sign, it is possible to calculate the actual amount of change in the luminance value at each pixel.

In addition, for the pixel data stored in the increase frame memory 24a and the decrease frame memory 24b, the brightness increases and decreases in the imaging device 12 by the number of changes in the brightness value of each pixel within a predetermined period of time. By integrating the threshold value Th1 used for the determination, it is possible to calculate the cumulative increase amount and cumulative decrease amount of the brightness value in each pixel.
By configuring frame data using the pixel data calculated in this way, it is possible to generate an image in which a change in the state of the subject is mapped (here, an image representing a change in brightness).
By using the image generated in this way alone or in combination with a color image, it is possible to improve performance in environment recognition, object detection, etc.

Note that when executing an application, the processing unit 23 can perform processing using either a single thread or multi-thread. Threads refer to different processes running in parallel.
For example, when processing by a single thread, the processing unit 23 acquires pixel data from the imaging unit 10 and acquires pixel data from the output data storage unit 24 within that thread (within the same process). Reading etc. can be performed simultaneously.
In addition, when performing multi-thread processing, the processing unit 23 transfers pixel data between different threads (different processes) via memory etc. pixel data can be acquired, pixel data can be read out from the output data storage unit 24 by the second thread, and the like.

The sensor unit 30 includes various sensors for detecting movement, such as an acceleration sensor or a gyro sensor. The sensor unit 30 can be attached to the imaging unit 10 or to the information processing unit 20 integrated with the imaging unit 10, and is installed to detect movement of the imaging unit 10. The detection results of the sensor unit 30 are output to the information processing unit 20.

[motion]
Next, the operation of the imaging device 1 will be explained.
[Data reception processing]
FIG. 4 is a flowchart illustrating the flow of data reception processing executed by the information processing unit 20.
The data reception process is started when the imaging device 1 is started, and is repeatedly executed.
In step S1, the data receiving unit 21 determines whether pixel data has been received from the imaging device 12.
If pixel data has not been received from the imaging device 12, a negative determination is made in step S1, and the process of step S1 is repeated.
On the other hand, when pixel data is received from the imaging device 12, the determination in step S1 is YES, and the process moves to step S2.

In step S2, the data receiving unit 21 stores the received pixel data in a buffer for received data.
In step S3, the data receiving section 21 determines whether the timing for outputting the data to the data storage control section 22 has arrived.
If it is not the output timing to the data storage control unit 22, the determination in step S3 is NO, and the process moves to step S1.
On the other hand, if it is the output timing to the data storage control unit 22, the determination in step S3 is YES, and the process moves to step S4.

In step S4, the data receiving section 21 outputs the pixel data stored in the received data buffer to the data storage control section 22.
After step S4, the data reception process is repeated.

[Threshold adjustment processing]
FIG. 5 is a flowchart illustrating the flow of threshold adjustment processing executed by the information processing unit 20.
The threshold value adjustment process is started when the imaging device 1 is started, and is repeatedly executed.
In step S11, the data receiving unit 21 determines whether the amount of received data per unit time is within a reference range with respect to the amount of movement of the imaging unit 10 represented by the detection result of the sensor unit 30.
If the amount of received data per unit time is not within the reference range with respect to the amount of movement of the imaging unit 10 represented by the detection result of the sensor unit 30, the determination in step S11 is NO, and the process moves to step S13.
On the other hand, if the amount of received data per unit time is within the reference range with respect to the amount of movement of the imaging unit 10 represented by the detection result of the sensor unit 30, YES is determined in step S11, and the process moves to step S12. do.

In step S12, the data receiving unit 21 determines whether an instruction to change the threshold Th1 used for determining increase or decrease in brightness in the imaging device 12 has been input from the processing unit 23.
If an instruction to change the threshold value Th1 used for determining an increase or decrease in brightness in the imaging device 12 has not been input from the processing unit 23, the determination in step S12 is NO, and the process moves to step S11.
On the other hand, if an instruction to change the threshold value Th1 used for determining an increase or decrease in brightness in the imaging device 12 is input from the processing unit 23, the determination in step S12 is YES, and the process moves to step S13.

In step S13, the data receiving unit 21 outputs an instruction to increase or decrease the threshold Th1 to the imaging device 12.
After step S13, the threshold value adjustment process is repeated.

[Data storage processing]
FIG. 6 is a flowchart illustrating the flow of data storage processing executed by the information processing unit 20.
The data storage process is started when the imaging device 1 is started, and is repeatedly executed.
In step S21, the data storage control unit 22 determines whether pixel data has been input from the data receiving unit 21.
If no pixel data has been input from the data receiving unit 21, a negative determination is made in step S21, and the process of step S21 is repeated.
On the other hand, if pixel data is input from the data receiving section 21, the determination in step S21 is YES, and the process moves to step S22.

In step S22, the data storage control unit 22 determines whether the data representing the state of the pixel among the input pixel data represents an increase (+1).
If the data representing the state of the pixel among the input pixel data does not represent an increase (that is, represents a decrease (-1)), the determination in step S22 is NO, and the process moves to step S24. .
On the other hand, if the data representing the state of the pixel among the input pixel data indicates an increase, YES is determined in step S22, and the process moves to step S23.

In step S23, the data storage control unit 22 stores time information of the input pixel data in the frame memory for increase 24a.
In step S24, the data storage control unit 22 stores time information of the input pixel data in the frame memory for reduction 24b.
After step S23 and step S24, the data storage process is repeated.

[Application execution process]
FIG. 7 is a flowchart illustrating the flow of application execution processing executed by the information processing unit 20.
The application execution process is started by performing an operation that instructs the start of the application execution process.
In step S31, the processing unit 23 reads the specified application program. Note that the application program is stored in the storage unit 20B or a storage medium such as a removable medium (not shown).

In step S32, the processing unit 23 executes processing based on the designated application program (hereinafter referred to as "designated application processing").
After step S32, the application execution process ends.

As described above, in the imaging device 1 according to the present embodiment, the imaging unit 10 asynchronously outputs pixel data including data representing the state of the pixel for each pixel, and the information processing unit 20 outputs the pixel data. The output data is stored in the output data storage section 24.
In the specified application processing, the information processing unit 20 reads and uses the pixel data stored in the output data storage section 24 to execute various applications while detecting changes in the subject in the viewing angle.
As a result, as soon as a change is detected for each pixel, data representing the state of the pixel is output asynchronously. Compared to other techniques, it is possible to detect changes in the subject in terms of angle of view at high speed and with a small amount of data.
Furthermore, the imaging device 1 can read out pixel data stored in a predetermined storage format and use it for various applications.
Therefore, according to the imaging device 1, it is possible to realize more appropriate functions in information processing technology using digital images.

Further, the output data storage section 24 is formed with a two-dimensional memory (an increase frame memory 24a and a decrease frame memory 24b) having an address corresponding to the address of the image captured by the imaging unit 10. The information processing unit 20 then updates the pixel in the increase frame memory 24a or the decrease frame memory 24b depending on whether the data representing the state of the pixel represents an increase or a decrease. Time information in pixel data is stored in an address corresponding to the address.
Thereby, pixel data can be saved in a format that makes it easy to refer to necessary data. Also, of the pixel data (time information, pixel address, data representing the state of each pixel (+1 or -1)), only the time information needs to be stored at a predetermined address. In comparison, the amount of data to be stored can be reduced. Further, since the sizes of the increase frame memory 24a and the decrease frame memory 24b are determined in advance, it is possible to easily estimate the storage capacity required to store pixel data.

[Modification 1]
In the embodiments described above, the image sensor included in the imaging device 12 can be any sensor in which a plurality of image sensors are arranged, but it is also possible to provide the image sensor with additional functions.
For example, a filter (such as a color filter) that transmits light of a specific wavelength may be arranged in each image sensor in the image sensor, and the imaging unit 10 may output data representing the state of the pixel corresponding to the light of the specific wavelength. good.

FIG. 8 is a schematic diagram showing a configuration example of an image sensor in which a filter that transmits light of a specific wavelength is disposed on an image sensor.
Note that R in FIG. 8 indicates a red filter, G indicates a green filter, and B indicates a blue filter.
In the image sensor shown in FIG. 8, when the amount of change in charge accumulated in each image sensor reaches a set threshold, "+1" indicating an increase is added as pixel data, as in the case of the above-described image sensor 1. Alternatively, "-1" indicating a decrease is output.
In the data storage control section 22 of the information processing unit 20, the address of the image captured by the imaging unit 10 and the type of filter (wavelength of transmitted light) arranged in the imaging sensor at that address are registered in association with each other. It is possible to store and read out pixel data associated with the type of filter.

By configuring the image sensor in this manner, it becomes possible, for example, to generate a frame representing the state of a pixel for each color.
For example, when the wavelength of the light source is biased, it is possible to detect a change in the subject by selecting pixel data in which a filter that transmits light with a complementary color wavelength is arranged. In this case, since the change in the subject appears more clearly in the frame, it becomes possible to detect the change in the subject more appropriately.
That is, the configuration of Modification 1 can provide a more convenient technique in applications where processing that reflects the wavelength of light is effective.

[Modification 2]
In the embodiment described above, the output data storage section 24 includes a two-dimensional memory (frame memory 24a for increase and frame memory 24b for decrease) having an address corresponding to the address of the image captured by the imaging unit 10. , to the address corresponding to the address of the pixel in the increase frame memory 24a or the decrease frame memory 24b, depending on whether the data representing the state of the pixel represents an increase or a decrease. Time information attached to pixel data is stored.

On the other hand, it is possible to use the following pixel data storage format.
(Save format 1)
The storage format of pixel data is (time t, image address (x, y), polarity).
Note that "polarity" is a value (+1 or -1, etc.) indicating an increase or decrease as the state of a pixel.
When the storage format 1 is used, the storage area of the output data storage section 24 does not need to be a two-dimensional memory, and can be one-dimensional continuous addresses.

FIG. 9 is a schematic diagram showing an example of a state in which pixel data of storage format 1 is stored in a storage area having one-dimensional continuous addresses.
As shown in FIG. 9, the storage area is assigned memory addresses that increase by 1 starting from address 0 as one-dimensional continuous addresses, and the storage format is assigned to memory addresses with smaller addresses in the order in which data is received by the data receiving unit 21. 1 pixel data is stored.
When pixel data is stored in storage format 1, it becomes easier to retain and manage the data, and the storage area can be flexibly expanded or reduced depending on the amount of received data.

(Save format 2)
The pixel data is stored in a format in which the latest pixel data (for example, data other than addresses) is stored in a storage area prepared for each pixel.
For example, the time and polarity at which pixel data was received can be overwritten and saved in a storage area prepared for each pixel (that is, a storage area dedicated to each pixel).
Note that in the storage format 2, the storage area of the output data storage unit 24 does not need to be a two-dimensional memory as in the above embodiment, but is a one-dimensional continuous address similar to the storage area shown in FIG. be able to.
However, similar to the embodiment described above, a storage area for increasing and a storage area for decreasing are formed independently, and only time information is stored in one of the storage areas depending on the polarity value. It may also be a thing.

(Save format 3)
A format is used in which pixel data for multiple times is stored in a data structure that can be reconstructed in one area for storing pixel data.
Specifically, pixel data for a plurality of times are allocated and saved with respect to the data width of one area for storing pixel data. For example, if one area for storing pixel data has 64 bits, and time information is represented by 16 bits, the 64-bit storage area can store time information for four times. can.

When using such a data storage format, when saving the time information of newly received pixel data, for example, 64 bits of data in the target storage area are read out, and the time information to be deleted (the most 64-bit data is created by excluding old time information, new time information, etc., and written back to the target storage area.
Note that by storing older time information in higher digits of the storage area, time information can be easily replaced.

(Save format 4)
In place of time information, information on the number of times pixel data has been output within a certain period of time is stored in the area for storing pixel data.
Specifically, in each storage format in the above-described embodiment and this modification, the number of times the state of a pixel changes within a certain period of time (that is, the number of times pixel data is output) is calculated without storing time information. save.
If the time set as a certain period of time is very short and specific time information is not necessary, approximate time information can be obtained by dividing the set time by the number of times pixel data has been output.
Moreover, since the data to be saved is only a numerical value (an integer within a predetermined range) representing the number of times, the required storage capacity can be reduced compared to the case where time information is saved.
Therefore, by using the storage format 4, it is possible to reduce the storage capacity and the load of numerical calculations, so that the device specifications necessary for the imaging device 1 can be suppressed.

[Modification 3]
In the embodiment and each modification described above, it is possible to control whether or not to hold the pixel data stored in the output data storage section 24 according to a predetermined policy.
As a predetermined policy, it is possible to determine whether or not to hold data mainly based on time, and to determine whether or not to hold data based on the address of pixel data.
Specifically, the following retention policy can be adopted.

(Retention policy 1)
Delete past data for a certain period of time (set threshold time) or more.
That is, when pixel data becomes older by a set threshold time or more from the current time, that data is deleted.
This makes it possible to prevent excessively old pixel data from continuing to be held when there is no change in the state of the pixel.

(Retention policy 2)
Delete past data equal to or greater than a certain number of elements (set number of pixel data).
That is, if the pixel data becomes older than a certain number of recent elements (the set number of pixel data), that data is deleted.
Thereby, when the state of a pixel changes and pixel data is received one after another, past pixel data that is considered unnecessary can be deleted.

(Retention policy 3)
If pixel data within a certain amount of time in the past exists at the same address, that pixel data is retained and other pixel data are deleted.
When a change occurs in pixel data, there is a high possibility that a change opposite to the change will occur within a certain period of time, so it is appropriate to retain that pixel data.
For example, when detecting an object in a surveillance camera, when an intruder passes within the field of view, the pixels corresponding to the intruder's body will become darker than before at night, and will become brighter again when the intruder passes. Situations may arise.
Therefore, useful pixel data can be retained when a frame is generated from pixel data when an object is detected.

(Retention policy 4)
If there is pixel data held at a nearby address, the central pixel data is held; if there is no pixel data held at a nearby address, that pixel data is deleted.
In this case, pixel data that is considered to have actually changed in the subject can be retained, and pixel data that has been discretely generated due to noise or the like can be deleted.
That is, it is possible to retain valid pixel data and delete unnecessary pixel data.

[Modification 4]
In the above-described embodiment and each modification, the period for determining whether the amount of received data per unit time is within a reference range may be changed according to preset conditions.
If the threshold value Th1 used to determine an increase or decrease in brightness in the imaging device 12 is inappropriate, changes in the subject cannot be appropriately detected, so it is useful to change (correct) the threshold value Th1.
On the other hand, in order to correct the threshold value Th1 sequentially, if it is judged frequently whether the amount of received data per unit time is within the standard range, there is a possibility that the processing load will increase and the system will become unstable. be.
Therefore, based on various information input to the imaging device 1, conditions regarding the cycle for determining whether or not the amount of received data per unit time is within the reference range are set, and the determination is made according to this condition. This allows the threshold Th1 to be corrected more efficiently.
Note that it may be determined whether the amount of received data per unit time is within the reference range or not for each pixel, and the threshold Th1 may also be set for each pixel.
Thereby, it is possible to determine whether the amount of received data per hour is within the reference range for each pixel at a determination period according to the state of the pixel, and to set a more appropriate threshold Th1.

[Specific application example 1]
Next, a specific application example of the imaging device 1 will be described.
In this application example, it is assumed that the imaging device 1 is installed in a car and used as an imaging device for automatic driving.
In this case, the imaging device 1 acquires an imaging result obtained by imaging the real world and an imaging result obtained by imaging a virtual space using the virtual imaging device 1, and realizes a function of generating data of the space in which the car travels. The imaging device 1 also realizes a function of simulating automatic driving of a car in the generated space and analyzing the simulation results.
To realize these functions, a library is prepared in the imaging device 1 to easily execute the desired process, and the necessary library is used as appropriate when executing various processes.

[composition]
FIG. 10 is a schematic diagram showing the overall configuration of the imaging device 1 in Specific Application Example 1.
The configuration shown in FIG. 10 differs from the overall configuration shown in FIG. 1 in that it includes an imaging unit 100 that takes a digital image (a still image or a moving image captured as a color image of the real world, etc.). .
The imaging unit 100 can be configured with an existing digital video camera that images a subject within an angle of view as a two-dimensional color image.
The imaging unit 100 includes a lens 111 and an imaging device 112.
The lens 111 focuses light rays from the subject within the field of view and forms an image of the subject on the image sensor of the imaging device 112.

The imaging device 112 includes an imaging sensor in which a plurality of imaging elements are arranged, and a processing circuit that processes output signals of the imaging sensor.
Note that the angles of view (shooting ranges) of the imaging unit 10 and the imaging unit 100 match (or overlap), and the imaging unit 10 outputs pixel data representing changes in the subject imaged by the imaging unit 100.
Further, since the imaging unit 100 is provided, the data receiving section 21 of the information processing unit 20 has the function of receiving pixel data output from the imaging unit 10 as well as the function of receiving pixel data output from the imaging unit 100. (pixel data group for each frame) at a predetermined period (for example, a 100 [ms] period). The data receiving section 21 outputs the pixel data received from the imaging unit 100 to the data storage control section 22, and stores it in the output data storage section 24 as data representing a frame.

[Functional configuration]
Next, the functional configuration of the information processing unit 20 will be explained.
FIG. 11 is a block diagram showing the functional configuration of the information processing unit 20.
FIG. 11 shows the functional configuration formed in the processing section 23 and storage section 20B of the information processing unit 20.
As shown in FIG. 11, the imaging device 1 in this application example includes a spatial data acquisition section 23a, a simulation processing section 23b, and an analysis processing section 23c in the processing section 23. Furthermore, in addition to the output data storage section 24, the storage section 20B includes a library storage section 25 and a 3D model database (3D model DB) 26.

The library storage unit 25 stores a plurality of libraries that compile various functions that are frequently used in the imaging device 1. In this embodiment, when a library is used, a necessary library is called as an API (Application Program Interface) from an application being executed, and a response based on the function of the called library is returned to the application.

For example, the library storage unit 25 stores a library that implements the following functions.
(Library 1) Library for switching between reading output data as an image or reading output data as pixel data (Library 2) Pixels whose polarity in pixel data is negative are blue, and pixels whose polarity is positive in pixel data are blue. As red, the library (Library 3) that displays the image represented by the output data; the Library (Library 4) that switches the storage format of the output data (two-dimensional memory or any of storage formats 1 to 4); the increase and decrease in brightness during imaging; A library that normalizes data according to the threshold value Th1 used for the determination. Note that the library 4 is used, for example, when output data from the imaging device 1 is analyzed later.
(Library 5) Library for switching whether or not to zoom in (or focus) on a specific area (such as an area with large changes) during shooting (Library 6) Adjusts the amount of output data to the reference range according to the movement of the subject. Library for changing whether or not to change the threshold Th1 so that

In an application program, by calling these libraries and writing higher-level commands, lower-level controls for realizing various functions in the imaging device 1 can be easily executed.
Note that, in addition to a library, the various functions that are frequently used in the imaging device 1 can be organized in the form of middleware or an application.

The 3D model DB 26 stores three-dimensional model data of a virtual space for simulating automatic driving. In this embodiment, the 3D model DB 26 stores three-dimensional model data representing a real space (a real space that constitutes a driving environment) in which automated driving is simulated. As a result, even if it is not possible to travel in real space, it is possible to obtain imaging results obtained by imaging the virtual space with the virtual imaging device 1 using the 3D model data stored in the 3D model DB 26. Become.
Note that the 3D model DB 26 includes not only 3D model data representing the real space in which automated driving simulation is performed (real space configuring the driving environment), but also 3D model data representing the virtual space configuring the imaginary driving environment. It is also possible to memorize. In this case, it is possible to simulate not only real space but also virtual driving in various driving environments.

The spatial data acquisition unit 23a executes a process for constructing spatial data representing a simulation environment for automatic driving (environment construction process).
That is, the spatial data acquisition unit 23a acquires pixel data (hereinafter appropriately referred to as "actual data") obtained by capturing an image of the real space within the driving range of the vehicle using the imaging unit 10 and the imaging unit 100 installed in the vehicle. The output data is stored in the output data storage section 24. In addition, the spatial data acquisition unit 23a uses the three-dimensional model data stored in the 3D model DB 26 to obtain pixel data (hereinafter referred to as ""virtualdata") is acquired asynchronously for each pixel and stored in the output data storage section 24. At this time, the spatial data acquisition unit 23a can call the above-mentioned library 7 and set the parameters (viewpoint position, aperture value, sensitivity, etc.) of the virtual camera that images the inside of the 3D space. Moreover, the spatial data acquisition unit 23a can call the above-mentioned library 3 and change the storage format of output data as appropriate.

Then, the spatial data acquisition unit 23a combines the acquired real data and virtual data to construct spatial data representing the automated driving simulation environment. At this time, the spatial data acquisition unit 23a calls the above-mentioned library 1 and reads out the data output by the imaging unit 10 as an image, thereby constructing spatial data representing the automated driving simulation environment. Further, the spatial data acquisition unit 23a stores data of the constructed space in a memory (not shown) or a storage area of the storage unit 20B.

Here, the actual data constituting the spatial data representing the automated driving simulation environment includes pixel data including data representing the state of pixels imaged by the imaging unit 10 and pixels representing scenery imaged by the imaging unit 100. The components are data. Since the imaging unit 10 outputs pixel data at a faster cycle than the imaging unit 100, pixels including pixel data representing the scenery imaged by the imaging unit 100 and data representing the state of the pixels imaged by the imaging unit 10. There are timings when all data is acquired together and timings when only pixel data including data representing the state of the pixel imaged by the imaging unit 10 is acquired. Therefore, when the spatial data acquisition unit 23a constructs spatial data representing a simulation environment of automatic driving, pixel data representing the scenery imaged by the imaging unit 100 and data representing the state of the pixels imaged by the imaging unit 10 are used. The shape of the object is interpolated using the pixel data including the data representing the state of the pixel imaged by the imaging unit 10 at times other than when the pixel data including the data is also acquired, and the spatial data representing the automated driving simulation environment is obtained. Convert to something with higher precision.

Specifically, spatial data representing the automated driving simulation environment can be configured according to the following procedure.
(Procedure 1) Imaged by the imaging unit 10, targeting pixel data representing the scenery imaged by the imaging unit 100 and pixel data including data representing the state of the pixels imaged by the imaging unit 10, which were also acquired. The shape of the object calculated based on the pixel data representing the landscape is associated with the shape calculated from the pixel data including data representing the state of the pixel imaged by the imaging unit 10.
(Step 2) At a timing when only pixel data including data representing the state of the pixel imaged by the imaging unit 10 is acquired, pixel data including data representing the state of the pixel imaged by the imaging unit 10 is retrieved from the past. Integration is performed over a predetermined period of time to temporally interpolate the shape of the object.
(Step 3) Based on the results of steps 1 and 2, pixel data representing the scenery imaged by the imaging unit 100 and pixel data including data representing the state of the pixels imaged by the imaging unit 10 are acquired. The data on the shape of the object at the timing when the image is taken by the imaging unit 10 and the data on the shape of the object at the time when only the pixel data including the data representing the state of the pixel imaged by the imaging unit 10 are acquired are combined to create a simulation environment for automatic driving. Construct data for the space to represent.

FIG. 12 is a schematic diagram showing the concept of interpolating the shape of an object using pixel data including data representing the state of pixels imaged by the imaging unit 10.
Note that FIG. 12 shows a state in which a car in which the imaging device 1 is installed moves in the order of time ta → time tb → time tc → time td and images a building adjacent to a road.
In FIG. 12, it is assumed that the imaging unit 100 performs imaging at time ta and time td according to its imaging cycle. At this time, although the building serving as the object is entirely included in the image taken at time ta, it is almost not captured in the image taken at time td.
On the other hand, since the imaging unit 10 can perform imaging (output pixel data) in a shorter imaging cycle than the imaging unit 10, the state of the pixels that imaged the building can be changed even between time ta and time td. Pixel data representing changes can be sequentially output.
Therefore, at time tb and time tc between time ta and time td, the pixel data output by the imaging unit 10 is used to synthesize and interpolate the data of the shape of the object (building) to create an autonomous driving simulation environment. It is possible to construct spatial data representing .

The simulation processing unit 23b refers to the space data representing the automated driving simulation environment constructed by the spatial data acquisition unit 23a, and performs a process (simulation) of simulating automated driving in this space according to the set automated driving algorithm. process). At this time, the simulation processing unit 23b calls the above-mentioned library 1 and reads out the data output by the imaging unit 10 as pixel data. Thereby, the simulation processing unit 23b performs a simulation of automatic driving while asynchronously acquiring data representing the state of the pixels. In addition, when simulating automatic driving, the simulation processing unit 23b calls the above-mentioned library 5 and determines whether or not to zoom in (or focus) on a specific area (such as an area with large changes) during imaging. Can be switched. Furthermore, when performing the automatic driving simulation, the simulation processing unit 23b calls the above-mentioned library 6 and switches whether or not to change the threshold Th1 so that the output data amount falls within the reference range according to the movement of the subject. I can do it. Furthermore, the simulation processing unit 23b can call the above-mentioned library 3 and change the storage format of output data as appropriate when performing a simulation of automatic driving.

Further, the simulation processing unit 23b stores data of driving history (simulation results) acquired by the automatic driving simulation in a memory (not shown) or a storage area of the storage unit 20B. By setting various automatic driving algorithms to be tested, etc., it is possible to evaluate the performance of the automatic driving algorithm based on the automatic driving simulation results.
Further, the simulation processing unit 23b can perform machine learning on the automatic driving algorithm so that the result obtained by analyzing the automatic driving simulation result by the analysis processing unit 23c becomes more appropriate.

The analysis processing unit 23c executes analysis processing (analysis processing) for performance evaluation of automatic driving algorithms, etc., based on the simulation result data acquired by the simulation processing unit 23b.
At this time, the analysis processing unit 23c calls the library 2 described above, and in the pixel data including data representing the state of the pixel imaged by the imaging unit 10, pixels with negative polarity are colored blue, and pixels with polarity are colored positive. An image represented by output data can be displayed with a certain pixel as red. In addition, the analysis processing unit 23c normalizes the data according to the threshold Th1 used to determine the increase and decrease in brightness during imaging by calling the library 4 described above, and Changes in the state of pixels can be graphed and displayed in chronological order.

[motion]
Next, the operation of the imaging device 1 of this application example will be explained.
FIG. 13 is a schematic diagram showing an example of a route on which automatic driving simulation is performed.
The route shown in Figure 13 starts from home (point A), passes through a residential area, turns right at point B, enters the national highway, then turns left at point C, enters a shopping district, and passes through a railroad crossing. and then reach the park (point D). However, the section from point C to point D is a section where actual driving is not possible due to construction or other reasons.
Hereinafter, assuming that spatial data representing a simulation environment for autonomous driving is constructed for the route illustrated in FIG. Explain the operation.

[Environment construction processing]
FIG. 14 is a flowchart illustrating the flow of environment construction processing executed by the imaging device 1.
The environment construction process is started by performing an operation that instructs the start of the environment construction process.
In step S41, the spatial data acquisition unit 23a acquires pixel data (actual data) captured by the imaging unit 10 and the imaging unit 100 installed in the vehicle to capture the real space in the driving range of the vehicle, and stores the acquired pixel data (actual data) in the output data storage unit 24. Remember. At this time, the spatial data acquisition unit 23a acquires pixel data representing the landscape imaged by the imaging unit 100 and pixel data including data representing the state of the pixels imaged by the imaging unit 10 at the same time. Pixel data is also acquired. Moreover, the spatial data acquisition unit 23a acquires only the pixel data output by the imaging unit 10 at a timing when only pixel data including data representing the state of the pixel imaged by the imaging unit 10 can be acquired.

In step S42, the spatial data acquisition unit 23a uses the 3D model data stored in the 3D model DB 26 to obtain pixel data (virtual data ) is acquired and stored in the output data storage section 24. At this time, the spatial data acquisition unit 23a calculates pixel data to be output from the imaging unit 10 at the same timing as the imaging cycle of the imaging unit 100, and calculates pixel data to be output from the virtual imaging unit 10 and the imaging unit 100. Obtain at the same time.
Note that in step S42, the spatial data acquisition unit 23a can call the above-mentioned library 7 and set parameters (viewpoint position, aperture value, sensitivity, etc.) of the virtual camera that images the inside of the 3D space. Moreover, in step S41 and step S42, the spatial data acquisition unit 23a can call the above-mentioned library 3 and change the storage format of the output data as appropriate.

In step S43, the spatial data acquisition unit 23a combines the acquired real data and virtual data to construct spatial data representing the automated driving simulation environment. At this time, the spatial data acquisition unit 23a calls the above-mentioned library 1 and reads out the data output by the imaging unit 10 as an image, thereby constructing spatial data representing the automated driving simulation environment. Note that the real data of the space representing the automated driving simulation environment includes pixel data including data representing the state of the pixel imaged by the imaging unit 10 and pixel data representing the scenery imaged by the imaging unit 100. It is. In the process of step S43, the spatial data acquisition unit 23a simultaneously acquires pixel data representing the scenery imaged by the imaging unit 100 and pixel data including data representing the state of the pixels imaged by the imaging unit 10. In addition to the timing, the shape of the object is interpolated using pixel data including data representing the state of the pixels captured by the imaging unit 10, and the spatial data representing the automated driving simulation environment is converted into more accurate data.

In step S44, the spatial data acquisition unit 23a stores data of the constructed space in a memory (not shown) or a storage area of the storage unit 20B.
After step S44, the environment construction process ends.

[Simulation processing]
FIG. 15 is a flowchart illustrating the flow of simulation processing executed by the imaging device 1.
The simulation process is started by performing an operation that instructs the start of the simulation process.
In step S51, the simulation processing unit 23b acquires spatial data representing a simulation environment of automatic driving.
In step S52, the simulation processing unit 23b starts movement (simulation) from the starting point within a space representing a simulation environment of automatic driving.

In step S53, the simulation processing unit 23b moves in a space representing the automated driving simulation environment according to the set automated driving algorithm, and records the response (driving history) of the vehicle as a simulation result.
In step S54, the simulation processing unit 23b determines whether the destination has been reached within the space representing the automated driving simulation environment.
If the destination has not been reached within the space representing the automated driving simulation environment, the determination in step S54 is NO, and the process proceeds to step S53.
On the other hand, if the destination is reached within the space representing the automated driving simulation environment, YES is determined in step S54, and the simulation process ends.

[Analysis processing]
FIG. 16 is a flowchart illustrating the flow of analysis processing executed by the imaging device 1.
The analysis process is started by performing an operation that instructs the start of the analysis process.
In step S61, the analysis processing unit 23c reads data on the driving history (simulation result) of the vehicle.

In step S62, the analysis processing unit 23c reads out images with the imaging device 1 as a viewpoint in chronological order and displays them as landscape images. By calling the library 2 described above, the analysis processing unit 23c sets pixels whose polarity is negative in blue and pixels whose polarity is positive in pixel data including data representing the state of pixels imaged by the imaging unit 10. The image represented by the output data can be displayed as red. Note that in step S62, different analysis results may be displayed depending on the content of the analysis, such as graphing changes in the states of pixels detected in the automatic driving simulation in time series.

In step S63, the analysis processing unit 23c determines whether an instruction has been given to end the analysis process.
If the end of the analysis process has not been instructed, the determination in step S63 is NO, and the process moves to step S62.
On the other hand, if an instruction to end the analysis process is given, a YES determination is made in step S63, and the analysis process ends.

As described above, the imaging device 1 in this application example acquires real data obtained by imaging the real space in the driving range of the car using the imaging unit 10 and the imaging unit 100 installed in the car, and also stores the data in the 3D model DB 26. Using the three-dimensional model data, virtual data obtained by capturing an image of a travel range set in a virtual space with a virtual imaging device 1 is obtained. Then, the imaging device 1 combines the acquired real data and virtual data to construct spatial data representing the automated driving simulation environment.
Therefore, at times other than when pixel data representing the scenery imaged by the imaging unit 100 and pixel data including data representing the state of the pixels imaged by the imaging unit 10 are also acquired, the image captured by the imaging unit 10 By interpolating the shape of an object using pixel data that includes data representing the state of pixels, it is possible to convert spatial data representing the automated driving simulation environment into more accurate data.
Furthermore, the imaging device 1 refers to the data of the constructed space and executes simulation processing in this space according to the set automatic driving algorithm.
Therefore, it is possible to set various automatic driving algorithms to be tested, etc., and obtain automatic driving simulation results.

The imaging device 1 also executes analysis processing for evaluating the performance of automatic driving algorithms, etc., based on data of simulation results.
This makes it possible to use the constructed space to analyze the results of automated driving based on the set automated driving algorithm, and to perform machine learning on the automated driving algorithm.
Furthermore, it will be possible to use automated driving algorithms optimized through machine learning for automated driving in actual cars.

In addition, the simulation result data can be visualized and analyzed, and reflected in the improvement of the performance of autonomous driving algorithms.
Furthermore, with the imaging device 1 of this application example, changes in the subject can be detected early using the pixel data output from the imaging unit 10, and by using the pixel data output from the imaging unit 10 and the imaging unit 100, It is possible to understand and construct the shape of the object more accurately. Furthermore, the imaging device 1 can speed up situational understanding and response during automatic driving.

Further, the imaging device 1 of this application example is provided with a library that compiles various frequently used functions.
As a result, in an application, by calling the library and writing higher-level commands, lower-level control for realizing various functions in the imaging device 1 can be easily executed.

Note that the imaging device 1 is capable of acquiring pixel data output by the imaging unit 10 and detecting changes in a subject when a movable device, not only a car, moves autonomously or by manual operation. For example, by using the imaging device 1, it is possible to acquire pixel data output by the imaging unit 10 and detect changes in the subject during movement of a walking robot, a flying object such as a drone, a ship, a submarine, etc. can. Furthermore, the imaging device 1 can be used for simulation or actual operation of autonomous control of these movable devices, including automatic driving of automobiles.
Furthermore, the object of analysis in the analysis process is not limited to the simulation results of automatic driving, but can also be data on the driving history of actual automatic driving.

Furthermore, the various processes described as being executed by the imaging device 1 in the above-described embodiments may be performed by an information processing device other than the imaging device 1.
Furthermore, part of the various processes described as being executed by the imaging device 1 in the above-described embodiments may be executed by an information processing device other than the imaging device 1, and the imaging device 1 may acquire the execution results. .
Further, in the above-described embodiment, various data used by the imaging device 1 (such as three-dimensional model data in real space) are stored in an information processing device other than the imaging device 1, and necessary data is downloaded as appropriate for imaging. It may also be stored in the device 1.

The imaging device 1 configured as described above includes an imaging unit 10 and an information processing unit 20. The information processing unit 20 includes a data receiving section 21, a data storage control section 22, a processing section 23, and a storage section 20B.
The imaging unit 10 transmits data representing a change in the state of a pixel corresponding to an image sensor based on a change in the output of an image sensor included in an image sensor that images a subject.
The data receiving unit 21 asynchronously receives data representing a change in the state of a pixel transmitted by the imaging unit 10 for each pixel.
The storage unit 20B stores data representing a change in the state of a pixel, time information regarding the data representing a change in the state of the pixel, and an address of the pixel in a preset storage format.
The data storage control unit 22 causes the storage unit 20B to store the data representing the change in the state of the pixel received by the data reception unit 21 as data in a preset storage format.
As a result, as soon as a change is detected for each pixel, data representing the state of the pixel is output asynchronously. Compared to other techniques, it is possible to detect changes in the subject in terms of angle of view at high speed and with a small amount of data.
Furthermore, the imaging device 1 can read out pixel data stored in a predetermined storage format and use it for various applications.
Therefore, according to the imaging device 1, it is possible to realize more appropriate functions in information processing technology using digital images.

The storage unit 20B also includes an increase frame memory 24a that stores data when the data representing a change in the pixel state indicates an increase, and an increase frame memory 24a that stores data representing a change in the pixel state when the data represents a decrease. A frame memory 24b for reduction is provided to store the data in case of a reduction.
Thereby, pixel data can be saved in a format that makes it easy to refer to necessary data. In addition, for example, of the pixel data (time information, pixel addresses, data representing the state of each pixel, etc.), only the time information needs to be stored at a predetermined address, so compared to the case where all of these data are stored, storage The amount of data to be processed can be reduced. Further, since the sizes of the increase frame memory 24a and the decrease frame memory 24b are determined in advance, it is possible to easily estimate the storage capacity required to store pixel data.

Further, based on the amount of data transmitted from the imaging unit 10, a threshold value for determining a change in the output of the imaging element is changed.
Thereby, the amount of data transmitted from the imaging unit 10 can be converged within the set range.

The data storage control unit 22 associates data representing a change in the state of a pixel corresponding to the image sensor with a threshold value used for determining a change in the output of the image sensor with respect to the data, and stores the data in the storage unit 20B. let
This makes it possible to calculate the amount of change in the state of the pixel by referring to the threshold value used when determining the change in the output of the image sensor.

Further, the information processing unit 20 controls focus or zoom in the imaging unit 10 in accordance with the distribution of data representing changes in the state of pixels corresponding to the imaging device transmitted from the imaging unit 10.
Thereby, it is possible to detect the state of the subject by focusing on an area where there is a large change within the angle of view.

Further, the image sensor includes a color filter in the image sensor.
Data representing a change in the state of a pixel corresponding to an image sensor represents a state of a pixel corresponding to a wavelength of light transmitted through a color filter provided in the image sensor.
This makes it possible to provide a more convenient technology in applications where processing that reflects the wavelength of light is effective.

Further, the imaging device 1 includes an imaging unit 100.
The imaging unit 100 acquires data of an image representing a landscape.
The processing unit 23 associates the data representing a change in the state of a pixel corresponding to the image sensor transmitted from the imaging unit 10 with the data of the image acquired by the imaging unit 100, and executes processing by the application.
As a result, in addition to constructing the shape of the object from the data of the image acquired by the imaging unit 100, the shape of the object is interpolated using pixel data including data representing the state of the pixels imaged by the imaging unit 10, and the shape of the object is further improved. It is possible to construct highly accurate spatial data.

The processing unit 23 switches between reading data representing a change in state of a pixel corresponding to the image sensor stored in the storage unit 20B as an image or as pixel data.
This makes it possible to read data representing changes in pixel states in an appropriate format depending on the processing content.

Further, the imaging device 1 as an information processing device includes a spatial data acquisition section 23a in the information processing unit 20.
The spatial data acquisition unit 23a asynchronously acquires data representing a change in the state of a pixel corresponding to a virtual image sensor when a subject is virtually imaged in a three-dimensional virtual space, for each pixel.
As a result, even if it is not possible to actually image the real space, it is possible to obtain data that virtually detects the state of the subject.

The imaging device 1 also includes an imaging unit 10, a data receiving section 21, a data storage control section 22, a processing section 23, and a storage section 20B.
The imaging unit 10 transmits data representing a change in the state of a pixel corresponding to an image sensor based on a change in the output of an image sensor included in an image sensor that images a subject.
The data receiving unit 21 asynchronously receives data representing a change in the state of a pixel transmitted by the imaging unit 10 for each pixel.
The storage unit 20B stores data representing a change in the state of a pixel acquired by the spatial data acquisition unit 23a, data representing a change in the state of a pixel received by the data receiving unit 21, and data representing a change in the state of the pixel. Time information and pixel addresses related to the image are stored in a preset storage format.
The data storage control unit 22 stores the data representing the change in the state of the pixel acquired by the spatial data acquisition unit 23a and the data representing the change in the state of the pixel received by the data receiving unit 21 in a preset storage format. The data is stored in the storage unit 20B as data.
As a result, as soon as a change is detected for each pixel, data representing the state of the pixel is output asynchronously. Compared to other techniques, it is possible to detect changes in the subject in terms of angle of view at high speed and with a small amount of data.
Furthermore, the imaging device 1 can read out pixel data stored in a predetermined storage format and use it for various applications.
Moreover, more appropriate processing can be performed by mutually interpolating data that virtually detects the state of the subject and data that represents changes in the state of pixels detected by imaging.
Therefore, according to the imaging device 1, it is possible to realize more appropriate functions in information processing technology using digital images.

The spatial data acquisition unit 23a acquires spatial data representing a three-dimensional space based on at least one of the acquired data representing a change in the state of a pixel and the data representing a change in the state of a pixel received by the data receiving unit 21. To construct.
This makes it possible to interpolate data that virtually detects the state of the subject and data representing changes in the pixel state detected through imaging, thereby making the data in three-dimensional space more accurate. can.

The imaging device 1 also includes a simulation processing section 23b.
The simulation processing unit 23b detects a change in the subject visually recognized from the movable device based on the spatial data constructed by the spatial data acquisition unit 23a, and performs a simulation for controlling the movement.
This makes it possible to set various algorithms for controlling the movable device and obtain simulation results.

The imaging device 1 also includes an analysis processing section 23c.
The analysis processing section 23c performs learning regarding an algorithm for controlling movement based on the results of the simulation performed by the simulation processing section 23b.
As a result, a more appropriate algorithm can be generated by repeatedly performing simulations using the constructed spatial data.

The analysis processing unit 23c converts the data representing the change in the state of the pixel acquired by the spatial data acquisition unit 23a and the data representing the change in the state of the pixel received by the data receiving unit 21 into an output of the image sensor for the data. The pixel changes are displayed after normalization based on the threshold used to determine the change.
Thereby, changes in the state of the acquired pixels can be displayed in a form that can be objectively analyzed.

It should be noted that the present invention is not limited to the above-described embodiments, and any modifications, improvements, etc. that can achieve the purpose of the present invention are included in the present invention.
That is, the present invention can be applied to various electronic devices having an imaging function or various electronic devices having an information processing function.

The series of processes described above can be executed by hardware or software.
In other words, the configurations in FIGS. 1 and 10 and the functional configuration in FIG. 11 are merely examples and are not particularly limited. That is, it is sufficient that the imaging device 1 has a function that can execute the series of processes described above as a whole, and there is no particular limitation on what kind of functional block is used to realize this function.
Further, one functional block may be configured by a single piece of hardware, a single piece of software, or a combination thereof.
The functional configuration in this embodiment is realized by a processor that executes arithmetic processing, and processors that can be used in this embodiment include various processing units such as a single processor, a multiprocessor, and a multicore processor. In addition, it includes combinations of these various processing devices and processing circuits such as ASICs (Application Specific Integrated Circuits) and FPGAs (Field-Programmable Gate Arrays).

When a series of processes is executed by software, a program constituting the software is installed on a computer or the like from a network or a recording medium.
The computer may be a computer built into dedicated hardware. Further, the computer may be a computer that can execute various functions by installing various programs, such as a general-purpose personal computer.

A recording medium containing such a program is not only configured as a removable medium distributed separately from the device itself in order to provide the program to the user, but also provided to the user in a state that is pre-installed in the device body. Consists of recording media, etc. The removable medium is, for example, a magnetic disk (including a floppy disk), an optical disk, a magneto-optical disk, or the like. Optical disks include, for example, CD-ROMs (Compact Disk-Read Only Memory), DVDs (Digital Versatile Disks), Blu-ray (registered trademark) Discs, and the like. The magneto-optical disk is composed of an MD (Mini-Disk) or the like. Further, the recording medium provided to the user in a state that is pre-installed in the main body of the apparatus includes, for example, a ROM (Read Only Memory) in which a program is recorded, a semiconductor memory included in the storage section 20B, and the like.

Although several embodiments of the present invention have been described above, these embodiments are merely illustrative and do not limit the technical scope of the present invention. The present invention can take various other embodiments, and various changes such as omissions and substitutions can be made without departing from the gist of the present invention. These embodiments and their modifications are included within the scope and gist of the invention described in this specification and the like, as well as within the scope of the invention described in the claims and its equivalents.

1 Imaging device, 10,100 Imaging unit, 11,111 Lens, 12,112 Imaging device, 20 Information processing unit, 20A Processor, 20B Storage unit, 21 Data receiving unit, 22 Data storage control unit, 23 Processing unit, 23a Space Data acquisition section, 23b Simulation processing section, 23c Analysis processing section, 24 Output data storage section, 24a Frame memory for increase, 24b Frame memory for decrease, 25 Library storage section, 26 3D model database, 30 Sensor unit

Claims (8)

In a three-dimensional virtual space, the present invention is characterized by comprising a virtual data acquisition means that asynchronously acquires data representing a change in the state of a pixel corresponding to a virtual image sensor when a subject is virtually imaged for each pixel in a three-dimensional virtual space. Information processing equipment. Imaging means for transmitting data representing a change in the state of a pixel corresponding to the image sensor based on a change in the output of an image sensor included in an image sensor that images a subject;
data receiving means for asynchronously receiving data representing a change in the state of the pixel transmitted by the imaging means for each pixel;
data representing a change in the state of the pixel acquired by the virtual data acquisition means, data representing a change in the state of the pixel received by the data receiving means, time information regarding the data representing a change in the state of the pixel, and a storage means in which the pixel address is stored in a preset storage format;
Storage for storing data representing a change in the state of the pixel acquired by the virtual data acquisition means and data representing a change in the state of the pixel received by the data receiving means as data in the storage format in the storage means. control means;
The information processing device according to claim 1, further comprising: Spatial data representing a three-dimensional space based on at least one of data representing a change in the state of the pixel acquired by the virtual data acquisition means and data representing a change in the state of the pixel received by the data receiving means. 3. The information processing apparatus according to claim 2, further comprising spatial data construction means for constructing spatial data. A claim characterized by comprising a simulation means for detecting a change in a subject visible from a movable device and performing a simulation for controlling movement based on the spatial data constructed by the spatial data construction means. The information processing device according to item 3. 5. The information processing apparatus according to claim 4, further comprising a learning means for learning an algorithm for controlling movement based on a result of the simulation by the simulation means. Determining a change in the output of the image sensor with respect to data representing a change in the state of the pixel acquired by the virtual data acquisition means and data representing a change in the state of the pixel received by the data receiving means. The information processing apparatus according to any one of claims 1 to 5, further comprising an analysis unit that normalizes the pixel based on a threshold value used for the change and displays the change in the pixel. A virtual data acquisition step of asynchronously acquiring data representing a change in the state of a pixel corresponding to a virtual image sensor when a subject is virtually imaged in a three-dimensional virtual space for each pixel. information processing method. to the computer,
In a three-dimensional virtual space, we aim to realize a virtual data acquisition function that asynchronously acquires data representing changes in the state of pixels corresponding to a virtual image sensor when a subject is virtually imaged for each pixel. Featured programs.

Images