JP2020198535A - Imaging device, information processing device, imaging method, information processing method, and program - Google Patents

Abstract

To enable more appropriate function to be realized in an information processing technique using a digital image.SOLUTION: In an imaging unit 1, an imaging unit 10 transmits data representing a change in the state of a pixel corresponding to the imaging element on the basis of a change in the output of the imaging element provided in an imaging sensor that images a subject. A data receiving unit 21 asynchronously receives the data representing the change in the state of the pixels transmitted by the imaging unit 10 for each pixel. A storage unit 20B stores the data representing the change in the state of the pixel, time information regarding the data representing the change in the state of the pixel, and an address of the pixel in a preset storage format. A data storage control unit 22 stores the data representing the change in the state of the pixels received by the data receiving unit 21 in the storage unit 20B as data in a storage format.SELECTED DRAWING: Figure 1

Description

The present invention relates to an image pickup apparatus, an information processing apparatus, an imaging method, an information processing method and a program.

In recent years, digital images have come to be used in a wide range of fields such as automatic driving of automobiles and AI (Artificial Intelligence).
As the digital image, a still image or a moving image obtained by capturing the real world as a color image is generally used.
Patent Document 1 discloses a technique for recognizing an object such as a vehicle using a digital image.

JP-A-2018-0976766

However, when constructing an application using a digital image, a conventionally used digital image (a still image or a moving image obtained by capturing a real world as a color image, etc.) may not be suitable for the desired function.
For example, when trying to detect an obstacle in automatic driving of an automobile or the like, an object closer to the camera moves faster, so that the captured digital image is more likely to be blurred. Therefore, an object that is closer to the camera and has a higher degree of attention becomes more difficult to detect accurately, and an obstacle may not be detected properly.
Further, since the conventionally used digital image is output after a process of generating a one-frame image based on the data output from the image sensor, sufficient processing is performed in an application or the like that requires high-speed processing. It can be difficult to ensure speed.
That is, there are cases where an appropriate function cannot be realized by the conventional information processing technology using a digital image.

An object of the present invention is to make it possible to realize more appropriate functions in an information processing technique using a digital image.

In order to solve the above problems, the imaging device according to the embodiment of the present invention is
An imaging means that transmits data representing a change in the state of pixels corresponding to the image sensor based on a change in the output of the image sensor provided in the image sensor that images the subject.
A data receiving means that asynchronously receives data representing a change in the state of the pixel transmitted by the imaging means for each pixel.
A storage means in which data representing a change in the state of the pixel, time information relating to the data representing the change in the state of the pixel, and the address of the pixel are stored in a preset storage format.
A storage control means for storing data representing a change in the state of the pixel received by the data receiving means in the storage means as data in the storage format.
It is characterized by having.

According to the present invention, it is possible to realize more appropriate functions in the information processing technology using digital images.

It is a schematic diagram which shows the structure of the whole image pickup apparatus 1 which concerns on this embodiment. It is a schematic diagram which shows the structure of the storage area of the output data storage unit 24. It is a schematic diagram which shows the reference range of the received data amount (the output data amount of the imaging device 12) per unit time with respect to the movement amount of the imaging unit 10. It is a flowchart explaining the flow of the data reception process executed by the information processing unit 20. It is a flowchart explaining the flow of the threshold value adjustment process executed by the information processing unit 20. It is a flowchart explaining the flow of the data storage process executed by the information processing unit 20. It is a flowchart explaining the flow of application execution processing executed by information processing unit 20. It is a schematic diagram which shows the structural example of the image pickup sensor in which the filter which transmits light of a specific wavelength is arranged in the image pickup element. It is a schematic diagram which shows an example of the state in which the pixel data of a storage format 1 is stored in the storage area which has a one-dimensional continuous address. It is a schematic diagram which shows the structure of the whole image pickup apparatus 1 in the specific application example 1. FIG. It is a block diagram which shows the functional structure of an information processing unit 20. It is a schematic diagram which shows the concept of interpolating the shape of an object by the pixel data including the data which shows the state of the pixel imaged by the image pickup unit 10. It is a schematic diagram which shows an example of the path where the simulation of automatic driving is performed. It is a flowchart explaining the flow of the environment construction process executed by the image pickup apparatus 1. It is a flowchart explaining the flow of the simulation process executed by the image pickup apparatus 1. It is a flowchart explaining the flow of the analysis process executed by the image pickup apparatus 1.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
[First Embodiment]
[Constitution]
FIG. 1 is a schematic view showing the configuration of the entire image pickup apparatus 1 according to the present embodiment.
The image pickup device 1 detects a state for each pixel (here, it is a change in the brightness value) in the image pickup unit 10 having an image pickup sensor in which the image pickup elements are arranged, and the information processing unit 20 detects the detection result data ( Hereinafter, it is appropriately referred to as “pixel data”) and stores it in a storage area in a predetermined storage format. Further, the information processing unit 20 reads out the detection result data (pixel data) of the imaging unit 10 stored in the storage area and uses it in various applications, and also receives imaging parameters (pixel data) in the imaging unit 10 in response to a request of the application or the like. Change the threshold for detecting the state of each pixel, etc.).
Therefore, the image pickup apparatus 1 can detect a change in the subject at the angle of view for each pixel and output the changed pixel as pixel data indicating that the change has occurred.
This makes it possible for the imaging device 1 to detect changes in the subject at an angle of view at high speed with a small amount of data.
Further, the image pickup apparatus 1 can read pixel data stored in a predetermined storage format and use it for various applications.
Therefore, according to the image pickup apparatus 1, it is possible to realize more appropriate functions in the information processing technology using digital images.
Hereinafter, the configuration of the image pickup apparatus 1 will be specifically described.

As shown in FIG. 1, the image pickup apparatus 1 includes an image pickup unit 10, an information processing unit 20, and a sensor unit 30, and the information processing unit 20 is a wired or wireless communication network with the image pickup unit 10 and the sensor unit 30. Connected by. It is also possible to integrally configure two or all of the image pickup unit 10, the information processing unit 20, and the sensor unit 30.

The image pickup unit 10 includes a lens 11 and an image pickup device 12.
The lens 11 focuses the light rays from the subject within the angle of view and forms a subject image on the image sensor of the image pickup device 12.
The image pickup device 12 includes an image pickup sensor in which a plurality of image pickup elements are arranged, and a processing circuit for processing an output signal of the image pickup sensor. The image pickup device 12 outputs a signal representing the state of each part of the subject image imaged by the lens 11 for each pixel corresponding to the image pickup element. That is, pixel data including data representing the state of each pixel is output from the imaging device 12 by the signal representing the state of each part of the subject image. In the present embodiment, a change in the brightness value of each pixel is used as an example of a parameter representing the state of each pixel. That is, when the amount of change in the electric charge accumulated for each image sensor reaches a set threshold value, the image pickup device 12 sets "+1" or decrease as data (polarity) representing the state of each pixel. The representative "-1" is output. Specifically, the imaging device 12 outputs data (+1) indicating an increase when the increase in the brightness detected for each pixel becomes the set threshold Th1 or more, and the brightness detected for each pixel. When the decrease of is equal to or higher than the set threshold value Th1, the data (-1) indicating the decrease is output. The threshold value Th1 used for determining the increase and decrease of the brightness is set by the information processing unit 20.
With the above configuration, the imaging device 12 asynchronously outputs pixel data including data (+1 or -1) representing the pixel state for each pixel.

The information processing unit 20 is composed of an information processing device such as a PC (Personal Computer), a microcomputer, or a smartphone, and includes a processor 20A and a storage unit 20B.
The processor 20A is composed of an arithmetic processing unit such as a CPU (Central Processing Unit). When the processor 20A executes a program for information processing, the processor 20A is provided with a data receiving unit 21, a data storage control unit 22, and a processing unit 23 as functional configurations.

The storage unit 20B is composed of a storage device such as a hard disk or a semiconductor memory, and an output data storage unit 24 is formed in a part of the storage area.
The output data storage unit 24 stores the pixel data output from the image pickup unit 10.
FIG. 2 is a schematic diagram showing the configuration of the storage area of the output data storage unit 24.
As shown in FIG. 2, the output data storage unit 24 is formed with a two-dimensional memory (frame memory) having an address corresponding to the address of the image captured by the image pickup unit 10. Specifically, in the output data storage unit 24, the frame memory (frame memory 24a for increasing) used when the data representing the state of each pixel is increasing, and the data representing the state of each pixel are reduced. The frame memory (frame memory for reduction 24b) used in this case is formed independently.

As will be described later, the pixel data output from the image pickup unit 10 is stored in the increasing frame memory 24a and the decreasing frame memory 24b by the data storage control unit 22.
The stored contents of the output data storage unit 24 (frame memory 24a for increasing and frame memory 24b for decreasing) are reset every predetermined time (for example, every 10 [ms]). However, since the past pixel data can be referred to even after the reset, the increasing frame memory 24a and the decreasing frame memory 24b for backup may be formed in different storage areas.

The data receiving unit 21 asynchronously receives the pixel data output from the imaging device 12 of the imaging unit 10, and temporarily stores the received pixel data in a buffer for receiving data. Further, the data receiving unit 21 stores pixel data stored in the buffer for receiving data at a predetermined timing (for example, every 1 [ms]) that defines the internal operation of the information processing unit 20. Output to 22. Specifically, when the data receiving unit 21 receives the pixel data from the imaging device 12, the data receiving unit 21 receives the received time information, the pixel address, and the data (+1 or -1) representing the state of each pixel as a buffer for the received data. Remember in. Then, the data receiving unit 21 associates the time information stored in the buffer for the received data, the pixel address, and the data (+1 or -1) representing the state of each pixel with the data at a predetermined timing. Output to the storage control unit 22.

Further, in the present embodiment, the data receiving unit 21 changes the threshold value Th1 used for determining the increase and decrease of the brightness in the imaging device 12 based on the received data amount per unit time and the movement amount of the imaging unit 10. ..
FIG. 3 is a schematic diagram showing a reference range of the amount of received data (the amount of output data of the imaging device 12) per unit time with respect to the amount of movement of the imaging unit 10.
As shown in FIG. 3, the larger the movement amount (rotational movement and translational movement) of the imaging unit 10, the larger the reference value (broken line in FIG. 3) of the received data amount per unit time. Further, the reference range of the received data amount per unit time is set within a certain amount of data from the reference value.
According to FIG. 3, an appropriate range (reference range) as the amount of received data per unit time is determined according to the amount of movement of the imaging unit 10, and when the range deviates from the reference range, the brightness of the imaging device 12 increases. It is considered that the threshold value Th1 used for determining the decrease is an inappropriate value (excess or under).

That is, when the amount of received data per unit time exceeds the reference range, it is considered appropriate to increase the threshold Th1 used for determining the increase and decrease of the brightness in the imaging device 12. Therefore, the data receiving unit 21 outputs an instruction to increase the threshold value Th1 to the imaging device 12. Further, when the amount of received data per unit time is less than the reference range, it is considered appropriate to reduce the threshold Th1 used for determining the increase and decrease of the brightness in the imaging device 12. Therefore, the data receiving unit 21 outputs an instruction to reduce the threshold value Th1 to the imaging device 12.
The appropriate range (reference range) for the amount of received data per unit time is calibrated by imaging various subjects in a state where the image pickup unit 10 does not move, a state where the movement is small, a state where the movement is large, and the like. Can be performed to set a specific numerical value.

Further, even when the data receiving unit 21 inputs a change instruction of the threshold value Th1 used for determining the increase and decrease of the brightness in the image pickup device 12 from the processing unit 23, the data receiving unit 21 sets the threshold value Th1 according to the content of the change instruction. An instruction to increase or decrease is output to the imaging device 12.
The data receiving unit 21 may focus on or zoom in on a specific area within the angle of view based on the distribution of received data (address bias, etc.) output from the imaging device 12. .. For example, when an object is detected by a surveillance camera, it is considered that the data representing the pixel state is concentrated and output in the area where the intruder is captured, so that the address of the pixel data received by the data receiving unit 21 is When the data is unevenly distributed, the data receiving unit 21 (or another functional block of the information processing unit 20) can output an instruction to zoom in to the vicinity of the address to the imaging unit 10.

The data storage control unit 22 stores the pixel data (time information, pixel address, data representing the state of each pixel (+1 or -1)) input from the data reception unit 21 in the output data storage unit 24.
In the pixel data input from the data receiving unit 21, when the data representing the pixel state is "+1" indicating the increase, the data storage control unit 22 has the pixel address of the pixel data in the frame memory 24a for increasing. The time information is stored in the address corresponding to. Further, when the data representing the pixel state is "-1" indicating the decrease in the pixel data input from the data receiving unit 21, the data storage control unit 22 sets the pixel data in the frame memory 24b for reduction. Time information is stored in the address corresponding to the pixel address. In the present embodiment, when the pixel data is output a plurality of times in the same pixel, the time information of the pixel is overwritten in the increasing frame memory 24a or the decreasing frame memory 24b.

As a result, the time information is sequentially written to the address of the pixel in which the increase amount of the brightness value of the pixel is the threshold Th1 or more in the frame memory 24a for increase, and the frame memory 24b for decrease is the pixel. The time information is sequentially written to the addresses of the pixels in which the amount of decrease in the luminance value is equal to or greater than the threshold value Th1.
The data storage control unit 22 may acquire the value of the threshold Th1 used for the determination when the pixel data is output, and store the value of the threshold Th1 in the pixel address in addition to the time information. Good.

The processing unit 23 reads the pixel data stored in the output data storage unit 24 and executes various processes according to the application. The processing unit 23 can execute various applications using the pixel data output by the imaging unit 10, for example, an application for environment recognition in automatic operation and an object detection in a surveillance camera. Applications, as well as machine learning and data processing related to them, can be executed.

Further, the processing unit 23 can appropriately output a change instruction of the threshold value Th1 used for determining the increase and decrease of the brightness in the imaging device 12 to the data receiving unit 21 in response to a request of the application being executed.
The processing unit 23 can read the pixel data stored in the output data storage unit 24 and perform various processing for use.
For example, the brightness of the imaging device 12 is increased or decreased by the number of changes in the brightness value of each pixel within a predetermined time for the entire pixel data stored in the frame memory 24a for increasing and the frame memory 24b for decreasing. By integrating the threshold value Th1 used in the determination of the above with a code, the actual amount of change in the luminance value in each pixel can be calculated.

Further, for the pixel data stored in each of the increasing frame memory 24a and the decreasing frame memory 24b, the brightness of the imaging device 12 is increased or decreased by the number of changes in the brightness value of each pixel within a predetermined time. By integrating the threshold value Th1 used for the determination of, the cumulative increase amount and the cumulative decrease amount of the luminance value in each pixel can be calculated respectively.
By constructing the frame data from the pixel data calculated in this way, it is possible to generate an image (here, an image showing the change in brightness) in which the change in the state of the subject is mapped.
By using the image generated in this way alone or in combination with a color image, it is possible to improve the performance of environment recognition, object detection, and the like.

When executing the application, the processing unit 23 can perform processing by either single thread or multi-thread. Threads mean different processes that run in parallel.
For example, when the processing unit 23 is performing processing by a single thread, the processing unit 23 acquires pixel data from the imaging unit 10 and obtains pixel data from the output data storage unit 24 in the thread (in the same process). Reading and the like can be performed at the same time.
Further, when the processing unit 23 is performing processing by multithreading, the processing unit 23 transfers pixel data between different threads (between different processes) via a memory or the like from the imaging unit 10 by the first thread. It is possible to acquire the pixel data of the above and read the pixel data from the output data storage unit 24 by the second thread.

The sensor unit 30 includes various sensors for detecting movement, such as an acceleration sensor or a gyro sensor. The sensor unit 30 can be attached to the image pickup unit 10 or to the information processing unit 20 integrated with the image pickup unit 10, and is installed to detect the movement of the image pickup unit 10. The detection result of the sensor unit 30 is output to the information processing unit 20.

[motion]
Next, the operation of the image pickup apparatus 1 will be described.
[Data reception processing]
FIG. 4 is a flowchart illustrating a flow of data reception processing executed by the information processing unit 20.
The data reception process is started at the start of the imaging device 1 and is repeatedly executed.
In step S1, the data receiving unit 21 determines whether or not pixel data has been received from the imaging device 12.
When the pixel data is not received from the image pickup device 12, NO is determined in step S1, and the process of step S1 is repeated.
On the other hand, when the pixel data is received from the image pickup device 12, YES is determined in step S1, and the process proceeds to step S2.

In step S2, the data receiving unit 21 stores the received pixel data in the received data buffer.
In step S3, the data receiving unit 21 determines whether or not the output timing is set to the data storage control unit 22.
If it is not the output timing to the data storage control unit 22, NO is determined in step S3, and the process proceeds to step S1.
On the other hand, when the output timing is set to the data storage control unit 22, YES is determined in step S3, and the process proceeds to step S4.

In step S4, the data receiving unit 21 outputs the pixel data stored in the buffer for the received data to the data storage control unit 22.
After step S4, the data reception process is repeated.

[Threshold adjustment processing]
FIG. 5 is a flowchart illustrating a flow of the threshold value adjustment process executed by the information processing unit 20.
The threshold adjustment process is started at the start of the imaging device 1 and is repeatedly executed.
In step S11, the data receiving unit 21 determines whether or not the received data amount per unit time is within the reference range with respect to the movement amount of the imaging unit 10 represented by the detection result of the sensor unit 30.
If the amount of received data per unit time is not within the reference range with respect to the amount of movement of the imaging unit 10 represented by the detection result of the sensor unit 30, NO is determined in step S11, and the process proceeds to step S13.
On the other hand, if the amount of received data per unit time is within the reference range with respect to the amount of movement of the imaging unit 10 represented by the detection result of the sensor unit 30, YES is determined in step S11, and the process proceeds to step S12. To do.

In step S12, the data receiving unit 21 determines whether or not a change instruction of the threshold Th1 used for determining the increase or decrease of the brightness in the imaging device 12 is input from the processing unit 23.
If no instruction for changing the threshold value Th1 used for determining the increase or decrease of the brightness in the image pickup device 12 is input from the processing unit 23, NO is determined in step S12, and the process proceeds to step S11.
On the other hand, when the processing unit 23 inputs an instruction to change the threshold value Th1 used for determining the increase and decrease of the brightness in the imaging device 12, it is determined as YES in step S12, and the process proceeds to step S13.

In step S13, the data receiving unit 21 outputs an instruction to increase or decrease the threshold Th1 to the imaging device 12.
After step S13, the threshold adjustment process is repeated.

[Data storage process]
FIG. 6 is a flowchart illustrating a flow of data storage processing executed by the information processing unit 20.
The data storage process is started at the start of the imaging device 1 and is repeatedly executed.
In step S21, the data storage control unit 22 determines whether or not pixel data has been input from the data reception unit 21.
When the pixel data is not input from the data receiving unit 21, NO is determined in step S21, and the process of step S21 is repeated.
On the other hand, when the pixel data is input from the data receiving unit 21, YES is determined in step S21, and the process proceeds to step S22.

In step S22, the data storage control unit 22 determines whether or not the data representing the pixel state among the input pixel data represents an increase (+1).
If the input pixel data that represents the pixel state does not represent an increase (that is, represents a decrease (-1)), it is determined as NO in step S22, and the process proceeds to step S24. ..
On the other hand, when the data representing the pixel state among the input pixel data represents an increase, YES is determined in step S22, and the process proceeds to step S23.

In step S23, the data storage control unit 22 stores the time information of the input pixel data in the frame memory 24a for increasing time.
In step S24, the data storage control unit 22 stores the time information of the input pixel data in the frame memory 24b for reduction.
After step S23 and step S24, the data storage process is repeated.

[Application execution process]
FIG. 7 is a flowchart illustrating a flow of application execution processing executed by the information processing unit 20.
The application execution process is started by performing an operation instructing the start of the application execution process.
In step S31, the processing unit 23 reads the designated application program. The application program is stored in a storage medium such as a storage unit 20B or a removable medium (not shown).

In step S32, the processing unit 23 executes a process based on the designated application program (hereinafter, referred to as “designated application process”).
After step S32, the application execution process ends.

As described above, the image pickup apparatus 1 according to the present embodiment asynchronously outputs pixel data including data representing a pixel state by the image pickup unit 10 for each pixel, and the information processing unit 20 outputs the pixel data. It is stored in the output data storage unit 24.
Then, the information processing unit 20 executes various applications while detecting a change in the subject at the angle of view by reading and using the pixel data stored in the output data storage unit 24 in the designated application processing.
As a result, as soon as a change is detected for each pixel, data representing the state of the pixel is output asynchronously. Therefore, a conventional digital image for each frame captured is output and used for applications such as object detection. Compared to technology, it is possible to detect changes in the subject at an angle of view at high speed with a small amount of data.
Further, the image pickup apparatus 1 can read pixel data stored in a predetermined storage format and use it for various applications.
Therefore, according to the image pickup apparatus 1, it is possible to realize more appropriate functions in the information processing technology using digital images.

Further, the output data storage unit 24 is formed with a two-dimensional memory (frame memory 24a for increase and frame memory 24b for decrease) having an address corresponding to the address of the image captured by the image pickup unit 10. Then, the information processing unit 20 of the pixel in the increasing frame memory 24a or the decreasing frame memory 24b, depending on whether the data representing the pixel state represents an increase or a decrease. The time information in the pixel data is stored in the address corresponding to the address.
As a result, the pixel data can be saved in a form in which the necessary data can be easily referred to. Further, of the pixel data (time information, pixel address, data representing the state of each pixel (+1 or -1)), only the time information needs to be stored at a predetermined address, so that all of them need to be stored. In comparison, the amount of data to be saved can be reduced. Further, since the sizes of the frame memory 24a for increasing and the frame memory 24b for decreasing are determined in advance, it is possible to easily estimate the storage capacity required for storing the pixel data.

[Modification 1]
In the above-described embodiment, the image sensor included in the image pickup device 12 can be adopted as long as a plurality of image pickup elements are arranged, but the image pickup sensor can also be provided with further functions.
For example, a filter (color filter or the like) that transmits light of a specific wavelength may be arranged in each image sensor of the image sensor, and the image pickup unit 10 may output data representing the state of pixels corresponding to the light of the specific wavelength. Good.

FIG. 8 is a schematic diagram showing a configuration example of an image sensor in which a filter that transmits light of a specific wavelength is arranged in the image sensor.
In FIG. 8, R indicates a red filter, G indicates a green filter, and B indicates a blue filter.
In the image sensor shown in FIG. 8, when the amount of change in the electric charge accumulated for each image sensor reaches a set threshold value, as in the case of the image sensor 1 described above, the pixel data is “+1” indicating an increase. Alternatively, "-1" indicating a decrease is output.
In the data storage control unit 22 of the information processing unit 20, the address of the image captured by the image pickup unit 10 and the type of filter (wavelength of transmitted light) arranged in the image pickup sensor at that address are registered in association with each other. Therefore, it is possible to store and read pixel data associated with the type of filter.

With such an image sensor configuration, for example, it is possible to generate a frame representing the state of pixels for each color.
As an example, when the wavelength of the light source is biased, it is possible to detect a change in the subject by selecting pixel data in which a filter that transmits light having a wavelength of the complementary color system is arranged. In this case, the change in the subject appears more clearly in the frame, so that the change in the subject can be detected more appropriately.
That is, according to the configuration of the first modification, it is possible to provide a more convenient technique in an application in which processing reflecting the wavelength of light is effective.

[Modification 2]
In the above-described embodiment, the output data storage unit 24 is formed with two-dimensional memories (frame memory 24a for increase and frame memory 24b for decrease) having an address corresponding to the address of the image captured by the image pickup unit 10. Depending on whether the data representing the state of the pixel represents an increase or a decrease, the address corresponding to the address of the pixel in the frame memory 24a for increase or the frame memory 24b for decrease The time information attached to the pixel data is stored.

On the other hand, the following pixel data storage format can be used.
(Save format 1)
The storage format of the pixel data is (time t, image address (x, y), polarity).
In addition, "polarity" is a value (+1 or -1 etc.) indicating an increase or decrease as a pixel state.
When the storage format 1 is used, the storage area of the output data storage unit 24 does not need to be a two-dimensional memory, and can be a one-dimensional continuous address.

FIG. 9 is a schematic diagram showing an example of a state in which pixel data of the storage format 1 is stored in a storage area having a one-dimensional continuous address.
As shown in FIG. 9, as a one-dimensional continuous address, a memory address that increases by 1 from address 0 is assigned to the storage area, and the storage format is stored in the memory address with the smaller address in the order received by the data receiving unit 21. The pixel data of 1 is stored.
When the pixel data is saved in the storage format 1, the data retention management becomes easy, and the storage area can be flexibly expanded or reduced according to the amount of received data.

(Save format 2)
The pixel data storage format is a format in which the latest pixel data (for example, data other than an address) is stored in a storage area prepared for each pixel.
For example, the time and polarity at which the pixel data is received can be overwritten and stored in the storage area prepared for each pixel (that is, the storage area dedicated to each pixel).
In the storage format 2, the storage area of the output data storage unit 24 does not need to be a two-dimensional memory as in the above embodiment, and is a one-dimensional continuous address as in the storage area shown in FIG. be able to.
However, as in the above-described embodiment, the storage area for increase and the storage area for decrease are formed independently, and only the time information is stored in one of the storage areas according to the value of polarity. It may be that.

(Save format 3)
A format is used in which pixel data for a plurality of times is stored in a reconstructable data structure in one area for storing pixel data.
Specifically, the pixel data for a plurality of times is allocated and saved with respect to the data width of one area for storing the pixel data. For example, if one area for storing pixel data has 64 bits and the time information is represented by 16 bits, the time information for four times can be stored in the 64-bit storage area. it can.

When using such a data storage format, when storing the time information of newly received pixel data, for example, 64 bits of data in the target storage area is read out and the time information to be deleted (most). After excluding old ones, etc.), 64-bit data with new time information added is constructed and written back to the target storage area.
It should be noted that the time information can be easily replaced by storing the time information older than the upper digit of the storage area.

(Save format 4)
In the area for storing pixel data, instead of time information, information on the number of times pixel data is output within a certain period of time is stored.
Specifically, in each of the storage formats in the above-described embodiment and this modification, the number of times the pixel state changes within a certain period of time without storing the time information (that is, the number of times the pixel data is output) is calculated. save.
If the time set as a fixed time is very short and specific time information is not required, the approximate time information can be obtained by dividing the set time by the number of times the pixel data is output.
Further, since the data to be stored is only a numerical value (integer in a predetermined range) representing the number of times, the required storage capacity can be reduced as compared with the case of storing the time information.
Therefore, by using the storage format 4, the storage capacity and the load of numerical calculation can be reduced, so that the device specifications required for the image pickup device 1 can be suppressed.

[Modification 3]
In the above-described embodiment and each modification, it is possible to control whether or not to retain the pixel data stored in the output data storage unit 24 according to a predetermined policy.
As a predetermined policy, it is possible to determine the necessity of holding mainly based on the time and the necessity of holding based on the address of the pixel data.
Specifically, the following retention policy can be adopted.

(Retention policy 1)
Delete past data for a certain period of time (set threshold time) or longer.
That is, when the pixel data becomes older than the set threshold time from the current time, the data is deleted.
This makes it possible to prevent excessively old pixel data from being retained when there is no change in the pixel state.

(Retention policy 2)
Delete past data over a certain number of elements (the number of set pixel data).
That is, when the pixel data is older than the latest fixed number of elements (the number of set pixel data), the data is deleted.
As a result, when the pixel state changes and the pixel data is sequentially received, it is possible to delete the past pixel data that is considered unnecessary.

(Retention policy 3)
If pixel data within a certain period of time exists at the same address, the pixel data is retained and other pixel data is deleted.
When a change occurs in the pixel data, there is a high possibility that a change opposite to the change will occur within a certain period of time, so it is appropriate to retain the pixel data.
For example, in object detection by a surveillance camera, when an intruder passes through the angle of view, the pixels corresponding to the intruder's body become darker than before at night, and become brighter again when the intruder passes through. A situation can occur.
Therefore, it is possible to retain useful pixel data when a frame is generated from the pixel data when an object is detected.

(Retention policy 4)
If there is pixel data held at a nearby address, the center pixel data is held, and if there is no pixel data held at a nearby address, the pixel data is deleted.
In this case, it is possible to retain the pixel data that is considered to have actually changed in the subject, and to delete the pixel data that is discretely generated due to noise or the like.
That is, it is possible to retain valid pixel data and delete unnecessary pixel data.

[Modification example 4]
In the above-described embodiment and each modification, the cycle for determining whether or not the amount of received data per unit time is within the reference range may be changed according to preset conditions.
When the threshold value Th1 used for determining the increase and decrease of the brightness in the image pickup device 12 is inappropriate, the change of the subject cannot be detected appropriately. Therefore, it is useful to change (correct) the threshold value Th1.
On the other hand, since the threshold Th1 is sequentially corrected, if it is frequently determined whether or not the amount of received data per unit time is within the reference range, the processing load may increase and the system may become unstable. is there.
Therefore, based on various information input to the image pickup apparatus 1, a condition regarding a cycle for determining whether or not the amount of received data per unit time is within the reference range is set, and the determination is performed according to this condition. Therefore, the threshold value Th1 can be corrected more efficiently.
It should be noted that the determination as to whether or not the amount of received data per unit time is within the reference range may be performed for each pixel, and the threshold Th1 may also be set for each pixel.
As a result, it is possible to determine whether or not the amount of received data per position time is within the reference range in the determination cycle according to the state of the pixel for each pixel, and set a more appropriate threshold Th1.

[Specific application example 1]
Next, a specific application example of the image pickup apparatus 1 will be described.
In this application example, the image pickup device 1 is installed in an automobile and used as an image pickup device for automatic driving.
In this case, the image pickup device 1 realizes a function of acquiring the image pickup result of capturing the real world and the image pickup result of the virtual space captured by the virtual image pickup device 1 and generating data of the space in which the automobile travels. Further, the image pickup device 1 realizes a function of simulating the automatic driving of an automobile in the generated space and analyzing the result of the simulation.
In order to realize these functions, the image pickup apparatus 1 prepares a library for easily executing the target process, and the necessary library is appropriately used when executing various processes.

[Constitution]
FIG. 10 is a schematic view showing the configuration of the entire image pickup apparatus 1 in the specific application example 1.
The configuration shown in FIG. 10 differs from the overall configuration shown in FIG. 1 in that an imaging unit 100 for capturing a digital image (a still image or a moving image obtained by capturing a real world as a color image) is provided. ..
The image pickup unit 100 can be configured by an existing digital video camera that captures a subject within the angle of view as a two-dimensional color image.
The image pickup unit 100 includes a lens 111 and an image pickup device 112.
The lens 111 focuses the light rays from the subject within the angle of view and forms a subject image on the image sensor of the image pickup device 112.

The image pickup device 112 includes an image pickup sensor in which a plurality of image pickup elements are arranged, and a processing circuit that processes an output signal of the image pickup sensor.
The angles of view (shooting ranges) of the imaging unit 10 and the imaging unit 100 are the same (or overlapped), and pixel data representing changes in the subject imaged by the imaging unit 100 is output by the imaging unit 10.
Further, since the image pickup unit 100 is provided, the data receiving unit 21 of the information processing unit 20 has a function of receiving pixel data output from the image pickup unit 10 and pixel data output from the image pickup unit 100. It has a function of receiving (pixel data group for each frame) at a predetermined cycle (for example, 100 [ms] cycle). The data receiving unit 21 outputs the pixel data received from the imaging unit 100 to the data storage control unit 22, and stores it in the output data storage unit 24 as data representing a frame.

[Functional configuration]
Next, the functional configuration of the information processing unit 20 will be described.
FIG. 11 is a block diagram showing a functional configuration of the information processing unit 20.
FIG. 11 shows a functional configuration formed in the processing unit 23 and the storage unit 20B of the information processing unit 20.
As shown in FIG. 11, the imaging apparatus 1 in this application example includes a spatial data acquisition unit 23a, a simulation processing unit 23b, and an analysis processing unit 23c in the processing unit 23. Further, in the storage unit 20B, in addition to the output data storage unit 24, a library storage unit 25 and a 3D model database (3D model DB) 26 are formed.

The library storage unit 25 stores a plurality of libraries that summarize various functions that are frequently used in the image pickup apparatus 1. In the present embodiment, when a library is used, a library required as an API (Application Program Interface) is called from a running application, and a response by the function of the called library is returned to the application.

For example, the library storage unit 25 stores a library that realizes the following functions.
(Library 1) A library that switches between reading output data as an image and reading output data as pixel data (Library 2) Pixels with negative polarity in pixel data are blue, and pixels with positive polarity in pixel data are Library that displays the image represented by the output data as red (Library 3) Library that switches the storage format of the output data (either two-dimensional memory or storage formats 1 to 4 etc.) (Library 4) Increases and decreases in brightness during imaging A library that normalizes the data according to the threshold Th1 used for the determination of the above. The library 4 is used when the output data of the image pickup apparatus 1 is analyzed later.
(Library 5) A library that switches whether to zoom in (or focus) on a specific area (area with large changes, etc.) during shooting (Library 6) The amount of output data is the reference range according to the movement of the subject. Library for switching whether or not to change the threshold Th1 (library 7) A library for setting parameters (viewpoint position, aperture value, sensitivity, etc.) of a virtual camera that images in 3D space.

By calling these libraries in the application program, if higher-level instructions are described, lower-level controls for realizing various functions in the image pickup apparatus 1 can be easily executed.
In addition to the library, middleware or an application can be used as a form for collecting various functions frequently used in the image pickup apparatus 1.

The 3D model DB 26 stores 3D model data of a virtual space for simulating automatic driving. In the present embodiment, the 3D model DB 26 stores 3D model data representing a real space (a real space constituting a driving environment) in which an automatic driving simulation is performed. As a result, even when the vehicle cannot travel in the real space, it is possible to acquire the imaging result obtained by imaging the virtual space with the virtual imaging device 1 by using the three-dimensional model data stored in the 3D model DB 26. Become.
The 3D model DB 26 is not limited to the 3D model data representing the real space (the real space constituting the driving environment) in which the simulation of automatic driving is performed, but also the 3D model data of the virtual space constituting the fictitious driving environment. It is also possible to memorize. In this case, it is possible to perform a simulation of virtually traveling in various driving environments, not limited to the real space.

The spatial data acquisition unit 23a executes a process (environment construction process) for constructing spatial data representing a simulation environment for automatic driving.
That is, the spatial data acquisition unit 23a acquires pixel data (hereinafter, appropriately referred to as “real data”) obtained by imaging the real space in the traveling range of the automobile by the imaging unit 10 and the imaging unit 100 installed in the automobile. It is stored in the output data storage unit 24. In addition, the spatial data acquisition unit 23a uses the three-dimensional model data stored in the 3D model DB 26 to capture pixel data in which the traveling range set in the virtual space is captured by the virtual imaging device 1 (hereinafter, appropriately “). (Referred to as "virtual data") is acquired asynchronously for each pixel and stored in the output data storage unit 24. At this time, the spatial data acquisition unit 23a can call the above-mentioned library 7 and set parameters (viewpoint position, aperture value, sensitivity, etc.) of the virtual camera that images the inside of the 3D space. Further, the spatial data acquisition unit 23a can call the above-mentioned library 3 and appropriately switch the storage format of the output data.

Then, the spatial data acquisition unit 23a synthesizes the acquired real data and virtual data, and constructs spatial data representing the simulation environment of automatic driving. At this time, the spatial data acquisition unit 23a calls the above-mentioned library 1 and reads out the data output by the imaging unit 10 as an image to construct the spatial data representing the simulation environment of the automatic driving. Further, the spatial data acquisition unit 23a stores the constructed spatial data in a memory (not shown) or a storage area of the storage unit 20B.

Here, the actual data constituting the space data representing the simulation environment of the automatic operation includes pixel data including data representing the state of the pixels imaged by the imaging unit 10 and pixels representing the landscape imaged by the imaging unit 100. Data is a component. Since the imaging unit 10 outputs pixel data at a cycle earlier than that of the imaging unit 100, pixels including pixel data representing the landscape imaged by the imaging unit 100 and data representing the state of the pixels imaged by the imaging unit 10. There is a timing at which the data is acquired together and a timing at which only the pixel data including the data representing the state of the pixels imaged by the imaging unit 10 is acquired. Therefore, when the spatial data acquisition unit 23a constructs spatial data representing the simulation environment for automatic operation, pixel data representing the landscape imaged by the imaging unit 100 and data representing the state of the pixels imaged by the imaging unit 10 are used. Except for the timing when the included pixel data is also acquired, the shape of the object is interpolated by the pixel data including the data representing the state of the pixels imaged by the imaging unit 10, and the spatial data representing the simulation environment of automatic operation is obtained. Convert to a more accurate one.

Specifically, according to the following procedure, spatial data representing a simulation environment for autonomous driving can be configured.
(Procedure 1) The pixel data representing the landscape imaged by the imaging unit 100 and the pixel data including the data representing the state of the pixels imaged by the imaging unit 10 are captured by the imaging unit 10. The shape of the object calculated based on the pixel data representing the landscape is associated with the shape calculated from the pixel data including the data representing the state of the pixels imaged by the imaging unit 10.
(Procedure 2) At the timing when only the pixel data including the data representing the state of the pixel imaged by the imaging unit 10 is acquired, the pixel data including the data representing the state of the pixel imaged by the imaging unit 10 is stored in the past. It integrates in a predetermined period and interpolates the shape of the object in time.
(Procedure 3) Based on the results of steps 1 and 2, pixel data including data representing the landscape imaged by the imaging unit 100 and data representing the state of the pixels imaged by the imaging unit 10 are also acquired. A simulation environment for automatic operation is created by synthesizing the data of the shape of the object at the timing when only the pixel data including the data representing the state of the pixels imaged by the imaging unit 10 is acquired. Construct the data of the space to be represented.

FIG. 12 is a schematic diagram showing a concept of interpolating the shape of an object by pixel data including data representing the state of pixels imaged by the imaging unit 10.
Note that FIG. 12 shows a state in which the automobile in which the image pickup apparatus 1 is installed advances in the order of time ta → time tb → time tk → time td and images a building adjacent to the road.
In FIG. 12, the imaging unit 100 shall perform imaging at time ta and time td according to the imaging cycle. At this time, although the entire building to be the subject is included in the image captured at time ta, it is hardly captured in the image captured at time td.
On the other hand, since the imaging unit 10 can perform imaging (output pixel data) in a shorter imaging cycle than the imaging unit 10, the state of the pixels that imaged the building is also between the time ta and the time td. Pixel data representing changes can be output sequentially.
Therefore, at time tb and time tk between time ta and time td, the pixel data output by the imaging unit 10 is used to synthesize and interpolate the shape data of the object (building), and the simulation environment for automatic operation is performed. It is possible to construct the data of the space representing.

The simulation processing unit 23b refers to the data in the space representing the simulation environment of the automatic driving constructed by the spatial data acquisition unit 23a, and performs a process (simulation) of simulating the automatic driving in this space according to the set automatic driving algorithm. Process) is executed. At this time, the simulation processing unit 23b calls the above-mentioned library 1 and reads out the data output by the imaging unit 10 as pixel data. As a result, the simulation processing unit 23b asynchronously acquires data representing the state of the pixels and simulates the automatic operation. Further, the simulation processing unit 23b calls the above-mentioned library 5 when simulating the automatic operation, and determines whether or not to zoom up (or focus) on a specific area (area with a large change, etc.) during shooting. You can switch. Further, the simulation processing unit 23b calls the above-mentioned library 6 when simulating the automatic operation, and switches whether or not to change the threshold Th1 so that the output data amount becomes the reference range according to the movement of the subject. Can be done. Further, the simulation processing unit 23b can call the above-mentioned library 3 and appropriately switch the storage format of the output data when simulating the automatic operation.

Further, the simulation processing unit 23b stores the operation history (simulation result) data acquired by the simulation of the automatic operation in a memory (not shown) or a storage area of the storage unit 20B. By setting various algorithms to be tested or the like as the algorithm of the automatic driving, it is possible to evaluate the performance of the algorithm of the automatic driving based on the simulation result of the automatic driving.
Further, the simulation processing unit 23b can perform machine learning on the automatic operation algorithm so that the result of analyzing the simulation result of the automatic operation by the analysis processing unit 23c becomes more appropriate.

The analysis processing unit 23c executes a process (analysis process) for performing an analysis for performance evaluation of an automatic driving algorithm or the like based on the simulation result data acquired by the simulation processing unit 23b.
At this time, by calling the library 2 described above, the analysis processing unit 23c calls the above-mentioned library 2, and in the pixel data including the data representing the state of the pixels imaged by the image pickup unit 10, the pixel having a negative polarity is blue and the polarity is positive. The image represented by the output data can be displayed with a certain pixel as red. Further, the analysis processing unit 23c calls the library 4 described above to normalize the data according to the threshold Th1 used for determining the increase and decrease of the brightness at the time of imaging, and is detected in the simulation of automatic operation. Changes in the state of pixels can be graphed and displayed in chronological order.

[motion]
Next, the operation of the image pickup apparatus 1 of this application example will be described.
FIG. 13 is a schematic diagram showing an example of a route on which an automatic driving simulation is performed.
The route shown in FIG. 13 departs from home (point A), passes through a residential area, turns right at point B to enter the national highway, then turns left at point C to enter the shopping district, and passes through a railroad crossing. Then it reaches the park (point D). However, the section from point C to point D is a section in which actual driving is not possible due to reasons such as construction.
Hereinafter, assuming a case where data of a space representing an automatic driving simulation environment is constructed for the route illustrated in FIG. 13 and automatic driving simulation and simulation result analysis are performed on the route, the imaging device 1 The operation of is explained.

[Environment construction process]
FIG. 14 is a flowchart illustrating the flow of the environment construction process executed by the image pickup apparatus 1.
The environment construction process is started by performing an operation instructing the start of the environment construction process.
In step S41, the spatial data acquisition unit 23a acquires pixel data (actual data) obtained by imaging the real space in the traveling range of the automobile by the imaging unit 10 and the imaging unit 100 installed in the automobile, and causes the output data storage unit 24 to acquire the pixel data (actual data). Remember. At this time, at the timing when the spatial data acquisition unit 23a can acquire pixel data including the pixel data representing the landscape imaged by the image pickup unit 100 and the state of the pixels imaged by the image pickup unit 10 together, these Pixel data is also acquired. Further, the spatial data acquisition unit 23a acquires only the pixel data output by the imaging unit 10 at the timing when only the pixel data including the data representing the state of the pixels imaged by the imaging unit 10 can be acquired.

In step S42, the spatial data acquisition unit 23a uses the three-dimensional model data stored in the 3D model DB 26 to capture pixel data (virtual data) in which the traveling range set in the virtual space is captured by the virtual imaging device 1. ) Is acquired and stored in the output data storage unit 24. At this time, the spatial data acquisition unit 23a calculates the pixel data output from the imaging unit 10 at the same timing as the imaging cycle of the imaging unit 100, and the pixel data output by the virtual imaging unit 10 and the imaging unit 100. Is also acquired.
In step S42, the spatial data acquisition unit 23a can call the above-mentioned library 7 and set parameters (viewpoint position, aperture value, sensitivity, etc.) of the virtual camera that images the inside of the 3D space. Further, in step S41 and step S42, the spatial data acquisition unit 23a can call the above-mentioned library 3 and appropriately switch the storage format of the output data.

In step S43, the spatial data acquisition unit 23a synthesizes the acquired real data and virtual data, and constructs spatial data representing a simulation environment for automatic driving. At this time, the spatial data acquisition unit 23a calls the above-mentioned library 1 and reads out the data output by the imaging unit 10 as an image to construct the spatial data representing the simulation environment of the automatic driving. The actual data in the space representing the simulation environment for automatic operation includes pixel data including data representing the state of the pixels imaged by the imaging unit 10 and pixel data representing the landscape imaged by the imaging unit 100. It has been. In the process of step S43, the spatial data acquisition unit 23a also acquires pixel data representing the landscape imaged by the imaging unit 100 and pixel data including data representing the state of the pixels imaged by the imaging unit 10. Other than the timing, the shape of the object is interpolated by the pixel data including the data representing the state of the pixels imaged by the imaging unit 10, and the space data representing the simulation environment of automatic operation is converted into a more accurate one.

In step S44, the spatial data acquisition unit 23a stores the constructed spatial data in a memory (not shown) or a storage area of the storage unit 20B.
After step S44, the environment construction process ends.

[Simulation processing]
FIG. 15 is a flowchart illustrating a flow of simulation processing executed by the image pickup apparatus 1.
The simulation process is started by performing an operation instructing the start of the simulation process.
In step S51, the simulation processing unit 23b acquires the data of the space representing the simulation environment of the automatic operation.
In step S52, the simulation processing unit 23b starts moving (simulation) from the starting point in the space representing the simulation environment for automatic driving.

In step S53, the simulation processing unit 23b moves in the space representing the simulation environment of automatic driving according to the set automatic driving algorithm, and records the response (driving history) of the automobile as the simulation result.
In step S54, the simulation processing unit 23b determines whether or not the destination has been reached in the space representing the simulation environment for automatic driving.
If the destination has not been reached in the space representing the simulation environment for automatic driving, NO is determined in step S54, and the process proceeds to step S53.
On the other hand, when the destination is reached in the space representing the simulation environment for automatic driving, YES is determined in step S54, and the simulation process ends.

[Analysis processing]
FIG. 16 is a flowchart illustrating a flow of analysis processing executed by the image pickup apparatus 1.
The analysis process is started by performing an operation instructing the start of the analysis process.
In step S61, the analysis processing unit 23c reads out the data of the driving history (simulation result) of the automobile.

In step S62, the analysis processing unit 23c reads out the images with the image pickup device 1 as the viewpoint in time series and displays them as landscape images. By calling the library 2 described above, the analysis processing unit 23c calls the pixel data including the data representing the state of the pixels imaged by the imaging unit 10 to display pixels having a negative polarity as blue and pixels having a positive polarity. The image represented by the output data can be displayed as red. In step S62, different analysis results may be displayed depending on the content of the analysis, such as graphing the changes in the state of the pixels detected in the simulation of automatic driving in time series.

In step S63, the analysis processing unit 23c determines whether or not the end of the analysis processing is instructed.
If the end of the analysis process is not instructed, NO is determined in step S63, and the process proceeds to step S62.
On the other hand, when the end of the analysis process is instructed, YES is determined in step S63, and the analysis process ends.

As described above, the imaging device 1 in this application example acquires actual data obtained by imaging the real space in the traveling range of the automobile by the imaging unit 10 and the imaging unit 100 installed in the automobile, and is stored in the 3D model DB 26. Using the three-dimensional model data, the virtual data obtained by capturing the traveling range set in the virtual space with the virtual imaging device 1 is acquired. Then, the image pickup apparatus 1 synthesizes the acquired real data and virtual data, and constructs the data of the space representing the simulation environment of the automatic driving.
Therefore, the image was taken by the image pickup unit 10 except at the timing when the pixel data including the pixel data representing the landscape imaged by the image pickup unit 100 and the data representing the state of the pixels imaged by the image pickup unit 10 were acquired together. The shape of the object can be interpolated by the pixel data including the data representing the pixel state, and the space data representing the simulation environment of automatic operation can be converted into a more accurate one.
Further, the image pickup apparatus 1 refers to the data of the constructed space and executes the simulation process in this space according to the set automatic driving algorithm.
Therefore, as an algorithm for automatic driving, various algorithms to be tested or the like can be set, and simulation results of automatic driving can be obtained.

Further, the image pickup apparatus 1 executes an analysis process for evaluating the performance of the algorithm of the automatic driving based on the data of the simulation result.
As a result, the constructed space can be used to analyze the result of automatic driving by the set automatic driving algorithm, and machine learning for the automatic driving algorithm can be performed.
In addition, it becomes possible to use an automatic driving algorithm optimized by machine learning for automatic driving in an actual automobile.

In addition, the simulation result data can be visualized and analyzed, and reflected in the high performance of the automatic driving algorithm.
Further, the image pickup device 1 of this application example can detect a change in the subject at an early stage by the pixel data output by the image pickup unit 10, and by using the pixel data output by the image pickup unit 10 and the image pickup unit 100 together, the subject can be detected. It is possible to more accurately grasp and construct the shape of the object. Further, the image pickup device 1 can speed up the situation grasping and the response at the time of automatic operation.

Further, in the image pickup apparatus 1 of this application example, a library that summarizes various functions that are frequently used is prepared.
As a result, in the application, if the upper instruction is described by calling the library, the lower control for realizing various functions in the image pickup apparatus 1 can be easily executed.

The imaging device 1 can acquire pixel data output by the imaging unit 10 and detect changes in the subject when the movable device is autonomously moved or moved manually, not limited to the automobile. For example, by using the image pickup device 1, it is possible to acquire pixel data output by the image pickup unit 10 and detect a change in a subject when moving a walking robot, a flying object such as a drone, a ship, a submersible, or the like. it can. Further, the image pickup device 1 can be used for simulation or actual operation of autonomous control of these movable devices, including automatic driving of an automobile.
Further, the target of analysis in the analysis process is not limited to the simulation result of the automatic operation, but can be the data of the operation history of the actual automatic operation.

Further, the various processes described as those executed by the image pickup apparatus 1 in the above-described embodiment may be performed by an information processing apparatus other than the image pickup apparatus 1.
Further, a part of the various processes described as being executed by the image pickup apparatus 1 in the above-described embodiment may be executed by an information processing apparatus other than the image pickup apparatus 1, and the image pickup apparatus 1 may acquire the execution result. ..
Further, in the above-described embodiment, various data (three-dimensional model data in real space, etc.) used by the imaging device 1 are stored in an information processing device other than the imaging device 1, and necessary data is appropriately downloaded and imaged. It may be stored in the device 1.

The image pickup apparatus 1 configured as described above includes an image pickup unit 10 and an information processing unit 20. The information processing unit 20 includes a data receiving unit 21, a data storage control unit 22, a processing unit 23, and a storage unit 20B.
The image pickup unit 10 transmits data representing a change in the state of the pixels corresponding to the image pickup element based on the change in the output of the image pickup element provided in the image pickup sensor that images the subject.
The data receiving unit 21 asynchronously receives the data representing the change in the state of the pixels transmitted by the imaging unit 10 for each pixel.
The storage unit 20B stores data representing the change in the state of the pixel, time information regarding the data representing the change in the state of the pixel, and the address of the pixel in a preset storage format.
The data storage control unit 22 stores the data representing the change in the state of the pixels received by the data reception unit 21 in the storage unit 20B as data in a preset storage format.
As a result, as soon as a change is detected for each pixel, data representing the state of the pixel is output asynchronously. Therefore, a conventional digital image for each frame captured is output and used for applications such as object detection. Compared to technology, it is possible to detect changes in the subject at an angle of view at high speed with a small amount of data.
Further, the image pickup apparatus 1 can read pixel data stored in a predetermined storage format and use it for various applications.
Therefore, according to the image pickup apparatus 1, it is possible to realize more appropriate functions in the information processing technology using digital images.

Further, in the storage unit 20B, when the data representing the change in the pixel state represents an increase, the frame memory 24a for increasing that stores the data and the data representing the change in the pixel state represent a decrease. A reduced frame memory 24b for storing the data in some cases is provided.
As a result, the pixel data can be saved in a form in which the necessary data can be easily referred to. Further, for example, among the pixel data (time information, pixel address, data representing the state of each pixel, etc.), only the time information needs to be stored at a predetermined address, so that it is saved as compared with the case where all of them are saved. The amount of data to be generated can be reduced. Further, since the sizes of the frame memory 24a for increasing and the frame memory 24b for decreasing are determined in advance, it is possible to easily estimate the storage capacity required for storing the pixel data.

Further, the threshold value for determining the change in the output of the image sensor is changed based on the amount of data transmitted from the image sensor 10.
As a result, the amount of data transmitted from the imaging unit 10 can be converged to a set range.

The data storage control unit 22 stores the data representing the change in the state of the pixel corresponding to the image sensor and the threshold value used for determining the change in the output of the image sensor for the data in the storage unit 20B in association with each other. Let me.
This makes it possible to calculate the amount of change in the pixel state by referring to the threshold value used when determining the change in the output of the image sensor.

Further, the information processing unit 20 controls the focus or zoom in the image pickup unit 10 according to the distribution of data representing the change in the state of the pixels corresponding to the image pickup element transmitted from the image pickup unit 10.
As a result, the state of the subject can be detected by focusing on a region or the like where the change is large within the angle of view.

In addition, the image sensor includes a color filter in the image sensor.
The data representing the change in the state of the pixel corresponding to the image sensor represents the state of the pixel corresponding to the wavelength of the light transmitted through the color filter provided in the image sensor.
As a result, it is possible to provide a more convenient technique in an application in which processing that reflects the wavelength of light is effective.

Further, the image pickup device 1 includes an image pickup unit 100.
The image pickup unit 100 acquires image data representing a landscape.
The processing unit 23 associates the data representing the change in the state of the pixels corresponding to the image sensor transmitted from the image sensor 10 with the image data acquired by the image sensor 100, and executes the processing by the application.
As a result, in addition to constructing the shape of the object from the image data acquired by the image pickup unit 100, the shape of the object is interpolated by the pixel data including the data representing the state of the pixels captured by the image pickup unit 10. Highly accurate spatial data can be constructed.

The processing unit 23 switches between reading the data representing the change in the state of the pixel corresponding to the image sensor stored in the storage unit 20B as an image or reading it as pixel data.
As a result, it is possible to read data representing a change in the pixel state in an appropriate format according to the processing content.

Further, the image pickup device 1 as an information processing device includes a spatial data acquisition unit 23a in the information processing unit 20.
In the three-dimensional virtual space, the spatial data acquisition unit 23a asynchronously acquires data representing a change in the state of the pixels corresponding to the virtual image sensor when the subject is virtually imaged for each pixel.
As a result, even when the real space cannot be actually imaged, it is possible to acquire the data that virtually detects the state of the subject.

Further, the imaging device 1 includes an imaging unit 10, a data receiving unit 21, a data storage control unit 22, a processing unit 23, and a storage unit 20B.
The image pickup unit 10 transmits data representing a change in the state of the pixels corresponding to the image pickup element based on the change in the output of the image pickup element provided in the image pickup sensor that images the subject.
The data receiving unit 21 asynchronously receives the data representing the change in the state of the pixels transmitted by the imaging unit 10 for each pixel.
The storage unit 20B includes data representing a change in the state of the pixel acquired by the spatial data acquisition unit 23a, data representing the change in the state of the pixel received by the data receiving unit 21, and data representing the change in the state of the pixel. Time information and pixel addresses related to the data are stored in a preset storage format.
The data storage control unit 22 stores the data representing the change in the pixel state acquired by the spatial data acquisition unit 23a and the data representing the change in the pixel state received by the data receiving unit 21 in a preset storage format. It is stored in the storage unit 20B as data.
As a result, as soon as a change is detected for each pixel, data representing the state of the pixel is output asynchronously. Therefore, a conventional digital image for each frame captured is output and used for applications such as object detection. Compared to technology, it is possible to detect changes in the subject at an angle of view at high speed with a small amount of data.
Further, the image pickup apparatus 1 can read pixel data stored in a predetermined storage format and use it for various applications.
Further, it is possible to perform more appropriate processing by interpolating the data that virtually detects the state of the subject and the data that represents the change in the state of the pixels detected by imaging.
Therefore, according to the image pickup apparatus 1, it is possible to realize more appropriate functions in the information processing technology using digital images.

The spatial data acquisition unit 23a obtains spatial data representing a three-dimensional space based on at least one of data representing a change in the state of the acquired pixel and data representing a change in the state of the pixel received by the data receiving unit 21. To construct.
As a result, the data that virtually detects the state of the subject and the data that represents the change in the state of the pixels detected by imaging can be interpolated with each other to make the data in the three-dimensional space more accurate. it can.

Further, the imaging device 1 includes a simulation processing unit 23b.
The simulation processing unit 23b detects a change in the subject visually recognized from the movable device based on the spatial data constructed by the spatial data acquisition unit 23a, and performs a simulation for controlling the movement.
Thereby, various algorithms for controlling the movable device can be set and the simulation result can be acquired.

Further, the image pickup apparatus 1 includes an analysis processing unit 23c.
The analysis processing unit 23c learns about an algorithm for controlling the movement based on the result of the simulation by the simulation processing unit 23b.
As a result, a more appropriate algorithm can be generated by repeating the simulation using the constructed spatial data.

The analysis processing unit 23c outputs data representing the change in the pixel state acquired by the spatial data acquisition unit 23a and data representing the change in the pixel state received by the data receiving unit 21 to the output of the image sensor for the data. Normalize based on the threshold used to determine the change and display the pixel change.
As a result, it is possible to display the change in the state of the acquired pixel in a form that can be objectively analyzed.

The present invention is not limited to the above-described embodiment, and modifications, improvements, and the like within the range in which the object of the present invention can be achieved are included in the present invention.
That is, the present invention can be applied to various electronic devices having an imaging function or various electronic devices having an information processing function.

The series of processes described above can be executed by hardware or software.
In other words, the configurations of FIGS. 1 and 10 and the functional configurations of FIG. 11 are merely examples and are not particularly limited. That is, it is sufficient that the image pickup apparatus 1 is provided with a function capable of executing the above-mentioned series of processes as a whole, and what kind of functional block is used to realize this function is not particularly limited.
Further, one functional block may be configured by a single piece of hardware, a single piece of software, or a combination thereof.
The functional configuration in the present embodiment is realized by a processor that executes arithmetic processing, and the processor that can be used in the present embodiment is composed of various processing units such as a single processor, a multiprocessor, and a multicore processor. In addition to the above, it includes a combination of these various processing units and processing circuits such as an ASIC (Application Specific Integrated Circuit) and an FPGA (Field-Programmable Gate Array).

When a series of processes are executed by software, the programs constituting the software are installed on a computer or the like from a network or a recording medium.
The computer may be a computer embedded in dedicated hardware. Further, the computer may be a computer capable of executing various functions by installing various programs, for example, a general-purpose personal computer.

The recording medium containing such a program is not only composed of removable media distributed separately from the device main body in order to provide the program to the user, but is also provided to the user in a state of being preliminarily incorporated in the device main body. It is composed of a recording medium and the like. The removable media is composed of, for example, a magnetic disk (including a floppy disk), an optical disk, a magneto-optical disk, or the like. The optical disk is composed of, for example, a CD-ROM (Compact Disk-Read Only Memory), a DVD (Digital Versaille Disk), a Blu-ray (registered trademark) Disc (Blu-ray Disc), or the like. The magneto-optical disk is composed of MD (Mini-Disk) or the like. Further, the recording medium provided to the user in a state of being incorporated in the apparatus main body in advance includes, for example, a ROM (Read Only Memory) in which the program is recorded, a semiconductor memory included in the storage unit 20B, and the like.

Although some embodiments of the present invention have been described above, these embodiments are merely examples and do not limit the technical scope of the present invention. The present invention can take various other embodiments, and various modifications such as omission and substitution can be made without departing from the gist of the present invention. These embodiments and modifications thereof are included in the scope and gist of the invention described in the present specification and the like, and are also included in the scope of the invention described in the claims and the equivalent scope thereof.

1 Imaging device, 10,100 Imaging unit, 11,111 lens, 12,112 Imaging device, 20 Information processing unit, 20A processor, 20B storage unit, 21 Data receiving unit, 22 Data storage control unit, 23 Processing unit, 23a space Data acquisition unit, 23b simulation processing unit, 23c analysis processing unit, 24 output data storage unit, 24a frame memory for increase, 24b frame memory for decrease, 25 library storage unit, 26 3D model database, 30 sensor unit

Claims (18)

An imaging means that transmits data representing a change in the state of pixels corresponding to the image sensor based on a change in the output of the image sensor provided in the image sensor that images the subject.
A data receiving means that asynchronously receives data representing a change in the state of the pixel transmitted by the imaging means for each pixel.
A storage means in which data representing a change in the state of the pixel, time information relating to the data representing the change in the state of the pixel, and the address of the pixel are stored in a preset storage format.
A storage control means for storing data representing a change in the state of the pixel received by the data receiving means in the storage means as data in the storage format.
An imaging device characterized by comprising. The storage means has an increasing storage area for storing the data when the data representing the change in the state of the pixel represents an increase, and a storage area for increasing when the data representing the change in the state of the pixel represents a decrease. It is equipped with a storage area for reduction to store the data.
When the data representing the change in the state of the pixel represents an increase, the storage control means stores the data in the storage area for increase in the storage means, and the data representing the change in the state of the pixel is stored. The imaging apparatus according to claim 1, wherein when the data represents a decrease, the data is stored in the storage area for reduction in the storage means. The imaging device according to claim 1 or 2, wherein a threshold value for determining a change in the output of the imaging element is changed based on the amount of data transmitted from the imaging means. The storage control means provides the storage means with data representing a change in the state of pixels corresponding to the image sensor and a threshold value used for determining a change in the output of the image sensor for the data. The image pickup apparatus according to any one of claims 1 to 3, wherein the image sensor is stored. A claim comprising an image pickup control means for controlling focus or zoom in the image pickup means according to a distribution of data representing a change in the state of pixels corresponding to the image pickup element transmitted from the image pickup means. The image pickup apparatus according to any one of 1 to 4. The image sensor includes a color filter in the image sensor.
Claims 1 to 5 are characterized in that the data representing the change in the state of the pixel corresponding to the image sensor represents the state of the pixel corresponding to the wavelength of the light transmitted through the color filter provided in the image sensor. The image pickup apparatus according to any one of the above items. Image acquisition means for acquiring image data representing landscapes,
A processing means for executing processing by an application by associating data representing a change in the state of pixels corresponding to the image pickup element transmitted from the image pickup means with the data of the image acquired by the image acquisition means. ,
The imaging device according to any one of claims 1 to 6, wherein the image pickup apparatus is provided with. The seventh aspect of the present invention is characterized in that the processing means switches between reading out data representing a change in the state of the pixels corresponding to the image sensor stored in the storage means as an image or reading out as pixel data. The imaging device described. In a three-dimensional virtual space, it is characterized by providing a virtual data acquisition means for asynchronously acquiring data representing a change in the state of pixels corresponding to a virtual image sensor when a subject is virtually imaged for each pixel. Information processing device. An imaging means that transmits data representing a change in the state of pixels corresponding to the image sensor based on a change in the output of the image sensor provided in the image sensor that images the subject.
A data receiving means that asynchronously receives data representing a change in the state of the pixel transmitted by the imaging means for each pixel.
Data representing the change in the state of the pixel acquired by the virtual data acquisition means, data representing the change in the state of the pixel received by the data receiving means, time information relating to data representing the change in the state of the pixel, and time information. A storage means in which pixel addresses are stored in a preset storage format, and
A storage in which the data representing the change in the state of the pixel acquired by the virtual data acquisition means and the data representing the change in the state of the pixel received by the data receiving means are stored in the storage means as the data in the storage format. Control means and
9. The information processing apparatus according to claim 9. Spatial data representing a three-dimensional space based on at least one of data representing a change in the state of the pixel acquired by the virtual data acquisition means and data representing a change in the state of the pixel received by the data receiving means. The information processing apparatus according to claim 10, further comprising a spatial data construction means for constructing the data. A claim comprising a simulation means for detecting a change in a subject visually recognized from a movable device and performing a simulation for controlling the movement based on the spatial data constructed by the spatial data construction means. Item 11. The information processing apparatus according to item 11. The information processing apparatus according to claim 12, further comprising a learning means for learning about an algorithm for controlling movement based on the result of simulation by the simulation means. The data representing the change in the state of the pixel acquired by the virtual data acquisition means and the data representing the change in the state of the pixel received by the data receiving means are used to determine the change in the output of the image pickup element for the data. The information processing apparatus according to any one of claims 9 to 13, further comprising an analysis means for normalizing based on a threshold value used for the purpose and displaying a change in the pixel. An imaging step of transmitting data representing a change in the state of pixels corresponding to the image sensor based on a change in the output of the image sensor provided in the image sensor that images the subject.
A data receiving step in which data representing a change in the state of the pixel transmitted in the imaging step is asynchronously received for each pixel, and
In the data receiving step, the data representing the change in the state of the pixel, the time information regarding the data representing the change in the state of the pixel, and the address of the pixel are received by the storage means stored in the preset storage format. A storage control step for storing the data representing the change in the state of the pixel as the data in the storage format, and
An imaging method comprising: In a three-dimensional virtual space, a virtual data acquisition step of asynchronously acquiring data representing a change in the state of pixels corresponding to a virtual image sensor when a subject is virtually imaged for each pixel,
An information processing method characterized by including. On the computer
Based on the change in the output of the image sensor provided in the image sensor that images the subject, the change in the state of the pixel transmitted by the image pickup means that transmits data representing the change in the state of the pixel corresponding to the image sensor. A data reception function that receives the represented data asynchronously for each pixel,
The data representing the change in the state of the pixel, the time information related to the data representing the change in the state of the pixel, and the address of the pixel are received by the data receiving function in a storage means stored in a preset storage format. A storage control function for storing the data representing the change in the state of the pixel as the data in the storage format, and
A program characterized by realizing. On the computer
A virtual data acquisition function that asynchronously acquires data representing changes in the state of pixels corresponding to a virtual image sensor when a subject is virtually imaged in a three-dimensional virtual space, for each pixel.
A program characterized by realizing.

Images