KR102424382B1 - Image processing device configured to regenerate timestamp and electronic device including the same - Google Patents

Abstract

The image processing apparatus of the present invention includes a vision sensor and a processor. The vision sensor may generate a plurality of events and timestamps in which the intensity of light is changed. The processor may group the timestamps according to the time the event occurred. Furthermore, the processor of the present invention may regenerate the timestamp of the pixel in which the abnormal event occurred based on the temporal correlation of the events.

Description

IMAGE PROCESSING DEVICE CONFIGURED TO REGENERATE TIMESTAMP AND ELECTRONIC DEVICE INCLUDING THE SAME

The present invention relates to an image processing device including a vision sensor, and more particularly, to an image processing device configured to recreate a timestamp of an event that occurred in a bad pixel.

In general, an image sensor may be largely divided into an image sensor operating synchronously and an image sensor operating asynchronously. A representative example of an image sensor operating synchronously is a complementary metal-oxide semiconductor (CMOS) image sensor. A typical example of an image sensor that operates asynchronously is a vision sensor such as a dynamic vision sensor (DVS).

CMOS image sensors process images on a frame-based basis. Therefore, frames must be continuously generated at regular intervals. Accordingly, since information about a part (eg, background, etc.) not of interest to the user is also continuously generated at regular intervals, the amount of data that the processor has to process may rapidly increase. This may be a factor that degrades the performance of the image processing apparatus.

A vision sensor, on the other hand, processes images on an event-based basis. Event-based means that data is not generated at regular intervals, but information about the event is generated and delivered to the user as soon as an event (eg, change in intensity of light) occurs. Since the light intensity change mainly occurs in the outline of the object, unnecessary background information is not generated by the vision sensor. Therefore, the amount of data that the processor has to process can be drastically reduced.

On the other hand, the size of pixels constituting the vision sensor is significantly larger than the pixel size of a conventional image sensor such as a CMOS image sensor. Therefore, a defect in the pixel itself constituting the vision sensor is an important issue in terms of not only the quality of the image provided to the user, but also the yield in the manufacturing process. However, since there is a limit to improving the yield only by improving the pure manufacturing process, it is very important to improve the bad pixels using hardware or software (firmware).

An object of the present invention is to improve the performance of an image processing apparatus by regenerating the timestamp of an event occurring in a bad pixel.

The image processing apparatus according to an embodiment of the present invention generates a plurality of events in which the intensity of light changes through at least some of a plurality of pixels including a target pixel and adjacent pixels around the target pixel, and the plurality of events a vision sensor configured to generate a plurality of timestamps relating to a time when a processor configured to group at least one second timestamp having a second value of and determine a direction in which events occur and determine whether a third timestamp having a third value among timestamps corresponding to the adjacent pixels is due to an abnormal event, wherein the processor is configured to: It may be further configured to replace the stamp with one of the timestamps corresponding to pixels adjacent to the pixel that caused the abnormal event.

According to the present invention, it is an object to improve the performance of an image processing apparatus by regenerating the timestamp of an event occurring in a bad pixel.

Furthermore, according to the present invention, since bad pixels can be treated similarly to normal pixels, the effect of improving the yield of the pixel array included in the image processing apparatus can be achieved.

1 is a block diagram illustrating an image processing apparatus according to an embodiment of the present invention.
FIG. 2 is a block diagram exemplarily showing the configuration of the vision sensor shown in FIG. 1 .
FIG. 3A is a block diagram showing an exemplary configuration of the vision sensor shown in FIG. 2 .
4A is a block diagram showing an exemplary configuration of the vision sensor shown in FIG. 2 .
5A is a block diagram showing an exemplary configuration of the vision sensor shown in FIG. 2 .
FIG. 5B is a diagram exemplarily showing the format of information output from the configuration of the exemplary vision sensor illustrated in FIG. 5A .
6A is a block diagram showing an exemplary configuration of the vision sensor shown in FIG. 2 .
FIG. 6B is a diagram exemplarily showing the format of information output from the configuration of the exemplary vision sensor illustrated in FIG. 6A .
7A is a block diagram showing an exemplary configuration of the vision sensor shown in FIG. 2 .
FIG. 7B is a diagram conceptually illustrating an exemplary format of a frame output from the vision sensor of FIG. 7A .
8 is a block diagram schematically illustrating the configuration of the pixel illustrated in FIGS. 3A to 7B .
9 is a circuit diagram exemplarily illustrating the configuration of the pixel shown in FIG. 8 .
10A is a cross-sectional view illustrating a semiconductor package of the image processing apparatus described with reference to FIGS. 1 to 9 .
10B is a cross-sectional view illustrating a semiconductor package of the image processing apparatus described with reference to FIGS. 1 to 9 .
10C is a cross-sectional view illustrating a semiconductor package of the image processing apparatus described with reference to FIGS. 1 to 9 .
10D is a cross-sectional view illustrating a semiconductor package of the image processing apparatus described with reference to FIGS. 1 to 9 .
11 is a block diagram exemplarily illustrating an operation of the processor shown in FIG. 1 .
FIG. 12 is a diagram conceptually illustrating that the timestamp generator shown in FIG. 11 regenerates the timestamp of a noisy pixel or a hot pixel.
13 is a diagram conceptually illustrating that the timestamp generator according to the present invention determines a noisy pixel or a hot pixel.
14 is a diagram conceptually illustrating that the processor of the present invention determines a noise pixel or a hot pixel.
15 is a diagram conceptually illustrating that the timestamp generator of the present invention regenerates timestamps of noisy pixels or hot pixels.
16 is a diagram conceptually illustrating that the timestamp generator of the present invention regenerates the timestamp of a dead pixel.
17 is a diagram conceptually illustrating that the processor of the present invention determines a dead pixel.
18 is a flowchart illustrating a timestamp regeneration scheme according to an embodiment of the present invention.
19 and 20 are diagrams illustrating a process in which the timestamp regeneration scheme of the present invention is applied to a subsampling process.
21 is a block diagram illustrating an electronic device to which an image processing apparatus according to an embodiment of the present invention is applied.

Below, embodiments of the present invention will be described clearly and in detail to the extent that those skilled in the art can easily practice the present invention.

1 is a block diagram illustrating an image processing apparatus 100 according to an embodiment of the present invention. The image processing apparatus 100 may be configured to process a synchronous event as well as an asynchronous event. For example, the image processing apparatus 100 may generate an event-related asynchronous packet as well as an event-related synchronous frame. The image processing apparatus 100 may include a vision sensor 110 and a processor 120 .

The vision sensor 110 may output an event signal by detecting a change in the intensity of incident light. For example, when an event in which the intensity of light increases occurs, the event-based sensor 110 may output an on-event corresponding thereto. Conversely, when an event in which the intensity of light decreases occurs, the event-based sensor 110 may output an off-event.

The vision sensor 110 may be an event-based vision sensor. For example, the vision sensor 110 may output an event signal by accessing a pixel in which a change in light intensity is detected. For example, the change in light intensity may be due to the movement of an object photographed by the vision sensor 110 or may be due to the movement of the vision sensor 110 itself. In this case, the event signal detected by the vision sensor 110 or output from the vision sensor 110 may be an asynchronous event signal.

Alternatively, the vision sensor 110 may be a frame-based vision sensor. For example, the vision sensor 110 may scan all pixels constituting the vision sensor 110 every reference period to output event signals. However, unlike a typical CMOS image sensor, the vision sensor 110 may not output event signals to all pixels, and may output event signals only to pixels in which the intensity of light is sensed. In this case, the event signal output from the vision sensor 110 may be converted into a synchronous event signal by a processor or the like.

The processor 120 may process signals sensed by the vision sensor 110 . The processor 120 may regenerate timestamps of noise pixels, hot pixels, or dead pixels by using temporal correlation of timestamps of adjacent pixels. have. A scheme for regenerating such a timestamp will be described later in detail.

The processor 120 may include an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a dedicated microprocessor, a microprocessor, etc. implemented to execute a timestamp regenerating scheme. Alternatively, the processor 120 may include a general purpose processor. In this case, the scheme for regenerating the timestamp may be performed by a host processor (eg, an application processor, etc.) connected to the processor 120 .

The image processing apparatus 100 of the present invention may improve the performance of the image processing apparatus 100 by regenerating a timestamp of an event signal output from a bad pixel among pixels constituting the vision sensor 110 . Since a bad pixel can be treated similarly to a normal pixel by this scheme, the effect of improving the yield of the pixel array constituting the vision sensor 110 can be achieved.

FIG. 2 is a block diagram exemplarily showing the configuration of the vision sensor 110 shown in FIG. 1 . Referring to FIG. 2 , the vision sensor 110 may include a pixel array 111 and an event detection circuit 112 . The event sensing circuit 112 may be configured to process events in which the intensity of light increases or decreases, which is sensed by the pixel array 112 . For example, the event detection circuit 112 may include at least one or more of various components such as an address event representation (AER), a sampler, a packetizer, and a scanner. Specific configurations of the event detection circuit 112 will be described in detail with reference to FIGS. 3A to 7A .

The pixel array 111 may include a plurality of pixels PX arranged in a matrix form along M rows and N columns. The pixel array 111 may include a plurality of pixels configured to detect events in which the intensity of light increases or decreases. For example, each pixel may be connected to the event sensing circuit 112 through a column line in a column direction and a row line in a row direction. A signal indicating that an event has occurred in the pixel may be transmitted to the event detection circuit 112 through, for example, a column line. Polarity information of an event occurring in each pixel (ie, whether it is an on-event in which the intensity of light increases or an off-event in which the intensity of light decreases) is, for example, determined by the event detection circuit 112 through a column line. ) can be transferred to

The event sensing circuit 112 may be configured to process the events that have occurred. For example, the event detection circuit 112 may generate a timestamp including time information when the event occurred. For example, the event detection circuit 112 may reset the pixel by transferring the reset signal RST to the pixel in which the event has occurred. Furthermore, the event detection circuit 112 may generate a packet or frame including polarity information of the event, an address ADDR of a pixel in which the event occurred, and a timestamp. The packets or frames generated by the event sensing circuitry 112 may be processed by a processor ( FIGS. 1 , 120 ) configured to implement the timestamp regeneration scheme to be described herein.

According to these exemplary configurations, events occurring in the pixel array 111 may be processed in units of pixels, may be processed in units of pixel groups including a plurality of pixels, may be processed in units of columns, or may be processed in units of frames. . However, these embodiments only mean that events sensed through the pixel array 111 may be processed in various ways, and the technical concept described herein is not limited to these configurations.

FIG. 3A is a block diagram showing an exemplary configuration of the vision sensor 110 shown in FIG. 2 . The vision sensor 110a may include a pixel array 111a, a column AER 113a, a row AER 115a, and a packetizer and input/output circuit 117a.

Among the plurality of pixels constituting the pixel array 111a, a pixel in which an event has occurred may transmit a signal (column request; CR) indicating that an event of increasing or decreasing light intensity has occurred to the column AER 113a. .

The column AER 113a may transmit a response signal ACK to the pixel in which the event occurs in response to the column request CR received from the pixel in which the event occurs. The pixel receiving the response signal ACK may transmit polarity information Pol of the event to the row AER 115a. Furthermore, the column AER 113a may generate the column address C_ARRD of the pixel in which the event occurs based on the column request CR received from the pixel in which the event occurs.

The row AER 115a may receive polarity information Pol from a pixel in which an event has occurred. The row AER 115a may generate a Timestamp TS including information about a time when an event occurs based on the polarity information Pol. For example, the timestamp may be generated by the timestamp 116a provided in the row AER 113a. For example, the time stamper 116a may be implemented using a time tick generated in units of several to tens of microseconds. The row AER 115a may transmit the reset signal RST to the pixel in which the event occurs in response to the polarity information Pol. The reset signal RST may reset a pixel in which an event has occurred. Furthermore, the row AER 115a may generate a row address R_ARRD of a pixel in which an event has occurred.

The row AER 115a may control a period in which the reset signal RESET is generated. For example, the AER logic may control the period in which the reset signal RESET is generated so that the event does not occur during a specific period in order to prevent the workload from increasing due to the occurrence of too many events. That is, the AER logic may control the refractory period of event generation.

The packetizer and input/output circuit 117a may generate a packet based on the timestamp TS, the column address C_ADDR, the row address R_ADDR, and the polarity information Pol. The packetizer and input/output circuit 117a may add a header indicating the start of the packet to the front end of the packet, and a tail indicating the end of the packet to the rear end of the packet.

FIG. 3B is a diagram exemplarily showing the format of information output from the configuration of the exemplary vision sensor illustrated in FIG. 3A . For better understanding, reference is also made to FIG. 3A.

The timestamp TS may include information about the time when the event occurred. For example, the timestamp TS may consist of 32 bits, but is not limited thereto.

Each of the column address C_ADDR and the row address R_ADDR may consist of 8 bits. Therefore, it is possible to support a vision sensor comprising a plurality of pixels arranged in up to 8 rows and 8 columns. However, this is only an example, and the number of bits of the column address (C_ADDR) and the row address (R_ADDR) may vary according to the number of pixels.

The polarity information Pol may include information about an on-event and an off-event. For example, the polarity information Pol may include one bit including information on whether an on-event has occurred and one bit including information about whether an off-event has occurred. For example, a bit indicating an on-event and a bit indicating an off-event may not both be '1', but may both be '0'.

A packet output from the packetizer and input/output circuit 117a may include a timestamp TS, a column address C_ADDR, a row address R_ADDR, and polarity information Pol. However, the arrangement order is not limited thereto.

FIG. 4A is a block diagram showing an exemplary configuration of the vision sensor 110 shown in FIG. 2 . The vision sensor 110b may include a pixel array 111b , a column AER 113b , a row AER 115b , a control logic 114b , and a packetizer and input/output circuit 117b .

Unlike the previous embodiment of FIG. 3A , the plurality of pixels may be grouped into a plurality of groups each including at least two or more pixels. Illustratively, a Kth pixel group including 8 pixels arranged in the same column is shown. Among pixels belonging to the K-th pixel group, a pixel in which an event has occurred may transmit a column request CR indicating that an event has occurred to the column AER 113b.

The column AER 113b may transmit a response signal ACK to the pixel in which the event occurs in response to the column request CR received from the pixel in which the event occurs. Pixels belonging to the K-th pixel group to which the pixel receiving the response signal ACK belongs may transmit polarity information Pol to the row AER 115b. Furthermore, the column AER 113b may generate the column address C_ARRD of the pixel group in which the event has occurred based on the column request CR received from the pixel in which the event has occurred.

The control logic 114b may transmit the column address C_ADDR to the row AER 115b.

The row AER 115b may receive polarity information Pol(s) from pixels of a pixel group to which the pixel in which the event occurs belongs. The raw AER 115b (eg, the time stamper 116b) is based on the polarity information (Pol(s)), the timestamps (TS(s)) including information about the time when the events occurred. can create The row AER 115b may transmit the reset signals RST(s) to the pixels of the pixel group to which the pixel in which the event occurs in response to the polarity information Pol(s), respectively. The reset signals RST(s) may reset pixels belonging to the Kth pixel group. The row AER 115b may generate a group address GP_ARRD of the Kth pixel group. The row AER 115b may control a period in which the reset signal RESET is generated.

Furthermore, the row AER 115b may generate a group address GP_ADDR of a group to which the pixel in which the event occurs belongs, based on the column address C_ADDR. A correspondence relationship between the group address GP_ADDR of the K-th pixel group and column/row addresses of pixels belonging to the K-th pixel group may be predetermined. For example, the group address of the first pixel group may be predetermined as indicating addresses of pixels from the first row of the first column to the eighth row of the first column. Therefore, as in the present embodiment, when the group address GP_ADDR is used, a separate column address or row address may not be included in the packet.

The packetizer and input/output circuit 117b may generate a packet based on the timestamps TS(s), the column address C_ADDR, the group address GP_ADDR, and the polarity information Pol.

4B is a diagram exemplarily showing the format of information output from the configuration of the exemplary vision sensor shown in FIG. 4A . For better understanding, reference is also made to FIG. 4A.

A packet output from the packetizer and input/output circuit 117b may include timestamps TS(s), a group address GP_ADDR, on-event information, and off-event information. However, the arrangement order is not limited thereto. The on-event information and the off-event information may be referred to as polarity information.

The timestamps TS(s) may include time information of an event occurring in each pixel belonging to the Kth group. The group address GP_ADDR may indicate a unique address of a pixel group to which a pixel in which an event occurs belongs. For example, although the group address GP_ADDR is illustrated as being composed of 6 bits, the present invention is not limited thereto. That is, the number of bits of the group address GP_ADDR may vary according to the number of pixels constituting the pixel array 111b.

Each of the on-event and the off event may consist of 8 bits. For example, a first bit among bits indicating an on-event may indicate event information of a first pixel belonging to a Kth pixel group (eg, a pixel indicated by '1' among pixels of FIG. 4A ). Similarly, an 8th bit among bits indicating an on-event may indicate event information of an 8th pixel belonging to a Kth pixel group (eg, a pixel indicated by '8' among pixels in FIG. 4A ). . Off-events are similar.

FIG. 5A is a block diagram showing an exemplary configuration of the vision sensor 110 shown in FIG. 2 . The vision sensor 110c may include a pixel array 111c, a column AER 113c, a control logic 114c, a row sampler 115c, and a packetizer and input/output circuit 117c.

Unlike the previous embodiment of FIG. 3A , a plurality of pixels may be grouped in a column unit. For example, the pixel in which the event has occurred may transmit a column request CR indicating that the event has occurred to the column AER 113c.

The column AER 113c may transmit a response signal ACK to the pixel in which the event occurs in response to the column request CR received from the pixel in which the event occurs. Pixels disposed in the same column as the column to which the pixel receiving the response signal ACK belongs may transmit polarity information Pol to the row sampler 115c. Furthermore, the column AER 113c may generate a column address C_ARRD of a column to which the event pixel belongs based on the column request CR received from the event pixel.

The control logic 114c may request the row sampler 115c to sample pixels arranged in the same column as the pixel in which the event occurred. The control logic 114c may transmit the column address C_ADDR.

The row sampler 115c may receive polarity information Pol(s) from pixels arranged in the same column as the pixel in which the event occurred. The raw sampler 115c (eg, the time stamper 116c) is based on the polarity information Pol(s). Timestamps TS(s) including information about the time at which events occurred can create The row sampler 115c may transmit reset signals RST(s) to pixels disposed in the same column as the pixel in which the event occurred in response to the polarity information Pol(s), respectively. The reset signals RST(s) may reset pixels disposed in the same column as a pixel in which an event occurs. The row sampler 115c may control a period in which the reset signal RESET is generated.

FIG. 5B is a diagram exemplarily showing the format of information output from the configuration of the exemplary vision sensor illustrated in FIG. 5A . For better understanding, reference is also made to FIG. 5A.

A packet output from the packetizer and input/output circuit 117c may include timestamps TS(s), a column address C_ADDR, on-event information, and off-event information. However, the arrangement order is not limited thereto.

Each of the on-event and the off event may consist of M bits. For example, a first bit among bits representing the on-event may represent event information of a pixel in one row among pixels disposed in the same column as a pixel in which the event occurs. Similarly, an M-th bit among bits representing the on-event may represent event information of a pixel in row M among pixels disposed in the same column as a pixel in which the event occurs. Off-events are similar.

FIG. 6A is a block diagram showing an exemplary configuration of the vision sensor 110 illustrated in FIG. 2 . The vision sensor 110d may include a pixel array 111d, a column scanner 113d, a control logic 114d, a row sampler 115d, and a packetizer and input/output circuit 117d.

Unlike the previous embodiments, the pixel in which the event has occurred may not transmit a signal (eg, a column request) indicating that the event has occurred to the column scanner 113d. And, unlike the previous embodiments, in the present embodiment, pixels are not grouped.

The column scanner 113d may sequentially and periodically scan from column 1 to row N of the pixel array 111d. In addition, the column scanner 113d may generate a column address C_ADDR of a column to be scanned and transmit it to the control logic 114d.

The control logic 114d may request the row sampler 115d to sample pixels arranged in a column scanned by the column scanner 113d. Furthermore, the control logic 114d may transmit the column address C_ADRR of the scanned column to the row sampler 115d.

The row sampler 115d may receive polarity information Pol(s) from pixels arranged in a scanned column. The raw sampler 115d (eg, the time stamper 116d) is based on the polarity information Pol(s). Timestamps TS(s) including information about the time at which events occurred can create The row sampler 115d may transmit reset signals RST(s) to pixels arranged in a column to be scanned in response to the polarity information Pol(s), respectively. The reset signals RST(s) may reset pixels arranged in a column to be scanned. The row sampler 115d may control a period in which the reset signal RESET is generated.

FIG. 6B is a diagram exemplarily showing the format of information output from the configuration of the exemplary vision sensor illustrated in FIG. 6A . For better understanding, reference is also made to FIG. 6A.

A packet output from the packetizer and input/output circuit 117d may include information related to all events occurring in the pixel array 111d. For example, the packet may include information related to the first column to information related to the Nth column. For simplicity of illustration, among the packets output from the packetizer and the input/output circuit 117d, information related to the first column and information related to the N-th column are shown.

First, a portion of a packet related to the first column may include timestamps TS(s), a first column address 1st C_ADDR, on-event information, and off-event information. However, the arrangement order is not limited thereto.

The timestamps TS(s) may include time information of events occurring in pixels disposed in the first column. Each of the on-event and the off event may consist of M bits. For example, a first bit among bits representing the on-event may represent event information of a pixel in row 1 of the first column. Similarly, an M-th bit among bits indicating an on-event may indicate event information of a pixel in an M-th row of the first column. Off-events are similar.

The above-described configuration may be equally applied to a portion corresponding to the second column to a portion corresponding to the N-th column of the packet. Therefore, detailed description will be omitted.

FIG. 7A is a block diagram showing an exemplary configuration of the vision sensor 110 shown in FIG. 2 . FIG. 7B is a diagram conceptually illustrating an exemplary format of a frame output from the vision sensor of FIG. 7A .

The vision sensor 110e may include a pixel array 111e, a column scanner 113e, a control logic 114e, a row sampler 115e, and a frame generator and input/output circuit 117e. The configuration and operation of the vision sensor 110e are substantially similar to the vision sensor described above with reference to FIG. 6A . Therefore, the overlapping description will be omitted. However, the vision sensor 110e may be configured to generate a synchronous-based frame instead of an asynchronous-based packet.

For example, the frame generator and input/output circuit 117e includes polarity information (TS(s)), column addresses (C_ADDR(s)), which are periodically obtained by the column scanner 113e and the row sampler 115e, A frame may be generated using the row addresses R_ADDR(s) and the polarity information Pol(s).

For example, when the column scanner 113e scans the pixel array 111e with a constant cycle, frames generated by the frame generator and the input/output circuit 117e will also be generated periodically. Therefore, the vision sensor 110e may operate synchronously. In an embodiment, when the vision sensor 110e operates synchronously, the low sampler 115e may control a period in which the reset signal RESET is generated. For example, if the frame rate is 60 frames/sec, the generation period of the reset signal RST may be 1/60 second.

However, the frame-based vision sensor 110e does not always operate synchronously. For example, the amount of events that occur may be different each time. For example, one frame may contain a very small amount of events, while another frame may contain a fairly large amount of events. For example, if the amount of generated events is small, the generation period of the reset signal RST may be reduced to increase the frame generation rate. On the other hand, if the amount of generated events is large, the frame generation rate may be reduced by increasing the generation period of the reset signal RST. In this case, since the generation period of the frame may be changed at any time, the frames may be output asynchronously.

Some examples of the event detection circuit 112 shown in FIG. 2 have been exemplarily described with reference to FIGS. 3A to 7A . However, these embodiments only mean that events sensed through the pixel array may be processed in various ways, and the technical concept described herein is not limited to these configurations.

8 is a block diagram schematically illustrating the configuration of the pixel illustrated in FIGS. 3A to 7B . The pixel PX may include a sensing unit 118 and an in-pixel circuit 119 . In some embodiments, the sensing unit 118 may be referred to as a functional block for sensing a change in light. In addition, the in-pixel circuit 119 may be referred to as a functional block that processes a change in the sensed light as an analog signal or a digital signal. For example, the in-pixel circuit 119 may be configured as an analog circuit, a digital circuit, or a combination thereof.

9 is a circuit diagram exemplarily illustrating the configuration of the pixel shown in FIG. 8 . A pixel may include a sensing unit 118 and an in-pixel circuit 119 .

The sensing unit 118 may include a photodiode PD. The photodiode PD may convert light energy into electrical energy. That is, electrical energy also changes due to a change in the intensity of light, and whether an event occurs may be determined by sensing the changed electrical energy.

The in-pixel circuit 119 may include a capacitor C, a differential amplifier DA, a comparator CMP, and a readout circuit. The in-pixel circuit 119 may further include a transfer gate transistor TG for supplying power and a switch SW for resetting the pixel after processing for one event is completed.

The capacitor C may store electrical energy generated by the photodiode PD. For example, the capacitance of the capacitor C may be appropriately selected in consideration of the shortest time (eg, a refractory period) between two events that may continuously occur in one pixel.

When the switch SW is switched on by the reset signal RESET, the charge stored in the capacitor C is discharged, so that the pixel may be initialized. For example, the reset signal RST may be received from a row AER ( FIGS. 3A and 4A ), a row sampler ( FIGS. 5A , 6A , 7A ), etc. described with reference to FIGS. 3A to 7A .

The differential amplifier DA may amplify the voltage level by the charge stored in the photodiode PD. This is to facilitate the operation of determining the type of event, performed by the comparator CMP.

The comparator CMP may compare the output voltage of the differential amplifier FA with the level of the reference voltage Vref to determine whether an event occurring in the pixel is an on-event or an off-event. For example, when the intensity of light increases, the comparator CMP may output a signal ON indicating an on-event. Conversely, when the intensity of light decreases, the comparator CMP may output a signal OFF indicating that it is an off-event.

The read circuit may transmit information about an event occurring in the pixel (ie, whether it is an on-event or an off-event). On-event information or off-event information, referred to as polarity information, may be transmitted to the row AER ( FIGS. 3A and 4A ), the row sampler ( FIGS. 5A, 6A, 7A ), etc. described through FIGS. 3A to 7A . .

The configuration of the pixel shown in this embodiment is exemplary. That is, a configuration that detects a change in light (eg, a photodiode), a configuration that stores electrical energy (eg, C), a configuration that determines the type of event generated from the stored electrical energy (eg, CMP), and a configuration related to the event An example of a configuration (eg, a read circuit) for generating the most basic information is illustrated, and various configurations of pixels for determining the type of event by sensing a change in light intensity may be applied to the present invention.

10A is a cross-sectional view illustrating a semiconductor package of the image processing apparatus described with reference to FIGS. 1 to 9 . Referring to FIG. 10A , the semiconductor package 10 may include a lower package 11 and an upper package 15 .

The lower package 11 may include a lower substrate 12 and a lower semiconductor chip 13 . For example, the lower semiconductor chip 13 includes components (eg, 113a, 115a, 117a, 113b, 114b, 115b, 117b, 113c, 114c, 115c, 117c, 113d, 114d, 115d, 117d, 113e, 114e, 115e, 117e) and the in-pixel circuit 119 of FIG. 9 . The lower package 11 may be a flip chip device in which the lower semiconductor chip 13 is mounted face down on the lower substrate 12 .

The lower substrate 12 may be a printed circuit board (PCB) having a circuit pattern. The external terminal provided on the lower surface of the lower substrate 12 may electrically connect the upper semiconductor chip 17 and/or the lower semiconductor chip 13 to an external electric device (not shown). For example, the upper semiconductor chip 17 and the external terminal may be electrically connected through internal wirings and/or through silicon via (TSV). For example, when the TSV is provided, the upper substrate 16 on which the upper semiconductor chip 17 is mounted may be omitted.

The lower semiconductor chip 13 may be mounted on the lower substrate 12 . The lower semiconductor chip 13 may be sealed by a molding layer (not shown). The molding layer may include an insulating polymer material such as an epoxy molding compound.

The upper package 15 may include an upper substrate 16 and an upper semiconductor chip 17 . For example, the upper semiconductor chip 17 may include the sensing unit 118 of FIG. 3 . That is, the upper semiconductor chip 17 may include a component (eg, a photodiode) that detects a change in light among the pixel array 111 of FIG. 2 .

The upper substrate 16 may be a printed circuit board (PCB) having a circuit pattern. The upper semiconductor chip 17 may be sealed by a molding layer (not shown).

10B is a cross-sectional view illustrating a semiconductor package of the image processing apparatus described with reference to FIGS. 1 to 9 . Referring to 10b , the semiconductor package 20 may include a lower package 21 and an upper package 22 . The lower package 21 may include a lower substrate 22 and a lower semiconductor chip 23 . The upper package 25 may include an upper substrate 26 and an upper semiconductor chip 27 . The semiconductor package of this embodiment is substantially similar to the semiconductor package described with reference to FIG. 10A except that it is packaged by a wire bonding method. Therefore, detailed description will be omitted.

10C is a cross-sectional view illustrating a semiconductor package of the image processing apparatus described with reference to FIGS. 1 to 9 . Referring to FIG. 10C , the semiconductor package 30 may include a package substrate 31 , a first semiconductor chip 33 , a second semiconductor chip 35 , and a third semiconductor chip 37 . The semiconductor package of this embodiment is substantially similar to the semiconductor package described with reference to FIG. 10A , except that a semiconductor chip including a memory device is further packaged. Therefore, the overlapping description will be omitted.

The first semiconductor chip 33 may include a memory. Data necessary for the image processing apparatus of the present invention to perform an operation may be stored in a memory implemented by the first semiconductor chip 33 . For example, data processed by the image processing apparatus may be stored in a memory implemented by the first semiconductor chip 33 . For example, the first semiconductor chip 33 may include a volatile memory such as a dynamic RAM (DRAM), a synchronous RAM (SDRAM), and/or a phase-change RAM (PRAM), a magneto-resistive RAM (MRAM), a resistive RAM (ReRAM), or the like. RAM), non-volatile memory such as Ferro-electric RAM (FRAM).

The second semiconductor chip 35 includes components (eg, 113a, 115a, 117a, 113b, 114b, 115b, 117b, 113c, 114c, 115c, 117c, 113d, 114d, 115d, 117d, 113e, 114e, 115e, 117e) and the in-pixel circuit 119 of FIG. 9 . The third semiconductor chip 37 may include pixel arrays 111a, 111b, 111c, 111d, and 111e among the components of the vision sensor of FIGS. 3A to 7B .

10D is a cross-sectional view illustrating a semiconductor package of the image processing apparatus described with reference to FIGS. 1 to 9 . Referring to FIG. 10D , the semiconductor package 40 may include a package substrate 41 , a first semiconductor chip 43 , a second semiconductor chip 45 , and a third semiconductor chip 47 . For example, the first semiconductor chip 43 , the second semiconductor chip 45 , and the third semiconductor chip 47 include the second semiconductor chip 35 , the first semiconductor chip 33 , and the third semiconductor chip 37 , respectively. That is, except for the order in which the semiconductor chips are stacked, this embodiment is substantially similar to the embodiment described with reference to FIG. 10C . Therefore, the overlapping description will be omitted.

11 is a block diagram exemplarily illustrating an operation of the processor 120 shown in FIG. 1 . In addition to the processor 120 , a memory 130 is also shown. For example, the memory 130 may be a memory implemented with the first semiconductor chip 33 or the second semiconductor chip 45 illustrated in FIG. 10C or 10D . Alternatively, the memory 130 may be a buffer or cache memory provided together with the processor 120 . The processor 120 may include a timestamp generator 122 . In order to help the understanding of the description, it will be described with reference to FIGS. 1 to 9 together.

The memory 130 may store packets or frames received from the vision sensor 110 . For example, as described above with reference to FIGS. 3B to 7B, a packet or frame includes a timestamp (TS), a column address (C_ADDR), a row address (R_ADDR), a group address (GP_ADDR), and on-event information and It may include some of the polarity information Pol including off-event information.

The processor 120 may generate an on-event map and an off-event map using the timestamp TS, the address ADDR, and the polarity information Pol. The on-event map and the off-event map may be stored in the memory 130 . The on-event map may include coordinates of pixels in which the on-event in which the intensity of light increases, and information about the time when the event occurred. In Ti-1 and j of the on-event map, i-1 and j denote coordinates of pixels constituting the pixel array 111, and T denotes a time when the on-event occurs. The case of off-events would be similar.

The timestamp generator 122 uses a temporal correlation of timestamps of adjacent pixels to remove a bad pixel such as a noise pixel, a hot pixel, or a dead pixel. You can recreate the timestamp of the event that occurred. A scheme for recreating the timestamp will be described in detail below.

12 is a diagram conceptually illustrating that the timestamp generator 122 shown in FIG. 11 regenerates timestamps of noise pixels or hot pixels. Illustratively, timestamps of 9 pixels arranged in three by three are shown as part of the on-event map or the off-event map shown in FIG. 11 . An event marked with time stamp '1' may indicate an event that occurred during a first time period, and an event marked with time stamp '2' may indicate an event that occurred during a second time period. In order to help the understanding of the description, it will be described with reference to FIGS. 1 to 11 together.

Among the nine pixels, the shaded pixels, arranged in row 2, column 2, indicate the target pixel. The noise pixel referred to in the present embodiment refers to a pixel in which an event is displayed even though an event has not actually occurred due to noise generated outside the pixel itself or the pixel. In addition, a hot pixel refers to a pixel that is always displayed as an event occurring due to a defect in the pixel itself. Since the noise pixel and the hot pixel are similar to each other in that an event is displayed even though the event has not actually occurred, they will be treated similarly in this specification.

The processor 120 may determine a temporal correlation between an event occurring in the target pixel and events occurring in pixels surrounding the target pixel by observing timestamps of the target pixel and eight pixels surrounding the target pixel. For example, if a first event occurs in a target pixel during a first time period, and after a reference time elapses, a second event occurs during a second time period in adjacent pixels around the target pixel, the first event and The second event may be referred to as having a weak temporal correlation. In this embodiment, “adjacent pixels” may be referred to to refer to eight pixels around a target pixel.

The processor 120 may group the timestamps appropriately based on temporal correlation. The processor 120 may group pixels marked with a timestamp '1' arranged in one row and three columns into a first group. Hereinafter, in describing the pixels disposed in the x-row and the y-column in the present specification, the x-row and the y-column will be denoted as [x, y]. Pixels placed in [1, 2] and [2, 3] have timestamp '2', so events that occurred at the target pixel and pixels placed at [1, 2] and [2, 3] Events have the same temporal correlation. Therefore, the processor 120 may group the pixels disposed in [1, 2] and [2, 3] into the second group. Pixels placed at [1, 1], [2, 2], [3, 3] have timestamp '0', so the event that occurred at the target pixel, [1, 1], [2, 2], Events occurring in pixels arranged in [3, 3] have the same temporal correlation. Therefore, the processor 120 may group the pixels disposed in [1, 1], [2, 2], and [3, 3] into a third group.

The processor 120 may determine the outline of the object based on the classified groups. For example, the addresses of pixels marked with timestamp '2' placed at [1, 2], [2, 3], and at [1, 1], [2, 2], [3, 3] Based on the addresses of the pixels marked with the arranged timestamp '0', the outline of the object moving from the left to the lower right may be determined.

Furthermore, the processor 120 may determine the movement direction of the object based on the timestamp of the classified group. For example, the processor 120 determines that the object is moved from the top right to the bottom left on the basis that the timestamps of the pixels belonging to the first group are '1' and the timestamps of the pixels belonging to the second group are '2'. moving can be considered. Also, the processor 120 may determine the velocity and/or acceleration of the object by referring to the moving direction of the object and the positions and/or values of the grouped timestamps.

Alternatively, the processor 120 may determine the moving direction of the vision sensor 110 based on the timestamp of the classified group. This is because the same timestamp as shown in FIG. 12 can be obtained even when the object is stationary and the vision sensor 110 moves from the lower left to the upper right. In this case, the processor 120 may determine the speed of the vision sensor 110 by referring to the moving direction of the vision sensor 110 and positions and/or values of the grouped timestamps. However, this is only due to relativity, and the present invention can be equally applied even when only the vision sensor 110 moves or both the object and the vision sensor 110 move.

The processor 120 may determine a temporal correlation between an event occurring in the target pixel and events occurring in pixels included in group 3 to which the target pixel belongs. For example, since the timestamps of events occurring in the target pixel and the pixels arranged in [1, 1] and [3, 1] are all '0', the processor 120 determines that the occurrence of the event has ended. can

Furthermore, the processor 120 may determine a temporal correlation between the event occurring in the target pixel and the event occurring in the pixel disposed at [3, 1]. However, considering that the object is moving from the top right to the bottom left and the event occurring in the pixels belonging to the second group is marked with timestamp '2' and the occurrence of the event is terminated, the event occurring in the target pixel and The temporal correlation of events occurring in the pixels placed in [3, 1] will be very low. That is, an event occurring in the pixel placed in [3, 1] is most likely due to noise or a defect in the pixel itself (ie, a hot pixel). As a result, the processor 120 may determine the pixel disposed at [3, 1] as a noise pixel or a hot pixel based on the temporal correlation.

The timestamp generator 122 may replace the timestamp (ie, 4) of the pixel determined to be a noisy pixel or a hot pixel with the timestamp (ie, 0) of the target pixel. That is, the timestamp generator 122 may update the timestamp of the on-event map or the off-event map stored in the memory 130 .

However, when the direction of movement of the object (or vision sensor) is considered, the timestamp of the pixel arranged at [3, 1] shown in FIG. 12 is suddenly marked as '4', so this is an abnormality caused by a defect in the pixel itself. It can be seen that this is due to the event. However, there may be cases where it is difficult to determine whether the timestamp is due to the occurrence of a normal event, or whether it is due to a noisy pixel or a hot pixel. In this case, the operation of the timestamp generator 122 will be described with reference to FIGS. 13 and 14 .

13 is a diagram conceptually showing that the timestamp generator 122 of the present invention determines a noise pixel or a hot pixel. Unlike the exemplary embodiment described above with reference to FIG. 12 , in order to more accurately determine a noisy pixel or a hot pixel, adjacent pixels around the target pixel may be considered. In this embodiment, “adjacent pixels” may be referred to to refer to 24 pixels around the target pixel.

The processor 120 may determine the temporal correlation by observing the timestamps of the target pixel and 24 pixels around the target pixel. Similar to that described above with reference to FIG. 12 , pixels in which an event marked with timestamp '1' occurred may be grouped into group 1, and pixels in which an event marked with timestamp '2' occurred may be grouped into group 2. Also, pixels in which an event marked with a timestamp '3' occurs may be grouped into group 3.

The processor 120 may determine the outline of the object based on that pixels belonging to the second group have timestamp '2' and pixels belonging to the third group have timestamp '3' . Furthermore, the processor 120 may determine that the object moves from the top right to the bottom left, based on the timestamps belonging to the first to third groups.

The processor 120 may determine a temporal correlation between the event occurring in the target pixel and the event occurring in the pixels belonging to the first to third groups, and the occurrence of the event after the events occur in the pixels belonging to the third group It can be considered that this has ended.

Furthermore, the processor 120 may determine a temporal correlation between the event occurring in the target pixel and the event occurring in the pixel disposed in [4, 2]. If, as in the embodiment described above with reference to FIG. 12 , if the processor 120 considers only the timestamps of 8 pixels around the target pixel, the timestamp '2' of the pixel disposed in [4, 2] is noise It is likely due to pixels or hot pixels.

However, the processor 120 places pixels at [4, 1], [5, 2] and [5, 1] to prevent the pixels placed at [4, 2] from being incorrectly judged as noise pixels or hot pixels. Time stamps can be further observed. According to the timestamps shown in Fig. 13, it is reasonable to assume that a new event has occurred in the pixels arranged in [4, 2], [4, 1], [5, 2] and [5, 1]. Rather, considering the direction in which the object moves, it is possible that the target pixel placed at [3, 3] is a dead pixel.

In conclusion, regardless of whether the target pixel is a dead pixel, the pixel placed in [4, 2] should not be judged as a noise pixel or a hot pixel. The processor 120 may determine a pixel disposed in [4, 2] as a normal pixel by expanding the range of pixels to be observed. In this case, the timestamp generator 122 will not work.

14 is a diagram conceptually illustrating that the processor 120 of the present invention determines a noise pixel or a hot pixel. Similar to FIG. 13 above, 24 pixels around the target pixel may be considered.

The processor 120 may group the pixels similarly to the previous embodiments. The processor 120 may determine that the outline of the object and the object move from top to bottom left in consideration of temporal correlations between events occurring in the target pixel and events occurring in pixels around the target pixel.

In addition to the timestamps of the pixels disposed in [4, 2], the processor 120 further observes the timestamps of pixels disposed in [4, 1], [5, 2] and [5, 1] for temporal correlation can be judged For example, since the timestamp of an event occurring in the target pixel is '0' and the timestamp of events occurring in the pixels disposed in [4, 2] is '1', the event occurring in the target pixel and the event occurring in [4, 2] There is some temporal correlation between the events that occurred in the pixels. That is, the processor 120 may determine that an event caused by a minute movement of a very small object has occurred in the pixel arranged in [4, 2].

However, in a special case, the event detected in the pixel arranged in [4, 2] may be due to very fine noise or a defect (hot pixel) of the pixel itself. Therefore, the processor 120 determines that very fine noise has occurred in the pixel disposed at [4, 2] because the timestamps of the pixel disposed at [4, 2] and the 8 pixels adjacent to it are all '0'. may be

The timestamp generator 122 may replace the timestamp of the pixel disposed in [4, 2] with '0' based on the determination of the processor 120 . This is because if the pixel placed in [4, 2] is a noisy pixel or a hot pixel, the timestamp should be replaced with '0'. Even if it is an event caused by a minute movement of a very small object in the pixel disposed in [4, 2], it is considered that the event has not occurred or an approximation is possible.

15 is a diagram conceptually illustrating that the timestamp generator 122 of the present invention regenerates timestamps of noise pixels or hot pixels.

The processor 120 may appropriately group timestamps similarly to the above-described embodiments. The processor 120 may determine the direction in which the object moves, and determine that the event occurring in the target pixel is an abnormal event based on a temporal correlation between the event occurring in the target pixel and events occurring in pixels around the target pixel. can do.

The timestamp generator 122 may replace the timestamp (ie, 4) of the target pixel with timestamps (ie, 0) of pixels adjacent to the target pixel based on the determination result of the processor 120 . Such replacement may be performed by updating an on-event map or an off-event map stored in memory 130 .

16 is a diagram conceptually illustrating that the timestamp generator 122 of the present invention regenerates the timestamp of a dead pixel. Illustratively, timestamps of 9 pixels arranged as [3, 3] are shown as part of the on-event map or the off-event map shown in FIG. 11 . In order to help the understanding of the description, it will be described with reference to FIGS. 1 to 11 together.

The shaded pixel, placed at [2, 2] of the 9 pixels, represents the target pixel. The dead pixel referred to in the present embodiment refers to a pixel in which an event is not displayed even though an event has actually occurred due to the same reason as a defect in the pixel itself.

The processor 120 may group the pixels disposed in [1, 3] into group 1 . The processor 120 may group the pixels arranged in [1, 2] and [2, 3] in which the event marked with the timestamp '2' occurs into group 2 . The processor 120 may determine that the outline of the object and the object move from the top right to the bottom left, based on the time stamp changes of the classified groups and pixels belonging to the first and second groups.

The processor 120 may determine a temporal correlation between the target pixel and events occurring in pixels belonging to the second group and adjacent pixels (excluding the target pixel). That is, the processor 120 may determine a temporal correlation between the target pixel and events occurring in pixels disposed in [1, 1] and [3, 3]. Since the timestamp of the event that occurred in the target pixel is '0', and the timestamp of the events that occurred in the pixels placed in [1, 1] and [3, 3] is '4', the processor determines that the event that occurred in the target pixel is '0'. It can be considered as an unusual event. (i.e. dead pixels)

The timestamp generator 122 may replace the timestamp (ie, 0) of the target pixel determined as the dead pixel with a timestamp having a larger value among surrounding timestamps (ie, 4). This is based on the fact that the probability that timestamps of events occurring in two pixels adjacent to each other will change abruptly is very low in consideration of the motion and outline of the object.

17 is a diagram conceptually illustrating that the processor 120 of the present invention determines a dead pixel. Unlike the embodiment described above with reference to FIG. 16 , 24 pixels around the target pixel may be considered. This is in consideration of the fact that, although the target pixel is normal in the embodiment of FIG. 16 , the possibility that the pixels disposed in [1, 1] and [3, 3] are dead pixels cannot be excluded.

The processor 120 may group the pixels similarly to those described in the previous embodiments. For example, the processor 120 may group a pixel in which an event marked with a timestamp '1' occurs into group 1. The processor 120 may group pixels in which events marked with timestamp '2' occur into group 2 . Groups 3 to 4 are similar.

The processor 120 may determine the outline and movement direction of the object in consideration of timestamps of events occurring in pixels belonging to each group and changes in timestamps. For example, the processor 120 may determine that the object is moving from top to bottom left, and the outline of the object is a diagonal line from left to bottom right.

The processor 120 may determine a temporal correlation between an event occurring in the target pixel and events occurring in pixels adjacent to the target pixel. Considering the movement direction and outline of the object, the event occurring in the pixels arranged in [2, 2] and [4, 4] is an abnormal event, or the target pixel, [1, 1] and [5, 5] Events that occurred in pixels placed in ] will be abnormal events. In this case, the processor 120 may determine that the event occurring in the pixels disposed in [2, 2] and [4, 4] is an abnormal event. (i.e. noise pixels or hot pixels). This is because the defect is more likely to be in 2 pixels than 3 pixels.

The timestamp generator 122 replaces the timestamps in [2, 2] and [4, 4] determined to be noisy pixels or hot pixels with '0' based on the determination result of the processor 120. can do.

According to the scheme described above with reference to FIGS. 12 to 17 , an effect such that a normal event is generated from a noise pixel, a hot pixel, and a dead pixel can be achieved. Therefore, the performance of the image processing apparatus can be improved. In addition, since an effect such as improving the yield of the pixel array can be achieved, the manufacturing cost of the image processing apparatus can be lowered.

18 is a flowchart illustrating a timestamp regeneration scheme according to an embodiment of the present invention. In order to help the understanding of the description, it will be described with reference to FIGS. 12 to 17 .

In step S110, a plurality of timestamps may be generated. The timestamp may represent information about the time when an event occurred in the pixel.

In step S120 , a plurality of timestamps may be grouped into a plurality of groups. For example, events occurring in pixels disposed in [1, 2], [2, 3], [3, 4], and [4, 5] may be grouped into a fourth group. The first through third groups will each contain pixels grouped in a similar manner.

An outline and a movement direction of the object may be determined based on a timestamp belonging to each group and a change trend of the timestamp. For example, referring to FIG. 16 , it may be determined that the object is moving from the top right to the bottom left, and the outline of the object is a diagonal line from the top left to the bottom right.

In operation S130 , a temporal correlation between an event occurring in the target pixel and events occurring in pixels belonging to each group may be determined. Furthermore, a temporal correlation between an event occurring in the target pixel and events occurring in a pixel adjacent to a group in which the last event occurred may be further determined. [1, 1], [2, 2], [3, 3], [4, 4], [5 , 5] may be abnormal events. In this case, the processor 120 may determine that the pixels disposed in [2, 2] and [4, 4] are bad pixels (eg, noise pixels or hot pixels).

In operation S140 , a timestamp of the bad pixel may be regenerated based on the determination result performed in operation S130 . For example, the timestamp generator 122 of FIG. 17 may replace the timestamps of pixels disposed in [2, 2] and [4, 4] with '0'.

19 and 20 are diagrams illustrating a process in which the timestamp regeneration scheme of the present invention is applied to a sub sampling process. In order to help the understanding of the description, it will be described with reference to FIGS. 1 to 11 together.

First, referring to FIG. 19 , a plurality of pixels may be grouped into a plurality of groups GP1 to GP9. For example, the first group GP1 includes pixels disposed in [1, 1], [1, 2], [2, 1], and [2, 2]. The second group GP2 includes pixels arranged in [1, 3], [1, 4], [2, 3], and [2, 4]. Each of the remaining groups GP3 to GP9 may similarly include a plurality of pixels. Although exemplarily four pixels are shown to be grouped into one group, the number of pixels belonging to each group may be determined to match the resolution to be implemented through sub-sampling.

In the present invention, sub-sampling may refer to a technique for reducing data throughput by reducing the number of pixels in a pixel. For example, if 6×6 (six by six) pixels are grouped into a plurality of groups (GP1 to GP9) and a representative value is selected from timestamp values of pixels included in each group, 3×3 ( It can be treated as an event occurring at pixels of three by three).

For example, by subsampling, a timestamp of a most recent event among timestamps of pixels belonging to each group may be selected as a representative value of the group. However, a timestamp of an event that occurs first may be selected as a representative value, or a timestamp of an intermediate value may be selected as a representative value. For example, among the events occurring in the pixels belonging to the first group GP1, the most recent event occurred in the pixels marked '6'. Therefore, the sub-sampled timestamp of the first group GP1 may be '6'. Timestamps may be sub-sampled for the remaining groups GP2 to GP9 in a similar manner. The sub-sampled timestamps of each group are indicated by circled numbers.

Meanwhile, it is assumed that pixels disposed in [6, 2] belonging to the seventh group GP7 are noise pixels or hot pixels. Although the occurrence of events in the pixels marked with the timestamp '6' ends, the sub-sampled timestamp of the seventh group GP7 becomes '4' by the sub-sampling described above. Even when sub-sampling is applied, the timestamp regeneration scheme of the present invention may be equally applied.

Referring to FIG. 20 , the subsampled timestamps of the second group GP3 are '3', the subsampled timestamps of the second and sixth groups GP2 and GP6 are '5', and the first, Considering that the sub-sampled timestamps of the 5 and 9 groups GP1, GP5, and GP9 are '6', the processor 120 may determine the movement direction of the object (or the vision sensor). In addition, considering that the subsampled timestamps of the fourth and eighth groups GP4 and GP8 are '0', the processor 120 may determine that the occurrence of the event has ended.

Furthermore, the processor 120 may determine a temporal correlation between timestamps of the plurality of groups GP1 to GP9. For example, the subsampled timestamps of the fourth and eighth groups GP4 and GP8 are '0', but based on the point that the subsampled timestamps of the seventh group GP7 are '4', the processor 120 may determine that the sub-sampled timestamp '4' of the seventh group GP7 is due to an abnormal event. This is because it is abnormal for the sub-sampled timestamp to rapidly change from '0' to '4' when the direction of movement of the object (or vision sensor) is considered.

The timestamp generator 122 may regenerate the subsampled timestamp of the seventh group GP7 having an abnormal representative value. For example, the timestamp generator 122 may replace the subsampled timestamp of the seventh group GP7 with an adjacent subsampled timestamp. For example, the subsampled timestamp '4' of the seventh group GP7 is the subsampled timestamp '0' of the fourth and eighth groups GP4 and GP8 adjacent to the seventh group GP7. will be replaced

21 is a block diagram illustrating an electronic device to which an image processing apparatus according to an embodiment of the present invention is applied. For example, the electronic device 1000 may be implemented as a smart phone, a tablet computer, a desktop computer, a laptop computer, or a wearable device. Furthermore, the electronic device 1000 may be implemented as one of various types of electronic devices required to operate an unmanned security system, the Internet of Things, and an autonomous vehicle.

The electronic device 100 may include an image processing device 1100 , a main processor 1200 , a working memory 1300 , a storage 1400 , a display 1500 , a communication block 1600 , and a user interface 1700 . can

The image processing apparatus 1100 may be an image processing apparatus implemented to execute the scheme described above with reference to FIGS. 1 to 20 .

Meanwhile, the timestamp regeneration scheme may be performed as software or firmware by the main processor 1200 instead of the processor 1120 . In this case, the timestamp generator 1310, which is firmware or software for realizing the timestamp regeneration scheme, may be loaded into the working memory 1300, and the main processor 1200 may drive the timestamp generator 1310. have. In this case, since the timestamp regeneration scheme is driven/processed by the main processor 1200 , in this case, the processor 1120 may be omitted.

The working memory 1300 may store data used for the operation of the electronic device 1000 . For example, the working memory 1300 may temporarily store packets or frames processed by the processor 1120 . For example, the working memory 1300 is a volatile memory such as DRAM (Dynamic RAM), SDRAM (Synchronous RAM), and/or PRAM (Phase-change RAM), MRAM (Magneto-resistive RAM), ReRAM (Resistive RAM). , a non-volatile memory such as a ferro-electric RAM (FRAM).

The storage 1400 may store firmware or software for performing a timestamp regeneration scheme. Firmware or software for performing the timestamp regeneration scheme may be read from the storage 1400 according to a request or command of the main processor 1200 , and may be loaded into the working memory 1300 . The storage 1400 may include a nonvolatile memory such as flash memory, PRAM, MRAM, ReRAM, and FRAM.

The display 1500 may include a display panel and a display serial interface (DSI) peripheral circuit. For example, the display panel may be implemented as various devices such as a liquid crystal display (LCD) device, a light emitting diode (LED) display device, an organic LED (OLED) display device, an active matrix OLED (AMOLED) display device, and the like. The DSI host built into the main processor 1200 may perform serial communication with the display panel through the DSI. The DSI peripheral circuit may include a timing controller, a source driver, etc. necessary for driving the display panel.

The communication block 1600 may exchange signals with an external device/system through an antenna. Transceiver 1610 and MODEM (Modulator / Demodulator, 1620) of the communication block 1600 is LTE (Long Term Evolution), WIMAX (Worldwide Interoperability for Microwave Access), GSM (Global System for Mobile communication), CDMA (Code Division Multiple) Access), Bluetooth, Near Field Communication (NFC), Wi-Fi (Wireless Fidelity), RFID (Radio Frequency Identification), etc., may process a signal exchanged with an external device/system according to a wireless communication protocol.

The user interface 1700 may include at least one of input interfaces such as a keyboard, a mouse, a keypad, a button, a touch panel, a touch screen, a touch pad, a touch ball, a gyroscope sensor, a vibration sensor, and an acceleration sensor.

Components of the electronic device 1000 are USB (Universal Serial Bus), SCSI (Small Computer System Interface), PCIe (Peripheral Component Interconnect Express), M-PCIe (Mobile PCIe), ATA (Advanced Technology Attachment), PATA (Parallel) One or more of various interface protocols such as ATA), SATA (Serial ATA), SAS (Serial Attached SCSI), IDE (Integrated Drive Electronics), EIDE (Enhanced IDE), NVMe (Nonvolatile Memory Express), UFS (Universal Flash Storage), etc. data can be exchanged based on

The contents described above are specific examples for carrying out the present invention. The present invention may include not only the above-described embodiments, but also simple design changes or easily changeable embodiments. In addition, the present invention will also include techniques that can be easily modified and implemented in the future using the above-described embodiments.

100: image processing unit 110: vision sensor
111: pixel array 113: column decoder
115: row decoder 116: sensing unit
117: address decoder 118: in-pixel circuit
119: output circuit 120: processor
122: buffer 124: timestamp
126: timestamp regenerator

Claims (20)

In the device:
Receive information about a plurality of events through at least one pixel of a plurality of pixels of a vision sensor, wherein the information includes a plurality of timestamps associated with times when the plurality of events occurred, the plurality of pixels includes a target pixel and adjacent pixels around the target pixel;
monitor timestamps between an event pertaining to the target pixel and events pertaining to the adjacent pixels;
group at least one first timestamp having a first value among the plurality of timestamps into a first group;
group at least one second timestamp having a second value among the plurality of timestamps into a second group;
determining a direction in which the plurality of events occur and an outline of an object underlying the plurality of events based on the first value and the second value;
determine a temporal correlation based on a timestamp of the target pixel and the at least one second timestamp; and
a processor for replacing the timestamp of the event of the target pixel with one of the timestamps of the adjacent pixels based on the determined outline and the determined temporal correlation,
The information includes polarity information regarding the plurality of events. The method of claim 1, wherein:
wherein the processor synchronously receives the plurality of events. 3. The method of claim 2,
The polarity information indicates an on-event in which the intensity of light increases. 4. The method of claim 3,
and the processor includes a timestamp generator to generate the timestamp. 4. The method of claim 3,
and the processor generates an on-event map based on the timestamps of the target pixel and the adjacent pixels. 6. The method of claim 5,
wherein the processor stores the on-event map in a memory. 7. The method of claim 6,
the processor marks a first number on the on-event map for an event that occurs during a first time period, and marks a second number on the on-event map for an event that occurs during a second time period;
wherein the first number and the second number are natural numbers and are different from each other. The method of claim 1,
The vision sensor is:
a pixel array including the target pixel and the adjacent pixels;
a column AER configured to receive a column request from a pixel in which an event has occurred among the target pixel and the adjacent pixels, and generate a column address based on the column request;
a row AER for receiving the polarity information from the target pixel and a pixel in which an event occurs among the adjacent pixels, generating a timestamp including information about a time when the event occurs, and generating a row address; and
and a packetizer and input/output circuitry configured to generate a packet based on the timestamp, the column address, the row address, and the polarity information. 9. The method of claim 8,
Each of the target pixel and the adjacent pixels is:
photodiode;
a capacitor for storing electrical energy converted by the photodiode;
a switch switched on by a reset signal to discharge charges stored in the capacitor;
a differential amplifier for amplifying a voltage level by the charges stored in the capacitor;
a comparator comparing the level of the output voltage of the differential amplifier with the level of a reference voltage and outputting an on-signal representing an on-event or an off-signal representing an off-event; and
and a read circuit configured to output the information based on the on-signal or the off-signal. In the device:
Receive information about a plurality of events through at least one pixel of a plurality of pixels of a vision sensor, wherein the information includes a plurality of timestamps associated with times when the plurality of events occurred, the plurality of pixels includes a target pixel and adjacent pixels around the target pixel;
monitor timestamps between an event pertaining to the target pixel and events pertaining to the adjacent pixels;
group at least one first timestamp having a first value among the plurality of timestamps into a first group;
group at least one second timestamp having a second value among the plurality of timestamps into a second group;
determining a direction in which the plurality of events occur and an outline of an object underlying the plurality of events based on the first value and the second value;
determine a temporal correlation based on a timestamp of the target pixel and the at least one second timestamp; and
a processor for replacing the timestamp of the event of the target pixel with one of the timestamps of the adjacent pixels based on the determined outline and the determined temporal correlation,
The information includes polarity information about the plurality of events,
wherein the processor asynchronously receives the plurality of events. 11. The method of claim 10,
The polarity information indicates an on-event in which the intensity of light increases. 12. The method of claim 11,
and the processor includes a timestamp generator to generate the timestamp. 12. The method of claim 11,
and the processor generates an on-event map based on the timestamps of the target pixel and the adjacent pixels. 14. The method of claim 13,
wherein the processor stores the on-event map in a memory. 15. The method of claim 14,
the processor marks a first number on the on-event map for an event that occurs during a first time period, and marks a second number on the on-event map for an event that occurs during a second time period;
wherein the first number and the second number are natural numbers and are different from each other. 11. The method of claim 10,
The vision sensor is:
a pixel array including the target pixel and the adjacent pixels;
a column AER configured to receive a column request from a pixel in which an event has occurred among the target pixel and the adjacent pixels, and generate a column address based on the column request;
a row AER for receiving the polarity information from the target pixel and a pixel in which an event occurs among the adjacent pixels, generating a timestamp including information about a time when the event occurs, and generating a row address; and
and a packetizer and input/output circuitry configured to generate a packet based on the timestamp, the column address, the row address, and the polarity information. 17. The method of claim 16,
Each of the target pixel and the adjacent pixels is:
photodiode;
a capacitor for storing electrical energy converted by the photodiode;
a switch switched on by a reset signal to discharge charges stored in the capacitor;
a differential amplifier for amplifying a voltage level by the charges stored in the capacitor;
a comparator comparing the level of the output voltage of the differential amplifier with the level of a reference voltage and outputting an on-signal representing an on-event or an off-signal representing an off-event; and
and a read circuit configured to output the information based on the on-signal or the off-signal. delete delete delete

Images