TW202022795A - Image processing circuit - Google Patents

Abstract

An image processing circuit is provided. A memory control unit first moves a reference image and parameters from a first memory to a third memory. A convolution neural network circuit obtains the parameters from the third memory. The memory control unit also moves at least one portion of the reference image from the third memory to a second memory, in which a storage capacity of the third memory is larger than that of the second memory. A depth decoder obtains the reference image from the second memory for calculating depth information according to the reference image and a structured image received from an infrared sensor, and stores the depth information into the second memory.

Description

Image processing circuit

The present invention relates to an image processing circuit, and particularly relates to a memory architecture of a convolutional neural network circuit.

In recent years, the technology of convolutional neural networks has received a lot of attention from academia and industry, and has made breakthrough developments in many fields, especially in image processing. When a convolutional neural network is used for image processing applications, a large amount of memory is often required to store images and parameters in the convolutional neural network. Therefore, how to design an appropriate hardware architecture to store or transport these data? This is a topic of concern to those skilled in the art.

An embodiment of the present invention provides an image processing circuit including a first memory, a second memory, a memory control unit, a convolutional neural network circuit, and a depth decoder. The first memory stores a reference image and a plurality of parameters, and the storage space of the second memory is smaller than that of the first memory. The memory control unit is electrically connected to the first memory and the second memory, the convolutional neural network circuit is electrically connected to the second memory, and the depth decoder is electrically connected to the second memory. The depth decoder is used to receive the structured image. In the first mode: the memory control unit moves the reference image and parameters from the first memory to the third memory, where the third memory is electrically connected to the convolutional neural network circuit, the memory control unit and the second Memory, and the storage space of the third memory is greater than the storage space of the second memory; the convolutional neural network circuit obtains parameters from the third memory, and the memory control unit transfers at least a part of the reference image from the third memory The body moves to the second memory; and the depth decoder obtains part of the reference image from the second memory, calculates the depth information based on the reference image and the structured image of this part, and stores the depth information in the second memory .

In some embodiments, the image processing circuit further includes a bus and a multiplexer. The bus is electrically connected to the memory control unit, the second memory and the convolutional neural network circuit, and the multiplexer is electrically connected to the bus, the third memory and the convolutional neural network circuit. In the first mode, the convolutional neural network circuit obtains parameters from the third memory through the multiplexer.

In some embodiments, in the first mode, the memory control unit also moves a part of the parameter from the third memory to the second memory through the bus; and when the memory control unit transfers the reference image through the bus While moving the part from the third memory to the second memory, the convolutional neural network circuit directly obtains the above-mentioned parameters from the second memory.

In some embodiments, the above-mentioned parameters correspond to the layers of the convolutional neural network.

In some embodiments, the second memory also receives the grayscale image from the infrared sensor. The convolutional neural network circuit also obtains depth information and grayscale images from the second memory, and based on the grayscale images, depth information and parameters Perform object detection process or object recognition process.

In some embodiments, the part of the reference image includes at least one column of pixels.

In some embodiments, the aforementioned second memory is a static random access memory, and the third memory is a dynamic random access memory.

In some embodiments, the first memory is a flash memory.

In some embodiments, the memory control unit is switched between the first mode and the second mode. In the second mode: the memory control unit moves the reference image and parameters from the first memory to the second memory; the convolutional neural network circuit obtains the parameters from the second memory; and the depth decoder from the second memory Obtain the reference image, calculate the depth information based on the reference image and the structured image, and store the depth information in the second memory.

In some embodiments, the memory control unit switches between the first mode and the second mode according to physical switches, detection circuits, or firmware settings.

In the above-mentioned image processing circuit, the space requirement of the memory can be reduced, or different memory structures can be adopted in different modes.

In order to make the above-mentioned features and advantages of the present invention more obvious and understandable, the embodiments are specifically described below and described in detail in conjunction with the accompanying drawings.

100‧‧‧Electronic device

110‧‧‧Infrared sensor

111‧‧‧Infrared projector

120‧‧‧Image processing circuit

121‧‧‧First memory

122‧‧‧Second memory

123‧‧‧Third memory

130‧‧‧Memory Control Unit

140‧‧‧Convolutional Neural Network Circuit

150‧‧‧Depth Decoder

160‧‧‧Bus

170‧‧‧Multiplexer

[Fig. 1] is a schematic diagram showing a circuit of an electronic device according to an embodiment.

[Fig. 2] is a schematic diagram showing a circuit of an electronic device according to an embodiment.

Regarding the "first", "second", ... etc. used in this text, it does not specifically refer to the order or sequence, but only to distinguish elements or operations described in the same technical terms.

[First Embodiment]

FIG. 1 is a schematic diagram of a circuit of an electronic device according to an embodiment. Please refer to FIG. 1, the electronic device 100 includes an infrared sensor 110, an infrared projector 111 and an image processing circuit 120. The electronic device 100 may be implemented as a mobile phone, a tablet computer, a notebook computer, or other suitable electronic devices. In some embodiments, the electronic device 100 may also include other components, such as a camera, etc. The invention is not limited thereto. The image processing circuit 120 includes a first memory 121, a second memory 122, a third memory 123, a memory control unit 130, a convolutional neural network circuit 140, a depth decoder 150, a bus 160, and a multiplexer.器170. Wherein, the second memory 122 is electrically connected to the infrared sensor 110, the depth decoder 150, and the convolutional neural network circuit 140. The bus 160 is electrically connected to the second memory 122, the memory control unit 130, and the multiplexer 170. The first memory 121 is electrically connected to the memory control unit 130, and the multiplexer 170 is electrically connected to the convolutional neural network circuit 140 and the third memory 123.

In this embodiment, the image processing circuit 120 is used to execute an image processing procedure, such as an object detection procedure or an object recognition procedure. The depth map of the scene is needed when executing the image processing program. This depth map is based on comparing the displacement between specific patterns in two infrared images (disparity) is calculated, and these two images are referred to as the reference image and the structured image here. The pattern in the reference image is a preset, which is also called a golden pattern. The preset pattern is, for example, a plurality of dots or a bar graph randomly set. The present invention does not limit the preset pattern. The structured image is projected into the scene through the infrared projector 111 with the aforementioned preset pattern, and then the infrared sensor 110 senses the resulting image. In other words, the structured image also has the aforementioned preset pattern. However, due to the influence of the depth in the scene, the preset pattern in the structured image will have a corresponding displacement. Therefore, by calculating the displacement of a certain position in the reference image and the structured image, the depth of field at that position can be calculated. In this embodiment, the above-mentioned reference image is stored in the first memory 121.

On the other hand, the convolutional neural network circuit 140 is used to implement a convolutional neural network. Generally speaking, a convolutional neural network has multiple layers, and each layer has multiple parameters, which are determined by training in advance. In this embodiment, the aforementioned parameters are also stored in the first memory 121.

In some embodiments, the storage space of the first memory 121 and the third memory 123 are both greater than the storage space of the second memory 122, but the read speed of the second memory 122 is greater than that of the first memory 121 and the first memory. 3. Reading speed of memory 123. For example, the first memory 121 is a flash memory, the second memory 122 is a static random access memory, and the third memory 123 is a dynamic random access memory, but the invention is not limited thereto. In particular, the aforementioned parameters and reference images are first moved from the first memory 121 to the third memory 123, and then from the third memory 123 to the second memory 122.

Specifically, the memory control unit 130 converts the reference image The AND parameter moves from the first memory 121 to the third memory 123, and at this time, the multiplexer 170 selects the input terminal "0". Next, the multiplexer 170 can continue to select the input “0”, and at this time the memory control unit 130 moves at least part of the reference image from the third memory 123 to the second memory 122. The depth decoder 150 obtains this part of the reference image from the second memory 122, calculates the depth information based on the reference image and the structured image received from the infrared sensor 110, and finally stores the depth information in the second memory体122中. In some embodiments, since the depth decoder 150 calculates the depth information row by row, the above-mentioned part of the reference image only needs to include one or more rows of pixels. In other words, the memory control unit 130 performs one or more Multiple rows of pixels are moved from the third memory 123 to the second memory 122, so it is not necessary to load all the reference images into the second memory 122 at once, and the space requirement of the second memory 122 can be reduced.

On the other hand, the convolutional neural network circuit 140 obtains parameters from the third memory 123 in two ways. The first method is that the multiplexer 170 selects the input terminal “1”, and the convolutional neural network circuit 140 reads the parameters from the third memory 123 through the multiplexer 170. The second method is that the multiplexer 170 selects the input terminal "0", the memory control unit 130 first moves the parameters from the third memory 123 to the second memory 122, and then the convolutional neural network circuit 140 directly moves from the first The parameters are read from the second memory 122. At the same time, the memory control unit 130 can move the reference image from the third memory 123 to the second memory 122 through the bus 160, so that the parameters and the reference image can be read Simultaneously. In some embodiments, the parameters moved to the second memory 122 correspond to one layer (not all layers) of the convolutional neural network, so that it can also The space requirement of the second memory 122 is reduced.

In some embodiments, the infrared sensor 110 not only senses the structured image, but also senses a grayscale image. There is no projected pattern in the grayscale image, which means that the pixels of the grayscale image only reflect the objects in the scene, and the grayscale image will be sent to the second memory 122. The convolutional neural network circuit 140 obtains parameters, depth information, and grayscale images from the second memory 122, and performs object detection procedures or object recognition procedures, such as face recognition, based on these grayscale images, depth information, and parameters The process or the face recognition process, however, the present invention does not limit the content of the object detection process and the object recognition process.

[Second Embodiment]

FIG. 2 is a schematic diagram of a circuit of an electronic device according to an embodiment. In the embodiment of FIG. 2, the image processing circuit 120 does not have the third memory 123, so the above-mentioned reference image and parameters are moved from the first memory 121 to the second memory 122. The convolutional neural network circuit 140 can obtain parameters from the second memory. The depth decoder 150 obtains the reference image from the second memory 122, calculates depth information based on the reference image and the structured image, and stores the depth information in the second memory 122.

Compared with the first embodiment in FIG. 1, the second embodiment in FIG. 2 is more suitable for images with a smaller resolution, so all parameters and the entire reference image can be directly stored in the second memory 122. However, the present invention is not limited to this, and the first embodiment and the second embodiment can be applied to images of any resolution. It is worth noting that the difference between FIG. 1 and FIG. 2 is only the third memory 123, and other components are the same, so the image processing circuit 120 does not need Too many modifications can be applied to different products. The above-mentioned first embodiment is called the first mode, and the second embodiment is called the second mode. The memory control unit 130 can be switched between the first mode and the second mode. In some embodiments, a detection circuit (not shown) may be provided in the image processing circuit 120, and the detection circuit is used to determine whether the third memory 123 exists, and if the third memory 123 exists, the memory control unit 130 will execute the first mode, otherwise execute the second mode. Alternatively, the memory control unit 130 can also switch between the first mode and the second mode according to a physical switch or firmware setting. The physical switch can be set on the circuit board or any other position, and the firmware setting can be preset In the burning memory control unit 130 or other controllers, the present invention is not limited thereto.

Although the present invention has been disclosed as above by the embodiments, it is not intended to limit the present invention. Any person with ordinary knowledge in the technical field can make some changes and modifications without departing from the spirit and scope of the present invention. The scope of protection of the present invention shall be subject to the scope defined in the appended patent application.

100‧‧‧Electronic device

110‧‧‧Infrared sensor

111‧‧‧Infrared projector

120‧‧‧Image processing circuit

121‧‧‧First memory

122‧‧‧Second memory

123‧‧‧Third memory

130‧‧‧Memory Control Unit

140‧‧‧Convolutional Neural Network Circuit

150‧‧‧Depth Decoder

160‧‧‧Bus

170‧‧‧Multiplexer

Claims (10)

An image processing circuit includes: a first memory storing a reference image and a plurality of parameters; a second memory, wherein the storage space of the second memory is smaller than the storage space of the first memory; a memory The control unit is electrically connected to the first memory and the second memory; a convolutional neural network circuit is electrically connected to the second memory; and a depth decoder is electrically connected to the second memory Memory for receiving a structured image from an infrared sensor, wherein in a first mode: the memory control unit moves the reference image and the parameters from the first memory to a third memory Body, wherein the third memory is electrically connected to the convolutional neural network circuit, the memory control unit and the second memory, and the storage space of the third memory is larger than the storage of the second memory Space; the convolutional neural network circuit obtains the parameters from the third memory, the memory control unit moves at least part of the reference image from the third memory to the second memory; and the The depth decoder obtains the at least part of the reference image from the second memory, calculates depth information based on the at least part of the reference image and the structured image, and stores the depth information in the second memory body. The image processing circuit described in item 1 of the scope of patent application further includes: a bus, electrically connected to the memory control unit, the second memory and the convolutional neural network circuit; and a multiplexer , Electrically connected to the bus, the third memory and the convolutional neural network circuit, wherein in the first mode, the convolutional neural network circuit is connected to the third memory through the multiplexer Obtain these parameters. For the image processing circuit described in claim 2, wherein in the first mode: the memory control unit also moves a part of the parameters from the third memory to the first mode through the bus Two memories; and when the memory control unit moves the at least part of the reference image from the third memory to the second memory through the bus, the convolutional neural network circuit directly The second memory obtains the part of the parameters. For the image processing circuit described in item 3 of the scope of patent application, the part of the parameters corresponds to a layer of a convolutional neural network. According to the image processing circuit described in claim 1, wherein the second memory also receives a gray-scale image from the infrared sensor, and the convolutional neural network circuit also obtains the second memory from the second memory. Depth information and the grayscale image, and execute an object detection process or an object recognition process based on the grayscale image, the depth information, and the parameters. In the image processing circuit described in claim 1, wherein the at least a part of the reference image includes at least one row of pixels in the reference image. In the image processing circuit described in claim 1, wherein the second memory is a static random access memory, and the third memory is a dynamic random access memory. The image processing circuit described in item 7 of the scope of patent application, wherein the first memory is a flash memory. According to the image processing circuit described in claim 1, wherein the memory control unit switches between the first mode and a second mode, in the second mode: the memory control unit uses the reference image And the parameters are moved from the first memory to the second memory; the convolutional neural network circuit obtains the parameters from the second memory; And the depth decoder obtains the reference image from the second memory, calculates the depth information according to the reference image and the structured image, and stores the depth information in the second memory. According to the image processing circuit described in claim 9, wherein the memory control unit is switched between the first mode and the second mode according to physical switches, detection circuits, or firmware settings.

Images