JP2021026465A - Information processing device, program and imaging system - Google Patents

Abstract

To provide an information processing device which has the high recognition force even when the number of objects is arbitrary and large, and can estimate the number of objects or ratio (density) of occupation in an image, and also provide a program and an imaging system.SOLUTION: In a control unit of an imaging system, a communication part receives image data including a plurality of objects, and a cell division part divides the image data into a plurality of cells. A cell group extraction part extracts a cell group including a desired cell and a surrounding cell from the plurality of cells, where the desired cell is one of the plurality of cells, and the surrounding cell is the cell located in the periphery of the desired cell. An estimation part estimates the number or density of objects included in the image data by inputting each cell included in the cell group to a machine learning architecture. The machine learning architecture includes a CNN layer which performs the convolution of the same weight matrix to each cell included in the cell group, and an LSTM layer which can process a plurality of feature intermediate values output from the CNN layer in a time series manner.SELECTED DRAWING: Figure 2

Description

The present invention relates to an information processing device, a program, and an imaging system.

In various fields, there is a demand for measuring the number of desired objects and their proportion (density) in an image from an image taken by a camera. For example, in an imaging system including a camera such as a surveillance camera, a person, an automobile, or the like is treated as a desired object. For example, Patent Document 1 discloses an object recognition device that detects a predetermined object. Such an object recognition device aims to improve the recognition rate by adjusting a camera control value when a predetermined object (here, a road sign) is detected.

Japanese Unexamined Patent Publication No. 2019-125022

However, the object recognition device disclosed in Patent Document 1 does not mention how to detect an object in the first place. If it is a road sign, the form is fixed in the first place, and it is estimated that the number of road signs is limited at one time of shooting. In other words, under such conditions, some results can be expected even with existing algorithms, but if the object is to be extended to an arbitrary object and there are many, a better algorithm is required. it is conceivable that.

The present invention has been made in view of such circumstances, has high cognitive ability even if the number of objects is arbitrary and large, and can estimate the number of such objects or the proportion (density) of the objects in an image. It is an object of the present invention to provide various information processing devices, programs, and imaging systems.

According to one aspect of the present invention, the information processing apparatus includes a reception unit, a cell division unit, a cell group extraction unit, and an estimation unit, and the reception unit includes image data including a plurality of objects. The cell division unit is configured to be able to divide the image data into a plurality of cells, and the cell group extraction unit is configured to select a cell group including a desired cell and surrounding cells from the plurality of cells. It is configured to be extractable, where the desired cell is one of the plurality of cells, the peripheral cell is a cell located around the desired cell, and the estimation unit is included in the cell group. By inputting each cell into the machine learning architecture, the number or density of objects contained in the image data can be estimated, and the machine learning architecture is the same for each cell included in the cell group. A CNN layer that convolves the weighting matrix of the above, and an LSTM layer that can sequentially process a plurality of feature intermediate values output from the CNN layer are provided.

According to the information processing apparatus according to one aspect of the present invention, even if the number of objects is arbitrary and large, the number of such objects or the ratio (density) occupied in the image is estimated while having high cognitive ability. It has the advantageous effect of being able to do it.

A block diagram showing an outline of the hardware configuration of an imaging system. FIG. 2A is a block diagram showing an outline of a hardware configuration of an information processing device, and FIG. 2B is a functional block diagram showing a function of a control unit. The figure which shows an example of the image taken by the camera. The figure which shows an example which divided the image shown in FIG. 3 into a plurality of cells. FIG. 5A is a diagram showing an example in which an arbitrary cell group is extracted from a plurality of cells shown in FIG. 4, and [FIG. 5B] a diagram showing desired cells and surrounding cells constituting the cell group shown in FIG. 5A. .. An activity diagram showing the flow of operation of the imaging system. A grayscale image of one of the mall datasets (available from http: // personal.ie.kuhk.edu.hk/ ~ ccloy / downloads_mall_dataset.html).

Hereinafter, embodiments of the present invention will be described with reference to the drawings. The various features shown in the embodiments shown below can be combined with each other. In particular, in the present specification, the "part" may include, for example, a combination of hardware resources implemented by circuits in a broad sense and information processing of software that can be concretely realized by these hardware resources. .. Further, various information is handled in this embodiment, and these information are represented by high and low signal values as a bit set of binary numbers composed of 0 or 1, and communication / calculation is executed on a circuit in a broad sense. Can be done.

Further, a circuit in a broad sense is a circuit realized by at least appropriately combining a circuit (Circuit), circuits (Circuitry), a processor (Processor), a memory (Memory), and the like. That is, an integrated circuit for a specific application (Application Special Integrated Circuit: ASIC), a programmable logic device (for example, a simple programmable logic device (Simple Programmable Logic Device: SPLD), a composite programmable logic device (Complex Program)) It includes a programmable gate array (Field Programmable Gate Array: FPGA) and the like.

1. 1. Overall Configuration In Section 1, the components included in this embodiment will be described in sequence.

1.1 Imaging system 1
FIG. 1 is a block diagram showing an outline of the hardware configuration of the imaging system 1 according to the present embodiment. The imaging system 1 includes a camera 2 and an information processing device 3. In the present embodiment, the camera 2 and the information processing device 3 are connected via the network N. Such a connection may be wired or wireless.

1.2 Camera 2
The camera 2 optically captures information in the outside world, and is preferably captured as a moving image. The resolution and frame rate of the camera 2 may be appropriately selected according to the application (object O to be imaged), and are not particularly limited in the present embodiment. For example, the resolution may be full HD, WQHD, 4k, etc., and the frame rate may be 30 fps, 60 fps, 120 fps, 250 fps, or the like. The image IM captured by the camera 2 is transmitted to the information processing device 3 described later. FIG. 3 shows an example of the image IM captured by the camera 2, so please refer to it. Here, the case where the object O is a human is illustrated.

1.3 Information processing device 3
The information processing device 3 is realized by a computer as hardware and a program as software. The information processing device 3 may be realized by having the computer user U install the program on the computer via the Internet or the like (download), or the information processing device may be realized by installing the program on the computer in advance. 3 may be realized.

FIG. 2A shows an outline of the hardware configuration of the information processing device 3, and FIG. 2B is a functional block diagram showing the functions of the control unit 33. The information processing device 3 includes a communication unit 31, a storage unit 32, a control unit 33, a display unit 34, and an input unit 35, and these components provide a communication bus 30 inside the information processing device 3. It is electrically connected via. Hereinafter, each component will be further described.

<Communication unit 31>
Although wired communication means such as USB, IEEE1394, Thunderbolt, and wired LAN network communication are preferable, the communication unit 31 performs wireless LAN network communication, mobile communication such as LTE / 3G, Bluetooth (registered trademark) communication, and the like as necessary. May be included. These are examples, and a dedicated communication standard may be adopted. That is, it is more preferable to carry out as a set of these plurality of communication means.

In particular, as shown in FIG. 1, the communication unit 31 configures the information processing device 3 so that it can communicate with the camera 2 via the network N. For example, the communication unit 31 receives the image IM captured by the camera 2 via the network N. In other words, the communication unit 31 (an example of the “reception unit”) is configured to be able to receive data of an image IM including a plurality of objects O.

<Memory unit 32>
The storage unit 32 is a volatile or non-volatile storage medium that stores various information. This is, for example, as a storage device such as a solid state drive (SSD), or a random access memory (Random Access Memory:) that stores temporarily necessary information (arguments, arrays, etc.) related to program operations. It can be implemented as a memory such as RAM). Moreover, these combinations may be used.

In particular, the storage unit 32 stores a program that causes the computer to function as the information processing device 3. It should also be noted that the storage unit 32 stores the machine learning (particularly deep learning) architecture described below.

<Control unit 33>
The control unit 33 processes and controls the entire operation related to the information processing device 3. The control unit 33 is realized as, for example, a central processing unit (CPU) (not shown). The control unit 33 realizes various functions related to the information processing device 3 by reading a predetermined program stored in the storage unit 32. Specifically, a cell division function that divides the image IM data received by the communication unit 31 into a plurality of cells C, and extracts a cell group CG including desired cells C0 and surrounding cells C1 to C8 from the plurality of cells C. The cell group extraction function, the estimation function for estimating the number or density of the object O included in the image IM, and the like by inputting each cell C included in the cell group CG into the machine learning architecture are applicable.

That is, the information processing by the software (stored in the storage unit 32) is specifically realized by the hardware (control unit 33), so that the cell division unit 331, the cell group extraction unit 332, and the estimation unit 333 can be used. Can be executed. Although it is described as a single control unit 33 in FIG. 2A, it is not actually limited to this, and it may be implemented so as to have a plurality of control units 33 for each function. Moreover, it may be a combination thereof. Hereinafter, the cell division unit 331, the cell group extraction unit 332, and the estimation unit 333 will be described in more detail.

[Cell division unit 331]
In the cell division unit 331, information processing by software (stored in the storage unit 32) is specifically realized by hardware (control unit 33). The cell division unit 331 divides the image IM imaged by the camera 2 and received via the communication unit 31 into a plurality of cells C. FIG. 4 shows an example in which the image IM shown in FIG. 3 is divided into a plurality of cells C (here, 8 × 8 = 64), so please refer to it.

[Cell group extraction unit 332]
In the cell group extraction unit 332, information processing by software (stored in the storage unit 32) is specifically realized by hardware (control unit 33). The cell group extraction unit 332 extracts a cell group CG including a desired cell C0 and surrounding cells C1 to C8 from a plurality of cells C divided by the cell division unit 331. FIG. 5A shows an example in which an arbitrary cell group CG is extracted from a plurality of cell Cs shown in FIG. 4, and FIG. 5B shows a cell group CG shown in FIG. 5A (here, 3 × 3 = 9). The desired cells C0 and the surrounding cells C1 to C8 that constitute the above are shown, so refer to them. Here, the desired cell C0 is one of the plurality of cells C, and the peripheral cells C1 to C8 are cells C located around the desired cell C0. In other words, the cell group CG is composed of a total of nine cells, one desired cell C0 and eight surrounding cells C1 to C8.

[Estimating unit 333]
In the estimation unit 333, information processing by software (stored in the storage unit 32) is specifically realized by hardware (control unit 33). The estimation unit 333 estimates the number or density of the objects O included in the image IM by inputting each cell C included in the cell group CG into the machine learning architecture stored in the storage unit 32. Here, the machine learning architecture adopted in the present embodiment is output from a CNN (Convolutional Neural Network) layer that convolves the same weighting matrix for each cell C included in the cell group CG, and the CNN layer. It should be noted that it includes an LSTM (Long short-term memory) layer capable of sequentially processing a plurality of feature maps.

The CNN layer is a neural network having a plurality of layers. Specifically, the CNN layer includes a convolutional layer that extracts features such as edge extraction from the image IM (to be exact, cell C), a pooling layer that ensures robustness such as translation, and an output value (“feature intermediate”). It is provided with a binding layer that gives an example of "value"). That is, the desired cells C0 and the surrounding cells C1 to C8 that constitute the cell group CG shown in FIG. 5B are input to the CNN layer, respectively, and a plurality of feature intermediate values are obtained. The plurality of feature intermediate values obtained are not limited to the same number of cells C included in the cell group CG. Further, the convolution matrix in the convolutional layer is not particularly limited, but it should be noted that the convolutional weight CNN convolves the same weighting matrix for each cell C.

The LSTM layer is a machine learning model suitable for classification, processing, and prediction based on series data (particularly time series data). For example, referring to FIG. 4, it can be confirmed that the object O occupies the plurality of cells C in the image IM. Based on these characteristics, the inventors have conceived that the desired cell C0 and the surrounding cells C1 to C8 are treated as series data and machine-learned to exhibit excellent recognition of the object O. It should also be noted that, as shown in FIG. 4, even if the objects O overlap each other, they exhibit excellent recognizability.

The LSTM layer is configured to feed back the output value of one cell C to the input value of the next cell C. Then, with the feature intermediate value corresponding to the desired cell C0 as an input, the last LSTM layer in the sequence order among the plurality of LSTM layers outputs the number or density of the objects O in the desired cell C0.

<Display unit 34>
The display unit 34 may be included in the housing of the information processing device 3, or may be externally attached, for example. The display unit 34 displays a screen of a graphical user interface (GUI) that can be operated by the user U. For example, it is preferable to use display devices such as a CRT display, a liquid crystal display, an organic EL display, and a plasma display properly according to the type of the information processing device 3. The display device can selectively display the GUI screen in response to a predetermined control signal by the control unit 33. As a supplement to the GUI, a specific interface such as an input form or a radio button may be displayed.

The display unit 34 displays the image IM received by the communication unit 31 from the camera 2. Further, the display unit 34 displays the number or density of the object O estimated by the estimation unit 333 in the control unit 33.

<Input unit 35>
The input unit 35 may be included in the housing of the information processing device 3 or may be externally attached. For example, the input unit 35 can be implemented as a touch panel integrally with the display unit 34. If it is a touch panel, the user U can input a tap operation, a swipe operation, and the like. Of course, instead of the touch panel, a switch button, a mouse, a QWERTY keyboard, or the like may be adopted. That is, the input unit 35 receives the operation input made by the user U. The input is transferred as a command signal to the control unit 33 via the communication bus 30, and the control unit 33 can execute a predetermined control or calculation as needed.

2. 2. Description of Each Function Section 2 describes the operation flow of the imaging system 1 according to the present embodiment. FIG. 6 is an activity diagram showing an operation flow of the imaging system 1. Hereinafter, each activity in the activity diagram shown in FIG. 6 will be described.

[from here]
(Activity A01)
The camera 2 in the image pickup system 1 takes an image as an image IM including the object O. Such imaging is continuously performed based on the frame rate. Each frame is sequentially transmitted to the information processing apparatus 3 via the network N.

(Activity A11)
The communication unit 31 in the information processing device 3 receives the image IM captured in the activity A1 (the image IM is received by the information processing device 3).

(Activity A12)
After the completion of the activity A11, the cell division unit 331 in the information processing apparatus 3 divides the image IM received in the activity A11 into a plurality of cells C.

(Activity A13)
After the completion of the activity A12, the cell group extraction unit 332 in the information processing apparatus 3 uses a part of the plurality of cells C divided in the activity A12 as the cell group CG (desired cell C0 and surrounding cells C1 to C8). Extract.

(Activity A14)
After the completion of the activity A13, the estimation unit 333 in the information processing device 3 inputs the desired cells C0 and the surrounding cells C1 to C8, and the number or density of the objects O is based on the machine learning architecture stored in the storage unit 32. Start estimating. Here, a plurality of feature intermediate values are output by introducing a CNN layer sharing a convolutional matrix with desired cells C0 and surrounding cells C1 to C8 as inputs.

(Activity A15)
After the completion of the activity A14, the density of the object O in the desired cell C0 is estimatedly output by introducing the LSTM layer by using the plurality of feature intermediate values obtained in the activity A14 as a series input.

(Activity A16)
The number or density of the object O in the image IM can be estimated by performing the density estimation of the object O in the desired cell C0 obtained as a result of the activity A15 on the image IM as a whole.
[to this point]

3. 3. Example In Section 3, the example of the information processing apparatus 3 described in Section 1 will be described. Since this is just an evaluation experiment, a mall data set (Mall Dataset) widely used as a benchmark by those skilled in the art was adopted instead of the image IM imaged by the camera 2 (see FIG. 7). In the mall data set, a large amount of human beings, which are objects O, are included in the image IM.

Annotations for each head position in the mall dataset are open to the public. The first 800 frames were used for training and the remaining 1200 frames were reserved for testing. By dividing the entire frame into a grid of 32 x 32 size images, more training data can be added from the 800 frame dataset. Overall, 800x20x15 training images were generated and used to train the model. The model was then trained using the Adam optimizer.

In evaluating the deviation from the true value, the MAE (mean absolute value error) specified in [Equation 1] and the MSE (mean square error) specified in [Equation 2] were measured.

More specifically, MAE is an indicator of the accuracy of the estimate and MSE is an indicator of the robustness of the estimate. In the equation, N is the number of test samples, y_i is the true value count, and y_i'is the estimated count corresponding to the i-th sample. It can be said that the lower the MAE and MSE values, the higher the accuracy and the better the estimated value.

Furthermore, [Table 1] shows the results of the evaluation experiment. As a comparative example, the number or density estimation method of the object O according to the prior art is described. It has been shown that accuracy and estimation robustness are higher than those of conventional methods.

4. Conclusion As described above, according to the present embodiment, even if the number of objects O is arbitrary and large, the object O has high cognitive ability, and the number of such objects O or the ratio (density) occupied in the image IM is estimated. A possible information processing device 3 can be provided.

That is, the information processing device includes a reception unit, a cell division unit, a cell group extraction unit, and an estimation unit, and the reception unit is configured to be able to receive image data including a plurality of objects. The cell division unit is configured to be able to divide the image data into a plurality of cells, and the cell group extraction unit is configured to be able to extract a cell group including a desired cell and surrounding cells from the plurality of cells. The desired cell is one of the plurality of cells, the surrounding cell is a cell located around the desired cell, and the estimation unit uses each cell included in the cell group as a machine learning architecture. By inputting, the number or density of objects contained in the image data can be estimated, and here, the machine learning architecture convolves the same weighting matrix for each cell included in the cell group. And an LSTM layer capable of sequentially processing a plurality of feature intermediate values output from the CNN layer are provided.

Further, according to the present embodiment, even if the number of objects O is arbitrary and large, a program having high cognitive ability and capable of estimating the number of such objects O or the ratio (density) occupied in the image IM is provided. Can be provided. Such a program may be provided as a non-temporary recording medium that can be read by a computer (information processing device 3), may be provided as a downloadable program from an external server, or may be provided by an external computer. You may start a program and execute so-called cloud computing that can execute each function on the client terminal.

That is, a program that causes the computer to function as the information processing device is provided.

Further, according to the present embodiment, even if the number of objects O is arbitrary and large, the imaging system has high cognitive ability and can estimate the number of such objects O or the ratio (density) of the objects O in the image IM. 1 can be provided.

That is, it is an imaging system, which includes a camera and an information processing device, which are configured to be able to communicate with each other via a network, and the camera is configured to be capable of capturing image data including a plurality of objects. The information processing device is the information processing device, and a reception unit provided therein is provided so as to be capable of receiving the image data captured by the camera.

Although various embodiments according to the present invention have been described, they are presented as examples and are not intended to limit the scope of the invention. The novel embodiment can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the gist of the invention. The embodiment and its modifications are included in the scope and gist of the invention, and are included in the scope of the invention described in the claims and the equivalent scope thereof.

1: Imaging system 2: Camera 3: Information processing device 30: Communication bus 31: Communication unit 32: Storage unit 33: Control unit 331: Cell division unit 332: Cell group extraction unit 333: Estimating unit 34: Display unit 35: Input Part C: Cell C0: Desired cell C1: Surrounding cell C2: Surrounding cell C3: Surrounding cell C4: Surrounding cell C5: Surrounding cell C6: Surrounding cell C7: Surrounding cell C8: Surrounding cell CG: Cell group IM: Image N: Network O: Object

Claims (7)

It is an information processing device and includes a reception unit, a cell division unit, a cell group extraction unit, and an estimation unit.
The reception unit is configured to be able to receive image data including a plurality of objects.
The cell division unit is configured so that the image data can be divided into a plurality of cells.
The cell group extraction unit is configured to be able to extract a cell group including a desired cell and a surrounding cell from the plurality of cells, wherein the desired cell is one of the plurality of cells and is the peripheral cell. Is a cell located around the desired cell,
The estimation unit is configured to be able to estimate the number or density of objects included in the image data by inputting each cell included in the cell group into the machine learning architecture.
Here, the machine learning architecture can sequentially process a CNN layer that convolves the same weighting matrix for each cell included in the cell group, and a plurality of feature intermediate values output from the CNN layer. Those including the LSTM layer. In the information processing apparatus according to claim 1,
The CNN layer has a convolutional layer, a pooling layer, and a binding layer. In the information processing apparatus according to claim 1 or 2.
The cell group consists of a total of nine cells, one desired cell and eight surrounding cells. The information processing apparatus according to any one of claims 1 to 3.
The object is a human. In the information processing apparatus according to any one of claims 1 to 4.
The last LSTM layer in the series order of the LSTM layers outputs the number or density of the objects in the desired cell. It ’s a program
A device that causes a computer to function as an information processing device according to any one of claims 1 to 5. It is an imaging system, which includes a camera and an information processing device, and is configured so that they can communicate with each other via a network.
The camera is configured to be capable of capturing image data including a plurality of objects.
The information processing device
The information processing device according to any one of claims 1 to 5.
The reception unit provided therein is configured to be able to receive the image data captured by the camera.