JP7368995B2 - Image recognition system, imaging device, recognition device, and image recognition method - Google Patents

Description

The present invention relates to an image recognition system, an imaging device, a recognition device, and an image recognition method.

In recent years, the number of surveillance cameras installed has increased due to heightened awareness of crime prevention. As a result, it has become difficult for a supervisor to visually recognize images captured by an imaging device such as a surveillance camera and to recognize objects such as suspicious persons or objects. Therefore, there is an increasing demand for automatically recognizing objects by performing image recognition processing on such images.

In order to perform image recognition processing, it is necessary to temporarily store and/or transmit captured images, and in order to store and/or transmit images captured by multiple imaging devices, a large amount of storage capacity or /and transmission capacity is required. Therefore, it is preferable that the data volume of the image to be subjected to image recognition processing is suppressed while maintaining the accuracy of image recognition. Patent Document 1 discloses that the edge level of each block is estimated based on the strength of edges included in a plurality of blocks into which a captured image is divided, and each block is replaced with a low-resolution image based on the estimated edge level. Surveillance cameras have been disclosed. Patent Document 2 discloses a surveillance camera that detects a fine edge whose thickness is less than a standard and a fine structure region in the vicinity thereof, and replaces the outside of the detected fine structure region with a low-resolution image.

Japanese Patent Application Publication No. 2015-088817 Japanese Patent Application Publication No. 2015-088818

However, in the methods of Patent Documents 1 and 2, the data capacity of the replaced image changes depending on the edge strength of the image, etc., so it is difficult to predict the storage capacity and/or transmission capacity required for image recognition processing. The problem was that it was difficult. Therefore, it is desired to stably reduce the data volume of images that are objects of image recognition while maintaining the accuracy of image recognition.

The present invention has been made to solve the above-mentioned problems, and provides an image recognition system that makes it possible to stably reduce the data volume of images that are the object of image recognition while maintaining the accuracy of image recognition. The present invention aims to provide an imaging device, a recognition device, and an image recognition method.

The image recognition system according to the present invention includes: an imaging means for imaging a space in which a predetermined object can be imaged, and generating a first image consisting of pixels having gradation values within a first gradation range; converting the first image into a second image by a converter that, when input with an image, outputs a second image consisting of pixels having tone values within a second tone range smaller than the first tone range; Recognition that generates recognition results for the second image using a conversion means that performs conversion and a recognizer that performs processing for recognizing a target on the second image when the second image is input and outputs the recognition results. It is characterized by having a means.

Further, in the image recognition system according to the present invention, the converter and the recognizer have a tone value within the first tone range in the neural network coupled so that the output of the converter becomes the input of the recognizer. The recognition result that is output when the first learning image consisting of pixels is input is trained to be close to the learning recognition result that is set in advance as the recognition result that should be output for the first learning image. Preferably, it is a trained neural network.

Further, in the image recognition system according to the present invention, the converter and the recognizer are configured to generate an image having a second gradation range that is output by the converter when the first image for learning is input to the combined neural network. It is preferable that the neural network is a trained neural network that has been trained to bring the image closer to the edge image generated from the first image for training and to bring the recognition result output by the combined neural network closer to the recognition result for training. .

The image recognition system according to the present invention further includes an output means for outputting the converted second image to a predetermined transmission network, and an acquisition means for acquiring the second image from the transmission network, and the recognition means includes: Preferably, a recognition result is generated for the acquired second image.

An imaging device according to the present invention includes an imaging means for imaging a space in which a predetermined object can be imaged to generate a first image composed of pixels having a gradation value within a first gradation range; converting the first image into a second image by a converter that outputs a second image consisting of pixels having tone values within a second tone range smaller than the first tone range when The image forming apparatus is characterized by comprising a converting means for outputting the second image, and an output means for outputting the second image.

Further, in the imaging device according to the present invention, the converter performs processing for recognizing a target on the second image when the output of the converter is the second image and outputs a recognition result. When a first image for training consisting of pixels having tone values within the first tone range is input to a neural network connected to be input to the recognizer, the recognition result outputted as the first image for learning is Preferably, it is a trained neural network that has been trained so as to approach a training recognition result set in advance as a recognition result to be output for one image.

The recognition device according to the present invention is configured such that when a first image consisting of pixels having a gradation value within a first gradation range is input, pixels within a second gradation range smaller than the first gradation range are input. an acquisition unit that acquires a second image obtained by converting the first image generated by imaging using a converter that outputs a second image consisting of pixels having gradation values; The present invention is characterized by comprising a recognition unit that generates a recognition result for the second image using a recognizer that performs recognition processing on the second image and outputs the recognition result.

Furthermore, in the recognition device according to the present invention, the recognizer includes a neural network connected such that the output of the converter becomes the input of the recognizer. This is a trained neural network that has been trained so that the recognition result output when the first image for learning is input is closer to the recognition result for learning that is set in advance as the recognition result that should be output for the first image for learning. It is preferable that there is.

The image recognition method according to the present invention images a space in which a predetermined object can be imaged, generates a first image consisting of pixels having a gradation value within a first gradation range, and the first image is an input image. converting the first image into a second image by a converter that outputs a second image consisting of pixels having tone values within a second tone range smaller than the first tone range when The feature includes generating a recognition result for the second image by a recognizer that performs processing for recognizing a target on the second image when the second image is input and outputs the recognition result. shall be.

The image recognition system, imaging device, recognition device, and image recognition method according to the present invention make it possible to stably reduce the data volume of an image that is a target of image recognition while maintaining the accuracy of image recognition.

FIG. 1 is a schematic diagram for explaining the outline of the present invention. 1 is a diagram showing an example of a schematic configuration of an image recognition system 1. FIG. 1 is a diagram showing an example of a schematic configuration of a learning device 2. FIG. 1 is a diagram showing an example of a schematic configuration of an imaging device 3. FIG. 4 is a diagram showing an example of a schematic configuration of a recognition device 4. FIG. FIG. 2 is a schematic diagram for explaining the outline of a converter. FIG. 2 is a schematic diagram for explaining the outline of a discriminator. 2 is a diagram showing an example of a data structure of learning data 211. FIG. FIG. 3 is a flow diagram showing an example of the flow of learning processing. FIG. 2 is a sequence diagram showing an example of the flow of image recognition processing. 7 is a diagram showing an example of a recognition result screen 700. FIG.

Hereinafter, various embodiments of the present invention will be described with reference to the drawings. However, it should be noted that the technical scope of the present invention is not limited to these embodiments, but extends to the invention described in the claims and equivalents thereof.

(Summary of the present invention)
FIG. 1 is a schematic diagram for explaining the outline of the present invention. The image recognition system according to the present invention includes an imaging means, a conversion means, and a recognition means.

The imaging means images a space in which a predetermined object can be imaged, and generates a first image made up of pixels having gradation values within a first gradation range. The predetermined object is, for example, a person, and the first image is, for example, an image consisting of pixels having tone values of 0 to 255 for each of the three RGB channels. The converting means uses a converter to convert the first image generated by the imaging means into a second image made up of pixels having gradation values within a second gradation range smaller than the first gradation range. The second image is, for example, an image composed of pixels having a gradation value of 0 or 1. The recognition means uses the recognizer to generate a recognition result of a predetermined object for the second image converted by the conversion means. The recognition result is, for example, information indicating a rectangular area circumscribing the image of the person in the second image.

The converter and recognizer are trained neural networks created by training coupled neural networks such that the output of the converter becomes the input of the recognizer. In the learning, when a first learning image consisting of pixels having gradation values within a first gradation range is input, the recognition result output from the neural network is output for the first learning image. This is performed so as to approximate the learning recognition result set in advance as the expected recognition result.

In this manner, in the image recognition system, the converting means converts the first image into the second image using a converter that outputs the second image when the first image is input. By doing so, the image recognition system makes it possible to stably reduce the data volume of the image that is the object of image recognition while maintaining the accuracy of image recognition. That is, the data capacity of the second image obtained by converting the first image is determined by the second gradation range and the number of pixels, and does not depend on the contents of the first image. Therefore, by converting the first image into the second image, the image recognition system makes it possible to stably reduce the data capacity of the image regardless of the content of the first image.

In addition, a converter that outputs a second image when the first image is input is a neural network that is connected so that the output of the converter becomes the input of the recognizer, and the first image for learning and the recognition for learning. It is generated by learning using the results. By doing so, the image recognition system can cause the converter to output a second image that allows the recognizer to perform image recognition with high accuracy.

Note that the above description of FIG. 1 is merely an explanation for deepening understanding of the content of the present invention. Specifically, the present invention is implemented in each embodiment described below, and may be implemented by various modifications without substantially exceeding the principles of the present invention. All such variations are within the scope of the invention and disclosure herein.

(Schematic system configuration)
FIG. 2 is a diagram showing an example of a schematic configuration of the image recognition system 1. As shown in FIG. The image recognition system 1 includes a learning device 2, an imaging device 3, a recognition device 4, and a display device 5. The learning device 2, the imaging device 3, the recognition device 4, and the display device 5 are interconnected via a transmission network 6 such as the Internet or an intranet.

The learning device 2 is an information processing device such as a server or a PC (Personal Computer). The learning device 2 generates a converter and a recognizer, which are trained neural networks, through simultaneous learning. The converter is a neural network that outputs a binary image when a multivalued image is input. The recognizer is a neural network that, when a binary image is input, outputs a human region in the binary image. Note that simultaneous learning of a converter and a recognizer refers to training a neural network that combines a converter and a recognizer. Further, the multivalued image is an example of the first image, and the binary image is an example of the second image.

The imaging device 3 is, for example, a surveillance camera. For example, the imaging device 3 generates a multivalued image of a room in a building by capturing an image of the room. The imaging device 3 converts the multivalued image into a binary image using the converter generated by the learning device 2. The imaging device 3 outputs the converted binary image to the transmission network 6. Note that the above room is an example of a space where an object can be imaged.

The recognition device 4 is an information processing device such as a server or a PC. The recognition device 4 acquires the binary image output by the imaging device 3 from the transmission network 6. The recognition device 4 uses the recognizer generated by the learning device 2 to generate a recognition result for the binary image. The recognition device 4 outputs the generated recognition result to the transmission network 6. The data capacity of the obtained binary image is smaller than that of the original multivalued image, and is a fixed value. Therefore, the transmission capacity required by the recognition device 4 and the transmission network 6 can be reduced, and the storage capacity required for temporary storage and storage by the recognition device 4 can also be reduced. Furthermore, since the binary image is generated by a converter trained simultaneously with the recognizer, the transmission capacity and storage capacity can be reduced while maintaining high recognition accuracy through conversion.

The display device 5 is an information processing device such as a server or a PC. The display device 5 acquires the recognition result output by the recognition device 4 from the transmission network 6. The display device 5 displays the acquired recognition results on a display section such as a liquid crystal display included in the display device 5.

FIG. 3 is a diagram showing an example of a schematic configuration of the learning device 2. As shown in FIG. The learning device 2 includes a first storage section 21, a first communication section 22, and a first processing section 23.

The first storage unit 21 is a device for storing programs or data, and includes, for example, a semiconductor memory device. The first storage unit 21 stores operating system programs, driver programs, application programs, data, etc. used in processing by the first processing unit 23. The program can be executed using a known setup program or the like from a computer-readable, non-temporary, portable storage medium such as a CD (Compact Disc)-ROM (Read Only Memory) or a DVD (Digital Versatile Disc)-ROM. It is installed in the first storage unit 21.

Further, the first storage unit 21 stores learning data 211 and a learning model 212.

The first communication unit 22 includes a communication interface circuit that enables the learning device 2 to communicate with other devices. The communication interface circuit included in the first communication unit 22 is a communication interface circuit such as a wired LAN (Local Area Network) or a wireless LAN. The first communication unit 22 receives data transmitted from another device and supplies it to the first processing unit 23, and also transmits the data supplied from the first processing unit 23 to the other device.

The first processing unit 23 includes one or more processors and their peripheral circuits. The first processing unit 23 is, for example, a CPU (Central Processing Unit), and controls the operation of the learning device 2 in an integrated manner. The first processing unit 23 may be a GPU (Graphics Processing Unit), a DSP (Digital Signal Processor), an LSI (Large-Scaled IC), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or the like. The first processing unit 23 controls the operation of the first communication unit 22 so that various processes of the learning device 2 are executed in an appropriate procedure based on the program stored in the first storage unit 21, and Perform various processing. Further, the first processing unit 23 can execute multiple programs in parallel.

The first processing unit 23 includes learning model acquisition means 231 , learning data acquisition means 232 , edge image generation means 233 , learning means 234 , and output means 235 . Each of these means is a functional module realized by a program executed by the first processing unit 23. Each of these means may be implemented in the learning device 2 as firmware.

FIG. 4 is a diagram showing an example of a schematic configuration of the imaging device 3. As shown in FIG. The imaging device 3 includes a second storage section 31, a second communication section 32, an imaging section 33, and a second processing section 34.

The second storage unit 31 is a device for storing programs or data, and includes, for example, a semiconductor memory device. The second storage unit 31 stores operating system programs, driver programs, application programs, data, etc. used in processing by the second processing unit 34. The program is installed in the second storage unit 31 from a computer-readable, non-temporary, portable storage medium such as a CD-ROM or DVD-ROM using a known setup program or the like.

The second communication unit 32 includes a communication interface circuit that enables the imaging device 3 to communicate with other devices. The communication interface circuit included in the second communication unit 32 is a communication interface circuit such as a wired LAN or a wireless LAN. The second communication unit 32 receives data transmitted from another device and supplies it to the second processing unit 34, and also transmits the data supplied from the second processing unit 34 to the other device.

The imaging unit 33 includes an imaging optical system, an image sensor, an image processing unit, and the like. The imaging optical system is, for example, an optical lens, and forms an image of the light beam from the subject onto the imaging surface of the imaging element. The image sensor is, for example, a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor), and outputs an image signal of a subject image formed on an imaging surface. The image processing section generates image data in a predetermined format from the image signal generated by the image sensor and supplies the generated image data to the second processing section 34 .

The second processing unit 34 includes one or more processors and their peripheral circuits. The second processing unit 34 is, for example, a CPU, and controls the operation of the imaging device 3 in an integrated manner. The second processing unit 34 may be a GPU, DSP, LSI, ASIC, FPGA, or the like. The second processing unit 34 controls the operations of the second communication unit 32 and the imaging unit 33 so that various processes of the imaging device 3 are executed in appropriate procedures based on the programs stored in the second storage unit 31. Control and execute various processes. Further, the second processing unit 34 can execute multiple programs in parallel.

The second processing section 34 includes an imaging means 341, a conversion means 342, and a binary image output means 343. Each of these means is a functional module realized by a program executed by the second processing unit 34. Each of these means may be implemented in the imaging device 3 as firmware.

FIG. 5 is a diagram showing an example of a schematic configuration of the recognition device 4. As shown in FIG. The recognition device 4 includes a third storage section 41, a third communication section 42, and a third processing section 43.

The third storage unit 41 is a device for storing programs or data, and includes, for example, a semiconductor memory device. The third storage unit 41 stores operating system programs, driver programs, application programs, data, etc. used in processing by the third processing unit 43. The program is installed in the third storage unit 41 from a computer-readable, non-temporary, portable storage medium such as a CD-ROM or DVD-ROM using a known setup program or the like.

The third communication unit 42 includes a communication interface circuit that enables the recognition device 4 to communicate with other devices. The communication interface circuit included in the third communication unit 42 is a communication interface circuit such as a wired LAN or a wireless LAN. The third communication unit 42 receives data transmitted from another device and supplies it to the third processing unit 43, and also transmits the data supplied from the third processing unit 43 to the other device.

The third processing unit 43 includes one or more processors and their peripheral circuits. The third processing unit 43 is, for example, a CPU, and controls the operation of the recognition device 4 in an integrated manner. The third processing unit 43 may be a GPU, DSP, LSI, ASIC, FPGA, or the like. The third processing unit 43 controls the operation of the third communication unit 42 so that various processes of the recognition device 4 are executed in an appropriate procedure based on the program stored in the third storage unit 41, and Perform various processing. Further, the third processing unit 43 can execute multiple programs in parallel.

The third processing section 43 includes a binary image acquisition means 431, a recognition means 432, and a recognition result output means 433. Each of these means is a functional module realized by a program executed by the third processing unit 43. Each of these means may be implemented in the recognition device 4 as firmware.

(Overview of converter and discriminator)
FIG. 6 is a schematic diagram for explaining the outline of the converter. The converter is a convolutional neural network (CNN) that outputs a binary image when a multivalued image is input, and has an input layer, a hidden layer, and an output layer. Hidden layers include convolution layers, pooling layers, unpooling layers, and the like.

The input layer of the converter receives a plurality of multivalued images D1 as input. The multivalued image D1 is, for example, an image composed of pixels having tone values within a tone range of 0 to 255 for each of the three RGB channels.

The convolution layer P101 of the converter performs convolution processing using a plurality of filters having predetermined sizes and coefficients on the plurality of multivalued images D1 input to the input layer, and generates a feature map. The generated feature map has the same size as the multivalued image D1 and the same number of channels as the number of filters (256 channels if the number of filters is 256). The convolution layer P101 executes batch normalization processing on the generated feature map, and adjusts each channel so that the feature amount of the generated feature map has a predetermined average value and variance value for each channel. Correct the feature amount. The convolution layer P101 executes an activation process that applies an activation function to each feature corrected by the batch normalization process. The activation function is, for example, a ReLU (Rectified Linear Unit) function. The activation function may be a Hyperbolic Tangent function or a Sigmoid function. The convolution layer P101 may add a predetermined bias value to each feature amount before applying the activation function.

The pooling layer P102 performs pooling processing on the feature map that is the output data of the convolutional layer P101. Pooling processing is a process of reducing the size of a feature map, and for example, maximum value pooling that extracts the largest feature amount from among the feature amounts included in a region of a predetermined size (for example, 2 x 2) in the feature map. (Max Pooling) processing. The pooling process may be an average value pooling process. The pooling layer P102 outputs the feature map generated by the pooling process. The size of the feature map that is the output data of the pooling layer P102 is smaller than the size of the feature map that is the input data of the pooling layer P102, for example, it is half the size of the input data in each of the vertical and horizontal directions. .

The convolution layer P103 performs convolution processing, batch normalization processing, and activation processing on the output data of the pooling layer P102. The pooling layer P104 performs a pooling process on the output data of the convolutional layer P103. The size of the output data of the pooling layer P104 is, for example, half the size of the input data of the pooling layer P104 in both the vertical and horizontal directions.

The convolution layer P105 performs convolution processing, batch normalization processing, and activation processing on the output data of the pooling layer P104. The unpooling layer P106 performs unpooling processing on the output data of the convolutional layer P105. The unpooling process is an upsampling process that increases the size of the feature map. The size of the output data of the unpooling layer P106 is larger than the size of the input data of the unpooling layer P106, for example, twice the size of the input data in each of the vertical and horizontal directions.

The convolution layer P107 performs convolution processing, batch normalization processing, and activation processing on the output data of the unpooling layer P106. The addition layer P108 adds the output data of the convolution layer P107 and the output data of the convolution layer P103. By providing the addition layer P108, the absolute value of the gradient calculated when applying the error backpropagation method described later becomes large, and the learning speed is improved. The unpooling layer P109 performs unpooling processing on the output data of the addition layer P108. The size of the output data of the unpooling layer P109 is, for example, twice the size of the input data of the unpooling layer P109 in both the vertical and horizontal directions.

The convolution layer P110 performs convolution processing, batch normalization processing, and activation processing on the output data of the unpooling layer P109. The addition layer P111 adds the output data of the convolution layer P110 and the output data of the convolution layer P101.

The conversion layer P112 performs channel conversion processing on the output data of the addition layer P111. The conversion layer P112 generates and outputs a one-channel feature map based on feature amounts of multiple channels for each pixel. For example, if the output data of the addition layer P111 is an N-channel feature map, the conversion layer P112 convolves the output data of the addition layer P111 with only one N-channel filter to generate a 1-channel feature map. . Thereby, the conversion layer P112 reduces the data capacity of the feature map.

The active layer P113 executes an activation process that applies an activation function to the output data of the conversion layer P112. The activation function is, for example, a sigmoid function. The activation layer P113 may add a predetermined bias value to each feature amount before applying the activation function.

The threshold processing layer P114 performs threshold processing that applies a step function having a predetermined threshold to the output data of the active layer P113. The step function is a function that converts the feature amount included in the feature map that is the output of the active layer P113 to 1 if it is equal to or greater than a threshold value, and converts the feature amount to 0 if it is less than the threshold value. Thereby, the threshold processing layer P114 outputs a feature map in which the feature amount corresponding to each pixel is 0 or 1.

The output layer of the converter outputs a binary image D2 in which the feature amount of the feature map output from the threshold processing layer P114 is used as the tone value of each pixel. The binary image D2 has the same size as the multivalued image D1, and the tone value of each pixel is 0 or 1. In this way, the converter outputs the binary image D2 when the multivalued image D1 is input.

Note that during learning, the threshold processing layer P114 may superimpose noise on the feature map that is the output of the active layer P113 (not superimpose during recognition) before applying the step function. For example, the threshold processing layer P114 adds a random number generated based on a distribution such as a normal distribution and having a predetermined variance value to each feature amount of the feature map. As a result, the converter allows the gradation value of all pixels of the binary image D2 to be 0 or 1 even if all the feature quantities output from the active layer P113 are less than the threshold value or all the feature quantities are greater than or equal to the threshold value. Reduce the probability that only one of the following will occur. If the gradation values of all pixels of the binary image D2 become 0 or 1, learning cannot be performed even if the binary image D2 is input to the recognizer described later, so the learning speed decreases. . The converter can improve the learning speed by reducing the possibility of outputting such a binary image D2.

Further, in this case, the threshold processing layer P114 may superimpose noise of a size according to the feature amount of the feature map. For example, the threshold processing layer P114 determines, for each feature quantity, a distribution of random numbers such that when the random number is added to each feature quantity, the probability that the relationship with the threshold value changes is a predetermined probability (for example, 1/1000). . The relationship with the threshold value changes means that when a random number is added to a feature quantity that is less than the threshold value, the value becomes greater than or equal to the threshold value, or when a random number is added to a feature quantity that is greater than or equal to the threshold value, it becomes less than the threshold value. . The threshold processing layer P114 generates random numbers based on the distribution determined for each feature amount, and adds the generated random numbers to each feature amount. As a result, the converter outputs a binary image D2 that has no correlation with the multilevel image D1 due to noise while reducing the probability that the tone values of all pixels of the binary image D2 will be 0 or 1. The probability can be reduced.

In addition, the threshold processing layer P114 determines one distribution based on statistical values such as the average value and median value of each feature amount, and adds a random number generated based on the determined one distribution to each feature amount. You may. This allows the converter to superimpose noise with less computational load.

Note that the addition layers P108 and P111 may not be provided in the converter.

FIG. 7 is a schematic diagram for explaining the outline of the recognizer. The recognizer is a CNN that outputs the target area and target type when a binary image is input, and is, for example, an SSD (Single Shot Multibox Detector). The target area is information indicating a rectangular area circumscribing the target image in the input binary image. The target type is information indicating which of a plurality of preset target types the target included in the rectangular area corresponds to. The type of target is, for example, "person", "vehicle", or "chair". The type of target may be "upper body of a person" or the like. Note that if the type of target to be recognized is one type (for example, only "person"), the recognizer does not need to output the type of target.

The input layer of the recognizer receives the binary image D3 as input. Binary image D3 is binary image D2 output from the converter.

The base network P201 is a CNN having multiple convolutional layers and fully connected layers. The base network P201 may be any CNN used for image classification, such as VGG-16. The base network P201 outputs a feature map when receiving the binary image D3.

The feature layer P202 receives the output data of the base network P201 as input. The feature layer P202 performs convolution processing on the input feature map and outputs a feature map smaller in size than the input data. In addition, the feature layer P202 outputs region information indicating a rectangular region estimated from the feature amount of each pixel of the output feature map, and also outputs region information indicating a rectangular region estimated from the feature amount of each pixel of the output feature map. Output reliability information indicating the possibility of being included. The area information is, for example, information on the center coordinates of the rectangular area and the width and height of the rectangular area. The reliability information is, for example, a vector consisting of a plurality of variables corresponding to each type of target and represented by a value between 0 and 1, and the closer the value of each variable is to 1, the more targets of the corresponding type are included. Indicates that there is a high possibility that the

The feature layer P203 receives as input the feature map that is the output data of the feature layer P202. Similar to the feature layer P202, the feature layer P203 executes convolution processing and outputs a feature map smaller in size than the input data, as well as area information and reliability information about the feature map.

After the feature layer P203, any number of feature layers may be provided.

The post-processing unit P204 receives as input the area information and reliability information output from each feature layer. Based on the input reliability information, the post-processing unit P204 determines whether any type of target is included in the rectangular area indicated by each area information, and if so, which type of target is included. Determine whether it is included. The determination is, for example, whether the value of each variable included in the reliability information is greater than or equal to a predetermined value, and if there are multiple variables that are greater than or equal to the predetermined value, which variable has the largest value. It is carried out based on. The post-processing unit P204 integrates a plurality of rectangular areas that are determined to include objects of the same type and whose areas overlap by a predetermined ratio or more. For example, a method such as Non-Maximum Suppression is used to integrate the rectangular areas. As a result, one rectangular area is generated for one object. The post-processing unit P204 outputs the area information of the generated rectangular area as the target area D4 via the output layer, and also outputs the reliability information corresponding to the rectangular area as the target type D5.

(Data structure of various data)
FIG. 8 is a diagram showing an example of the data structure of the learning data 211 stored in the first storage unit 21 of the learning device 2. The learning data 211 is data in which a data ID, a learning multivalued image, and a learning recognition result are associated. Note that the multivalued image for learning is an example of the first image for learning.

The data ID is identification information for identifying the combination of the learning multivalued image and the learning recognition result. The multivalued learning image includes information on the gradation value of each pixel making up the image. In the example shown in FIG. 8, tone values from 0 to 255 are stored for each of the three RGB channels for each pixel. The learning recognition result is a recognition result set in advance to be output for the learning multivalued image, and includes a target area and a target type. The target area is information indicating a rectangular area circumscribing the target image in the learning multilevel image, and is, for example, information about the center coordinates of the rectangular area and the width and height of the rectangular area. The target type information is information indicating which of a plurality of preset target types the target included in the rectangular area indicated by the target area corresponds to. The target type is, for example, a so-called one-hot vector in which the value of a variable corresponding to the relevant type is 1, and the value of variables corresponding to other types is 0. Note that when there is only one type of target to be recognized, the learning recognition result does not need to include the type of target. Further, when a multivalued learning image includes a plurality of objects, information on a plurality of object regions and object types corresponding to each object may be included.

The learning data 211 is set in advance by the administrator of the learning device 2 and stored in the first storage unit 21.

(Processing flow)
FIG. 9 is a flow diagram showing an example of the flow of learning processing executed by the learning device 2. As shown in FIG. The learning process is realized by the first processing unit 23 cooperating with each component of the learning device 2 according to a program stored in the first storage unit 21.

First, the learning model acquisition unit 231 acquires a learning model from the first storage unit 21 (S101). The learning model is a CNN that is coupled such that the output of the converter becomes the input of the recognizer. The learning model acquisition means 231 may initialize parameters such as filter coefficients included in the acquired learning model using random numbers or the like.

Subsequently, the learning data acquisition means 232 acquires the learning data 211 from the first storage unit 21 (S102).

Subsequently, the edge image generation unit 233 generates an edge image from the learning multivalued image included in the learning data 211 (S103). An edge image is a binary image in which the tone values of edge pixels and the tone values of other pixels are different from each other. The edge image generation unit 233 applies Canny's edge detection method to the learning multi-value image to detect edge pixels from the learning multi-value image. The edge image generation unit 233 generates an image in which the tone value of the detected edge pixel is set to 1 and the tone values of other pixels are set to 0 in the learning multivalued image as an edge image.

Note that the edge image generation unit 233 may generate the edge image using a known edge detection filter such as a Sobel filter.

Subsequently, the learning unit 234 generates a recognition result by inputting the learning multivalued image to the learning model (S104). The recognition result is the region of the object and the type of the object output from the learning model. The recognition result may include a binary image output from a converter of the learning model.

Note that the learning unit 234 may input, to the learning model, an image obtained by adding noise to a multivalued learning image. Thereby, the learning device 2 can train the learning model to appropriately output recognition results even if the input multivalued image contains noise. However, in this case, it is preferable that the edge image generation unit 233 generates the edge image from the learning multivalued image before adding noise.

Subsequently, the learning means 234 calculates an error based on the generated recognition result and the learning recognition result (S105). The error is an index that indicates the degree of difference between the generated recognition result and the training recognition result used for learning the converter, and is the weighted sum of the error related to the target area and the error related to the target type. It is calculated by the error function. The error regarding the target area is, for example, a square error or a logarithmic square error of the center coordinate, width, and height between the rectangular area of the generated recognition result and the rectangular area of the learning recognition result. The error related to the target type is, for example, a cross-entropy error between the target type of the generated recognition result and the target type of the learning recognition result.

The error function may further include errors regarding the binary image. The error regarding the binary image is a squared error between the binary image output from the learning model converter and the edge image generated from the learning multivalued image, which is included in the recognition result. Thereby, the learning device 2 causes the learning model to learn so that the binary image output by the converter approaches the edge image, and the recognition result output by the learning model approaches the learning recognition result. The learning device 2 makes it easier for the user of the image recognition system 1 to visually recognize the target image in the binary image by bringing the binary image output by the converter closer to the edge image.

The error regarding the binary image may be a squared error between the binary image and the image with the edge image blurred. An image with a blurred edge image is an image obtained by applying a predetermined filter (for example, a Gaussian filter) to the edge image. Furthermore, the error regarding the binary image may be a squared error between the histogram of the binary image and the histogram of the edge image. For example, in a histogram, when each image is divided into regions of a predetermined size, the number of pixels whose gradation value is 0 (or pixels whose tone value is 1) included in each region is classified as a class, and the number of pixels corresponding to each class is This is a frequency distribution in which the frequency is the number of regions. The histogram may be a HOG (Histogram of Oriented Gradients) that indicates the frequency of gradients of tone values in each image.

Even if there is a slight difference in the position or shape of the edge between the binary image and the edge image, such a slight difference is unlikely to cause a problem when the user visually recognizes the target image in the binary image. By using an image in which the edge image is blurred, the learning device 2 makes it difficult for such slight differences in the position and shape of the edge to be reflected in the error function, thereby facilitating the learning of the converter.

In addition, the error function used for learning a transformer that includes a convolution layer and an activation layer that applies an activation function to an input based on the output of the convolution layer includes the coefficients of the filter applied in the convolution layer. The norm may be included. The norm of the coefficients of the filter is, for example, the sum of squares of the coefficients (L2 norm) or the spectral norm of the filter.

When the L2 norm of the filter coefficient is large, the absolute value of the filter coefficient applied in the convolution layer of the transformer is large, and therefore the absolute value of the feature quantity of the feature map input to the activation layer P113 of the transformer is also large. Prone. In this case, if the activation function applied by the activation layer P113 is a sigmoid function, most of the feature quantities output from the activation layer P113 have values close to 0 or close to 1, and 0 is in the middle. It does not have a value close to .5. When a feature map having such feature amounts is input to the threshold processing layer P114, there is a high possibility that the tone values of all pixels of the image output from the threshold processing layer P114 will be 0 or 1, and recognition Learning of the device is not performed, and the learning speed decreases.

Further, the spectral norm is the maximum ratio of the L2 norm of the feature map that is the output corresponding to each input to the L2 norm of the plurality of feature maps that are input to the convolutional layer. When the spectral norm is large, the absolute value of the feature amount of the output data of the convolutional layer is large, so there is a possibility that the output of the threshold processing layer P114 will be an image in which all pixels have tone values of 0 or 1. This increases the learning speed.

The learning device 2 makes the CNN learn to reduce the value of the L2 norm or the spectral norm by adding the L2 norm or the spectral norm to the error function. Thereby, the learning device 2 can reduce the possibility that the tone values of all pixels of the binary image output from the converter will be 0 or 1, and can improve the learning speed. Note that instead of adding the spectral norm of the filter of the convolutional layer to the error function, the coefficients of the filter normalized so that the spectral norm becomes 1 may be used in the convolution.

Additionally, the error function includes the percentage of pixels whose gradation value is 1 (or the percentage of pixels whose gradation value is 0) among the pixels that constitute the binary image output from the converter. You can. Further, the error function may include a squared error in tone values between each pixel forming the binary image output from the converter and a pixel adjacent to each pixel. By doing so, it is possible to improve the compression efficiency when compressing the binary image output from the converter.

Subsequently, the learning means 234 updates the parameters of the CNN (S106). The learning means 234 calculates the gradient of each layer of the CNN using the error backpropagation method, and updates the parameters so that the error becomes smaller using the stochastic gradient method based on the calculated gradient. The parameters to be updated are the coefficients of the filter applied in the convolution layer and the average value and variance value of each feature amount corrected by batch normalization processing in the convolution layer. The updated parameters may include bias values added to each feature before the activation function is applied in the convolution layer and the activation layer. The updated parameters may include a step function threshold applied in the threshold processing layer, and the like.

When applying the error backpropagation method to update the parameters of the converter, the learning means 234 applies the step function to the input included in the converter, using the gradient of another function different from the step function. It may also be used as the gradient of the threshold processing layer P114. The other functions are functions in which the interval in which the gradient is 0 is smaller than the step function, such as an identity function or a sigmoid function. By doing so, the learning device 2 can update the parameters to further reduce the error and improve the learning speed. That is, in the error backpropagation method, the gradient of the previous layer is calculated based on the gradient of each layer, and the parameters are updated by identifying the parameter that causes a large error. Therefore, if there is a layer that applies a function such as a step function in which sections where the slope is 0 is dominant, it becomes difficult to identify parameters that cause errors in layers before that layer. The learning device 2 uses, as the gradient of the threshold processing layer P114, the gradient of another function different from the step function, in which the section where the gradient is 0 is smaller than the step function, thereby identifying the parameter that causes the error. make it easier.

Subsequently, the learning means 234 determines whether the learning end condition is satisfied (S107). The learning termination condition is, for example, that the parameter has been updated a predetermined number of times or more, or that the amount of change of the updated parameter with respect to the pre-update parameter is less than or equal to a predetermined value.

If it is determined that the termination condition is not satisfied (S107-No), the learning means 234 advances the process to S102. If it is determined that the termination condition is satisfied (S107-Yes), the learning means 234 stores the CNN as a trained model in the first storage unit 21 (S108), and ends the series of processing.

In this way, the learning device 2 generates a converter and a recognizer through simultaneous learning. Thereby, the learning device 2 enables the converter to learn to output a binary image in which the recognizer has high recognition accuracy of the target object.

FIG. 10 is a sequence diagram showing an example of the flow of image recognition processing executed by the image recognition system 1. The image recognition process is performed by the first processing section 23, the second processing section 34, and the third processing section 43 based on the programs stored in the first storage section 21, the second storage section 31, and the third storage section 41. This is achieved by cooperating with the components of the device.

First, the output means 235 of the learning device 2 outputs the converter and the discriminator to the imaging device 3 and the recognition device 4 via the first communication unit 22 (S201). The output unit 235 generates a converter and a recognizer by separating the CNN, which is a trained model stored in the first storage unit 21 . The output means 235 transmits the converter to the imaging device 3 and the recognizer to the recognition device 4 via the first communication unit 22 . The imaging device 3 receives the converter and stores it in the second storage unit 31. The recognition device 4 receives the recognizer and stores it in the third storage unit 41.

Subsequently, the imaging means 341 of the imaging device 3 controls the imaging unit 33 to image one room in the building and generate a multivalued image (S202).

Subsequently, the converting means 342 converts the generated multivalued image into a binary image (S203). The conversion means 342 converts the multivalued image into a binary image by inputting the multivalued image into a converter stored in the second storage unit 31 and outputting the binary image.

Subsequently, the binary image output means 343 outputs the binary image to the transmission network 6 via the second communication unit 32 (S204). The binary image output means 343 may apply a predetermined reversible compression technique to the binary image and output it. Thereby, the imaging device 3 can suppress the transmission capacity of binary images.

Subsequently, the binary image acquisition means 431 of the recognition device 4 acquires the binary image from the transmission network 6 via the third communication unit 42 (S205).

Subsequently, the recognition unit 432 generates a recognition result for the binary image (S206). The recognition unit 432 generates a recognition result by inputting the binary image to a recognizer stored in the third storage unit 41 and outputting the recognition result.

Subsequently, the recognition result output means 433 outputs the generated recognition result to the display device 5 via the third communication unit 42 (S207), and the series of processing ends. For example, the recognition result output means 433 transmits display data to the display device 5 for the display device 5 to display a recognition result screen 700 based on the recognition result.

FIG. 11 is a diagram showing an example of a recognition result screen 700 displayed on the display device 5. As shown in FIG. The recognition result screen 700 includes a binary image 710, a target image 711, a circumscribed rectangle 720, and a type display object 721.

Binary image 710 is a binary image generated by imaging device 3. In the example shown in FIG. 11, pixels with gradation values of 1 and 0 are shown in black and white, respectively. In the example shown in FIG. 11, the target image 711 is a full-body image of a person. The circumscribed rectangle 720 is a rectangular object that circumscribes the target image 711 and is displayed based on the target area generated by the recognition device 4. The type display object 721 is an object that is displayed based on the type of the target generated by the recognition device 4 and indicates the type of the target using characters or the like. The type display object 721 indicates, for example, the type of target corresponding to the variable with the largest value among the variables included in the type of target generated by the recognition device 4.

As described above, in the image recognition system 1, the learning device 2 trains a CNN that is coupled so that the output of the converter becomes the input of the recognizer. Then, the imaging device 3 converts the multivalued image into a binary image using a converter that is a trained model, and the recognition device 4 generates a recognition result for the binary image using a recognizer that is a trained model. By doing so, the image recognition system 1 makes it possible to stably reduce the data volume of the image that is the object of image recognition while maintaining the accuracy of image recognition.

Note that in the above description, the input to the converter is an image having a plurality of channels, but the input to the converter may be a one-channel image (for example, a grayscale image).

Further, in the above description, it is assumed that the output of the converter is a one-channel binary image, but the present invention is not limited to this. The converter may output a plurality of binary images corresponding to each channel of the input multilevel image. For example, if the input multilevel image has three channels, RGB, the converter converts a binary image corresponding to the R channel, a binary image corresponding to the G channel, and a binary image corresponding to the B channel. Generate each.

In this case, the recognizer accepts multiple binary images as input. Furthermore, the edge image generation means 233 generates edge images corresponding to each channel based on the gradation values of each channel of the learning multilevel image. By doing so, the recognition accuracy of the recognizer is improved.

Further, the output of the converter may be a multi-value image having a smaller gradation range than the multi-value image input to the converter. This improves recognition accuracy in the recognizer.

Furthermore, in the above description, the image recognition system 1 includes one imaging device 3 and one recognition device 4, but the present invention is not limited to this. The image recognition system 1 may include a plurality of imaging devices 3 or recognition devices 4. In this case, the learning device 2 outputs a converter to each of the plurality of imaging devices 3 or outputs a recognizer to each of the plurality of recognition devices 4.

Further, the functions of the learning device 2 or the display device 5 may be realized by the imaging device 3 or the recognition device 4.

In addition, in the above explanation, the recognizer that recognizes the area of the object or the area and type of the object and the corresponding converter were exemplified, but the recognizer that recognizes attributes such as a person's age and gender and the corresponding converter are exemplified. It may be a recognition device that recognizes conditions such as the degree of crowding or posture of people or vehicles, and a corresponding converter, and can be applied to image recognition of various objects. In these cases, learning is performed by setting learning recognition results according to the target.

It should be understood that those skilled in the art can make various changes, substitutions, and modifications thereto without departing from the spirit and scope of the invention. For example, the processing of each part described above may be executed in a different order as appropriate within the scope of the present invention. Furthermore, the embodiments and modifications described above may be implemented in appropriate combinations within the scope of the present invention.

1 Image recognition system 2 Learning device 231 Learning model acquisition means 232 Learning data acquisition means 233 Edge image generation means 234 Learning means 235 Output means 3 Imaging device 341 Imaging means 342 Conversion means 343 Binary image output means 4 Recognition device 431 2 Value image acquisition means 432 Recognition means 433 Recognition result output means

Claims (6)

Imaging means for imaging a space in which a predetermined object can be imaged to generate a first image consisting of pixels having gradation values within a first gradation range;
A converter that outputs, when the first image is input, a second image consisting of pixels having tone values within a second tone range smaller than the first tone range, converts the first image into a second image. converting means for converting the image into the second image;
a recognition unit that generates a recognition result for the second image by a recognizer that performs processing for recognizing the target on the second image and outputs the recognition result when the second image is input; , comprising ;
The converter and the recognizer are configured to include a learning array consisting of pixels having tone values within the first tone range, which are connected to a neural network such that the output of the converter becomes the input of the recognizer. A trained neural network that is trained to bring the recognition result output when one image is input closer to the learning recognition result set in advance as the recognition result to be output for the first learning image. be,
An image recognition system characterized by: The converter and the recognizer are configured to select pixels having tone values within the second tone range that are output by the converter when the first training image is input to the combined neural network. a trained neural network that has been trained to bring an image closer to an edge image generated from the first learning image and to bring the recognition result output by the combined neural network closer to the learning recognition result. is,
The image recognition system according to claim 1 . output means for outputting the converted second image to a predetermined transmission network;
acquisition means for acquiring the second image from the transmission network;
Furthermore,
The recognition means generates a recognition result for the acquired second image,
The image recognition system according to claim 1 or 2 . Imaging means for imaging a space in which a predetermined object can be imaged to generate a first image consisting of pixels having gradation values within a first gradation range;
A converter that outputs, when the first image is input, a second image consisting of pixels having tone values within a second tone range smaller than the first tone range, converts the first image into a second image. converting means for converting the image into the second image;
output means for outputting the second image ,
The converter is configured such that the output of the converter is the input of a recognizer that performs processing on the second image to recognize the target when the second image is input and outputs a recognition result. When a first learning image consisting of pixels having a gradation value within the first gradation range is input to a neural network connected so that It is a trained neural network that has been trained to approximate the learning recognition result set in advance as the recognition result to be output for the
An imaging device characterized by: When a first image consisting of pixels having tone values within a first tone range is input, from pixels having tone values within a second tone range smaller than the first tone range. an acquisition unit that acquires a second image obtained by converting the first image generated by imaging using a converter that outputs a second image;
Recognition means that generates a recognition result for the second image by a recognizer that performs processing for recognizing a predetermined object on the second image and outputs the recognition result when the second image is input. and ,
The recognizer inputs a first image for learning consisting of pixels having tone values within the first tone range to a neural network coupled such that the output of the converter becomes the input of the recognizer. is a trained neural network that has been trained so that the recognition result output when
A recognition device characterized by: imaging a space in which a predetermined object can be imaged to generate a first image consisting of pixels having tone values within a first tone range;
A converter that outputs, when the first image is input, a second image consisting of pixels having tone values within a second tone range smaller than the first tone range, converts the first image into a second image. into the second image,
A recognition result for the second image is generated by a recognizer that performs processing for recognizing the target on the second image when the second image is input and outputs the recognition result. including,
The converter and the recognizer are configured to include a learning array consisting of pixels having tone values within the first tone range, which are connected to a neural network such that the output of the converter becomes the input of the recognizer. A trained neural network that is trained to bring the recognition result output when one image is input closer to the learning recognition result set in advance as the recognition result to be output for the first learning image. be,
An image recognition method characterized by:

Images