JP2020042367A - Learning system, server, and feature amount image drawing interpolation program - Google Patents

Abstract

To provide a learning system, a server, and a program that can reduce the number of cameras installed in each store of chain stores and reduce the costs of newly opening and maintaining each store of the chain stores.SOLUTION: In a learning system, a server 2 includes: a drawing processing unit that draws a feature amount image on the basis of feature amounts extracted from images captured by a plurality of cameras; a feature amount image interpolation unit that interpolates a missing portion of the feature amount image by applying a learned neural network to the entire feature amount image including the missing portion of the feature amount image caused by the missing of the captured images; and a machine learning unit that performs machine learning of the neural network before completion of the learning using a learning image based on the feature amount image drawn by the drawing processing unit on the basis of the feature amounts extracted from the images captured by all the cameras.SELECTED DRAWING: Figure 2

Description

  The present invention relates to a learning system, a server, and a feature image drawing interpolation program.

  2. Description of the Related Art Conventionally, in a store such as a store or a restaurant, cameras such as a surveillance camera and a so-called AI (Artificial Intelligence) camera are installed in various places in the store. Since these cameras are used to prevent crimes such as shoplifting and to grasp the behaviors of customers and employees (see Patent Literature 1 and the like), for example, drugstores each having a size of about 400 square meters are used in cameras. By installing about 50 cameras, the behaviors of customers and employees in the store can be converted into data and applied to various analyses.

JP 2018-93283 A

  However, since cameras such as the above-mentioned surveillance cameras and AI cameras are expensive, installing a large number of cameras in each store requires new store opening costs (initial costs) and store maintenance costs (running costs). ) Leads to a rise.

  The present invention is to solve the above problems, a learning system capable of reducing the number of cameras to be installed in each store of the chain store, and reducing the cost of new opening and maintenance of each store in the chain store, It is an object to provide a server and a feature quantity image drawing interpolation program.

  In order to solve the above-described problem, a learning system according to a first aspect of the present invention communicates with a plurality of cameras arranged in a chain store having stores having the same layout and belonging to a series, and the plurality of cameras. A learning system including a server, wherein the server is a drawing processing unit that draws a feature amount image based on a feature amount extracted from images captured by the plurality of cameras; and Since the number of arranged cameras is smaller than the number of cameras to be originally arranged, when the photographed images are missing, a missing portion of the feature amount image caused by the lack of these photographed images is included. A feature image interpolating unit that interpolates a missing part of the feature image by applying a learned neural network to the entire feature image; In another store belonging to the series having a camera missing from the certain store among the cameras to be drawn, the drawing is performed based on feature amounts extracted from images taken by all cameras arranged in the other store. A machine learning unit that performs machine learning of the neural network before learning is completed using a learning image based on the feature amount image drawn by the processing unit.

In this learning system, the other store is a store provided with all cameras to be originally arranged among stores other than the certain store belonging to the series,
The machine learning unit is configured to perform learning based on the feature amount image drawn by the drawing processing unit based on feature amounts extracted from images captured by all the cameras to be originally arranged, which are arranged in the another store. It is desirable to perform machine learning of the neural network before learning is completed using the image for training.

  In this learning system, it is preferable that the feature amount image is a heat map image in which feature amounts extracted from images captured by the plurality of cameras are visualized using colors.

  In this learning system, the feature quantity is multidimensional data composed of data of a plurality of types of feature quantities, and the feature quantity image is a color heat map image obtained by visualizing the multidimensional data using colors. It is desirable that

  In this learning system, it is preferable that the drawing processing unit draws the heat map image based on the feature amount for a predetermined time extracted from images captured by the plurality of cameras.

  In this learning system, the feature amount is an image itself taken by each of the plurality of cameras, and the drawing processing unit in the server combines these images based on the image taken by each of the plurality of cameras. An image in a store that is an image is drawn as a feature amount image, and the learned neural network has a smaller number of cameras arranged in the certain store than the number of cameras to be originally arranged. When the photographed image is missing, the image in the certain store may be completed by interpolating the missing photographed image.

  In this learning system, the neural network is a neural network of a generative model, and the feature image interpolating unit uses the learned neural network to generate a missing portion of the feature image. The missing part may be interpolated.

  A server according to a second aspect of the present invention is a server capable of communicating with a plurality of cameras arranged in a store having a chain store in which affiliated stores have the same layout, and is extracted from images taken by the plurality of cameras. The drawing processing unit that draws a feature amount image based on the obtained feature amount, and the number of cameras arranged in the store where the chain store is located is smaller than the number of cameras that should be originally arranged, so that the photographing is performed. By applying a learned neural network to the entire feature amount image including a missing portion of the feature amount image caused by the lack of these captured images when the image is missing, the feature amount is obtained. A feature amount image interpolating unit that interpolates a missing part of an image, and another store belonging to the series having a camera that is missing from the certain store among the cameras to be originally arranged. And, using a learning image based on the feature amount image drawn by the drawing processing unit based on the feature amount extracted from images captured by all the cameras arranged in the other stores, the learning before completion of the learning A machine learning unit that performs machine learning of the neural network.

  In the server, it is preferable that the feature amount image is a heat map image in which feature amounts extracted from images captured by the plurality of cameras are visualized using colors.

  The feature quantity image drawing interpolation program according to the third aspect of the present invention is a program for converting a computer belonging to a series into feature quantities extracted from images captured by a plurality of cameras arranged in a store having a chain store having the same layout. Based on the drawing processing unit that draws the feature amount image, the number of cameras arranged at the store where the chain store is located is smaller than the number of cameras that should be originally arranged. In this case, by applying a learned neural network to the entire feature amount image including the missing portion of the feature amount image caused by the lack of these captured images, the missing portion of the feature amount image In a feature amount image interpolating unit to be interpolated, and in another store belonging to the series having a camera lacking in the certain store among the cameras to be originally arranged, The neural network before learning is completed using a learning image based on the feature amount image drawn by the drawing processing unit based on the feature amount extracted from images captured by all cameras arranged in the other stores. Is a feature amount image drawing interpolation program for functioning as a machine learning unit that performs machine learning.

  In the feature amount image drawing interpolation program, it is preferable that the feature amount image is a heat map image in which feature amounts extracted from images captured by the plurality of cameras are visualized using colors.

  According to the learning system according to the first aspect of the present invention, the server according to the second aspect, and the feature quantity image drawing interpolation program according to the third aspect, the cameras that should be arranged and are missing from a certain store In another store belonging to the (same) series provided with, using a learning image based on a feature amount image drawn based on a feature amount extracted from images captured by all cameras arranged in the other store, Machine learning of neural network before learning is completed. Here, in a so-called chain store, stores belonging to the affiliate have the same layout. For this reason, as described above, drawing is performed based on the feature amounts extracted from the captured images of the other stores belonging to the (same) series having the camera missing in a certain store among the cameras to be originally arranged. By using the learning image based on the feature amount image and performing machine learning of the neural network, the number of cameras arranged in a store belonging to the same series is smaller than the number of cameras that should be originally arranged. Of the cameras that should be arranged, even if images captured by some cameras are missing, the missing part of the feature amount image caused by the lack of these captured images is obtained using a learned neural network, Can be interpolated.

  As a result, even when the number of cameras to be arranged at each chain store is smaller than the number of cameras to be originally arranged, it is inferior to the case where the same number of cameras to be originally arranged are arranged. It is possible to obtain a feature-value-free image. Therefore, it is possible to reduce the number of cameras installed in each chain store, thereby reducing the cost of opening and maintaining each chain store.

FIG. 1 is a block diagram showing a schematic configuration of a learning system according to an embodiment of the present invention. FIG. 2 is a functional block configuration diagram of an edge camera and a server included in the learning system. 9 is a flowchart of a process of preparing for learning of an image interpolation network (and a discriminator) in the learning system. FIG. 4 is an explanatory diagram of a photographing area in an edge camera of the learning system and a feature amount for each photographing area. FIG. 4 is an explanatory diagram of a photographing area when images photographed by the edge camera are arranged so as to reproduce a photographed image of the entire shop. FIG. 4 is an explanatory diagram of the learning preparation processing of FIG. 3. 9 is a flowchart of a process of generating a heat map image including a missing portion (missing heat map image) and an interpolation process of the missing heat map image in the learning system. FIG. 8 is an explanatory diagram of the missing heat map image generation processing and the missing heat map image interpolation processing of FIG. 7. Explanatory drawing of the image interpolation network in the learning system. FIG. 4 is an explanatory diagram of each layer included in the image interpolation network in the learning system. Explanatory drawing of the discriminator in the learning system. Explanatory drawing of each layer contained in the discriminator in the learning system. FIG. 4 is an explanatory diagram of machine learning of an image interpolation network and a discriminator in the learning system.

  Hereinafter, a learning system, a server, and a feature image drawing interpolation program according to an embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram illustrating a schematic configuration of a learning system according to the present embodiment. The learning system 10 includes a plurality of edge cameras 1 (so-called AI (Artificial Intelligence) cameras: corresponding to “cameras” in the claims), and a server 2 (on a cloud) that communicates with the edge cameras 1. ing. The above-described edge camera 1 is a camera having a so-called edge computing function, and stores belonging to affiliates are arranged in each chain store having the same layout. Although FIG. 1 shows only one edge camera 1, a plurality of edge cameras 1 are installed in each chain store. The number of edge cameras 1 that should be originally arranged in each chain store is, for example, 50 or more, but only some of the edge cameras 1 that should be arranged (only 50 or more) are provided in some stores. Edge cameras 1) are installed, and in many other stores, only a smaller number (for example, 5 to 10) of edge cameras 1 than the number of edge cameras 1 to be originally installed are installed.

  The edge camera 1 includes a camera unit 11, a CPU 12 that controls the entire apparatus and performs various calculations, and a communication unit 13. The CPU 12 includes a GPU (discrete GPU) not shown or an external GPU (not shown) equipped with a GPU for machine learning calculation that can be connected to the edge camera 1 by a graphics card, USB, or the like. Device).

  Further, the edge camera 1 includes a memory 14 for storing various data and programs. The programs stored in the memory 14 include an edge camera-side control program 15. A part or all of the edge camera-side control program 15 may be stored in a memory (not shown) in a GPU.

  The server 2 includes a CPU 21 that controls the entire apparatus and performs various calculations. The server 2 has a communication unit 22, and communicates with the edge camera 1 via the communication unit 22. The communication unit 22 includes a communication IC.

  The server 2 includes a hard disk 23 for storing various programs and data, a RAM 24 for loading programs and data to be executed when the various programs are executed, a display 25, and an operation unit used for various input instruction operations. 30. The hard disk 23 stores a heat map drawing interpolation program 26, an image interpolation network I, a discriminator D, and a training data set 29 for learning the image interpolation network I. The image interpolation network I and the discriminator D correspond to a generator and a discriminator of a GAN (more precisely, DCGAN (Deep Convolutional Advertisement Networks)) which is a kind of a generation model. The hard disk 23 also stores the parameter data 27 of the image interpolation network I and the parameter data 28 of the discriminator D. Although not shown in the figure, it is desirable that the server 2 also includes a GPU.

  FIG. 2 shows a functional block on the edge camera 1 side and a functional block on the server 2 side. The CPU 12 on the edge camera 1 side has a feature amount extraction unit 30. The feature amount extraction unit 30 extracts feature amounts such as the number of women, the number of men, and the residence time from an image captured by the own device (edge camera 1). The feature amount extraction unit 30 uses, for example, an R-CNN-based object detection engine to detect the number of men and women in the captured image. The R-CNN-based object detection engine has a function of extracting an object-like region in an input image, and applying CNN to the extracted region to determine to which class the image of the extracted region belongs. And the ability to classify.

  Further, the CPU 21 of the server 2 includes a drawing processing unit 31, a feature amount image interpolation unit 32, and a machine learning unit 33. The drawing processing unit 31 draws a heat map image (“feature amount image” in the claims) based on the feature amount extracted by the feature amount extraction unit 30 from images captured by the plurality of edge cameras 1. . This heat map image is an image obtained by visualizing feature amounts extracted from images captured by the plurality of edge cameras 1 using colors. When the feature amount extracted by the feature amount extraction unit 30 is multi-dimensional data composed of data of a plurality of types of feature amounts, the heat map image is obtained by converting the multi-dimensional data using colors. It becomes a heatmap image of the visualized color.

  Since the number of edge cameras 1 arranged in a store where a chain store is located is smaller than the number of edge cameras 1 which should be originally arranged, the feature amount image interpolation unit 32 lacks an image captured by the edge camera 1. In this case, a learned image interpolation network I ("learned neural network" in the claims) is applied to the entire heat map image including the missing part of the heat map image caused by the lack of these captured images. Is applied, the missing part of the heat map image is interpolated.

  The above-mentioned machine learning unit 33 performs, in the other stores belonging to the same series, the edge cameras 1 which are to be arranged, including the edge cameras 1 which are missing in a certain store, and all the other cameras arranged in the other stores. The image interpolation network I (and the discriminator) before learning is completed, using a learning image based on the heat map image drawn by the drawing processing unit 31 based on the above-described feature amount extracted from the image captured by the edge camera 1 of FIG. Perform the machine learning of D). To be more precise, the machine learning unit 33 extracts, from among the stores other than a certain store belonging to the affiliated system, the images captured by all the edge cameras 1 in the other stores having all the edge cameras 1 to be originally arranged. Based on the extracted feature amount, machine learning of the image interpolation network I (and discriminator D) before the completion of learning is performed using a learning image based on the heat map image drawn by the drawing processing unit 31. .

  The function of the feature amount extraction unit 30 of the edge camera 1 is realized by the CPU 12 of the edge camera 1 executing the edge camera control program 15. The function of each block (drawing processing unit 31, feature amount image interpolation unit 32, and machine learning unit 33) in the CPU 21 of the server 2 is realized by the CPU 21 executing the heat map drawing interpolation program 26. . However, the present invention is not limited to this configuration. For example, at least one of the functions of each block in the CPU 12 and the CPU 21 may be realized by individual hardware including an ASIC (Application Specific Integrated Circuit) or the like.

  Next, with reference to FIGS. 4 to 6 in addition to the flowchart of FIG. 3, a preparation process for learning of the image interpolation network I (and the discriminator D) will be described. In the learning preparation process, the server 2 generates a heat map image drawn by the drawing processing unit 31 using the feature amount at each time t transmitted from time to time from all edge cameras 1 to be originally arranged. This is a process of repeatedly storing in the training data set as (the base image of) the learning image.

In the learning preparation process, the feature amount extraction unit 30 of the CPU 12 of the edge camera 1 converts the image captured by the own device (edge camera 1) into a 4 × 4 (4 rows × 4 columns) shooting area AR shown in FIG. divided into _i, extracts features _{a it} of each imaging area AR _i at time t (determined) that the photographic area feature amount information _{c it} including the feature quantity _{a it,} repeats the process of transmitting to the server 2 (S1 ). Here, i indicates the number of the shooting area. The serial number of the shooting area included in the images shot by all the edge cameras 1 to be originally arranged is assigned to the shooting area number i. More specifically, as shown in FIG. 5, when the images captured by all the edge cameras 1 are arranged so as to reproduce the captured images of the entire store, the image i of the entire store is assigned to the number i of the capture area. A serial number of the shooting area included in the image is given. In the example shown in FIG. 5, 64 edge cameras having 8 rows and 8 columns are arranged in the store. In this case, the maximum value of the number i of the shooting area is (8 × 4) × (8 × 4). ) = 32 × 32 = 1024.

Imaging area feature amount information _{c it} above the imaging area per AR _i is expressed by the following equation (1).
c _it = {xi _, y _i , a _it } (1)

The above x _{i and} y _i represent the position of the imaging area AR _{i in} the x-coordinate direction (horizontal direction) and the y-coordinate direction (vertical direction), respectively, and a _it is the characteristic of each imaging area AR _{i at} time t. Represents the quantity a _it . The position of the x-coordinate direction of the imaging area AR _i (horizontal direction) and y coordinate direction (vertical direction), for example, the minimum value of the minimum value and the y coordinate of the x-coordinate of the imaging areas AR _i. Note that, as shown in Expression (1), the position x _i in the x coordinate direction and the position y _{i in the} y coordinate direction of the imaging area AR _i are unchanged regardless of the time t.

The feature amount a _{it for} each shooting area AR _{i at} the time t is, for example, the number of men (staying), the number of women, and the staying time included in the shooting area AR _{i at} the time t. . In this case, the feature amount a _it is represented by the following equation (2). Here, the residence time, for example, among of which (remains) who stays in imaging areas AR _i, the residence time is the longest human residence time. However, currently, the average value of the residence time of the people who are staying in the shooting area AR _i, may be as the value of the residence time.
a _it = {number of men _it , number of women _it , residence time _it } (2)

The lower part in FIG. 4 shows the feature amount a _it for each shooting area AR _{i at} each time. For example, as shown in FIG. _4, a it = For {1,1,4}, are included in the captured area AR _i at time t (staying) Number = 1 male, number of women = 1 person, dwell time = 4 seconds.

The server 2 converts the photographing area feature amount information c _{1t to} c _nt at time t of all the photographing areas AR _{1 to} AR _n acquired from all the edge cameras 1 into feature amount information as shown in the following equation (3). _Collected as ct.
c _t = {c _0t, c _1t,... , c _nt } (3)

Then, the server 2 generates all obtained from the edge camera 1, the time from the feature amount information _{c (t-T) ~c t} at time t from (t-T), heat map image HM _t at time t (S2). Here, T is a predetermined time, and is set to, for example, 30 seconds. The above heat map image HM _t is one in which the characteristic amount contained in the feature amount information c _(t-T) ~c _t of the predetermined time T min, and visualized by using color. More particularly, the feature amount a _1t ~a _nt included in the feature amount information c _t at time t, as shown in equation (2), for example, female persons, male persons, and residence time of the data Is multidimensional data composed of In the case of this example, for example, the number of males _it , the number of females _it , and the residence time _it included in the feature amount a _it are represented by R (red), B (blue), and G (green), respectively. heat map image HM _t of will color the heat map image.

However, heat map image HM _t, as described above, are those generated based on the feature amount information _{c (t-T)} to c _t for a predetermined time T minutes, the time (t-T) ~ time t generated from the feature amount information c _(t-T) ~c _t at each time, the time a (t-T) color heatmap image of the to time t, is one synthesized. When the color heat map images at each time are combined, the weighting of the R, G, and B luminance values in the heat map image to be combined becomes smaller as the time goes back (the latest time t , So that the weighting of the R, G, and B luminance values is the largest, and the weighting of the R, G, and B luminance values at the oldest time (t-T) is the smallest. Therefore, it is desirable to combine these heat map images.

As shown in FIG. 5, the number of shooting area, if it is 32 rows and 32 columns, expressed each shot area by one pixel, the size of the heat map image HM _t will 32 × 32 = 1024 pixels .

CPU21 of the server 2, the heat map image HM _t generated in the signal line S2 in FIG. 3 is added to the training data set 29 (see FIG. 2) as an image on which to base the learning image (S3 in FIG. 3).

When the processing of FIG. 3 is viewed as a whole in the learning system 10, as shown in FIG. 6, the learning system 10 performs the entire shooting from the captured images at the time (t−T) to the time t acquired by the all edge cameras 1. extracting the time (t-T) ~ time t feature amount information _{c (t-T)} to c _t of the area, the (predetermined) time T min of the feature amount information _{c (t-T)} to c _t based on, to generate a heat map image HM _t, the heat map image HM _t to be varies continuously generating repeats the process of adding to the training data set 29 as the learning images (images to be of the group). Then, at the time of learning, the image interpolation network I based on the heat map image (for example, the heat map image HM _t ) at each time stored in the training data set 29 and the random mask corresponding to the heat map image at each time. (Machine) learning of That is, (machine) learning of the image interpolation network I is performed using the heat map image on which the random mask is applied as a learning image. To be precise, in the present embodiment, a network corresponding to a DCGAN generator is used as the image interpolation network I. Therefore, when learning the image interpolation network I corresponding to the above-described generator, the discriminator D (Discriminator) is used. ) Is also learned. The (machine) learning processing of the image interpolation network I (and the discriminator D) will be described later in detail.

  Next, the interpolation process of the heat map image using the image interpolation network I after the machine learning (the learned image interpolation network I) will be described. The image interpolation processing by the learned image interpolation network I corresponds to a so-called neural network inference processing. In a store where only a smaller number of edge cameras 1 (for example, 5 to 10) are installed than the number of edge cameras 1 to be arranged, the number of edge cameras 1 to be arranged is determined by the number of edge cameras to be originally arranged. Since the number is smaller than 1, the captured image is missing. For this reason, when the number of edge cameras 1 arranged in the store is smaller than the number of edge cameras 1 to be originally arranged, the images taken from all the edge cameras 1 arranged in the store are shown in FIG. In the images arranged as shown in FIG. 8, a missing portion as shown in FIG. 8 occurs. The learned image interpolation network I, as described above, interpolates a missing portion of the heat map image caused by a missing captured image when the captured image is missing.

The details of the above-described heat map image interpolation processing will be described with reference to the flowchart of FIG. 7 and FIG. First, extraction each edge camera 1 in store small edge camera 1 than the number is arranged to be located originally, from the captured image of the own apparatus, the feature amount a ^L _it of each imaging area AR _i at time t ( determination) that the imaging area feature amount information ^c _{L it} including the feature quantity ^a _{L it,} repeats the process of transmitting to the server 2 (S11).

Imaging area feature amount information c ^L _it of the shooting area per AR _i is represented by the above formula (1) and similar following formula (4).
_{^{c L it = {x L i}} , y L i, a L it} ··· (4)

The above _{^{x L i, y L i,}} a L , respectively, corresponding to _{x i,} y _{i, a it} in the formula (1). Incidentally, the upper characteristic quantity a ^L _it, like the above feature quantity a _it, as a feature, male persons, female persons, and includes a retention time is represented by the formula similar to the formula (2) You.

Server 2 could be obtained from each edge camera 1, the imaging area feature amount information ^c _L 1t ^~c _{L mt} at time t, as shown in the following formula (5), the feature amount information ^c _{L t} As a summary. Here, the above m is a number smaller than n in the above equation (3).
_{^{c L t = {c L 0t}} , c L 1t, ···, c L mt} ··· (5)

Then, the server 2 obtains the missing heat map at time t from the feature amount information c ^L _{(t−T) to} c ^L _t from time (t−T) to time t, which could be obtained from each edge camera 1. generating an image ^HM _{L t} (S12). However, when the heat map image interpolation process is performed, since the captured image is missing as described above, the captured area feature amount information (at each time) transmitted from each edge camera 1 to the server 2 is transmitted. For example, the shooting area feature amount information c ^L _{1t to} c ^L _{nt at} time t also lacks. For this reason, as shown in FIG. 8, the missing heat map image generated from the feature amount information c ^L _{(t−T) to} c ^L _t that summarizes the imaging area feature amount information from time (t−T) to time t. also HM ^L _t, missing part occurs.

As shown in FIG. 8, the missing heat map image HM ^L _t, the input to the learned image interpolation network I (S13 in FIG. 7), the learned image interpolation network I is (interpolated) of the interpolation heatmap and outputs the image IHM _t (S14 in FIG. 7). That is, the learned image interpolation network I is interpolates the missing part of the missing heat map image HM ^L _t input.

As described above, by interpolating the missing heat map image at each time using the learned image interpolation network I, the number of edge cameras 1 to be arranged at each store in the chain store is determined by the edge to be originally arranged. Even if the number is smaller than the number of cameras 1, a heat map image (interpolated heat map image IHM _t ) is obtained which is comparable to the case where the same number of edge cameras 1 as the number of edge cameras 1 to be arranged is arranged. Can be. Therefore, it is possible to reduce the number of edge cameras 1 installed in each chain store, and to reduce the cost of newly opening and maintaining each chain store.

  Next, an image interpolation network I corresponding to a DCGAN generator and a discriminator D corresponding to a DCGAN discriminator will be described with reference to FIGS. The image interpolation network I is a Full Convolutional Network (FCN) -based neural network, and includes a number of convolution layers as shown in FIGS. 9 and 10. More precisely, behind each convolution layer in the image interpolation network I is a ReLU (Rectified Linear Unit) layer. The output layer (“OUTPUT” in FIG. 10) in the image interpolation network I is a convolution layer with a sigmoid function.

  Note that “type” in FIG. 10 indicates the type (type) of a layer, and “filter size” indicates the size of a kernel (filter for convolution operation). “Dilation” indicates the size of the interval in the Diluted Convolution (convolution in which the interval between the counterparts that take the product with the filter (kernel) is taken), and “stride” indicates the size of the window to which the filter is applied. Indicates the interval. The “number of output channels” in FIG. 10 indicates the number of output channels of each layer (corresponding to the number of filters). In the “type” column of FIG. 10, conv. , Deconv. , Dilated conv. , Output indicate a normal convolution layer, a layer that performs deconvolution (convolution after performing upsampling (amplifying)), a layer that performs the above-described Diluted Convolution, and an output layer (output layer).

  As shown in FIGS. 9 and 10, the image interpolation network I reduces the original image size H × W by setting the stride at the time of convolution to 2 × 2 in the second and fourth convolution layers. Are convolved while reducing the size to (H / 2) × (W / 2) and (H / 4) × (W / 4), respectively. As described above, by repeating the convolution process while performing the image reduction process, the position information of the feature in the image becomes ambiguous. After that, the number of receptive fields (input portions that affect a certain pixel) is increased by performing a Diluted Convolution (convolution with a space between the counterparts that take the product with the filter). Then, in the third and fifth convolutional layers (deconv.) From the end, the stride is set to (1/2) × (1/2), and deconvolution is performed, so that (H / 4) × The convolution process is repeated while restoring the size of the image (W / 4) to (H / 2) × (W / 2) and H × W, respectively. Note that H × W in FIG. 9 is, for example, 32 (pixels) × 32 (pixels).

When inference by trained image interpolation network I as shown in FIG. 8, the input image to the image interpolation network I is a missing heatmap image HM ^L _t. During this reasoning, the feature image interpolation unit 32 of the CPU21 of the server 2 side, after application of the image interpolation network I in for the missing heat map image HM ^L _t input, the missing portion of the missing heat map image HM ^L _t input Is replaced with the output image from the output layer of the image interpolation network I (the output heat map image in FIG. 9). That is, the feature quantity image interpolation unit 32 in the CPU21 of the server 2 side, the image of the missing portion in the missing heat map image HM ^L _t input, the output image (output heatmap image from the output layer of the image interpolation network I replaced with), but for the image of the portion other than the missing portion of the missing heat map image HM ^L _t, missing inputted heat map image HM L ^t _(partial image in), used as it is, the interpolation heat shown in FIG. 8 to generate a map image IHM _t.

On the other hand, when learning the image interpolation network I (and discriminator D) as shown in FIG. 6, the input image (learning image) to the image interpolation network I is a heat map image with random mask information. That is, the input image to the image interpolation network I during learning, the no missing portion heatmap image HM _t, is an image in which a mask area of the random position and size. At the time of this learning, the feature amount image interpolating unit 32 in the CPU 21 of the server 2 applies the image interpolation network I to the heat map image having the above mask area (hereinafter, “heat map image with mask”). Only the mask area in the input heat map image with the mask is replaced with the output image from the output layer of the image interpolation network I (the output heat map image in FIG. 9). That is, the feature amount image interpolating unit 32 in the CPU 21 of the server 2 determines the output image (from the output layer of the image interpolation network I) for the image of the mask area in the input heat map image with the mask as in the inference. The output heat map image) is replaced with the interpolation heat map image IHM _t for the image of the portion other than the mask region in the masked heat map image, using the input heat map image with the mask (partial image in) as it is. Generate Then, at the time of learning, the interpolation heat map image IHM _t is sent to the discriminator D, used in learning of the discriminator D.

Note that the image of the missing portion in the missing heat map image HM ^L _t which is entered when the above reasoning to the trained image interpolation network I, and the mask regions in the mask with a heat map image input to the image interpolation network I during learning For the images, before input to the image interpolation network I, the luminance values of R, G, and B at each pixel of these images (images of the missing portion and the mask area) are respectively set for all the pixels in the training data set 29. It is desirable that the average R, G, and B luminance values of the heat map image be set.

Next, the discriminator D will be described. The discriminator D is an input image based on a (real) heat map image (for example, a heat map image HM _t shown in FIG. 6) generated based on feature amount information acquired from all edge cameras 1. This is a network that identifies whether or not there is an interpolated heat map image generated (interpolated) using the image interpolation network I (for example, an interpolated heat map image IHM _t ). At the time of learning the image interpolation network I and the discriminator D, the discriminator D determines whether the input image is an interpolation heat map image generated (interpolated) using the image interpolation network I, or Not) Learn to correctly identify whether it is a real heatmap image. On the other hand, the image interpolation network I categorizes (identifies) the interpolated heat map image generated by interpolation so that the discriminator D is a real heat map image (not generated by interpolation). Do the learning. As described above, the learning is performed in such a manner that the image interpolation network I and the discriminator D compete with each other, so that the image interpolation network I can obtain an interpolation heat map close to a real heat map image (not generated by interpolation). Images can be generated. The details of this learning process will be described later.

  Here, the configuration of the discriminator D will be described with reference to FIGS. As shown in these figures, the discriminator D is a CNN (Convolutional Neural Network) -based network, and includes three convolutional layers and one fully connected layer (“FC” in FIG. 12). It is composed of Each of the above three convolutional layers reduces the size of the image (reduces image resolution) by employing a 2 (pixel) × 2 (pixel) stride, while reducing the number of output filters in each layer by Double the number of convolution layers. Note that the size (H × W) of the input image in FIG. 11 is 32 (pixels) × 32 (pixels). The output layer (“FC” in FIG. 12) of the discriminator D is a fully connected layer with a sigmoid function, and outputs the probability (a value corresponding to) that the input image is the real heat map image described above. . The meanings of “type”, “filter size”, “stride”, and “number of output channels” in FIG. 12 are the same as those in FIG. In the “type” column of FIG. 12, conv. , And FC denote a normal convolution layer and a fully connected layer, respectively.

  Next, the machine learning of the image interpolation network I and the discriminator D performed by the machine learning unit 33 of the CPU 21 of the server 2 will be described with reference to FIG. A predetermined number or more of heat map images drawn by the drawing processing unit 31 are stored in the training data set 29 using the feature amounts at each time instantly transmitted from all the edge cameras 1 to be originally arranged. After that, when the user instructs the server 2 to start machine learning using the operation unit 30, the machine learning unit 33 of the CPU 21 sets 1 to the number of repetitions IT, initializes it, and performs image interpolation. Initialization (setting of initial values) of parameters of the network I and the discriminator D (filter and bias for the convolutional layer) is performed (S21).

Then, the machine learning unit 33 of the CPU21, from the training data set 29, (missing no portion) of the mini-batch heatmap image selected at random (S23), for each heat map image X _i included in the mini-batch the mask MI _i is a binary mask representing an interpolation region (interpolation region mask) randomly generates (S24). Then, based on each heat map image X _i included in the mini-batch and the mask MI _i corresponding to the heat map image X _i , the machine learning unit 33 of the CPU 21 calculates a weighted sum of squares error of the following equation (6). The process of updating the parameters (filter and bias) of the image interpolation network I using the loss function of is performed in mini-batch units (S26). This mini-batch unit update process is a process performed to stabilize the learning of the image interpolation network I. The processes of S23 to S26 are repeated until the number of repetitions IT reaches IT _int1 (YES in S25). It is. Loss function of Equation (6) below, the loss function of the weighted squared error considering mask MI _i representative of the interpolation region.

In the above formula _{(6), L (X i} , MI i) includes a heat map image _{X i,} based on the mask MI _i corresponding to the heat map image _{X i,} represents the error (loss) obtained. C (X _i , MI _i ) is a functional form of an image interpolation network I when the heat map image X _i and the mask MI _i are used for the input image and the interpolation area mask. the input image and interpolation region mask, when using the heat map image X _i and the mask MI _i, corresponding to the output image from the image interpolation network I. |||| in Equation (6) represents a norm.

Incidentally, the parameter updating process in S26, the machine learning unit 33 of the CPU21, for each heat map image _{X i} included in the mini-batch, calculated error _{L (X} i, MI _i) obtained by the above formula (6) Above, the average value of all the obtained errors is calculated, and the parameter is updated in the direction of the gradient. The machine learning unit 33 of the CPU 21 repeats the parameter updating process of S26 for one IT _int while changing the mini-batch of the heat map image used for learning.

When the number of repetitions IT exceeds IT _int1 (NO in S25), the machine learning unit 33 of the CPU 21 temporarily stops updating the parameters of the image interpolation network I, and performs the discriminator D for two IT _ints. Is repeated. Specifically, the machine learning unit 33 of the CPU 21 adds the cross entropy error to the loss function in addition to the processing in S23 and S24 until the number of repetitions IT becomes (IT _int1 + IT _int2 ) (YES in S27). The process of updating the parameters of the discriminator D is performed on a mini-batch basis (S28). This updating process includes an output image (fake image) from the image interpolation network I when each heat map image X _i and the mask MI _i included in the mini-batch are used, and each heat map image X _i included in the mini-batch. (Real image) using a stochastic gradient descent method. Incidentally, the output image from the image interpolation network I (fake image) may be using the output heat map image in FIG. 9 may be used an interpolation heat map image IHM _t.

If the number of repetitions IT exceeds (IT _int1 + IT _int2 ) (NO in S27), the machine learning unit 33 of the CPU 21 performs learning (training) of both the image interpolation network I and the discriminator D as the final stage of learning. Enter the stage to do together. Here, the objective expression of DCGAN composed of the image interpolation network I and the discriminator D is represented by the following expression (7) corresponding to a general GAN objective expression and the following expression (6). The following equation (8) is obtained by combining the loss functions.

In the above equations (7) and (8), E [•] represents an expected value. Α is a hyperparameter for weighting. The loss function in the general GAN objective expression of Expression (7) is Expression (9) below. The loss function in the above equation (8) is the following equation (10). Since the loss function of the equation (10) is a combination of (combined with) the loss function of the weighted sum of squares error of the equation (6) and the GAN loss function of the equation (9), Equation (10) is referred to as a coupling loss function.

When the repetition number IT exceeds (IT _int1 + IT _int2 ) (NO in S27), the machine learning unit 33 of the CPU 21 _substitutes the coupling loss function of the above equation (10) into the loss function until the repetition number IT becomes IT _k. Then, a process of updating the parameters of the image interpolation network I and the discriminator D in mini-batch units is performed (S29). Every time the update process is completed once done addition of the number of repetitions IT (increment) is, (YES in S22) until the number of repetitions IT exceeds IT _k, update processing of S29 is repeated. When the parameters (weights and biases) of the image interpolation network I are represented by θ _c , the stochastic gradient of the coupling loss function for the parameter θ _c (“the coupling loss function for the parameter θ _c for all the heat map images X _i included in the mini-batch”) The average value of “slope” is expressed by the following equation (11). In the formula (11), m is the number of heat map image _{X i} included in the mini-batch.

Also, to represent the parameters of the discriminator D and theta _d, stochastic gradient coupling loss function for parameter theta _d is expressed by the following equation (12). M in the formula (12) is also a number of heat map image X _i included in the mini-batch.

In parameter update processing in S29, the machine learning unit 33 of the CPU21, for parameter theta _c of the image interpolation network I is to reduce the value of the coupling loss function of the above formula (10), the parameter theta _c the gradient direction Update only a small amount. On the other hand, with respect to the parameter θ _d of the discriminator D, the machine learning unit 33 of the CPU 21 sets the parameter of the discriminator D to θ _d in the gradient direction so as to increase the value of the coupling loss function of the above equation (10). Is updated by a very small amount. By repeating the updating of the parameter theta _d parameter theta _c and discriminator D of such image interpolation network I, to machine learning is completed image interpolation network I and discriminator D, image interpolation network Learned I can be obtained. As described above, the machine learning unit 33 of the CPU 21 performs the machine learning of the image interpolation network I and the discriminator D by the stochastic gradient descent method using the mini-batch.

As described above, according to the learning system 10 of the present embodiment, among the stores belonging to the series, from the images captured by all the edge cameras 1 in other stores having all the edge cameras 1 to be originally arranged. using the learning image based on the heat map image HM _t drawn based on the extracted feature quantity (heat map image with the mask), the machine learning image interpolation network I before learning completion (and discriminator D) I did it. Here, in a so-called chain store, stores belonging to the affiliate have the same layout. Therefore, as described above, using a learning image based on the heat map image HM _t drawn based on the feature amount extracted from the captured image in another store with all the edge camera 1 should be arranged originally Then, by performing the machine learning of the image interpolation network I (and the discriminator D), the number of the edge cameras 1 arranged at a certain store belonging to the same series is larger than the number of the edge cameras 1 which should be originally arranged. for small, of the edge camera 1 to be arranged originally, some even in the case where the photographed image by the edge camera 1 is missing, heatmap image (missing heat map image HM ^L caused by lack of these captured images The missing part of _t ) can be interpolated using the learned image interpolation network I.

Thereby, even if the number of edge cameras 1 to be arranged in each chain store is smaller than the number of edge cameras 1 to be arranged, the same number of edge cameras 1 as the number of edge cameras 1 to be arranged is required. A heat map image (interpolated heat map image IHM _t ) comparable to the case where the camera 1 is arranged can be obtained. Therefore, it is possible to reduce the number of edge cameras 1 installed in each chain store, and to reduce the cost of newly opening and maintaining each chain store.

  Further, the same effect as described above can be obtained by the server 2 and the heat map drawing interpolation program 26 of the present embodiment.

  Further, according to the learning system 10 of the present embodiment, an image to be drawn based on a feature amount extracted from an image captured by a plurality of edge cameras 1 (“feature amount image” in the claims) is output to the plurality of edge cameras 1. By using a feature map extracted from a photographed image by using a heat map image visualized by using colors, a user such as an employee can visually grasp the current feature in the corresponding store.

  Further, according to the learning system 10 of the present embodiment, the feature amount extracted from the images captured by the plurality of edge cameras 1 is multi-dimensional data composed of data of a plurality of types of feature amounts. An image to be drawn based on the image (“feature amount image” in the claims) was defined as a color heat map image obtained by visualizing the multidimensional data using colors. As a result, a user such as an employee can easily and visually grasp a plurality of types of characteristics at the present store at the present time.

Further, according to the learning system 10 of the present embodiment, based on the plurality of features of a predetermined time period which is extracted from the photographed image by the edge camera 1 (for example, feature amount information _{c (t-T) ~c t} ) Draw a heat map image. Accordingly, even if a captured image is missing because the number of edge cameras 1 arranged in a certain store is smaller than the number of edge cameras 1 that should be originally arranged, the camera is extracted from these captured images. In the missing heat map image HM ^L _t drawn based on the feature amount and the interpolated heat map image IHM _t after interpolation using the learned image interpolation network I, the feature amount for a predetermined time period before the present time t. Can be reflected. Therefore, it is possible for a user such as an employee to grasp features that are not reflected in the current captured image captured by the plurality of edge cameras 1.

Further, according to the learning system 10 of the present embodiment, the image interpolation network I and the discriminator D (“neural network” in the claims) are a neural network of DCGAN, which is a kind of generation model, and the feature image interpolation unit 32, using the image interpolation network I learned, by generating the missing part of the missing heat map image HM ^L _t, and to interpolate the missing part. DCGAN are the product model using the CNN suitable for deep learning image, using a neural network DCGAN, by generating the missing part of the missing heat map image HM ^L _t, heat map image after the interpolation The (interpolated heat map image IHM _t ) can be an appropriate image.

Modification:
The present invention is not limited to the configuration of each of the above embodiments, and various modifications can be made without departing from the spirit of the invention. Next, a modified example of the present invention will be described.

Modification 1
In the above embodiment, an example in which the “feature amount image” in the claims is a heat map image in which feature amounts extracted from images captured by the plurality of edge cameras 1 are visualized using colors has been described. The server extracts the feature amount extracted from the photographed image as the photographed image itself by each of the plurality of edge cameras, based on the photographed image by each of the plurality of edge cameras, and an image obtained by combining these images in the store. The image may be drawn as a feature image. In the case of this example, the learned neural network (corresponding to the “learned image interpolation network I” in the above embodiment) has the number of edge cameras arranged in a certain store, If the captured image is missing because the number is less than the number of images, the image in the store is completed by interpolating the missing captured image.

In the first modification as well, in the same manner as in the above-described embodiment, the interpolated image is captured by a feature amount for a predetermined time (in the first modification, the image is captured by a plurality of edge cameras for a predetermined time). Image). However, in the case of the above embodiment, heat map image (missing heat map image HM L _^t) itself having a missing part of the previous interpolation, the feature amount of a predetermined time period (e.g., the feature amount information c _{(t-T )} so to c _t) to be rendered based on, at the time of interpolation, it is not necessary to consider the historical lack heat map image, considering only the missing heat map image of the present time, the image after the interpolation, a predetermined It is possible to reflect the feature amount for time. However, in the first modification, since the image to be interpolated is the photographed image itself, if the past photographed image is not taken into account during the interpolation processing, the image after interpolation is displayed in the feature amount (photographed image) for a predetermined time. Image) cannot be reflected. If the interpolated image cannot reflect the feature amount for a predetermined time, a feature that appears only in a missing portion of a captured image at a certain time (for example, a person existing in the missing portion) Cannot be grasped by a user such as an employee.

  Therefore, as in the first modification, the feature amount extracted from the captured image is an image in which the server combines the images captured by each of the plurality of edge cameras, as the captured image itself by each of the plurality of edge cameras. When an image in a store is rendered as a feature amount image, it is desirable to use a CNN such as 3DCNN or the like, or a recurrent neural network (RNN), which is a recursive neural network. A neural network is a neural network that can reflect not only current input information but also past input information in current output information, such as the above-described 3DCNN or other CNN that is conscious of a time-series direction, or a recursive type. By using a neural network, multiple images after interpolation It is possible to reflect an image taken for a predetermined time by each of the edge cameras.

Modification Example 2:
In the above-described embodiment, an example in which the neural network in the claims is DCGAN, which is a kind of generative model, has been described. However, the neural network used for interpolation of a missing portion of a feature amount image such as a heat map image is However, the neural network may be a neural network of another type of generation model such as WGAN (Wasserstein GAN) or a neural network other than the generation model.

Modification 3:
In the above-described embodiment, an example in which the edge camera side extracts a feature amount from an image captured by the edge camera has been described. However, the present invention is not limited to this, and the server side extracts a feature amount from a captured image transmitted from the edge camera. You may make it. Further, in the above embodiment, an example in which the server on the cloud draws the feature amount image based on the feature amount extracted from the image captured by the edge camera has been described. However, the above-described feature amount image drawing processing may be performed, and the drawn feature amount image may be transmitted to a server on the cloud.

Modification 4:
In the above-described embodiment, an example has been described in which the “feature amount” in the claims is three-dimensional data composed of data of three types of feature amounts (number of men _it , number of women _it , stay time _it ). However, the “feature amount” in the claims may be data of any dimension, for example, four-dimensional data composed of data of four types of feature amounts.

1 edge camera (camera)
2 server 10 learning system 26 heat map drawing interpolation program (feature image drawing interpolation program)
31 drawing processing unit 32 feature amount image interpolation unit 33 machine learning unit T predetermined time

Claims (11)

A learning system including a plurality of cameras arranged in stores of chain stores in which stores belonging to the affiliate have the same layout, and a server that communicates with the plurality of cameras,
The server is
A drawing processing unit that draws a feature amount image based on the feature amount extracted from the images captured by the plurality of cameras,
When the number of cameras arranged in a certain store of the chain store is smaller than the number of cameras that should be originally arranged, when the photographed images are missing, the feature caused by the lack of these photographed images. A feature amount image interpolating unit that interpolates the missing portion of the feature amount image by applying a learned neural network to the entire feature amount image including the missing portion of the amount image;
In another store belonging to the series having a camera lacking in the certain store among the cameras to be originally arranged, a feature amount extracted from images captured by all cameras arranged in the other store is used. Using a learning image based on the feature amount image drawn by the drawing processing unit based on the machine learning unit that performs machine learning of the neural network before learning is completed,
A learning system with The other store is a store provided with all cameras to be originally arranged, among stores other than the certain store belonging to the affiliate,
The machine learning unit is configured to perform learning based on the feature amount image drawn by the drawing processing unit based on feature amounts extracted from images captured by all the cameras to be originally arranged, which are arranged in the another store. The learning system according to claim 1, wherein machine learning of the neural network before learning is completed is performed using an image for learning.   The learning system according to claim 1, wherein the feature amount image is a heat map image in which feature amounts extracted from images captured by the plurality of cameras are visualized using colors. The feature amount is multidimensional data composed of data of a plurality of types of feature amounts,
The learning system according to claim 3, wherein the feature amount image is a color heat map image obtained by visualizing the multidimensional data using colors.   5. The heat map image according to claim 3, wherein the drawing processing unit draws the heat map image based on the feature amount for a predetermined time extracted from images captured by the plurality of cameras. 6. Learning system. The feature amount is an image itself captured by each of the plurality of cameras,
The drawing processing unit in the server, based on the images captured by each of the plurality of cameras, draws an in-store image that is an image obtained by combining these images, as a feature amount image,
The learned neural network, when the number of cameras arranged in the certain store is smaller than the number of cameras to be originally arranged, if the photographed image is missing, the missing photographed image The learning system according to claim 1, wherein an image in the certain store is completed by interpolation.   The neural network is a neural network of a generative model, and the feature image interpolating unit interpolates the missing portion by generating a missing portion of the feature image using the learned neural network. The learning system according to any one of claims 1 to 6, wherein: A server that can communicate with a plurality of cameras arranged in a store where a store belonging to the affiliate has a chain store having the same layout,
A drawing processing unit that draws a feature amount image based on the feature amount extracted from the images captured by the plurality of cameras,
When the number of cameras arranged in a certain store of the chain store is smaller than the number of cameras that should be originally arranged, when the photographed images are missing, the feature caused by the lack of these photographed images. A feature amount image interpolating unit that interpolates the missing portion of the feature amount image by applying a learned neural network to the entire feature amount image including the missing portion of the amount image;
In another store belonging to the series having a camera lacking in the certain store among the cameras to be originally arranged, a feature amount extracted from images captured by all cameras arranged in the other store is used. Using a learning image based on the feature amount image drawn by the drawing processing unit based on the machine learning unit that performs machine learning of the neural network before learning is completed,
Server comprising:   9. The server according to claim 8, wherein the feature amount image is a heat map image in which feature amounts extracted from images captured by the plurality of cameras are visualized using colors. Computer
A drawing processing unit that draws a feature amount image based on feature amounts extracted from images captured by a plurality of cameras arranged in a store where a chain store having the same layout has a store belonging to the series;
When the number of cameras arranged in a certain store of the chain store is smaller than the number of cameras that should be originally arranged, when the photographed images are missing, the feature caused by the lack of these photographed images. A feature amount image interpolating unit that interpolates the missing portion of the feature amount image by applying a learned neural network to the entire feature amount image including the missing portion of the amount image;
In another store belonging to the series having a camera lacking in the certain store among the cameras to be originally arranged, a feature amount extracted from images captured by all cameras arranged in the other store is used. A feature amount image drawing interpolation program for functioning as a machine learning unit that performs machine learning of the neural network before completion of learning using a learning image based on the feature amount image drawn by the drawing processing unit based on the drawing.   The computer-readable storage medium according to claim 10, wherein the feature amount image is a heat map image in which feature amounts extracted from images captured by the plurality of cameras are visualized using colors.

Images