JP2022152023A - Feature quantity data generation device and method thereof, and machine learning device and method thereof - Google Patents

Abstract

To provide a feature quantity data generation device and a method thereof, and a machine learning device and a method thereof that reduce learning time in machine learning.SOLUTION: In a data processor including a first learning data acquisition unit, a first coupling unit, a first learning unit, a second learning data acquisition unit, a second coupling unit, and a second learning unit, the second learning unit 60 acquires, as input data (IN_B), second coupled data composed of image data of a plurality of images each including a recognition target object. By compressing the input data IN_B with a learned encoder 32a, compressed data (E_B) including each feature quantity of a plurality of recognition target objects in a plurality of images are generated. The second learning unit 60 uses the compressed data E_B as input data to a neural network (NN) 61 to perform machine learning of the NN 61.SELECTED DRAWING: Figure 9

Description

The present invention relates to a feature amount data generation device and method and a machine learning device and method.

Mini-batch learning is often used when training an inference model for image recognition. In mini-batch learning, image data of a plurality of learning images constituting learning data is divided into mini-batches having a predetermined mini-batch size, and learning is performed for each mini-batch. For example, when the number of pixels W in the horizontal direction and the number of pixels H in the vertical direction of the learning image are both 100 and an RGB format color image is used as the learning image, the data size of one learning image is ( W×H×3), and a mini-batch is formed by combining image data of 32 learning images in the size direction of the mini-batch. The mini-batch size in this case is (W x H x 3 x 32).

For example, if the learning data contains 10,240 learning images, by executing mini-batch learning 320 times from "10,240/32=320", one time of learning for all learning images will be completed. That is, the number of iterations (the number of repetitions) is 320, and 320 mini-batch learnings correspond to one epoch.

Japanese Patent Application Laid-Open No. 2020-71808

In the above method, if the number of learning images included in one mini-batch is increased, the mini-batch size is increased proportionally, but the number of mini-batch learning executions per epoch is decreased. For example, if the mini-batch size is (W×H×3×320), the mini-batch learning is performed 32 times to complete one-time learning for all the learning images. That is, one epoch is completed with 32 mini-batch learnings. By reducing the number of mini-batch learnings per epoch, the learning time of the inference model (for example, the time required for the value of the loss function to become equal to or less than a predetermined threshold) may be shortened.

However, the mini-batch size cannot be unconditionally increased due to limitations on the capacity of the memory installed in the device that performs machine learning. Although it depends on the size of each learning image, the upper limit of the number of learning images per mini-batch is often about 32 in practice. Therefore, if the number of learning images per mini-batch exceeds 32, the cost of the apparatus will increase as the required memory capacity increases. It would be beneficial if training time could be reduced without increasing memory requirements.

An object of the present invention is to provide a feature amount data generation device and method and a machine learning device and method that contribute to reduction in learning time.

A feature amount data generation device according to the present invention includes an image data acquisition unit for acquiring image data of a plurality of images each including a recognition target object, and compressing the image data of the plurality of images to a feature amount data generation unit that generates feature amount data including each feature amount of a plurality of recognition target objects (first configuration).

In the feature amount data generation device according to the first configuration, the plurality of images may be a configuration (second configuration) including two or more images captured temporally continuously by a predetermined camera. good.

A machine learning device according to the present invention includes: a first combining unit that generates first combined data by combining image data of a plurality of first input images in a channel direction; A first learning unit that trains an autoencoder having an encoder that compresses first combined data in the channel direction and a decoder that restores the compression, and combining image data of a plurality of second input images in the channel direction. Compressed data output from a second combining unit that generates second combined data and input of the second combined data to the encoder after learning by the first learning unit is input to a neural network. , and a second learning unit for learning the neural network thereby (third configuration).

In the machine learning device according to the third configuration, the second learning unit uses teacher data including a plurality of label data associated with the plurality of second input images to cause the neural network to learn. (Fourth configuration).

In the machine learning device according to the fourth configuration, the second learning unit may create an inference model capable of object detection by learning the neural network (fifth configuration).

In the machine learning device according to the fifth configuration, each first input image and each second input image may include a recognition target object in the object detection (sixth configuration).

In the machine learning device according to any one of the third to sixth configurations, in the first combined data, the image data of the plurality of first input images are arranged in the channel direction, and in the second combined data, The image data of the plurality of second input images are arranged in the channel direction, and the number of dimensions in the channel direction of the first combined data is reduced by the encoder in the learning by the first learning unit. The first combined data is compressed, and in learning by the second learning unit, the number of dimensions in the channel direction of the second combined data is reduced by the encoder after learning by the first learning unit. A configuration (seventh configuration) may be employed in which two-combined data is compressed to obtain the compressed data.

In the machine learning device according to the seventh configuration, the image data of each first input image and the image data of each second input image include image data for a plurality of colors, and in the first combined data, the channel direction in which the image data for the plurality of colors of each first input image are arranged, and in the second combined data, the image data for the plurality of colors of each second input image are arranged in the channel direction (eighth configuration).

A feature amount data generation method according to the present invention includes an image data acquisition step of acquiring image data of a plurality of images each including a recognition target object; and a feature amount data generation step of generating feature amount data including each feature amount of a plurality of recognition target objects (a ninth configuration).

A machine learning method according to the present invention includes a first combining step of generating first combined data by combining image data of a plurality of first input images in a channel direction; a first learning step of training an autoencoder having an encoder that compresses first combined data in the channel direction and a decoder that restores the compression; and combining image data of a plurality of second input images in the channel direction. a second combining step of generating second combined data; and inputting the second combined data into the encoder after learning in the first learning step, thereby inputting compressed data output from the encoder into a neural network. , and a second learning step for learning the neural network by this (a tenth configuration).

According to the present invention, it is possible to provide a data recording apparatus and method that contribute to improving the convenience of data recording.

1 is a configuration diagram of a data processing device according to an embodiment of the present invention; FIG. 4 is a configuration diagram of first learning data according to the embodiment of the present invention; FIG. 4 is a configuration diagram of second learning data according to the embodiment of the present invention; FIG. FIG. 4 is a diagram showing one input image and corresponding label data according to the embodiment of the present invention; 1 is a configuration diagram of one input image as a color image in RGB format according to an embodiment of the present invention; FIG. FIG. 4 is a configuration diagram of first combined data according to the embodiment of the present invention; FIG. 4 is an explanatory diagram of the configuration and operation of an autoencoder according to the embodiment of the present invention; FIG. 4 is a configuration diagram of second combined data according to the embodiment of the present invention; FIG. 10 is an explanatory diagram of learning operation of the second learning unit according to the embodiment of the present invention; FIG. 10 is an explanatory diagram of input data to the neural network of the second learning unit according to the embodiment of the present invention; FIG. 4 is a diagram for explaining the contents of teacher data according to the embodiment of the present invention; 4 is an operation flowchart of the data processing device according to the embodiment of the present invention; FIG. 4 is a diagram for explaining the effect of data compression according to the embodiment of the present invention; 1 is a configuration diagram of a feature amount data generation device according to an embodiment of the present invention; FIG.

Hereinafter, examples of embodiments of the present invention will be specifically described with reference to the drawings. In each figure referred to, the same parts are denoted by the same reference numerals, and redundant descriptions of the same parts are omitted in principle. In this specification, for simplification of description, by describing symbols or codes that refer to information, signals, physical quantities, or members, etc., the names of information, signals, physical quantities, or members, etc. corresponding to the symbols or codes are It may be omitted or abbreviated. For example, the second learning data acquisition unit (see FIG. 1) referred to by “40” to be described later may be referred to as the second learning data acquisition unit 40, or may be abbreviated as the acquisition unit 40. but they all refer to the same thing.

Although the details will be described later, in this embodiment, a trained encoder 32a capable of extracting a feature amount included in an image is generated using the first learning data (see FIG. 9). Next, the learned encoder 32a is used to generate data (compressed data) in which the feature amount of the recognition target object is extracted from the second learning data. When the learned encoder 32a extracts the feature amount of the recognition target object from the second learning data, a so-called compression technique is used. Next, the NN 61 is trained using the data (compressed data) obtained by extracting the feature amount of the object to be recognized. The NN 61 becomes an inference model for object detection through learning. Since the data (compressed data) obtained by extracting the feature amount of the object to be recognized is used for learning of the NN 61, the learning time of the NN 61 can be reduced. A detailed description will be given below.

FIG. 1 shows a block diagram of a data processing device 1 according to this embodiment. The data processing device 1 is an example of a machine learning device. The data processing device 1 includes a first learning data acquiring section 10 , a first combining section 20 , a first learning section 30 , a second learning data acquiring section 40 , a second combining section 50 and a second learning section 60 . The data processing apparatus 1 may be configured by a single computer device, or may be configured by a plurality of physically separated computer devices. The data processing device 1 may be configured using so-called cloud computing.

The first learning data acquisition unit 10 acquires first learning data including image data of a plurality of images. Since the image data of each image forming the first learning data is input to the first combining unit 20, each image forming the first learning data is referred to as a first input image. As shown in FIG. 2, the first learning data includes image data of a total of P first input images IA[1] to IA[P]. P is any integer greater than or equal to 2, and has a value of, for example, tens to thousands. Note that the first input image IA[i] may be simply referred to as input image IA[i]. i represents an arbitrary integer.

The second learning data acquisition unit 40 acquires second learning data including image data of a plurality of images. Since the image data of each image forming the second learning data is input to the second combining unit 50, each image forming the second learning data is referred to as a second input image. As shown in FIG. 3, the second learning data includes image data of a total of Q second input images IB[1] to IB[Q]. Q is any integer greater than or equal to 2, and has a value of, for example, thousands to tens of thousands. Note that the second input image IB[i] may be simply referred to as the input image IB[i]. In this embodiment, the number of images included in the second learning data is greater than the number of images included in the first learning data. That is, "P<Q" is established.

An arbitrary image such as the first input image or the second input image includes image data of the image and other data (hereinafter referred to as additional data). An arbitrary image may be an image captured by a camera, and additional data for a certain image includes data other than image data in the image, and shooting time information representing the shooting time of the image. including.

As will be described later, in the data processing device 1, an inference model (algorithm) is created through learning by the second learning unit 60, and the inference model can perform object detection as image recognition. In object detection, position identification for identifying the position of an object within an image to be recognized and class identification for identifying the class (type) of the object within the image for recognition are performed. Each first input image and each second input image includes one or more objects to be recognized. In the present embodiment, an object refers to a recognition target object that is a target of image recognition in object detection. A part of the first input image may not include the recognition target object. Similarly, some of the second input images may not include the recognition target object. Also, one or more first input images may include an object other than the recognition target object. Similarly, the one or more second input images may include objects other than the recognition target object.

In this embodiment, inclusion of image data of an object in an image may be expressed as inclusion or presence of the object in the image. Similarly, inclusion of image data of an object in an image area of interest (for example, an object area described later) in an image may be expressed as inclusion or presence of an object in the image area of interest.

The second training data includes label data for each second input image. In the second learning data, the label data associated with the second input image IB[i] is referenced by the symbol "LB[i]". The label data LB[i] includes position information specifying the position of each object included in the second input image IB[i] and class information specifying the class of the object.

An input image 610 is shown in FIG. Input image 610 is an example of second input image IB[i]. Input image 610 in FIG. 4 includes three objects 611-613. Objects 611, 612, and 613 are a vehicle, a person, and a traffic light, respectively, and are all objects to be recognized. Here, vehicles, humans, and traffic lights are classified into first, second, and third classes, and the inference model performs object detection for objects in multiple classes, including the first to third classes. It shall be possible. Here, an automobile that can run on roads is assumed as the vehicle.

An object region 611B surrounding the image of the object 611, an object region 612B surrounding the image of the object 612, and an object region 613B surrounding the image of the object 613 are set for the input image 610 in FIG. An object region of an object is a rectangular region (preferably the smallest rectangular region) surrounding the image of the object, also called a bounding box.

_Label information _{620 corresponding} to the input image ₆₁₀ in FIG. POS ₆₁₃ and class information CLS ₆₁₃ . If the input image 610 is the second input image IB[i], the label information 620 is the label information LB[i]. Position information POS ₆₁₁ , POS ₆₁₂ , and POS ₆₁₃ represent the positions of object regions 611B, 612B, and 613B in the input image 610, respectively. Specifically, the coordinate values of the two endpoints of one diagonal line in the rectangular region as the object region 611B (corresponding to the coordinate values (x ₁ , y ₁ ) and (x ₂ , y ₂ ) in FIG. 4) are the position information POS ₆₁₁ . The same applies to other location information. Class information CLS ₆₁₁ , CLS ₆₁₂ , and CLS ₆₁₃ represent the class to which object 611 belongs, the class to which object 612 belongs, and the class to which object 613 belongs, respectively. In the example of FIG. 4, the class information CLS ₆₁₁ , CLS ₆₁₂ , and CLS ₆₁₃ respectively represent the first class to which vehicles belong, the second class to which people belong, and the third class to which traffic lights belong.

For example, the first input images IA[1] to IA[P] and the second input images IB[1] to IB[Q] may be selected from images captured by a camera mounted on a vehicle such as an automobile. The first input images IA[1] to IA[P] and the second input images IB[1] to IB[Q] may partially overlap.

The first learning data acquisition unit 10 may be a functional block that itself creates the first learning data, or may acquire a previously created first learning data from an external device (not shown) different from the data processing device 1 through wired or wireless communication. 1 learning data may be input. Similarly, the second learning data acquisition unit 40 may be a functional block that creates the second learning data by itself, or an external device (not shown) different from the data processing device 1 through wired or wireless communication, The input of the second learning data may be received.

Each first input image and each second input image is a two-dimensional still image having horizontal and vertical dimensions. The one or more first input images may be frames of a moving image. Similarly, the one or more second input images may be frames of a moving image. Let W denote the number of pixels in the horizontal direction in each first input image and each second input image, and let H be the number of pixels in the vertical direction in each first input image and each second input image. Then each first input image and each second input image consists of (W×H) pixels. It is also assumed that the first input image and the second input image are color images expressed in RGB format. That is, each pixel of the first input image and each pixel of the second input image have an R signal representing a red signal component, a G signal representing a green signal component, and a B signal representing a blue signal component. .

Then, an input image 650, which is an arbitrary first input image or an arbitrary second input image, consists of (W×H) pixels and only the R signal, as shown in FIG. as color signals, a green grayscale image 650G consisting of (W×H) pixels and having only G signals as color signals, and a (W×H) pixels consisting only of B signals. Images 650R, 650G and 650B, which can be considered to consist of a blue-toned image 650B having as color signals, are arranged in channel directions different from both the horizontal and vertical directions. The number of types of color signals forming the image data of the input image 650 is represented by "C". Here, "C=3".

Referring to FIG. 1 again, the first combiner 20 divides the first input images IA[1] to IA[P] into mini-batches having a predetermined mini-batch size. Then, for each mini-batch, the first combining unit 20 generates first combined data by combining the plurality of first input images belonging to the mini-batch in the channel direction. Here, the mini-batch size is assumed to be the data size for N first input images. N is an arbitrary integer greater than or equal to 2, for example "N=32". (P/N) pieces of first combined data are formed from the first input images IA[1] to IA[P]. (P/N) is an arbitrary integer of 2 or more.

FIG. 6 shows one first combined data (that is, one mini-batch structure) generated by the first combining unit 20 . The first combined data shown in FIG. 6 is composed of the image data of the input images IA[i] to IA[i+N], and the image data of the input images IA[i] to IA[i+N] are arranged in the channel direction. . Each of the input images IA[i] to IA[i+N] is composed of a red grayscale image, a green grayscale image and a blue grayscale image arranged in the channel direction. Therefore, in the first combined data, in the channel direction, the image data of the input image IA[i] for a plurality of colors (the image data of the red grayscale image, the green grayscale image, and the blue grayscale image), the input image IA[i+1] Image data for a plurality of colors (image data for a red grayscale image, a green grayscale image, and a blue grayscale image), . and blue grayscale image data) are arranged.

Therefore, the first combined data corresponds to (C×N) monochromatic two-dimensional images, each consisting of (H×W) monochromatic pixels, arranged along the channel direction. The first combined data has a data amount for (W.times.H.times.C.times.N) monochrome pixels. In the first combined data, the number of channels is (C×N), so the number of dimensions in the channel direction is (C×N).

Referring to FIG. 1 again, the first learning unit 30 has a neural network 31 (hereinafter referred to as NN31), and performs machine learning of NN31 using first learning data. At this time, the machine learning of the NN 31 is performed on a mini-batch basis. That is, (P/N) pieces of first combined data based on the first learning data are sequentially used as input data to the NN 31, and machine learning of the NN 31 is performed on a mini-batch basis (mini-batch learning is performed). Machine learning in the first learning unit 30 may be classified as deep learning, so the NN 31 may be a deep neural network. Machine learning in the first learning unit 30 is unsupervised learning, and the NN 31 forms an autoencoder. That is, the first learning unit 30 learns the autoencoder (in other words, the autoencoder is created by learning the NN 31).

FIG. 7 shows the configuration of the autoencoder. The NN 31 forming an autoencoder comprises an encoder 32 and a decoder 33 . The type of autoencoder here is arbitrary, and may be, for example, a variational autoencoder (VAE) or a convolutional autoencoder (CAE). The first combined data is input to the encoder 32 as input data IN_A, and the encoder 32 compresses the input data IN_A to generate compressed data E_A. The decoder 33 obtains the output data OUT_A by restoring the compressed data E_A (that is, by restoring the compression by the encoder 32). In machine learning in the first learning unit 30, each parameter (bias and weight) of the NN 31 is adjusted so that the output data OUT_A matches the input data IN_A.

At this time, the encoder 32 is designed so that the input data IN_A (therefore, the first combined data) is compressed in the channel direction, and the decoder 33 is designed so that the compressed data E_A is restored in the channel direction. That is, compression by the encoder 32 corresponds to dimensionality reduction in the channel direction, and the encoder 32 reduces the number of dimensions in the channel direction of the input data IN_A (therefore, the first combined data) from "(C×N)" to "J". do. In other words, the number of dimensions in the channel direction of the input data IN_A (therefore, the first combined data) is "(C×N)". Compressed data E_A with the number of dimensions in the channel direction of "J" is obtained. It can also be considered that the encoder 32 reduces the number of channels from "(C×N)" to "J".

It is "CxN>J". For example, if "(C, N, J)=(3, 32, 3)", the encoder 32 changes the number of dimensions in the channel direction of the input data IN_A (therefore, the first combined data) from 96 to 3. In this case, the data size of the compressed data E_A is 1/32 of the data size of the input data IN_A from “3/(3×32)=1/32”.

The first learning unit 30 performs machine learning of the NN 31 until the training error (loss function value) of the NN 31 that functions as an autoencoder becomes equal to or less than a predetermined value. The encoder 32 after completing this machine learning is hereinafter particularly referred to as a learned encoder 32a (see FIG. 9).

Referring to FIG. 1 again, the second combining unit 50 divides the second input images IB[1] to IB[Q] into mini-batches having a predetermined mini-batch size. The mini-batch size at the second joint 50 is the same as the mini-batch size at the first joint 20 . Therefore, the mini-batch size in the second combiner 50 is the data size of N second input images (for example, "N=32"). Then, for each mini-batch, the second combining unit 50 generates second combined data by combining the plurality of second input images belonging to the mini-batch in the channel direction. (Q/N) pieces of second combined data are formed from the second input images IB[1] to IB[Q]. (Q/N) is an arbitrary integer of 2 or more, and has a value of several hundred to several thousands, for example.

FIG. 8 shows one second combined data (that is, one mini-batch structure) generated by the second combining unit 50 . The structure of the second combined data is similar to the structure of the first combined data. That is, the second combined data shown in FIG. 8 is composed of the image data of the input images IB[i] to IB[i+N], and the image data of the input images IB[i] to IB[i+N] are arranged in the channel direction. be done. Each of the input images IB[i] to IB[i+N] is composed of a red grayscale image, a green grayscale image and a blue grayscale image arranged in the channel direction. Therefore, in the second combined data, in the channel direction, the image data for a plurality of colors of the input image IB[i] (the image data of the red grayscale image, the green grayscale image, and the blue grayscale image), the input image IB[i+1] Image data for multiple colors (image data for a red grayscale image, green grayscale image, and blue grayscale image), and image data for multiple colors of the input image IB[i+N] (red grayscale image, green grayscale image) and blue grayscale image data) are arranged.

Therefore, the second combined data corresponds to (C×N) monochromatic two-dimensional images, each consisting of (H×W) monochromatic pixels, arranged along the channel direction. The second combined data has a data amount for (W.times.H.times.C.times.N) monochrome pixels. In the second combined data, the number of channels is (C×N), so the number of dimensions in the channel direction is (C×N).

Referring to FIG. 1 again, the second learning unit 60 has a neural network 61 (hereinafter referred to as NN61), and performs machine learning of NN61 using second learning data. Machine learning in the second learning unit 60 may be classified as deep learning, so the NN 61 may be a deep neural network. Machine learning in the second learning unit 60 is supervised learning, and the NN 61 forms an inference model for object detection. That is, the second learning unit 60 learns an inference model for object detection (in other words, an inference model capable of object detection is created by making the NN 61 learn).

Machine learning by the second learning unit 60 will be described with reference to FIG. The learned encoder 32a described above is used for machine learning by the second learning unit 60 . The second combined data is input to the learned encoder 32a as input data IN_B, and the learned encoder 32a compresses the input data IN_B to generate compressed data E_B. In compression by the trained encoder 32a, the number of dimensions in the channel direction of the input data IN_B (therefore, the second combined data) is reduced from "(C×N)" to "J". In other words, the number of dimensions in the channel direction of the input data IN_B (therefore, the second combined data) is "(C×N)", and the trained encoder 32a reduces the dimensionality of the input data IN_B in the channel direction. , the compressed data E_B with the number of dimensions in the channel direction of "J" is obtained. It can also be considered that the number of channels is reduced from "(C×N)" to "J" in the learned encoder 32a.

By sequentially inputting a plurality of second combined data based on the second input images IB[1] to IB[Q] as input data IN_B to the learned encoder 32a, a plurality of compressions based on the plurality of second combined data Data E_B is obtained.

The second learning unit 60 performs machine learning of the NN 61 using the compressed data E_B as input data to the NN 61 . At this time, machine learning of the NN 61 is performed in units of mini-batch (that is, mini-batch learning is performed). Although the mini-batch size in machine learning of NN61 and the mini-batch size in machine learning of NN31 may be different, here they are assumed to be the same. Then, the mini-batch size in machine learning of NN61 is the data size for N second input images, and the data size for N second input images is (W×H×C×N).

FIG. 10 shows an overview of the flow until input data to the NN 61 is generated. In FIG. 10, data DTa includes “(C×N)/J” sets of image data of N second input images. The data DTb consists of “(C×N)/J” pieces of input data IN_B, that is, “(C×N)/J” pieces of second combined data. Data DTc consists of "(C×N)/J" pieces of compressed data E_B based on data DTb.

Data DTb is obtained by inputting data DTa to the second combining unit 50 . That is, the sets of image data of the N second input images are sequentially input to the second combining unit 50 for “(C×N)/J” sets, so that the second combining unit 50 outputs “( C×N)/J″ pieces of second combined data are output. The data size of each second combined data is (W×H×C×N). Therefore, the data size of the data DTb is "(W.times.H.times.C.times.N).times.(C.times.N)/J". The same applies to the data size of data DTa.

Compressed data E_B is generated for each second coupled data by inputting each of the second coupled data as input data IN_B to the learned encoder 32a, and as a result, "(C×N)/J" pieces of compressed data E_B are generated. A data DTc consisting of is obtained. Since the number of dimensions in the channel direction is reduced from "(C×N)" to "J" in the trained encoder 32a, the data size of one piece of compressed data E_B is (W×H×J). Therefore, the data size of data DTc is (W×H×C×N).

Compressed data E_B corresponding to a data size of (W×H×C×N) is input to NN 61 as data for mini-batch learning per one time. This corresponds to inputting information of “N×(C×N)/J” pieces of input images to the NN 61 in the mini-batch learning per NN 61 . For example, in a numerical example where "(C, N, J) = (3, 32, 3)" and "Q = 10240" holds, in one mini- ^batch learning, the information of 322 input images is NN61 will be entered in the Then, from "10240/32 ² =10", by performing NN61 mini-batch learning 10 times, one-time learning using all the second input images that make up the second learning data will be completed ( That is, the number of iterations is 10).

In the hypothetical case in which the image data of N second input images themselves are input to the NN 61, in order to complete one-time learning using all the second input images that make up the second learning data, in the above numerical example, Mini-batch learning of the NN 61 needs to be performed 320 times, and the learning time is longer than that of the data processing device 1 .

In machine learning of NN 61 in second learning unit 60, NN 61 generates output data OUT_B based on compressed data E_B having a mini-batch size (see FIG. 9). The second learning unit 60 derives the value of the loss function corresponding to the error between the output data OUT_B and the teacher data for each mini-batch (for each mini-batch learning), and performs error back propagation so that the value of the loss function is reduced. is used to tune the parameters (weights and biases) of NN61. Machine learning of NN 61 (that is, machine learning of an inference model for object detection) is performed until the value of the loss function is equal to or less than a predetermined threshold.

In the NN61 mini-batch learning, the teacher data consists of label data for all the second input images used in the mini-batch learning. For example, in a certain mini-batch learning, if the data DTa (see FIG. 10) is composed of the image data of the second input images IB[1] to IB[1024], the teacher data in the mini-batch learning is label It consists of data LB[1] to LB[1024]. That is, for example, if the image data of the second input images IB[1] and IB[2] are included in the data DTa, as shown in FIG. 11, the corresponding label data LB[1] and LB[ 2] is included in the teacher data corresponding to the data DTa (only the information of the label data LB[1] and LB[2] is shown in FIG. 11).

The NN 61, which should function as an inference model, performs position identification and class identification of objects in each input image that constitutes the data DTa in machine learning using the data DTa, and outputs the results of the position identification and class identification as output data. Output as OUT_B. The value of the loss function is derived by comparing this output data OUT_B with teacher data corresponding to the data DTa.

FIG. 12 shows an operation flowchart of the data processing device 1. As shown in FIG. First, in step S<b>1 , first learning data is acquired by the first learning data acquisition unit 10 . Next, in step S2, first combined data is generated by the first combining unit 20 based on the first learning data. Next, in step S3, the first learning unit 30 learns the autoencoder based on the first combined data (that is, learns the NN 31), thereby creating the trained encoder 32a. Next, the second learning data is acquired by the second learning data acquiring section 40 in step S4. Note that the acquisition timing of the second learning data is arbitrary as long as it is before step S5.

After obtaining the second learning data, in step S5, the second combining unit 50 generates the second combined data based on the second learning data. At this time, the teacher data described above is also created. The teacher data may be created by the second combining unit 50 or the second learning unit 60 . Thereafter, in step S6, the second combined data is input to the learned encoder 32a to generate compressed data (E_B), and the second learning unit 60 performs inference for object detection based on the generated compressed data (E_B). Train the model (in other words, train the NN 61 to create an inference model for object detection).

In this embodiment, by compressing the learning data (second learning data) as described above, the information amount of data per mini-batch in the second learning unit 60 can be increased. That is, compared to the hypothetical case in which the image data of N second input images themselves are input to the NN 61, the information amount of data per mini-batch in the second learning unit 60 is "(C×N)/J" times. Increase (eg, increase 32-fold). Therefore, in comparison with the virtual case, the learning time in the second learning unit 60 (for example, the time required for the value of the loss function of the inference model by the NN 61 to become equal to or less than a predetermined threshold value) can be shortened. . From a different point of view, if the learning time is assumed to be constant, the required memory capacity can be reduced.

FIG. 13 shows the data size compression effect and the like when "(C×N)/J=3×32/3=32". In the numerical example of FIG. 13, by using the learned encoder 32a, the data size (data amount) of the input data IN_B is compressed to 1/32 to obtain the compressed data E_B. Therefore, assuming that the learning time per unit data amount of the data input to the NN 61 is constant, the learning time of the NN 61 is shortened to 1/32 compared to the hypothetical case. Also, the number of iterations in learning the NN 61 is reduced to 1/32 of the number of iterations required in the hypothetical case. On the other hand, if the learning time of the NN 61 according to this embodiment is the same as the learning time according to the virtual case, it is possible to reduce the required memory capacity to 1/32 of the virtual case.

Since each first input image includes the recognition target object of the inference model, the autoencoder extracts the feature amount of the recognition target object of each first input image from the first combined data (IN_A) and converts it into compressed data E_A. be included. That is, learning of the autoencoder progresses so that the feature amount of the recognition target object is extracted from the input image including the recognition target object, and the learned encoder 32a is configured. Therefore, if the second combined data (IN_B) based on the second input image including the recognition target object is input to the learned encoder 32a, the feature amount of the recognition target object in each second input image is obtained by the learned encoder 32a. It is extracted and included in the compressed data E_B. By inputting this compressed data E_B to the inference model (NN61), efficient learning that contributes to shortening the learning time becomes possible.

When considering the function of the data processing device 1 from the viewpoint of the feature amount of the recognition target object, it is considered that the data processing device 1 functions as the feature amount data generation device 2 in FIG. 14 or includes the feature amount data generation device 2. be able to. The feature amount data generation device 2 includes an image data acquisition unit 2A that acquires a plurality of images II each including a recognition target object, and a plurality of recognition targets in the plurality of images II by compressing the image data of the plurality of images II. and a feature amount data generation unit 2B that generates feature amount data including each feature amount of the object. The plurality of images II associated with acquisition units 2A and 2B correspond to the plurality of second input images. The acquisition unit 2A corresponds to the acquisition unit 40 in FIG. 1, and the generation unit 2B corresponds to a functional block including the combining unit 50 in FIG. 1 and the learned encoder 32a in FIG. The feature data generated by the generator 2B corresponds to the compressed data E_B.

That is, the feature amount data generation device 2 acquires a plurality of second input images each including a recognition target object, and the feature amount data (EN_B ). By learning an inference model (NN 61) for object detection using this feature amount data, efficient learning that contributes to shortening of the learning time becomes possible.

Here, the plurality of images II described above preferably include two or more images captured temporally continuously by a predetermined camera (not shown). That is, at least a part of the second input images IB[1] to IB[Q] may be two or more images captured temporally continuously by a predetermined camera. The predetermined camera captures a scene (subject) within its own capture area and generates image data of a camera image, which is a captured image. At this time, the predetermined camera periodically takes pictures at a predetermined frame rate. Then, a predetermined camera acquires a plurality of camera images arranged in time series at intervals of the reciprocal of the frame rate. A plurality of camera images arranged in time series (hereinafter referred to as a camera image sequence) correspond to two or more images temporally consecutively captured by a predetermined camera.

The predetermined camera may be a fixed point camera fixed at a fixed location. In this case, it is expected that the scenery (parts other than the object to be recognized) in the camera image hardly changes in the sequence of camera images, and only the vehicle or the person as the object to be recognized moves in the sequence of camera images. As a result, the compression effect of the encoder 32 (learned encoder 32a) increases, enabling efficient extraction of the feature amount of the object to be recognized, which in turn promotes efficient learning of the inference model (NN61). The predetermined camera may be a camera mounted on a moving object such as a vehicle.

The image data acquiring unit 2A selects the images shot by the predetermined camera from among the images shot by the predetermined camera, based on the shooting time information included in the additional data of each image shot by the predetermined camera. More than one captured image may be extracted. For example, when a first captured image and a second captured image are included in images captured by a predetermined camera, the time difference between the capturing time of the first captured image and the capturing time of the second captured image is equal to or less than a predetermined time. , the first and second captured images may be extracted as two images that are temporally consecutively captured.

Supplementary matters, applied techniques, modified techniques, etc. for the above configuration are shown below.

Each first input image and each second input image may be a monochrome image (a grayscale image without color information). In this case, "C=1".

The encoder 32 (including the trained encoder 32a) compresses the input data (IN_A or IN_B) in the channel direction, even if the input data is compressed in the horizontal or vertical direction of the input image. good.

The data processing device 1 includes, as hardware, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), etc., which are arithmetic processing devices. The data processing device 1 may realize the functions of the parts shown in FIG. Each process of steps S1 to S6 may be implemented.

An inference model created by the data processing device 1 may be applied to an in-vehicle device (not shown). An in-vehicle device is a type of electronic device mounted in a vehicle such as an automobile. In this case, it is preferable to apply the inference model formed by the NN 61 through the machine learning of the NN 61 by the second learning unit 60 to the in-vehicle device. Then, the in-vehicle device may be caused to detect an object using the inference model, and the inference result may be used for automatic driving or driving assistance that can be implemented in the vehicle.

Note that the data processing device 1 itself may be an in-vehicle device. Depending on the vehicle (for example, broadcast relay van), an on-vehicle device having abundant computational resources may be installed.

Also, part or all of the processing executed by the data processing device 1 may be realized by mixed processing of software and hardware. A computer program that causes a computer to execute the method described above and a computer-readable recording medium that records the program are included in the scope of this embodiment. Here, computer-readable recording media are, for example, flexible disks, hard disks, CD-ROMs, MOs, DVDs, DVD-ROMs, DVD-RAMs, large-capacity DVDs, next-generation DVDs, and semiconductor memories.

The embodiments of the present invention can be appropriately modified in various ways within the scope of the technical idea indicated in the scope of claims. The above embodiments are merely examples of the embodiments of the present invention, and the meanings of the terms of the present invention and each constituent element are not limited to those described in the above embodiments. The specific numerical values given in the above description are merely examples and can of course be changed to various numerical values.

1 data processing device 2 feature amount data generation device 2A image data acquisition unit 2B feature amount data generation unit 10 first learning data acquisition unit 20 first coupling unit 30 first learning unit 31 neural network (autoencoder)
32 encoder 33 decoder 40 second learning data acquisition unit 50 second coupling unit 60 second learning unit 61 neural network (inference model)

Claims (10)

an image data acquisition unit that acquires image data of a plurality of images each including a recognition target object;
a feature amount data generation unit configured to generate feature amount data including each feature amount of a plurality of recognition target objects in the plurality of images by compressing image data of the plurality of images. . 2. The feature amount data generation device according to claim 1, wherein said plurality of images includes two or more images captured temporally continuously by a predetermined camera. a first combining unit that generates first combined data by combining image data of a plurality of first input images in a channel direction;
a first learning unit that receives the supply of the first combined data and trains an autoencoder having an encoder that compresses the first combined data in the channel direction and a decoder that restores the compression;
a second combining unit that generates second combined data by combining image data of a plurality of second input images in the channel direction;
A second learning unit for inputting compressed data output from the encoder by inputting the second combined data to the encoder after learning by the first learning unit to a neural network, thereby learning the neural network. and a machine learning device. 4. The machine learning device according to claim 3, wherein said second learning unit makes said neural network learn using teacher data including a plurality of label data associated with said plurality of second input images. 5. The machine learning device according to claim 4, wherein said second learning unit creates an inference model capable of object detection by learning said neural network. 6. The machine learning device according to claim 5, wherein each first input image and each second input image include a recognition target object in said object detection. in the first combined data, the image data of the plurality of first input images are arranged in the channel direction;
in the second combined data, the image data of the plurality of second input images are arranged in the channel direction;
In learning by the first learning unit, the encoder reduces the number of dimensions of the first combined data in the channel direction, thereby compressing the first combined data,
In learning by the second learning unit, the encoder after learning by the first learning unit reduces the number of dimensions of the second combined data in the channel direction, thereby compressing the second combined data. 7. The machine learning device according to any one of claims 3 to 6, wherein said compressed data is obtained by: The image data of each first input image and the image data of each second input image include image data for a plurality of colors,
in the first combined data, the image data for the plurality of colors of each first input image are arranged in the channel direction;
8. The machine learning device according to claim 7, wherein in said second combined data, image data for said plurality of colors of each second input image are arranged in said channel direction. an image data acquisition step of acquiring image data of a plurality of images each including a recognition target object;
a feature amount data generating step of compressing image data of the plurality of images to generate feature amount data including each feature amount of a plurality of recognition target objects in the plurality of images. . a first combining step of generating first combined data by combining image data of a plurality of first input images in a channel direction;
a first learning step of receiving the first combined data and training an autoencoder having an encoder for compressing the first combined data in the channel direction and a decoder for restoring the compression;
a second combining step of generating second combined data by combining image data of a plurality of second input images in the channel direction;
A second learning step of inputting compressed data output from the encoder by inputting the second combined data into the encoder after learning in the first learning step to a neural network, thereby learning the neural network. and a machine learning method.

Images