JP2021012043A - Information processing device for machine learning, information processing method for machine learning, and information processing program for machine learning - Google Patents

Abstract

To improve the accuracy of machine learning using labeled measurement data.SOLUTION: An information processing device 100 for machine learning includes: a data input unit that inputs measurement data measured by measuring means that measures depth information and generates a measurement image based on the measurement data; a label information input unit that inputs label information that is added to a feature quantity related to a target included in the measurement image; an angular information input unit that inputs angular information indicating a measurement direction of the measurement means for each piece of the measurement data; a determination unit that determines the correlation of time series of the feature quantity related to the target included in the measurement image based on the angular information; and a label correction unit that corrects the label information added to the feature quantity related to the target based on a determination result of the correlation.SELECTED DRAWING: Figure 1

Description

The present invention relates to an information processing device for machine learning, an information processing method for machine learning, and an information processing program for machine learning.

For example, Patent Document 1 discloses the following as a technique for correcting label information of learning data used for machine learning with high accuracy. That is, Patent Document 1 states that "a plurality of sensors including a camera for capturing an image frame in the front direction of the autonomous vehicle, and vehicle control for controlling the autonomous vehicle". A system configured to (i) receive sensor inputs from the plurality of sensors and (ii) generate a vehicle control signal based on the sensor inputs, and independently of the vehicle control system. The image frame is assumed to represent a travelable space or a non-travelable space by performing image analysis on the received image frame in a step of receiving an image frame from the camera. Based on the step of labeling the region, at least a portion of the vehicle data, whether the autonomous vehicle may collide with the non-travelable space region represented in the received image frame. The step of determining, and the step of generating a collision avoidance signal when it is determined that the autonomous vehicle may collide with the non-travelable space region shown in the received image, the collision avoidance. A vehicle comprising a system configured to perform a step that prevents the autonomous vehicle from colliding with the non-travelable space region by a signal. " .. According to the technique disclosed in Patent Document 1, an object can be detected from a captured image by image recognition, and the shape of the image can be corrected so that the positional deviation is minimized.

Japanese Unexamined Patent Publication No. 2019-8996

For example, as an automatic determination of an object using underwater sonar, there is a method of improving the determination accuracy by machine learning using labeled sonar data. In this case, as with a general camera image, the feature amount in the measurement data has a constant continuity on the time axis, and the learning accuracy can be improved by applying this continuity to machine learning. On the other hand, there is a method of changing the azimuth or depression angle of the sonar sensor for the purpose of measuring a wide range with the sonar, but in this case, the continuity of the data is lost due to the change of the measurement direction.

However, Patent Document 1 described above does not disclose a method of correcting a label in consideration of continuity of measurement data when the measurement direction of the sensor changes. Therefore, in the above-mentioned Patent Document 1, there is a problem that the continuity of the measurement data cannot be estimated and the label information cannot be corrected, and the accuracy of machine learning using the labeled measurement data is lowered.

The present invention has been made in view of the above-mentioned problems, and it is intended to estimate the continuity of measurement data by a sensor, correct label information, and improve the accuracy of machine learning using the labeled measurement data. It has one purpose.

In order to solve such a problem, in the present invention, as one means for achieving one object, the machine learning information processing device inputs the measurement data measured by the measurement means for measuring the depth information, and inputs the measurement data into the measurement data. A data input unit that generates a measurement image based on the measurement image, a label information input unit that inputs label information to be given to a feature amount related to a target object included in the measurement image, and a measurement direction of the measurement means for each measurement data are shown. The angle information input unit for inputting angle information, the determination unit for determining the time-series correlation of the feature amount related to the target object included in the measurement image based on the angle information, and the determination unit based on the correlation determination result. It is characterized in that it is provided with a label correction unit that corrects the label information given to the feature amount related to the target object.

According to the present invention, for example, it is possible to estimate the continuity of measurement data by a sensor, correct the label information, and improve the accuracy of machine learning using the labeled measurement data.

FIG. 1 is a block diagram showing a configuration example of a machine learning information processing device according to an embodiment. FIG. 2 is a plan view showing an example of an arrangement of a plurality of sonar sensors constituting the sonar in the embodiment. FIG. 3 is a side view showing an example of the sonar and the sonar sensor in the embodiment. FIG. 4 is a diagram showing an example of sonar image and label information generated based on sonar data in the embodiment. FIG. 5 is a diagram showing an example of sonar image and label information when labeling fails. FIG. 6 is a diagram for explaining the correction of label information according to the embodiment. FIG. 7 is a flowchart showing information processing for machine learning according to the embodiment. FIG. 8 is a diagram showing an example of computer hardware that realizes the machine learning information processing device according to the embodiment.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. It should be noted that the embodiments described below do not limit the invention according to the claims, and all of the elements and combinations thereof described in the embodiments are essential for the means for solving the invention. Not necessarily. In addition, an embodiment in which various elements described in the embodiment are appropriately combined is also included in the embodiment disclosed in the present application.

(Configuration of Information Processing Device 100 for Machine Learning)
FIG. 1 is a block diagram showing a configuration example of the machine learning information processing apparatus 100 according to the embodiment. The machine learning information processing device 100 includes a sonar data input unit 101, a label information input unit 102, a sonar angle information input unit 103, a determination unit 104, a label correction unit 105, and a machine learning data output unit 106. The sonar data input unit 101, the label information input unit 102, the sonar angle information input unit 103, the determination unit 104, the label correction unit 105, and the machine learning data output unit 106 are connected via the bus 107. The bus 107 mediates the transmission of data, control information, and analysis information handled by each processing unit connected to the bus 107.

The machine learning information processing device 100 is connected to the sonar 110 via a bus 107. The machine learning information processing device 100 may be connected to the sonar 110 by wire or wirelessly. Although the sonar 110 is shown in FIG. 1 as an example provided outside the machine learning information processing device 100, it may be built in the machine learning information processing device 100.

The sonar data input unit 101 processes the sonar data measured by the sonar 110. For example, the sonar data input unit 101 generates array data from the sonar data input from the sonar 110 as a distribution of signal strength with respect to the measurement direction and depth.

The array data generated by the sonar data input unit 101 is one-dimensional array data of the signal strength with respect to the depth when the sonar 110 is composed of a single sensor, and the sonar 110 makes a plurality of sensors in the one-dimensional direction. When the sonar 110 is configured by arranging a plurality of sensors in the two-dimensional direction, it is a two-dimensional array data of the signal strength with respect to the one-dimensional measurement direction and the depth. Is a three-dimensional array data of the signal strength with respect to the two-dimensional measurement direction and depth. In the following description, unless otherwise specified, two-dimensional array data will be described as an example. Further, in the following description, the array data generated based on the sonar data will be referred to as a sonar image. The sonar image is an example of a measurement image generated based on the measurement data measured by the measurement means.

The label information input unit 102 inputs label information related to the target object included in the sonar image input from the sonar data input unit 101. The label information is information recorded as coordinate values of a position or range in a sonar image in which it is determined that a target object exists by human visual inspection, for example. This coordinate value is obtained from, for example, the respective positions of the target and the sonar 110 obtained from GPS or an aerial photograph and the relative positions of the respective positions.

The sonar angle information input unit 103 inputs angle information of an azimuth or depression angle indicating the measurement direction of the sonar 110. At the time of measurement, the sonar 110 has its azimuth or depression angle controlled from the outside or autonomously controlled. The sonar angle information input unit 103 inputs information on the azimuth or depression angle used in controlling the sonar 110. Alternatively, the sonar angle information input unit 103 inputs information on the azimuth or depression angle of the sonar 110 measured or estimated by, for example, a camera or other measuring means. Hereinafter, the information of the azimuth angle or the depression angle indicating the direction measured by the sonar 110 is referred to as the measurement direction of the sonar 110.

The determination unit 104 determines the time-series correlation of the image features related to the target object included in the sonar image based on the amount of change in the measurement direction of the sonar 110. That is, the determination unit 104 determines whether or not the image features related to the target object are continuous in time series based on the amount of change in the measurement direction of the sonar 110. The measurement directions of the sonar 110 are compared at two or more times, and when the amount of change in the measurement direction is less than a certain value a, it is determined that the target at each time is continuous, and the amount of change in the measurement direction is a constant value. When it is larger than b, it is determined that it is not continuous.

In determining the continuity, the sonar image and label information may be used in addition to the amount of change in the measurement direction of the sonar 110. For example, the image feature amount of the region corresponding to the target in the sonar image is detected by known image processing such as edge detection and histogram analysis, and the time series of the target included in the sonar image at two or more times. The correlation may be determined based on the image feature amount of.

For example, the larger the image correlation value based on the image feature amount of the target included in the sonar image, the higher the image correlation between the comparison images. Therefore, the smaller the inverse of the image correlation value, the higher the correlation between the comparison images. Will be shown. Therefore, when the sum (V + 1 / R) of the normalized amount of change V in the measurement direction of the sonar 110 and the normalized reciprocal 1 / R of the image correlation value R is less than a certain value A, the target object is in chronological order. It can also be determined that they are continuous, and when the sum (V + 1 / R) is larger than the constant value B, it is determined that they are not continuous. A weighted sum (αV + β / R) (where α, β> 0) may be adopted instead of the sum (V + 1 / R). By doing so, it is possible to improve the determination accuracy of the time-series correlation of the target object included in the sonar image.

The label correction unit 105 adds or deletes label information based on the time-series continuity of the image features related to the target object included in the sonar image determined by the determination unit 104. The label correction unit 105 adds label information when it is determined by the determination unit 104 that the targets included in the sonar image are continuous in time series and no label information is given to the targets. Further, the label correction unit 105 deletes the label information when the determination unit 104 determines that the target objects included in the sonar image are not continuous in time series and the label information is given to the target objects.

The machine learning data output unit 106 converts the learning data into a data format used in machine learning based on the sonar image generated by the sonar data input unit 101 and the corrected label data corrected by the label correction unit 105. And output. By using the learning data with the label information corrected for machine learning, the learning accuracy can be improved.

(Structure of sonar 110)
FIG. 2 is a plan view showing an example of an arrangement of a plurality of sonar sensors 201 constituting the sonar 110 in the embodiment. Each sonar sensor 201 measures the distance to the target object 211 based on the signal intensity of the reflected wave reflected by the target object 211 by the ultrasonic waves emitted within the range of each measurement range 202 shown by the broken line in FIG. .. The sonar 110 can capture the reflected wave reflected by the target 211 located in the emission direction of the ultrasonic waves emitted from at least one sonar sensor 201. As shown in FIG. 2, the arrangement direction of the sonar sensors 201 is the x-axis direction.

FIG. 3 is a side view showing an example of the sonar 110 and the sonar sensor 201 in the embodiment. FIG. 3 is a side view seen from a direction orthogonal to the plan view direction of FIG. As shown in FIG. 3, the plurality of sonar sensors 201 are located at the same height, and the depression angle θ from the reference is set in each measurement direction 203. The plurality of sonar sensors 201 emit ultrasonic waves within the measurement range 202 centered on the depression angle θ, capture the reflected waves, and measure the signal strength with respect to the distance from the sonar sensor 201. As shown in FIG. 3, the direction of the depth measured by the sonar sensor 201 is the y-axis direction.

FIG. 4 is a diagram showing an example of sonar image 401 and label information generated based on the sonar data measured by the sonar 110 in the embodiment. In the sonar image 401, the x-axis resolution is determined by the number of a plurality of sonar sensors 201, and the y-axis resolution is determined by the sampling time of each sonar sensor 201.

In the sonar image 401, the image feature 402 shows the reflected wave from the target 211. The sonar image 401 is recognized by a person or device as including, for example, the target 211 within the range of the rectangular region 403 defined by Ps (xs, ys) and Pe (xe, ye) in the sonar image 401. The image feature 402 is labeled with. At this time, Ps (xs, ys) and Pe (xe, ye) are label information. At the time of machine learning, the image feature amount within the range of the rectangular area 403 is analyzed and learned.

(Example of labeling failure)
FIG. 5 is a diagram showing an example of sonar image 401a and label information when labeling fails. The sonar image 401a of FIG. 5A shows an example in which the image feature 402a of the target object 211 is not labeled. The image feature 402a is not labeled because it is harder for a person to see than the image feature 402 in FIG. 4, but since the amount of change in depression angle Δθ from the sonar image 401 is less than a constant value a, it is a target object. It is determined that 211 is within the measurement range 202 of the sonar 110. In such a case, adding label information to the image feature 402a improves the accuracy of machine learning.

Further, the sonar image 401b of FIG. 5B shows an example in which the image feature of the target object 211 is not present but is labeled. As shown in FIG. 5B, even when the amount of change in depression angle Δθ from the sonar image 401 in FIG. 4 exceeds a certain value b and the target object 211 deviates from the measurement range 202 of the sonar 110. If the label information 403b is labeled with information such as GPS or aerial photograph, the image features that do not include the target object 211 are learned, so that the accuracy of machine learning is lowered. In such a case, the accuracy of machine learning is improved by deleting the label information 403b.

(Correction of label information)
FIG. 6 is a diagram for explaining the correction of label information according to the embodiment. In FIG. 6, the label information indicates the presence or absence of the target object and the above-mentioned Ps (xs, ys) and Pe (xe, ye) as ROI (Region of Interest). Although FIG. 6 shows a case where the amount of change in the depression angle is used, the amount of change in the azimuth angle and the image feature amount, or an appropriate combination of the depression angle, the azimuth angle, and the image feature amount may be used.

Table 601a shows an example in which the image feature 402a of the target object 211 of FIG. 5 (a) is not labeled. In FIG. 5A, there was no label information in the frame of t = n, but the amount of change Δθ of the depression angle from the previous frame of t = n-1 was less than a constant value a. , T = n and t = n-1 are determined to have continuity in the sonar images of each frame, and label information is added. By adding the label information in this way, the accuracy of machine learning is improved.

Table 601b shows an example in which the sonar image 401b of FIG. 5B is labeled despite the absence of image features of the target. In FIG. 5B, while the labels are given in the frame of t = n, the amount of change Δθ of the depression angle from the previous frame of t = n-1 exceeds a certain value b, so t = It is determined that there is no continuity with the sonar image of the frame of n-1, and the label information is deleted. By deleting the label information in this way, the accuracy of machine learning is improved.

FIG. 7 is a flowchart showing information processing for machine learning according to the embodiment. The information processing for machine learning may be real-time processing or batch processing.

First, in step S701, the machine learning information processing apparatus 100 determines whether or not the label correction processing of the sonar data for the required number of frames is completed. When the label correction processing of the sonar data for the required number of frames is completed (step S710Yes), the machine learning information processing apparatus 100 ends the machine learning information processing. On the other hand, when the label correction processing of the sonar data for the required number of frames is not completed (step S710No), the machine learning information processing apparatus 100 shifts the processing to step 702.

Next, in step S702, the machine learning information processing apparatus 100 inputs sonar data, label information, and sonar angle information by the sonar data input unit 101, the label information input unit 102, and the sonar angle information input unit 103.

Next, in step S703, the machine learning information processing apparatus 100 determines the amount of change in the sonar depression angle in each frame by the determination unit 104. The amount of change may be obtained from the sonar depression angle one frame before, or may be obtained from the average value of a plurality of frames.

When the amount of change in the sonar depression angle is smaller than the constant value a, the machine learning information processing apparatus 100 adds label information by the label correction unit 105 (step S704). The label information to be added may be a copy of the information one frame before, or may be obtained by averaging the information of two or more frames before and after. Further, when the change amount of the sonar depression angle of the machine learning information processing apparatus 100 is larger than a constant value b, the machine learning information processing apparatus 100 deletes the label information by the label correction unit 105 (step S705). When the step S704 or S705 is completed, the machine learning information processing apparatus 100 returns the process to step S701. Further, when the amount of change in the sonar depression angle is equal to or greater than a certain value a and equal to or less than a certain value b, the machine learning information processing apparatus 100 returns the process to step S701.

According to the above-described embodiment, when the measurement direction such as the azimuth angle and the depression angle of the sonar changes, the continuity of the sonar data is estimated and the label is corrected to perform machine learning using the labeled sonar data. It is possible to prevent a decrease in accuracy and further improve the accuracy.

(Other embodiments)
(1) In the above-described embodiment, the label information is information based on Classification that displays the coordinates of the ROI. However, the present invention is not limited to this, and for example, the position and shape of the segmented image are used as label information, and additional label information is generated by estimation or complementation, thereby adding and deleting the label information as in the above-described embodiment. Can be done.

(2) In the above-described embodiment, an example of correcting the label information of the sonar data measured by the sonar is shown. However, not limited to sonar, it is also possible to correct the label information of the data measured by the sensor or the measuring means for measuring the depth information. Sensors or measuring means for measuring depth information include laser radar, laser lidar, and millimeter wave radar.

(3) In the above-described embodiment, the label information is added or deleted based on the amount of change in the azimuth or depression angle of the sonar. However, the label information may be corrected not only by addition and deletion but also by complementation or correction of a value based on estimation.

(4) In the above-described embodiment, the label information is added or deleted according to the one-dimensional change amount of the azimuth or depression angle of the sonar. However, the present invention is not limited to this, and label information may be added or deleted according to the amount of two-dimensional change in the azimuth angle and the depression angle. That is, the label information may be corrected based on the sum of squares of the amount of change in the first axial direction and the second axial direction of the sensor or the measuring means or the result of determining the square root thereof as a threshold value.

(Computer hardware that realizes information processing equipment for machine learning)
FIG. 8 is a diagram showing an example of computer hardware that realizes the machine learning information processing apparatus 100 according to the embodiment. In the computer 500 that realizes the machine learning information processing device 100, an arithmetic unit 530 represented by a CPU, a memory 540 such as a RAM, an input device 560, and an output device 570 are interconnected through a memory controller 550. In the computer 500, a predetermined program is read from an external storage device 580 such as an SSD or HDD via the I / O controller 520 and executed in cooperation with the arithmetic unit 530 and the memory 540 for machine learning. The information processing device 100 is realized. The program for realizing the machine learning information processing apparatus 100 may be read from the distribution medium and acquired, or may be acquired from an external computer by communication via the network interface 510.

Although the embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments, and includes various modifications. For example, the above-described embodiment has been described in detail in order to explain the present invention in an easy-to-understand manner, and is not necessarily limited to the one including all the described configurations. Further, it is possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. In addition, it is possible to add, delete, replace, integrate, or distribute a part of the configuration of each embodiment. Further, each configuration and each process shown in the embodiment may be appropriately distributed or integrated based on the processing efficiency or the mounting efficiency.

100: Machine learning information processing device, 101: Sonar data input unit, 102: Label information input unit, 103: Sonar angle information input unit, 104: Judgment unit, 105: Label correction unit, 106: Machine learning data output unit , 110: sonar, 201: sonar sensor, 211: target, 401, 401a, 401b: sonar image, 500: computer

Claims (10)

A data input unit that inputs measurement data measured by a measurement means that measures depth information and generates a measurement image based on the measurement data.
A label information input unit for inputting label information to be given to an image feature related to a target object included in the measurement image, and a label information input unit.
An angle information input unit for inputting angle information indicating the measurement direction of the measurement means for each measurement data,
A determination unit that determines the time-series correlation of the image features based on the angle information,
A machine learning information processing apparatus including a label correction unit that corrects label information given to the image feature based on the correlation determination result. Claim 1 is characterized in that the determination unit determines the time-series correlation of the image features based on the amount of change of the angle information in the first axial direction of the measuring means between two or more times. Information processing device for machine learning described in. further,
The determination unit determines the time-series correlation of the image features based on the amount of change in the angle information in the second axial direction of the measuring means different from the first axial direction between two or more times. The information processing apparatus for machine learning according to claim 2, wherein the information processing apparatus is to be used. When the amount of change is less than the threshold value, the determination unit determines that the image features are continuous in time series.
When the determination unit determines that the image features are continuous in time series and the label information is not added to the image features, the label correction unit applies the label information to the image features. The information processing apparatus for machine learning according to claim 2, wherein the information processing device is provided. When the amount of change is larger than the threshold value, the determination unit determines that the image features are not continuous in time series.
The label correction unit deletes the label information when the determination unit determines that the image features are not continuous in time series and the label information is added to the image features. The information processing apparatus for machine learning according to claim 4, which is characterized. The measuring means is sonar.
The fifth aspect of claim 5, wherein the determination unit determines the time-series correlation of the image features based on the amount of change between the azimuth angle of the sonar or the depression angle of the sonar between two or more times. Information processing device for machine learning. further,
The first aspect of claim 1, wherein the determination unit determines the time-series correlation of the image features based on the time-series image feature amount of the target object included in the measurement image at two or more times. Information processing device for machine learning. The information processing apparatus for machine learning according to claim 1, further comprising a data output unit for machine learning that generates and outputs data used for machine learning based on the label information corrected by the label correction unit. A machine learning information processing method executed by a machine learning information processing device.
The machine learning information processing device
Enter the measurement data measured by the measuring means that measures the depth information,
A measurement image is generated based on the measurement data,
Input the label information to be given to the image feature related to the target object included in the measurement image, and enter the label information.
Input the angle information indicating the measurement direction of the measurement means for each measurement data,
Based on the angle information, the time-series correlation of the image features is determined.
An information processing method for machine learning, comprising each process of correcting label information given to the image feature based on the correlation determination result. Computer,
A data input unit that inputs measurement data measured by a measurement means that measures depth information and generates a measurement image based on the measurement data.
A label information input unit for inputting label information to be given to an image feature related to a target object included in the measurement image.
An angle information input unit for inputting angle information indicating the measurement direction of the measurement means for each measurement data.
A determination unit that determines the time-series correlation of the image features based on the angle information.
A machine learning information processing program for functioning as a label correction unit that corrects label information given to the image feature based on the correlation determination result.

Images