JP2020035199A - Learning data generation device and obstacle detection system - Google Patents

Abstract

To efficiently generate learning data for machine learning.SOLUTION: A learning data generation device for generating learning data used for machine learning, comprises a processing unit displaying an input image on a display unit. The processing unit stores the input image displayed on the display unit in a storage unit as the learning data when receiving an instruction from a user.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a learning data generation device and an obstacle detection system.

  Patent Document 1 below discloses that when a vehicle is traveling on a track laid on the ground, it is checked whether there is any obstacle in the track, and if there is an obstacle within the building limit, the vehicle is automatically turned on. An obstacle detection system for stopping is disclosed.

  By the way, in order to prevent the above-mentioned obstacle detection system from detecting an object which is not an obstacle as an obstacle (hereinafter, referred to as “over-detection”) and automatically stopping the vehicle, the vehicle detects obstacles. Is provided with a cancel button for disabling. With this, when the vehicle occupant determines that the obstacle detection of the obstacle detection system is overdetection, the passenger presses the cancel button to prevent the vehicle from being automatically stopped due to the overdetection. .

JP-A-2006-88183

  By the way, the inventors of the present application have obtained an idea to automate the determination of overdetection by the occupant of the vehicle using AI technology. However, in order to perform overdetection determination using the AI technology, a large amount of learning data for machine learning is required. Therefore, it is desired to efficiently create learning data for machine learning. Such a problem is not limited to the determination of overdetection, but is a problem common when creating learning data for machine learning.

  The present invention has been made in view of such circumstances, and an object of the present invention is to provide a learning data generation device and an obstacle detection system capable of efficiently creating learning data for machine learning.

  One embodiment of the present invention is a learning data generation device that generates learning data used for machine learning, including a processing unit that displays an input image on a display unit, wherein the processing unit has received an instruction from a user. In this case, the learning data generation device is characterized in that the input image displayed on the display unit is stored in the storage unit as the learning data.

  One embodiment of the present invention is the above-described learning data generation device, including an operation unit, wherein the processing unit receives an instruction from a user by detecting that the operation unit has been operated.

  One embodiment of the present invention is the above-described learning data generation device, wherein the input image is an image reflecting an object detected as an obstacle, and the learning data is an object over-detected as the obstacle. It is an image of.

  One embodiment of the present invention is the above-described learning data generation device, wherein the operation unit includes a plurality of switches corresponding to each of the obstacle types, and the processing unit includes: The type of the obstacle corresponding to the switch operated by the user and the input image displayed on the display unit when the switch is operated are linked and stored in the storage unit as the learning data. .

  One embodiment of the present invention is the above-described learning data generation device, wherein the processing unit detects the input image displayed on the display unit as the obstacle when the operation unit is operated. Only an image of the detected object is cut out and stored in the storage unit as learning data.

  One embodiment of the present invention includes an obstacle detection device that detects an obstacle based on a captured image from an imaging device mounted on a vehicle, the learning data generation device described above, and a learning data generation device. An obstacle detection system, wherein the processing unit causes the display unit to display, as the input image, an image of an object detected as an obstacle by the obstacle detection device from the captured image. It is an obstacle detection system.

  As described above, according to the present invention, learning data for machine learning can be efficiently created.

1 is a diagram illustrating an example of a schematic configuration of an obstacle detection system 1 including a learning data generation device according to an embodiment of the present invention. FIG. 4 is a diagram illustrating an example of mask information stored in a storage unit 7 according to an embodiment of the present invention in a table format. FIG. 4 is a diagram illustrating a method for generating an obstacle candidate image according to an embodiment of the present invention. FIG. 4 is a diagram illustrating an operation of a learning data generation method according to an embodiment of the present invention.

  Hereinafter, a learning data generation device and an obstacle detection system according to an embodiment of the present invention will be described with reference to the drawings.

  FIG. 1 is a diagram illustrating an example of a schematic configuration of an obstacle detection system 1 including a learning data generation device according to an embodiment of the present invention.

  An obstacle detection system 1 according to an embodiment of the present invention includes an obstacle detection process of detecting an obstacle based on a captured image captured from a traveling vehicle, and learning data for generating learning data from the captured image. This is a system that executes a generation process. For example, in the obstacle detection processing in the obstacle detection system 1, an image pickup apparatus mounted on the inspection vehicle is configured to run a certain vehicle (hereinafter, referred to as an “inspection vehicle”) along a track laid on a road surface. This is a process of detecting an obstacle in the trajectory from an image captured from the camera. The obstacle in the track is, for example, a falling object or a fallen tree.

  Note that the obstacle detection processing is desirably performed prior to the operation of the track running vehicle. For this reason, the obstacle detection system 1 is operated, for example, between the end of the operation of the track traveling vehicle and the start of the next operation (for example, at midnight). That is, the obstacle detection processing is performed by, for example, running the inspection vehicle at midnight.

  Hereinafter, a configuration of the obstacle detection system 1 according to an embodiment of the present invention will be described.

  As shown in FIG. 1, the obstacle detection system 1 includes an imaging device 2, an obstacle detection device 3, and a learning data generation device 4.

  The imaging device 2 is mounted on an inspection vehicle. For example, the imaging device 2 captures an image of the front of the inspection vehicle in the traveling direction, and outputs the captured image to the obstacle detection device 3. For example, the imaging device 2 includes a CCD image sensor using a semiconductor device such as a charge coupled device (CCD) or a MOS image using a complementary metal oxide semiconductor (CMOS). A solid-state image sensor such as a sensor is applied.

  The obstacle detection device 3 detects an obstacle based on an image captured by the imaging device 2 mounted on the vehicle.

  The learning data generation device 4 is a device that generates learning data used for machine learning, and generates learning data based on an image in which an obstacle is detected by the obstacle detection device 3.

  Hereinafter, a configuration of the obstacle detection device 3 according to an embodiment of the present invention will be described.

  The obstacle detection device 3 according to one embodiment of the present invention includes an object detection unit 5, a mask processing unit 6, a storage unit 7, and an obstacle determination unit 8.

  The object detection unit 5 performs known image processing on the captured image captured by the imaging device 2 and performs an object detection process of detecting an object appearing in the captured image. The image processing is not particularly limited as long as it is image processing necessary for the object detection processing. For example, edge detection processing using a DoG (Difference of Gaussian) filter, binarization processing, or the like, opening processing And labeling processing. For example, the object detection unit 5 can detect an object appearing in a captured image by using a process (trajectory detection process and object detection process) described in JP-A-2006-88183.

  Further, the object detection unit 5 acquires information (hereinafter, referred to as “specific information”) that can specify the detected object, and associates the specific information with the object detected by the object detection process.

  For example, the specific information is the position information of the object on the captured image, the position information on the track of the inspection vehicle when the captured image showing the object is obtained (hereinafter referred to as “track position information”), Information (hereinafter, referred to as “feature information”) indicating characteristics of the object such as the size (for example, width and height) and shape of the object on the captured image.

  For example, the object detection unit 5 acquires position information and feature information of the object on the captured image. The position information and the characteristic information can be obtained by the object detection processing. In addition, the object detection unit 5 acquires track position information of the inspection vehicle from a self-position acquisition unit provided in the inspection vehicle. For example, the self-position obtaining unit can obtain the position information of the inspected vehicle in global coordinates by using a GPS (Global Positioning System). When the inspection vehicle travels on the rails of the railway, the object detection unit 5 can obtain the track position information by an encoder or the like that detects the ground child signal from the ground child or the rotation of the wheel of the inspection vehicle.

  The mask processing unit 6 executes a mask process for preventing an object that meets a predetermined condition among the objects detected by the object detection unit 5 from being detected as an obstacle. For example, the mask processing unit 6 excludes an object present in a specific area (hereinafter, “mask area”) in the captured image from objects to be detected (hereinafter, referred to as “object to be detected”).

  Specifically, the mask processing unit 6 acquires information on a mask area (hereinafter, referred to as “mask information”) from a mask table stored in the storage unit 7 in advance. Then, the mask processing unit 6 sets one or more mask areas on the captured image based on the acquired mask information, and when an object exists in the set mask area, excludes the object from obstacle detection targets. I do.

The storage unit 7 stores mask information.
FIG. 2 is a diagram illustrating an example of the mask information stored in the storage unit 7 in a table format. The storage unit 7 is provided with a mask table for storing mask information for each mask area.

  The mask table stores mask region identification information (hereinafter, referred to as “mask region identification information”), orbital position, reference coordinates, width, and height in association with each other.

The mask area identification information is information for identifying each of the plurality of mask areas.
The track position indicates the position of the inspection vehicle on the track. For example, if the track is a rail of a railway, the track position is on the order of kilometers. The track position may be position information of the inspection vehicle obtained by using a GPS (Global Positioning System).

  The reference coordinates are coordinates that serve as references for setting a mask area on a captured image, and can be arbitrarily set by the user.

  The width is information indicating the width of the mask area on the captured image. The height is information indicating the height of the mask area on the two-dimensional image.

  The obstacle determining unit 8 determines, as the obstacle, an object that is not excluded from the obstacle detection targets by the mask processing unit 6 among the objects detected by the object detecting unit 5.

  Hereinafter, the configuration of the learning data generation device 4 according to an embodiment of the present invention will be described.

  The learning data generation device 4 includes a processing unit 9, a display unit 10, an operation unit 11, and a storage unit 12. The operation unit 11 may not be included in the learning data generation device 4.

  The processing unit 9 extracts an image in a predetermined range (hereinafter, referred to as an “obstacle candidate image”) in which the object determined to be an obstacle by the obstacle determination unit 8 appears from the captured image, and extracts the extracted image. The obstacle candidate image is associated with the specific information of the object. Then, the processing unit 9 causes the display unit 10 to display the obstacle candidate image. Note that the obstacle candidate image acquired from the obstacle determination unit 8 is an example of the “input image” of the present invention.

For example, as illustrated in FIG. 3, the object detection unit 5 performs an object detection process on the captured image G captured by the imaging device 2, detects the objects B <b> 1 to B <b> 3 in the captured image G, and performs a mask process. It is assumed that the unit 6 performs mask processing on the captured image G and sets mask areas M1 and M2. In this case, since the objects B1 and B2 are included in the mask areas M1 and M2, they are excluded from the obstacle detection targets. That is, the obstacle determination unit 8 determines, as the obstacle, the object B3 that is not excluded from the obstacle detection targets by the mask processing unit 6 among the objects B1 to B3 detected by the object detection unit 5.
Therefore, the processing unit 9 obtains the obstacle candidate image C of the object B3 by cutting out the object B3 determined as the obstacle by the obstacle determination unit 8 from the captured image G. Then, the processing unit 9 associates the obstacle candidate image C with the specific information of the object B3, and causes the display unit 10 to display the obstacle candidate image C.

  The processing unit 9 performs a tracking process using a known tracking technique on an object determined to be an obstacle over a plurality of captured images that are continuously acquired at predetermined time intervals, and the same object is detected. The obstacle candidate images may be tracked by associating the imaged areas with each other.

  Further, the processing unit 9 receives an instruction from the user. Specifically, the processing unit 9 is connected to the operation unit 11 by wire or wirelessly, and when acquiring an operation signal from the operation unit 11, determines that an instruction from the user has been received. The instruction from the user instructs not to detect the object displayed on the display unit 10 as an obstacle, that is, to determine that the object is overdetected. Therefore, when acquiring the operation signal from the operation unit 11, the processing unit 9 determines that the object displayed on the display unit 10 is not an obstacle, that is, that the overdetection has occurred, The object determination unit 8 is notified. Accordingly, the obstacle detection device 3 determines that the object (for example, the object being tracked) displayed on the display unit 10 is not an obstacle, and cancels the determination that the object is an obstacle. .

  Further, when the processing unit 9 obtains the operation signal from the operation unit 11, the processing unit 9 adds the specific information of the obstacle candidate image to the obstacle candidate image displayed on the display unit 10, and The image is stored in the storage unit 12 as learning data. That is, when the processing unit 9 determines that the overdetection has occurred, the processing unit 9 stores the obstacle candidate image to which the overdetected object is reflected and to which the specific information is added, as the learning data in the storage unit 12. . Therefore, the storage unit 12 stores an image of the object over-detected by the obstacle detection device 3 (an obstacle candidate image) as learning data.

  The display unit 10 displays the obstacle candidate image output from the processing unit 9 on a display screen. For example, the display unit 10 is a monitor for a personal computer. However, the display unit 10 only needs to be able to display an image, and may be a display device of a portable device such as a mobile phone or a head mounted display.

The operation unit 11 is operated by the user when overdetection occurs. The operation unit 11 outputs an operation signal to the processing unit 9 when operated by a user. Thus, if the user determines that the obstacle detected by the obstacle detection device 3 is overdetected, the user can operate the operation unit 11 to cancel the detection of the obstacle.
The operation unit 11 is not particularly limited as long as it can be operated by a user, and is, for example, a switch, a button, a touch panel, a pointing device, or the like.

  The storage unit 12 stores, for example, a database, in which the obstacle candidate image to which the specific information has been transmitted from the processing unit 9 is stored as learning data. The storage unit 12 is configured by a HDD (Hard Disk Drive), a memory, and the like.

  The learning device 100 performs machine learning (for example, deep learning or a support vector machine) using the obstacle candidate image stored in the storage unit 12 to extract a feature of an object that is likely to be overdetected. Based on this, the mask table stored in the storage unit 7 is created, updated, or extended. For example, the learning device 100 performs machine learning to extract features such as position information, shape, and size of an object that is likely to be overdetected, and adds the features to the mask table as mask information. Thereby, overdetection in the obstacle detection system 1 can be suppressed. The learning device 100 may be configured by a microprocessor such as a CPU or an MPU, a microcontroller such as an MCU, and the like.

  Hereinafter, a method of generating learning data of the obstacle detection system 1 according to an embodiment of the present invention will be described with reference to FIG. FIG. 4 is a diagram illustrating the operation of the learning data generation method according to one embodiment of the present invention.

  When the inspection vehicle on which the imaging device 2 is mounted starts traveling on a track, a process of imaging the front of the inspection vehicle in the traveling direction with the imaging device 2 is started (step S101). The obstacle detection device 3 acquires a captured image captured by the imaging device 2 at predetermined time intervals, and performs an obstacle detection process for each captured image (step S102). Specifically, the obstacle detection device 3 detects an object on the captured image by the object detection process and acquires specific information of the object. Next, the obstacle detection device 3 excludes an object that is clearly not an obstacle from the obstacle detection target by performing the mask processing. Then, the obstacle detection device 3 determines an object that is not excluded by the mask processing as an obstacle.

  The processing unit 9 of the learning data generation device 4 cuts out the image of the object determined as the obstacle by the obstacle detection device 3 from the captured image, and displays the cut-out image on the display unit 10 as an obstacle candidate image ( Step S103).

  Here, the crew of the inspection vehicle determines whether or not overdetection has occurred while checking the image displayed on the display unit 10. That is, the crew determines whether or not the object displayed on the display unit 10 is an obstacle, and if it determines that the object is not an obstacle, it determines that overdetection has occurred, and Operate.

  For example, it is assumed that this inspection vehicle is a vehicle that detects an obstacle on a rail of a railway. In this case, it is assumed that a ground child is displayed on the display unit 10 as an object determined by the obstacle detection device 3 as an obstacle. However, the ground child is always installed in the rail of the railway, and is not an obstacle. Therefore, when the ground child is displayed on the display unit 10, the crew determines that overdetection has occurred and operates the operation unit 11.

The processing unit 9 operates the operation unit 11 for a predetermined period while an image (an obstacle candidate image) in which an object determined to be an obstacle by the obstacle detection device 3 is displayed on the display unit 10. It is determined whether or not an operation has been performed (step S104), and if an operation signal has been obtained from the operation unit 11, it is determined that the operation unit 11 has been operated. On the other hand, if the processing unit 9 does not acquire an operation signal from the operation unit 11 for a predetermined period, the processing unit 9 determines that the operation unit 11 has not been operated, and returns to the process of step S103.
Here, when the operation unit 11 has not been operated for a predetermined period, the obstacle detection system 1 may determine that there is an obstacle in the track and automatically stop the inspection vehicle. However, when the operation unit 11 is operated during the predetermined period, the obstacle detection system 1 may determine that overdetection has occurred and cancel the automatic stop of the inspection vehicle.

  When the processing unit 9 determines that the operation unit 11 has been operated, the processing unit 9 adds the specific information of the object to the image of the object (obstacle candidate image) displayed on the display unit 10. Then, the processing unit 9 stores the obstacle candidate image to which the specific information is added in the storage unit 12 as learning data (Step S105). For example, the processing unit 9 stores the obstacle candidate image to which the specific information is added in the storage unit 12 as learning data for identifying an overdetection of the obstacle.

  The obstacle detection device 3 determines whether or not the traveling of the vehicle has been completed (step S106). If it is determined that the traveling of the vehicle has been completed, the obstacle detection device 3 ends the obstacle detection processing (step S106). S107), the generation of the learning data ends. On the other hand, if the obstacle detection device 3 determines that the traveling of the vehicle is continuing, the process returns to step S103.

  As described above, the embodiments of the present invention have been described in detail with reference to the drawings. However, the specific configuration is not limited to the embodiments and includes a design and the like within a range not departing from the gist of the present invention.

(Modification 1) The obstacle detection system 1 of the above embodiment further includes a millimeter-wave radar, a laser radar, and the like, and detects an object by using a detection result by the millimeter-wave radar or the laser radar and an image captured by the imaging device 2. Processing may be performed.

(Modification 2) In the above embodiment, the case where the operation unit 11 is included in the configuration of the learning data generation device 4 has been described. However, the present invention is not limited to this. May not be included in the configuration.

(Modification 3) In the above embodiment, the obstacle detection device 3 and the learning data generation device 4 have been described as separate devices, but the present invention is not limited to this. For example, the obstacle detection device 3 and the learning data generation device 4 may be configured integrally. That is, the obstacle detection system 1 may include the imaging device 2, a processing device (for example, configured by a microprocessor such as a CPU or an MPU, a microcontroller such as an MCU), and the operation unit 11. The processing device may include an object detection unit 5, a mask processing unit 6, a storage unit 7, an obstacle determination unit 8, a processing unit 9, a display unit 10, and a storage unit 12.

(Modification 4) In the above embodiment, the processing unit 9 cuts out an image of an object determined as an obstacle by the obstacle detection device 3 from the captured image, and uses the cut-out image as an obstacle candidate image on the display unit 10. However, the present invention is not limited to this. That is, the processing unit 9 only needs to display an image showing an object determined to be an obstacle by the obstacle detection device 3 on the display unit 10, and the display mode is not particularly limited. For example, the processing unit 9 displays the captured image on the display unit 10 and surrounds an object determined as an obstacle by the obstacle detection device 3 on the captured image with a predetermined frame (for example, a bounding box). Alternatively, an object determined to be an obstacle may be displayed on the display unit 10.
In this case, when the processing unit 9 obtains the operation signal from the operation unit 11, the processing unit 9 selects the object surrounded by the predetermined frame (that is, the over-detected object) in the image displayed on the display unit 10. ) Is cut out, and the cut out image is set as an obstacle candidate image. Then, the processing unit 9 may add the specific information of the object surrounded by the predetermined frame to the obstacle candidate image and store it as learning data in the storage unit 12.

(Modification 5) In the above embodiment, the learning device 100 creates, updates, or expands the mask table by performing machine learning using the obstacle candidate images stored in the storage unit 12. It is not limited to this. For example, the learning device 100 performs machine learning using the obstacle candidate image stored in the storage unit 12, so that the object detection unit 5 detects the object learning unit 5, and the mask processing unit 6 detects the obstacle detection target. It is also possible to generate a learning model (a learning model for identifying over-detection) that determines whether an object not excluded from is an obstacle. And the obstacle determination part 8 may perform an obstacle determination using this learning model.

(Modification 6) The operation unit 11 of the above embodiment may include a plurality of switches corresponding to each type of obstacle. The processing unit 9 then links the type of obstacle corresponding to the switch operated by the user among the plurality of switches with the input image displayed on the display unit 10 when the switch is operated. In addition, it may be stored in the storage unit 12 as learning data.
For example, in the case of detecting an obstacle in a rail of a railway, for example, a switch is provided corresponding to each of the easily detectable objects such as a rail, a ground child, and a light. Then, for example, when the user operates the switch corresponding to the ground child, the information of the ground child is linked to the obstacle candidate image displayed on the display unit 10 and stored in the storage unit 12 as learning data. . Thereby, the labor of labeling required for machine learning can be reduced.

(Modification 7) The learning data according to the above embodiment may be data with a correct answer or may not be data with a correct answer. That is, the machine learning performed by the learning device 100 may be supervised learning, unsupervised learning, or reinforcement learning. Furthermore, the machine learning according to the present embodiment can include deep learning.

(Modification 8) In the obstacle detection system according to the above-described embodiment, the obstacle detection device 3 and the learning data generation device 4 do not need to be mounted on the vehicle. The installation location is not particularly limited as long as a location where a captured image can be acquired from the imaging device 2. Furthermore, only the learning data generation device 4 may be installed outside the vehicle.

(Modification 9) In the above embodiment, the processing unit 9 receives an instruction from a user by acquiring an operation signal from the operation unit 11, but the present invention is not limited to this. For example, the processing unit 9 may receive an instruction from the user by detecting a voice input from the user. Further, the processing unit 9 may interpret the user's voice using a known voice recognition technology to identify whether or not the object displayed on the display unit 10 is an obstacle.

(Modification 10) In the above embodiment, the processing unit 9 adds the specific information to the obstacle candidate image and stores it in the storage unit 12, but the present invention is not limited to this. That is, the processing unit 9 may store the obstacle candidate image in the storage unit 12 as learning data, and the addition of the specific information is not necessarily essential.

  As described above, the learning data generation device 4 according to the present embodiment includes the processing unit 9 that causes the display unit 10 to display an input image. Then, when receiving an instruction from the user, the processing unit 9 stores the input image displayed on the display unit 10 in the storage unit 12 as learning data. At this time, the input image stored in the storage unit 12 as the learning data may be a part of the input image or the entire input image.

  According to such a configuration, learning data can be efficiently created.

  Further, the learning data generation device 4 may include the operation unit 11. Then, the processing unit 9 may determine that an instruction from the user has been received by detecting that the operation unit 11 has been operated.

  According to such a configuration, learning data can be automatically collected by the operation unit 11 and the number of steps for creating learning data can be reduced.

  The input image may be an image showing an object detected as an obstacle, and the learning data may be an image of an object over-detected as an obstacle.

  According to such a configuration, it is possible to automatically collect learning data by operating the operation unit 11 while checking for the presence or absence of an obstacle on the orbit, thereby reducing the number of steps for creating learning data. Furthermore, it is possible to machine-learn an object that is likely to be over-detected as an obstacle, thereby improving the accuracy of obstacle detection.

DESCRIPTION OF SYMBOLS 1 Obstacle detection system 2 Imaging device 3 Obstacle detection device 4 Learning data generation device 5 Object detection unit 6 Mask processing unit 7, 12 Storage unit 8 Obstacle determination unit 9 Processing unit 10 Display unit 11 Operation unit

Claims (6)

A learning data generation device that generates learning data used for machine learning,
A processing unit for displaying the input image on the display unit,
The learning data generation device, wherein, when receiving an instruction from a user, the processing unit stores the input image displayed on the display unit as the learning data in a storage unit. Equipped with an operation unit,
The learning data generation device according to claim 1, wherein the processing unit receives an instruction from a user by detecting that the operation unit has been operated. The input image is an image reflecting an object detected as an obstacle,
The learning data generation device according to claim 2, wherein the learning data is an image of an object that is overdetected as the obstacle. The operation unit includes a plurality of switches corresponding to each type of the obstacle,
The processing unit associates a type of the obstacle corresponding to a switch operated by a user among the plurality of switches with the input image displayed on the display unit when the switch is operated. The learning data generation device according to claim 3, wherein the learning data is stored in the storage unit as the learning data.   When the operation unit is operated, the processing unit cuts out only an image of an object that is overdetected as the obstacle from the input image displayed on the display unit, and stores the cutout image as learning data in the storage unit. The learning data generation device according to claim 3, wherein the learning data generation device stores the learning data. An obstacle detection device that detects an obstacle based on a captured image from an imaging device mounted on a vehicle, the learning data generation device according to any one of claims 1 to 5, and an obstacle detection system including the learning data generation device. And
The obstacle detection system, wherein the processing unit causes the display unit to display, as the input image, an image of an object detected as an obstacle by the obstacle detection device from the captured image.

Images