JP2021064139A - Information processing apparatus, information processing method and information processing program - Google Patents

Abstract

To improve detection accuracy of detection using machine-learned models.SOLUTION: An information processing apparatus (1) comprises a first detection unit (101) that inputs input data into a first learned model to detect a detection target, wherein the first learned model is capable of detecting a plurality of types of detection targets; and a second detection unit (102) that inputs the input data into a second learned model to detect a second detection target, wherein the second learned model is capable of detecting at least a part of the plurality of types of detection targets. The information processing apparatus determines the final detection result based on a detection result by both detection units.SELECTED DRAWING: Figure 1

Description

The present invention relates to an information processing device or the like that detects a detection target using a trained model constructed by machine learning.

In recent years, with the development of machine learning such as deep learning, the recognition and detection accuracy of objects on images has improved, and the applications using image recognition are expanding. However, since the current detection accuracy is not 100%, further ingenuity is required to further expand the application.

For example, the image recognition device described in Patent Document 1 below performs pattern matching on an image of an object using a plurality of types of templates. Then, this image recognition device determines that the pattern matches with a plurality of types of templates, and when the degree of overlap between the templates is equal to or greater than the threshold value, the recognition target object pertaining to at least one of the templates. Recognize that. This makes it possible to improve the recognition accuracy as compared with the case where pattern matching is performed by one type of template.

Japanese Unexamined Patent Publication No. 2008-165394 (published on July 17, 2008)

However, in the above-mentioned prior art, there is room for improving the detection accuracy. For example, when a pedestrian detection template and a signboard detection template are used, if a pedestrian detection omission occurs in the pedestrian detection template, the signboard detection template does not detect the pedestrian, so walking. There is no way to make up for the omission of detection by a person. Further, even when a false detection occurs in the template for pedestrians (for example, when a roadside tree is mistakenly recognized as a pedestrian), there is no means for compensating for the false detection. Further, when the detection target is not an object (for example, when sound data is input to a trained model and a predetermined sound component is detected), even when a plurality of templates as in Patent Document 1 are used. Similarly, there is room for improving the detection accuracy.

One aspect of the present invention is to realize an information processing apparatus or the like capable of improving the detection accuracy of detection using a machine-learned model.

In order to solve the above problems, the information processing apparatus according to one aspect of the present invention inputs input data to a first trained model that has been machine-learned so that a plurality of types of first detection targets can be detected. A second machine-learned unit that has been machine-learned so as to be able to detect a first detection unit that detects the first detection target and a second detection target that is at least a part of the plurality of types of first detection targets. A third trained machine-learned so that the input data is input to the model to detect the second detection target, or a third detection target different from the first detection target can be detected. A second detection unit that inputs the input data to the model and detects the third detection target is provided, and based on the detection result of the first detection unit and the detection result of the second detection unit, Confirm the final detection result.

In order to solve the above problems, the information processing method according to one aspect of the present invention is an information processing method executed by one or a plurality of information processing devices, and is a machine so as to be able to detect a plurality of types of detection targets. Input data is input to the trained first trained model, the first detection step of detecting the detection target from the input data, and machine learning so that at least a part of the plurality of types of detection targets can be detected. The input data is input to the second trained model that has been processed, or the third trained model that has been machine-learned so that a detection target different from the first trained model can be detected. A confirmation step for determining the final detection result based on the detection result of the first detection step and the detection result of the second detection step, including the second detection step of detecting the detection target from the data. including.

According to one aspect of the present invention, it is possible to improve the detection accuracy of detection using a machine-learned model.

It is an example of the functional block diagram of the control part of the information processing apparatus which concerns on Embodiment 1 of this invention. It is a block diagram which shows the structural example of the unsuitable substance detection system including the said information processing apparatus. It is a figure which shows a state that a garbage truck is dropping garbage into a garbage pit in a garbage incineration facility. It is a figure which shows the inside of a garbage pit. It is a figure explaining the flow of the process executed by the said information processing apparatus. It is a figure explaining construction and relearning of a trained model. It is a block diagram which shows the structural example of the control part provided in the information processing apparatus which concerns on Embodiment 2 of this invention. It is a figure explaining the flow of the process executed by the said information processing apparatus. It is a block diagram which shows the structural example of the control part provided in the information processing apparatus which concerns on Embodiment 3 of this invention. It is a figure explaining the flow of the process executed by the said information processing apparatus.

[Embodiment 1]
In recent years, the introduction of unsuitable materials for incineration (hereinafter simply referred to as unsuitable materials) into waste incineration facilities has become a problem. When unsuitable materials are put into the incinerator, combustion in the incinerator may deteriorate, the incinerator's ashing equipment may be blocked, and in some cases, the incinerator may be shut down urgently. In the past, employees of waste incineration facilities randomly selected the collected waste and manually checked whether the selected waste contained inappropriate substances, which was a heavy burden on the workers.

In addition, when trying to alert the person in charge of collecting garbage in order to reduce the unsuitable items transported to the garbage incineration facility, the unsuitable items are detected from the transported garbage and the detected unsuitable items are detected. A system is needed to present to the person in charge of collection. In this case, it is not preferable to present what is not actually unsuitable as unsuitable. Further, when the photographed image is shown to the person in charge as it is, it is not preferable because it is difficult to grasp at what timing and at what position the unsuitable object is captured.

The information processing device 1 according to the embodiment of the present invention can solve the above-mentioned problems. The information processing device 1 has a function of detecting unsuitable substances from the garbage carried into the garbage incineration facility. Specifically, the information processing device 1 detects an unsuitable object by using an image of the garbage being thrown into the garbage pit. The garbage pit will be described later based on FIG. In addition, unsuitable substances may be detected after the garbage is dropped. In addition, unsuitable substances are objects that should not be incinerated in an incinerator installed in a waste incineration facility. Specific examples of unsuitable materials will be described later.

〔System configuration〕
The configuration of the unsuitable object detection system according to the present embodiment will be described with reference to FIG. FIG. 2 is a block diagram showing a configuration example of the unsuitable object detection system 100. The unsuitable object detection system 100 includes an information processing device 1, a garbage photographing device 2, a vehicle information collecting device 3, a selection display device 4, and an unsuitable object display device 5.

Further, FIG. 2 also shows an example of the hardware configuration of the information processing apparatus 1. As shown in the figure, the information processing device 1 includes a control unit 10, a high-speed storage unit 11, a large-capacity storage unit 12, an image IF (interface) unit 13, a vehicle information IF unit 14, a selection display IF unit 15, and an unsuitable object display IF. The part 16 is provided. The information processing device 1 may be, for example, a personal computer, a server, or a workstation.

The control unit 10 controls each unit of the information processing device 1 in an integrated manner. The functions of each part of the control unit 10 described later based on FIG. 1 can be realized by a logic circuit (hardware) formed in an integrated circuit (IC chip) or the like, or can be realized by software. This software may include an information processing program that causes the computer to function as each unit included in the control unit 10 shown in FIGS. 1, 7, and 9, which will be described later. When realized by software, the control unit 10 may be configured by, for example, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or a combination of these. Further, in this case, the software is stored in the large-capacity storage unit 12. Then, the control unit 10 reads the software into the high-speed storage unit 11 and executes it.

Both the high-speed storage unit 11 and the large-capacity storage unit 12 are storage devices that store various data used by the information processing device 1. The high-speed storage unit 11 is a storage device capable of writing and reading data at a higher speed than the large-capacity storage unit 12. The large-capacity storage unit 12 has a larger data storage capacity than the high-speed storage unit 11. As the high-speed storage unit 11, for example, a high-speed access memory such as SDRAM (Synchronous Dynamic Random-Access Memory) can be applied. Further, as the large-capacity storage unit 12, for example, an HDD (Hard Disk Drive), an SSD (Solid-State Drive), an SD (Secure Digital) card, an eMMC (embedded Multi-Media Controller), or the like can be applied.

The image IF unit 13 is an interface for communicating and connecting the dust photographing device 2 and the information processing device 1. Further, the vehicle information IF unit 14 is an interface for communicating and connecting the vehicle information collecting device 3 and the information processing device 1. These IF units may be for wired communication or may be for wireless communication. For example, USB (Universal Serial Bus), LAN (Local-Area Network), wireless LAN, or the like can be applied as these IF units.

The selection display IF unit 15 is an interface for communicating and connecting the selection display device 4 and the information processing device 1. Further, the unsuitable object display IF unit 16 is an interface for communicating and connecting the unsuitable object display device 5 and the information processing device 1. These IF units may also be for wired communication or wireless communication. For example, HDMI (High-Definition Multimedia Interface, registered trademark), DisplayPort, DVI (Digital Visual Interface), VGA (Video Graphics Array) terminal, S terminal, RCA terminal, or the like can be applied as these IF units. ..

The garbage photographing device 2 photographs the garbage in the process of being dropped into the garbage pit, and transmits the photographed image to the information processing device 1. Hereinafter, this photographed image will be referred to as a garbage image. As an example, the dust photographing device 2 may be a high-speed shutter camera that shoots a moving image. The dust image may be a moving image or a time-series still image taken continuously. The garbage image is input to the information processing device 1 via the image IF unit 13. Then, the input garbage image can be processed by the control unit 10 as it is, or can be processed by the control unit 10 after being stored in the high-speed storage unit 11 or the large-capacity storage unit 12.

The vehicle information collecting device 3 collects identification information of a vehicle (so-called garbage truck) that carries in garbage and drops the garbage into a garbage pit, and transmits the identification information to the information processing device 1. The dropping of garbage into the garbage pit by the garbage truck will be described later based on FIG. This identification information is used by the carry-in vehicle identification unit 105 to identify the garbage carry-in subject. The identification information may be, for example, information indicating a license plate number or the like. In this case, the vehicle information collecting device 3 may take a picture of the license plate and transmit the taken image as identification information to the information processing device 1. Further, the vehicle information collecting device 3 may receive the input of the identification information of the garbage truck and transmit it to the information processing device 1.

The selection display device 4 displays an image of an unsuitable object detected by the information processing device 1. In the unsuitable object detection system 100, the information processing device 1 displays an image of the unsuitable object detected by the information processing device 1 on the selection display device 4 in consideration of the possibility that a non-inappropriate object is erroneously determined as an unsuitable object. Let them visually confirm whether or not what is shown in the image is inappropriate. Then, the person in charge of visual confirmation selects an image showing an unsuitable object from the images displayed on the selection display device 4.

The unsuitable object display device 5 is an image of an unsuitable object selected via the selection display device 4, that is, an image visually confirmed that the unsuitable object is shown, among the images of the unsuitable object detected by the information processing device 1. Is displayed. The unsuitable object display device 5 displays the above image for alerting the person in charge, the business operator, or the like who brought in the unsuitable object.

[Shooting garbage images]
FIG. 3 is a diagram showing a state in which a garbage truck 200 is dropping garbage into a garbage pit at a garbage incineration facility. FIG. 4 is a diagram showing the inside of the garbage pit. The garbage pit is a place to temporarily store the garbage collected in the garbage incineration facility, and the garbage in the garbage pit is sequentially sent to the incinerator and incinerated. As shown in FIG. 3, the waste incineration facility is provided with a plurality of doors such as doors 300A and 300B (hereinafter, collectively referred to as door 300 when it is not necessary to distinguish them). Further, as shown in FIG. 4, a dust pit is provided at the tip of the door 300. That is, when the door 300 is opened, a drop port for dropping garbage into the garbage pit appears. As shown in FIG. 3, the garbage truck 200 drops garbage into the garbage pit from the drop port.

The dust photographing apparatus 2 is attached to a position where dust flowing on the slope 600 of FIG. 4 can be photographed. For example, the dust photographing apparatus 2 may be attached to the attachment points 400 shown in FIGS. 3 and 4. Since the mounting location 400 is located on the surface of each door 300, when the dust photographing device 2 is mounted on the mounting location 400, the dust photographing device 2 is located above the slope 600 when the door 300 is opened. This position is suitable for taking pictures of garbage. Of course, the attachment location of the dust photographing apparatus 2 can be any position where the dust flowing on the slope 600 can be photographed.

Further, when the vehicle information collecting device 3 is a photographing device, the vehicle information collecting device 3 may also be attached to the attachment point 400. When the garbage truck 200 approaches the door 300, the door 300 is closed, so that the license plate of the garbage truck 200 can be photographed from the vehicle information collecting device 3 attached to the mounting location 400. Of course, the mounting location of the vehicle information collecting device 3 can be any position where the garbage truck 200 can be photographed, and may be mounted at a position different from that of the garbage capturing device 2. Further, the vehicle information collecting device 3 may be, for example, an information input device. In this case, the vehicle information collecting device 3 is attached to the operator room to receive the input of the identification information of the garbage truck 200 by the operator. May be good.

〔Device configuration〕
The configuration of the information processing device 1 will be described with reference to FIG. FIG. 1 is an example of a functional block diagram of the control unit 10 of the information processing device 1. The control unit 10 shown in FIG. 1 includes a first detection unit 101, a second detection unit 102, a learning unit 103, a selection display control unit 104, a carry-in vehicle identification unit 105, and an unsuitable object display control unit 106. .. Further, the large-capacity storage unit 12 shown in FIG. 1 includes an input data storage unit 121, a detection result storage unit 122, a learned model storage unit 123, and a teacher data storage unit 124.

The first detection unit 101 detects the first detection target by inputting input data into the first trained model that has been machine-learned so that a plurality of types of first detection targets can be detected. Note that detecting a plurality of types of detection targets means that there are a plurality of classifications (also referred to as classes) trained by the first trained model.

Further, the second detection unit 102 inputs the input data to the second trained model machine-learned so as to detect the second detection target which is at least a part of the first detection target of a plurality of types. The second detection target is detected. In the present embodiment, an example will be described in which a plurality of types of first detection targets are a plurality of types of unsuitable objects, and a second type of detection target is also unsuitable. The first detection target and the second detection target may include an object similar in appearance to the unsuitable object but not an unsuitable object.

The first trained model and the second trained model are read from the trained model storage unit 123, and the input data is read from the input data storage unit 121. Although the details will be described later, the first trained model and the second trained model are constructed by machine learning using the original teacher data 124a stored in the teacher data storage unit 124. Further, the first detection target and the second detection result are stored in the detection result storage unit 122. These detection results are used as additional teacher data 122a. The additional teacher data 122a is a copy of the additional teacher data 122a into the teacher data storage unit 124, which is the additional teacher data 124b. The re-learning of the first trained model and the second trained model is performed using the original teacher data 124a and the additional teacher data 124b stored in the teacher data storage unit 124.

The first trained model and the second trained model may be models constructed by machine learning. In this embodiment, an example will be described in which the first trained model and the second trained model are trained models of a neural network constructed by deep learning. More specifically, these trained models take an image as input data and output object information of a detection target appearing in the image. The object information may include information indicating an identifier indicating the classification of the object, a position, a size, a shape, and the like. Further, the object information may include a probability value indicating the certainty of the detection result. This probability value may be, for example, a numerical value of 0 to 1.

Further, the first detection unit 101 and the second detection unit 102 may perform object detection based on the above probability values. In this case, the detection threshold value may be set in advance, and an object having the above probability value larger than the threshold value may be detected. The larger the detection threshold, the higher the detection accuracy, but the more missed, and the smaller the detection threshold, the less missed, but the more false detections, so set an appropriate detection threshold according to the required detection accuracy, etc. do it. The construction of the trained model will be described later based on FIG.

The learning unit 103 relearns the first trained model and the second trained model. Further, the learning unit 103 may also construct the first trained model and the second trained model. The construction and re-learning of the trained model will be described later based on FIG. The learning unit 103 for the first trained model and the learning unit 103 for the second trained model may be provided separately.

The selection display control unit 104 displays the detection result (for example, an image determined to show an unsuitable object) determined based on the detection results of the first detection unit 101 and the second detection unit 102 on the selection display device 4. Let me. The person in charge of visual confirmation confirms whether or not an unsuitable object is reflected in the displayed image, and selects an image in which the unsuitable object is reflected. Then, the selection display control unit 104 accepts the selection of the image by the person in charge of visual confirmation. As a result, erroneous detection can be almost certainly avoided.

The carry-in vehicle identification unit 105 identifies the garbage carry-in vehicle (for example, the garbage collection vehicle 200 in FIG. 3) by using the identification information received from the vehicle information collection device 3. Then, when the information processing device 1 detects an unsuitable object from the garbage carried in by the carry-in vehicle specified by the carry-in vehicle identification unit 105, the unsuitable object display control unit 106 displays the image of the unsuitable object as the unsuitable object display device. Display on 5. As a result, it is possible to present an image of an unsuitable object to the person in charge of carrying in the garbage in the carry-in vehicle to alert the person in charge.

As described above, the information processing apparatus 1 inputs input data into the first trained model machine-learned so as to be able to detect a plurality of types of first detection targets, and detects the first detection target. The input data is input to the first detection unit 101 and the second trained model machine-learned so that the second detection target, which is at least a part of the first detection target of the plurality of types, can be detected. It includes a second detection unit 102 that detects the second detection target. Then, the information processing device 1 determines the final detection result based on the detection result of the first detection unit 101 and the detection result of the second detection unit 102. Specifically, in the information processing device 1, since the first detection unit 101 and the second detection unit 102 output the detection result to the common output destination, the detection result output to the common output destination is finally output. Use as the detection result.

According to the above configuration, since at least a part of the detection target of the first trained model and the second trained model overlaps, the possibility of erroneous detection of the overlapped part can be reduced. .. For example, a board that is unsuitable and a corrugated cardboard that is not unsuitable may have similar appearances. It is possible to do. According to the above configuration, one of the first trained model and the second trained model should be able to correctly detect the object on the other side even if a false detection such as the above-mentioned board and cardboard occurs. Then, in the end, the correct detection result of the object can be output. Therefore, it is possible to improve the detection accuracy of the detection using the machine-learned model.

Further, according to the above configuration, since the first detection unit 101 uses the first trained model machine-learned so that a plurality of types of first detection targets can be detected, a plurality of types of first detections are performed. Targets can be detected collectively and efficiently.

The method of determining the final detection result is not limited to the method of sharing the output destinations of the detection results of the first detection unit 101 and the second detection unit 102. For example, a block for integrating the detection results may be added to the control unit 10, and the detection results of the first detection unit 101 and the second detection unit 102 may be integrated by this block as the final detection result. Examples of the integration method include the following methods.

(1) The detection result of the second detection unit 102 is added to the detection result of the first detection unit 101 to obtain the final detection result (for example, the first detection unit 101 detects unsuitable objects A and B from a certain image, and then obtains the final detection result. When the second detection unit 102 detects unsuitable objects B and C from the same image, the final detection result from the image is defined as unsuitable objects A, B and C, etc.).

(2) The intersection of the detection result of the first detection unit 101 and the detection result of the second detection unit 102 is set as the final detection result (for example, from a certain image, the first detection unit 101 sets unsuitable objects A, B, and C. When the second detection unit 102 detects the unsuitable object B from the same image, the final detection result from the image is regarded as the unsuitable object B, etc.).

(3) The detection result of the second detection unit 102 is added to the detection result of the first detection unit 101, but the portion where both detection results do not match is excluded from the final detection result (for example, an image showing the object X and the object Y). The first detection unit 101 detects that the object X is an unsuitable object A and the object Y is an unsuitable object B, and the second detection unit 102 detects that the object X is an unsuitable object A and the object Y is an unsuitable object C. If this is the case, the final detection result will be the unsuitable object A, etc.).

[Processing flow]
FIG. 5 is a diagram illustrating a flow of processing executed by the information processing apparatus 1 of the present embodiment. The processing executed by the information processing apparatus 1 of the present embodiment and the execution order thereof can be defined by, for example, the setting file F1 shown in FIG. 5, and can also be represented by the flowchart of the figure.

(About the configuration file)
The setting file F1 shown in FIG. 5 is a data structure divided into sections. One section starts with the section name. In the example of FIG. 5, the character string surrounded by "[" and "]" is the section name, and specifically, [EX1] and [EX2] are the section names. One section ends with the start of the next section or the end of the configuration file. Define what to do at each stage in one section.

In this example, the execution of sections is in the section order, but the execution order is not limited to this example, and the execution order may be defined as part of the section name, for example. Also, different algorithms may be executed for each section. For example, an algorithm that performs processing using a neural network in which the number of intermediate layers is different for each section may be executed, or an algorithm that performs different internal processing may be executed. It should be noted that each of the above definitions may be performed by other means (for example, an argument when executing the flowchart of FIG. 5) instead of the setting file.

In the setting file F1, the definition is made for each variable <KEY> as follows. The value of <KEY> is <VALUE>.

<KEY> = <VALUE>
In <KEY> of this example, three "script", "src", and "dst" are defined for each section. Of these, "script" indicates the file name of the script to be executed. The script file is, for example, an algorithm for detecting an unsuitable object, and this algorithm may include processing other than the unsuitable object detection.

Further, "src" indicates input data used for executing the script. For example, in the case of a script that detects an object from an image, "src" may indicate a storage location of an image to be processed in the large-capacity storage unit 12 or a list of images to be processed. Then, "dst" indicates the output destination of the result of script execution. For example, "dst" may indicate a location on the large-capacity storage unit 12. Other parameters and the like used at the time of execution may be defined in, for example, "script".

The control unit 10 reads the setting file F1 and executes the script file whose file name is "ex1" defined in [EX1] of the setting file F1, so that the control unit 10 functions as the first detection unit 101. To do. Then, the first detection unit 101 identifies an image to be processed with reference to "ex_src", detects an object from the identified image, and records the result in "ex_dst". Next, the control unit 10 functions as the second detection unit 102 by executing the script file whose file name is "ex2" defined in [EX2]. Then, the second detection unit 102 identifies an image to be processed with reference to "ex_src", detects an object from the identified image, and records the result in "ex_dst".

“Ex1” in [EX1] of the setting file F1 is a script for detecting an object from an image which is input data by the above-mentioned first trained model. Further, "ex2" in [EX2] is a script for detecting an object from an image which is input data by the above-mentioned second trained model.

Further, [EX1] and [EX2] in the setting file F1 have the same values of "src" and "dst". That is, the first detection unit 101 and the second detection unit 102 have a common image to be processed, and also have a common output destination of the object detection result from the image.

As [EX1] in the setting file F1, for example, the script SC11 (script file name: ex1) shown in FIG. 5 may be applied. This script file is an example of a UNIX (registered trademark) format shell script including Linux (registered trademark), but may be in another format such as DOS (Disk Operating System) Batch format.

When this script file is used, as shown in FIG. 5, (2) the script is executed based on (1) the setting file items. Here, ex1 is the name of the script file to be executed as described above, and ex_src and ex_dst are passed to the script as arguments. The ex1 script (SC11) is composed of one line, and here the executable file (first detection command) of the first detection unit 101 is executed. In this example, this command uses three arguments. The argument ex_src of ex1 is passed to the first argument $ 1, the argument ex_dst of ex1 is passed to the second argument $ 2, and the third argument is the file name of the trained model (first trained model). .. As the first trained model, for example, a trained model in which all detection objects are machine-learned is used. The detection target may be, for example, corrugated cardboard, board, wood, goza, and a long object. Of these detection objects, boards, wood, rugs, and long objects are unsuitable. Corrugated cardboard is not unsuitable, but it is included in the detection target because it is similar in appearance to the board. In this way, by including dust having a similar appearance to the unsuitable object in the detection target object, it is possible to reduce the possibility of omission of detection or erroneous detection of the unsuitable substance.

Although the executable file of this example uses three arguments, for example, other setting items (threshold value to be detected, etc.) may be used as arguments if necessary. Further, the order of the arguments may be changed as needed. Although the SC 11 is composed of only one line, for example, pre-processing and post-processing commands (for example, pre-processing and post-processing commands) by the first detection unit 101 may be added as needed. For example, a command for organizing input data, a command for extracting necessary information from log data after execution, and the like may be added.

When the script SC11 is used, for example, a moving image file (hereinafter referred to as a moving image file) taken by the garbage photographing apparatus 2 or a plurality of still image files may be specified by src = ex_src. As a result, the detection result from the moving image file (for example, the image in which the object is detected and the position and size information of the object) is saved in "ex_dst".

Script SC12 may be applied as [EX2] in the setting file F1. The difference between SC11 and SC12 is the name of the executable file to be executed and the trained model to be used. Further, [EX2] may be set as the detection target of [EX1], in which the detection omission is particularly desired to be avoided. For example, when it is desired to avoid omission of detection of a tree, [EX2] may set the tree as a detection target. As a result, even if a tree is missed by [EX1], if the tree can be detected by [EX2], the tree will not be missed as a whole. Further, when [EX1] is to detect a part of all the detection objects, [EX2] may be to detect another part of all the detection objects. Also in this case, at least one of the detection target of [EX1] and the detection target of [EX2] are duplicated. Although different executable files are used for SC11 and SC12, the first detection command and the second detection command may be the same when only the trained model is different. That is, the first detection unit 101 and the second detection unit 102 may be the same.

(About the flowchart)
The processing (information processing method) of the flowchart shown in FIG. 5 will be described. This flowchart shows the flow of processing according to the setting file F1 shown in the figure. It is assumed that the moving image file photographed by the garbage photographing apparatus 2 is stored in the "ex_src" of the input data storage unit 121 before the processing of this flowchart is performed. Instead of the moving image file, a plurality of frame images extracted from the moving image file or a plurality of still image files taken in time series by the garbage photographing apparatus 2 may be stored.

In S11 (first detection step), the first detection unit 101 detects an object from the image stored in the input data storage unit 121 by the first learned model. Specifically, the first detection unit 101 uses a frame image extracted from a moving image file stored in "ex_src" of the input data storage unit 121 as input data, and the frame image has been trained with the script name "ex1". Input to the model and output the object information. Then, the first detection unit 101 determines whether or not the object is detected based on the object information, and if the object is detected, associates the frame image with the object information to obtain a detection result and detects the object. Record in "ex_dst" of the result storage unit 122. These processes are performed for each of the frame images extracted from the moving image file stored in "ex_src".

In S12 (second detection step, confirmation step), the second detection unit 102 detects an object from the same frame image as S11 by the second learned model. In this way, by setting the input data to be input to the first trained model and the second trained model to be the same data, it is possible to suppress the occurrence of detection omission. This is because even if the detection by one of the trained models causes an omission of detection of an unsuitable object, if the unsuitable object can be detected by the detection by the other, the detection omission does not occur as a whole.

In S12, the second detection unit 102 specifically uses the frame image extracted from the moving image file stored in the “ex_src” of the input data storage unit 121 as the input data, and uses the frame image as the input data with the script name “ex2”. Is input to the trained model of "" and the object information is output. Then, the second detection unit 102 determines whether or not the object is detected based on the object information, and if the object is detected, associates the frame image with the object information and obtains the detection result. The second detection unit 102 records this detection result in "ex_dst" of the detection result storage unit 122, which is an output destination common to the detection result of the first detection unit 101. These processes are performed for each of the frame images extracted from the moving image file stored in "ex_src". The data recorded in "ex_dst" of the detection result storage unit 122 at the end of the processing of S12 is the final detection result. That is, the final detection result is determined by the processing of S12.

In S13, the detection result is output, and the process ends. For example, the selection display control unit 104 may output the detection result. In this case, the selection display control unit 104 may display the frame image and the object information recorded in "ex_dst" by the processing of S11 and S12 on the selection display device 4. As a result, the user of the selection display control unit 104 can visually confirm whether or not the detection result of the information processing device 1 is correct, and input the confirmation result to the information processing device 1. In addition, the selection display control unit 104 can identify an image in which it is confirmed that an unsuitable object is captured according to the input confirmation result. Further, the selection display control unit 104 may correct the object information based on the visual confirmation result. Then, the image confirmed by visual inspection that the unsuitable object is reflected is displayed on the unsuitable object display device 5 by the unsuitable object display control unit 106, for example, when the person who carries in the unsuitable object brings in the garbage again. Is displayed.

The execution order of the processing of S11 and the processing of S12 is not limited to the example of FIG. 5, and the processing of S11 may be performed after the processing of S12, or these processes may be performed in parallel. In any case, the final detection result is determined when both the processes of S11 and S12 are completed. Further, in the example of FIG. 5, two trained models are used, but three or more trained models may be used. In this case, a section corresponding to the third and subsequent trained models may be added to the configuration file F1.

[About the resolution of input data]
When the input data for the trained model is image data as in the present embodiment, the first detection unit 101 may reduce the resolution of the image data and input it to the first trained model. Alternatively, the second detection unit 102 may reduce the resolution of the image data and input it to the second trained model. By lowering the resolution of the input image data, it is possible to reduce the amount of calculation of the object detection process using the trained model and shorten the required time.

It should be noted that which trained model to reduce the resolution of the input data may be determined according to the detection target of each trained model. For example, it is easy to detect a large object such as a tree or a board even with low resolution image data, but it is preferable to use high resolution image data to detect a small object such as a can. Therefore, among the trained models to be used, for a model having a large detection target size, a low-resolution captured garbage image may be used as input data. As a result, it is possible to speed up the detection process for a large-sized object without degrading the detection accuracy. In the trained model that uses low-resolution image data as input data, the same low-resolution image data as the input data is constructed as teacher data. Further, after the object is detected by using the low resolution image data as the input data, the resolution of the image data may be restored and output. The process of changing the resolution may be performed by the first detection unit 101 or the second detection unit 102, or a block for changing the resolution may be separately added to the control unit 10.

Further, each trained model may have a different number of intermediate layers depending on the detection target. For example, the number of intermediate layers of the trained model used to detect large objects may be less than the number of intermediate layers of the trained model used to detect smaller objects. Even in such a configuration, as in the above example, it is possible to speed up the detection process for a large-sized object without degrading the detection accuracy.

[Building and retraining trained models]
The construction and retraining of the first trained model and the second trained model described above will be described with reference to FIG. FIG. 6 is a diagram illustrating the construction and re-learning of the trained model. Here, an example of constructing a trained model of a neural network will be described. When using a neural network, there may be a plurality of intermediate layers, and machine learning in this case is deep learning. Of course, the number of intermediate layers may be one, and machine learning algorithms other than neural networks can be applied.

As shown, the trained model is built by initial training. Then, object detection is performed using the trained model constructed by the initial learning, re-learning is performed using the object detection result, and the trained model is updated.

In the initial learning, the image showing the detection target is used as the teacher image, and the object information of the detection target shown in the teacher image (for example, the identifier indicating the classification of the object, the position, the size, the shape, etc.) is shown. ) Is the correct answer data, and the teacher data is used. It is assumed that the teacher data is stored in the teacher data storage unit 124 of FIG. 1, but the teacher data used in the initial learning is only the original teacher data 124a. In machine learning, the learning unit 103 repeatedly inputs the teacher image to the neural network and updates the weight value so that the output value of the neural network approaches the correct answer data while changing the teacher image.

In machine learning, basically, as the number of repetitions increases, the weight value approaches the optimum value, but due to factors such as overfitting, the weight value may deviate from the optimum value after the repetition. In addition, when constructing a trained model that detects a plurality of detection objects, the detection accuracy of a certain detection target is high when a certain weight value is applied, but the detection accuracy of another detection target is low. possible.

Therefore, in the example of FIG. 6, a plurality of trained models 1 to I having different weight values are generated. Then, from these trained models, the one to be applied as the above-mentioned first trained model and the one to be applied as the above-mentioned second trained model are selected. For example, an image showing a detection target is input to each of the trained models 1 to I as test data, the detection accuracy of each detection target is calculated from the output value, and the calculated detection accuracy is used as a reference. The above selection may be made. These trained models are stored in the trained model storage unit 123.

Further, in this selection, it is preferable not to generate a detection object having low detection accuracy in both the first trained model and the second trained model. For example, when the first trained model has a low detection accuracy of a long object, it is preferable to use a model having a high detection accuracy of a long object as the second trained model. It is not necessary to select both the first trained model and the second trained model from the trained models 1 to I. For example, when the first trained model is selected from the trained models 1 to I, the second trained model may be selected from a plurality of trained models constructed separately.

Further, the trained model may be manually selected or may be selected by the information processing apparatus 1. In the latter case, the selection criteria of the trained model are set in advance, and the information necessary for determining whether or not the selection criteria are satisfied (for example, information indicating the detection accuracy of each detection target in each trained model) is provided. It may be input to the processing device 1 or calculated.

By performing object detection from the detection target image using the first trained model and the second trained model selected in this way, as described with reference to FIG. 5, the image in which the object is detected and the image Object information is recorded in the detection result storage unit 122. The learning unit 103 uses this image as a teacher image, and relearns using the teacher data to which the object information of this image is added as correct answer data and the original teacher data 124a used for the initial learning. It is preferable that the image and the object information used as the teacher data are confirmed to be correct by visual inspection through the selection display device 4. The teacher data selected here may be the additional teacher data 122a, and may be copied as the additional teacher data 124b to the teacher data storage unit 124 for re-learning. Although the additional teacher data 124b is a copy of the additional teacher data 122a here, 124b may be the same as 122a without copying.

The image and object information are recorded in the detection result storage unit 122, but when the input data is a still image, it is not necessary to save the image, and only the image file name of the input data may be recorded.

In the re-learning, the learning unit 103 learns using the original teacher data 124a and the additional teacher data 124b stored in the teacher data storage unit 124, and relearns a plurality of relearned models 1 to J having different weight values. To construct. Then, the first trained model and the second trained model are selected from them. The difference from the initial learning is that additional teacher data 124b is added to the teacher data used for machine learning. By adding teacher data, it is expected that the detection accuracy of the trained model will be improved. In the re-learning, the learning unit 103 does not need to use all of the object information recorded as a result of the object detection, and may select and use a part of the object information. Further, the learning unit 103 may repeat the re-learning a plurality of times. Further, each trained model of the second and subsequent embodiments can be constructed in the same manner as described above, and can be retrained.

[Embodiment 2]
Other embodiments of the present invention will be described below. For convenience of explanation, the same reference numerals will be added to the members having the same functions as the members described in the above embodiment, and the description will not be repeated. This also applies to the third embodiment described later.

[Configuration example of control unit]
A configuration example of the control unit 10 of the information processing device 1 of the present embodiment will be described with reference to FIG. FIG. 7 is a block diagram showing a configuration example of the control unit 10 included in the information processing device 1 according to the second embodiment. Further, in FIG. 7, the large-capacity storage unit 12 is also shown.

As shown in FIG. 7, the control unit 10 includes a first detection unit 201, a second detection unit 202A, and a second detection unit 202B. Since the learning unit 103, the selection display control unit 104, the carry-in vehicle identification unit 105, and the unsuitable object display control unit 106 are the same as those in the first embodiment, they are not shown.

The first detection unit 201 inputs input data into the first trained model machine-learned so that a plurality of types of first detection targets can be detected, and detects the first detection target. The first detection unit 201 has the same function as the first detection unit 101 of the first embodiment, but the second detection unit 202A and the second detection unit 202B are based on the detection result of the first detection unit 201. It differs from the first detection unit 101 in that the input data to be used is determined.

The second detection unit 202A inputs the input data (first) to the second trained model A machine-learned so as to be able to detect the second detection target A, which is at least a part of the first detection target of a plurality of types. The detection result of the detection unit 201) is input to detect the second detection target A. For the second detection target A, the detection result of the second detection unit 202A is the final detection result. The second detection unit 202A uses, as the input data for the second trained model A, the input data for which the first detection unit 201 has detected the detection target among the input data for the first detection unit 201. It is different from the second detection unit 102 of 1.

Similar to the second detection unit 202A, the second detection unit 202B inputs the input data in which the detection target is detected by the first detection unit 201 into the second learned model B, and the input data (first detection unit 202B). From the detection result of 201), the second detection target B, which is at least a part of the first detection target of a plurality of types, is detected. For the second detection target B, the detection result of the second detection unit 202B is the final detection result. The second detection target A and the second detection target B are different objects.

As described above, the information processing apparatus 1 inputs input data into the first trained model machine-learned so as to be able to detect a plurality of types of first detection targets, and detects the first detection target. The input data is input to the first detection unit 201 and the second trained model machine-learned so that the second detection target, which is at least a part of the first detection target of the plurality of types, can be detected. It includes a second detection unit 202A that detects the second detection target. Then, the information processing device 1 determines the final detection result based on the detection result of the first detection unit 201 and the detection result of the second detection unit 202A. Specifically, in the information processing device 1 of the present embodiment, the second detection unit 202A secondly uses the input data in which the first detection target is detected by the first detection unit 201 among the plurality of input data. The second detection target A is detected by inputting to the trained model of the above, and the detection result of the second detection unit 202A is set as the final detection result for the second detection target A. Further, the second detection unit 202B inputs the input data in which the first detection target is detected by the first detection unit 201 out of the plurality of input data to the second trained model B to perform the second detection. The target B is detected, and the detection result of the second detection unit 202B is set as the final detection result for the second detection target B.

According to the above configuration, since at least a part of the detection target of the first trained model and the second trained model A overlaps, the possibility of erroneous detection of the overlapped part can be reduced. it can. Similarly, since at least a part of the detection target of the first trained model and the second trained model B overlaps, the possibility of erroneous detection of the overlapped part can be reduced. Therefore, it is possible to improve the detection accuracy of the detection using the machine-learned model. Further, according to the above configuration, since the first detection unit 201 uses the first trained model machine-learned so that a plurality of types of first detection targets can be detected, a plurality of types of first detections are performed. Targets can be detected collectively and efficiently.

The method for determining the final detection result is not limited to the above method. For example, as described in the first embodiment, a block for integrating the detection results is added to the control unit 10, and the detection results of the first detection unit 201 and the second detection unit 202A are integrated by this block, and finally. It may be the detection result of. The same applies to the integration of the detection results of the first detection unit 201 and the second detection unit 202B. Further, as in the example described in "Resolution of input data" of the first embodiment, either the input data of the first trained model or the second trained model A and the second trained model B. Alternatively, image data having a reduced resolution may be used as both input data.

In the example of FIG. 7, two blocks, a second detection unit 202A and a second detection unit 202B, are described as blocks for detecting a part of the detection target object of the first detection unit 201. However, the first detection unit 201 may have only one block for detecting a part of the detection target object, or may have three or more blocks.

[Processing flow]
FIG. 8 is a diagram illustrating a flow of processing executed by the information processing apparatus 1 of the present embodiment. The processing executed by the information processing apparatus 1 of the present embodiment and the execution order thereof can be defined by, for example, the setting file F2 shown in FIG. 8 and can also be represented by the flowchart of the figure.

(About the configuration file)
In the configuration file F2, three sections [EX_all], [EX_goza], and [EX_tree] are defined in this order. The details of each script are omitted, but they are the same as those of SC11 and SC12. The trained model used in [EX_all] is the same as SC11, and is a section for detecting all detection objects (for example, cardboard, board, wood, goza, and long objects). [EX_goza] is a section of the detection target that is specialized for detecting goza, and [EX_tree] is a section of the detection target that is specialized for the detection of trees.

The src of [EX_goza] and [EX_tree] are both "all_res", which is the dst of [EX_all]. That is, [EX_goza] is an image in which at least one of the detection objects is detected by [EX_all], and [EX_tree] is an image in which at least one of the detection objects is detected by [EX_all]. Detect trees from. Then, the detection result of the goza by [EX_goza] is output to "goza_res", and the detection result of the tree by [EX_tree] is output to "tree_res". These detection results are the final detection results.

(About the flowchart)
The control unit 10 functions as the first detection unit 201 by executing the script file whose file name is "all" defined in [EX_all]. Further, the control unit 10 functions as the second detection unit 202A by executing the script file whose file name is "goza" defined in [EX_goza] after the execution of the script file is completed. Then, the control unit 10 functions as the second detection unit 202B by executing the script file whose file name is "tree" defined in [EX_tree] after the execution of the script file is completed.

Hereinafter, the processing (information processing method) executed by these processing units will be described based on the flowchart. It is assumed that the moving image file photographed by the garbage photographing apparatus 2 is stored in "all_src" of the input data storage unit 121 before the processing of this flowchart is performed. Instead of the moving image file, a plurality of frame images extracted from the moving image file or a plurality of still image files taken in time series by the garbage photographing apparatus 2 may be stored.

In S21 (first detection step), the first detection unit 201 uses the first trained model to detect objects for all the detection targets from the processing target image stored in the input data storage unit 121. .. Specifically, the first detection unit 201 uses a frame image extracted from a moving image file stored in "all_src" of the input data storage unit 121 as input data, and the frame image has been trained with the script name "all". Input to the model and output the object information. Then, when an object is detected from the frame image, the first detection unit 201 associates the frame image with the object information to obtain a detection result, and records the object in "all_res" of the detection result storage unit 122. These processes are performed for each of the frame images extracted from the moving image file.

As described above, the frame image recorded in "all_res" becomes the input data of [EX_goza] and [EX_tree], and the detection of the goza and the tree is attempted again. For this reason, it is preferable that the first detection unit 201 avoids overlooking the rough wood, that is, not being able to detect the rough wood in the frame image, even if the false detection of the rough wood increases. Therefore, in the object detection based on the output value of the first trained model, the detection threshold value to be compared with the probability value included in the output value may be set lower.

In S22 (second detection step, confirmation step), the second detection unit 202A is the second detection target A in this example by the second learned model A from the image in which the object is detected in S21. Perform detection. Specifically, the second detection unit 202A uses the frame image recorded in "all_res" of the detection result storage unit 122 as input data, and inputs the frame image into the trained model of the script name "goza". Output object information. Then, when the second detection unit 202A determines that the goza has been detected, the frame image and the object information are associated with each other to obtain a detection result, which is recorded in "goza_res" of the detection result storage unit 122. These processes are performed for each of the frame images stored in "all_res". The data recorded in "goza_res" of the detection result storage unit 122 at the end of the processing of S22 is the final detection result of the goza. That is, the final detection result of the goza is determined by the process of S22.

In S23, the second detection unit 202B detects the tree, which is the second detection target B in this example, by the second learned model B from the image in which the object is detected in S21. Specifically, the second detection unit 202B uses the frame image recorded in "all_res" of the detection result storage unit 122 as input data, and inputs the frame image into the trained model of the script name "tree". Output object information. Then, when the second detection unit 202B determines that the tree has been detected, the frame image and the object information are associated with each other to obtain a detection result, and the tree is recorded in the "tree_res" of the detection result storage unit 122. These processes are performed for each of the frame images stored in "all_res". The data recorded in the “tree_res” of the detection result storage unit 122 at the end of the processing of S23 is the final detection result for the tree. That is, the final detection result of the tree is determined by the processing of S23.

In S24, the detection result is output in the same manner as in S13 of FIG. 5, and the process ends. The detection result of the goza may be read from "goza_res" of the detection result storage unit 122, and the detection result of the tree may be read from "tree_res" of the detection result storage unit 122. Further, the detection result of the detection target object other than the goza and the tree may be read from "all_res" of the detection result storage unit 122.

The detection result of [EX_all] may include erroneous detection in which a non-tree is detected as a tree or a non-tree is detected as a goza. However, according to the above processing, the frame image in which some object is detected by [EX_all] is used for object detection by [EX_tree] and [EX_goza], so that it is possible to detect the rough wood with high accuracy. it can.

Further, according to the above configuration, the processing speed may be increased as compared with the case where the object is detected by using both [EX_tree] and [EX_goza]. For example, when 200 frame images are extracted from one video file, if two of [EX_tree] and [EX_goza] are used, 200 frame images are processed by each of [EX_tree] and [EX_goza]. .. In this case, the object detection process is performed 400 times in total. On the other hand, according to the above configuration, first, each of the 200 frame images is processed by [EX_all]. Here, assuming that an object is detected in 30 frame images, the number of frame images processed by each of [EX_tree] and [EX_goza] is 30, and the object detection process is performed 260 times in total. Therefore, the number of times the object detection process is executed can be greatly reduced, and the time required for the process can be significantly reduced.

[Embodiment 3]
A configuration example of the control unit 10 of the information processing device 1 of the present embodiment will be described with reference to FIG. FIG. 9 is a block diagram showing a configuration example of the control unit 10 included in the information processing device 1 according to the third embodiment. Further, in FIG. 9, the large-capacity storage unit 12 is also shown.

As shown in FIG. 9, the control unit 10 includes a dust image extraction unit 301, a first detection unit 302, a second detection unit 303, and a detection result integration unit 304. Since the learning unit 103, the selection display control unit 104, the carry-in vehicle identification unit 105, and the unsuitable object display control unit 106 are the same as those in the first embodiment, they are not shown.

The dust image extraction unit 301 extracts an image showing dust from an image to be detected as an object (for example, each frame image extracted from a moving image file). As a result, the images to be detected by the first detection unit 302 and the second detection unit 303 can be narrowed down to the images in which dust is reflected, so that the number of executions of the object detection process can be reduced and the process can be speeded up. It becomes possible to change. For example, when dust is not reflected in 3/4 of the frame images extracted from the moving image file, the first detection unit 302 and the second detection unit 303 are used to capture the dust in the frame image (of all frame images). The object detection process may be performed for 1/4). Therefore, as compared with the case where the object detection process is performed for all the frame images, the number of times the object detection process is executed can be greatly reduced, and the time required for the process can be greatly reduced.

The dust image extraction unit 301 may perform the above extraction using, for example, a trained model constructed by using an image in which dust is flowing on the slope 600 and an image in which dust is not flowing as teacher data. This trained model only needs to be able to identify the presence or absence of dust on the slope 600, and it is not necessary to discriminate the classification of dust. Therefore, when this trained model is used as a model of a neural network, the number of intermediate layers may be smaller than that of the trained model used by the first detection unit 302 and the second detection unit 303, which will be described later.

Further, the image used as the input data of the trained model used by the garbage image extraction unit 301 has a lower resolution than the image used as the input data of the trained model used by the first detection unit 302 and the second detection unit 303. It may be an image. It is preferable to input a higher resolution image to the first detection unit 302 and the second detection unit 303. Therefore, when the waste image is reduced in resolution and used as the input data of the waste image extraction unit 301, it is preferable that the waste image extraction unit 301 saves the image with the resolution before the reduction in resolution as the output result.

The dust image extraction unit 301 is not an essential component of the information processing device 1, but is included in order to improve the efficiency of object detection by the first detection unit 302 and the second detection unit 303. The waste image extraction unit 301 can also be applied to the information processing device 1 of the first and second embodiments.

The first detection unit 302 inputs input data into the first trained model that has been machine-learned so that a plurality of types of first detection targets can be detected, and detects the first detection target. Further, the second detection unit 303 inputs the input data to the third trained model machine-learned so as to detect a third detection target different from the first detection target, and the third detection unit 303 inputs the input data to the third trained model. Detect the detection target.

Also in the information processing device 1 of the present embodiment, the final detection result is determined based on the detection result of the first detection unit 302 and the detection result of the second detection unit 303, as in each of the above-described embodiments. To. Specifically, the detection result integration unit 304 determines the final detection result based on the detection result of the first detection unit 302 and the detection result of the second detection unit 303.

More specifically, the detection result integration unit 304 excludes the remainder of the detection target detected by the first detection unit 302 as the first detection target, excluding the detection target detected by the second detection unit 303 as the third detection target. , The detection result of the first detection target. In other words, the detection result integration unit 304 determines the detection result based on the first trained model when the detection result based on the first trained model and the detection result based on the third trained model do not match. Is invalid.

According to the above configuration, the detection targets of the first trained model and the third trained model are different. Therefore, when the first detection unit 302 detects a certain detection target as the first detection target, the second detection unit 303 may detect the same detection target as the third detection target. In such a case, it can be determined that either the first detection unit 302 or the second detection unit 303 has erroneously detected. Therefore, the remainder of the detection target detected by the first detection unit 302 as the first detection target, excluding the detection target detected by the second detection unit 303 as the third detection target, is the detection result of the first detection target. According to the above configuration, erroneous detection can be reduced. Therefore, according to the above configuration, it is possible to improve the detection accuracy of the detection using the machine-learned model. Further, since the first detection unit 302 uses the first trained model machine-learned so that a plurality of types of first detection targets can be detected, the plurality of types of first detection targets can be efficiently collectively detected. Can be detected.

Also in this embodiment, the resolution is set as the image data to be input to the first trained model or the third trained model in the same manner as in the example described in "About the resolution of the input data" of the first embodiment. The reduced image data may be used.

[Processing flow]
FIG. 10 is a diagram illustrating a flow of processing executed by the information processing apparatus 1 of the present embodiment. The processing executed by the information processing apparatus 1 of the present embodiment and the execution order thereof can be defined by, for example, the setting file F3 shown in FIG. 10, and can also be represented by the flowchart of the figure.

(About the configuration file)
In the configuration file F3, four sections [EX_trash], [EX_all], [EX_bag], and [EX_final] are defined in this order. [EX_all] is the same as the section [EX1] of FIG. 5, and is a section for detecting all detection objects (for example, corrugated cardboard, board, wood, goza, and long objects). [EX_trash] is a section for extracting an image showing dust from an image showing dust and an image not showing dust. Further, [EX_bag] is a section for detecting the garbage bag and the slope 600 (see FIG. 4), and [EX_final] is a section for outputting the result obtained by removing the result of [EX_bag] from the result of [EX_all]. is there.

The dst of [EX_trash] is "trash_res", and "trash_res" is the src of [EX_all]. That is, the detection target is detected by [EX_all] from the image extracted by [EX_trash].

Further, the dst of [EX_all] is "all_res", and "all_res" is the src of [EX_bag]. That is, the garbage bag and the slope 600 are detected by [EX_bag] from the image in which at least one of the detection objects is detected by [EX_all].

Then, the src of [EX_final] is "all_res", and the dst is "final_res". Since the detection result of [EX_all] and the detection result of [EX_bag] are recorded in "all_res", [EX_final] outputs the final detection result based on these detection results to "final_res".

Here, the reason why [EX_all], [EX_bag], and [EX_final] are used in combination in the present embodiment will be described. There are innumerable shapes and colors of garbage bags containing garbage, and therefore the shape of the part between the garbage bags shown in the garbage image is also diverse. For example, if the part between the garbage bag and the garbage bag looks like a long stick, it is erroneously detected that the unsuitable object is reflected based on the part where the unsuitable object does not actually exist. There is. In order to avoid such false detection, it is preferable to learn the garbage bags as well, but the number of garbage bags reflected in the garbage image may be, for example, several tens in one image, which is very large. Therefore, it is very troublesome to create correct answer data for all teacher images. The same applies to the slope 600, and the appearance of the slope 600 shown in the image changes depending on the condition of dust on the slope 600. Therefore, it is very troublesome to create correct answer data of the slope for all the teacher images.

Therefore, in the present embodiment, apart from [EX_all] that detects unsuitable substances and the like, [EX_bag] specialized for detecting the garbage bag and the slope 600 is used. The third trained model used in [EX_bag] can be constructed by machine learning using the teacher data of only the garbage bag and the slope 600.

[EX_final] is a section for confirming whether [EX_bag] has detected a garbage bag or a slope 600 at a place where an unsuitable substance is detected by [EX_all]. Specifically, [EX_final] outputs the detection result of the unsuitable material to "final_res" when neither the garbage bag nor the slope 600 is detected at the place where the unsuitable material is detected by [EX_all]. This is the section to do. As a result, among the detection results of unsuitable substances by [EX_all], the detection results excluding those that may have erroneously detected the garbage bag or slope 600 as unsuitable objects from the detection results of [EX_bag] are promptly selected. be able to. That is, it is possible to efficiently reduce the false detection of the garbage bag or the slope 600 as an unsuitable object in a short time process.

(About the flowchart)
The control unit 10 functions as the garbage image extraction unit 301 by executing the script file whose file name is "trash" defined in [EX_thrash]. Further, the control unit 10 functions as the first detection unit 302 by executing the script file whose file name is "all" defined in [EX_all] after the execution of the script file is completed. Then, the control unit 10 functions as the second detection unit 303 by executing the script file whose file name is "bag" defined in [EX_bag] after the execution of the script file is completed. Further, the control unit 10 functions as the detection result integration unit 304 by executing the script file whose file name is "final" defined in [EX_final] after the execution of the script file is completed.

Hereinafter, the processing (information processing method) executed by these processing units will be described based on the flowchart. It is assumed that the moving image file photographed by the garbage photographing apparatus 2 is stored in "all_src" of the input data storage unit 121 before the processing of this flowchart is performed. Instead of the moving image file, a plurality of frame images extracted from the moving image file or a plurality of still image files taken in time series by the garbage photographing apparatus 2 may be stored.

In S31, the dust image extraction unit 301 extracts an image in which dust is captured from the image to be processed. Specifically, the garbage image extraction unit 301 inputs the frame image extracted from the moving image file stored in the “all_src” of the input data storage unit 121 into the trained model of the script name “trash” to input the object information. Output. Then, the dust image extraction unit 301 records a frame image determined that dust has been detected based on the object information in the “trash_res” of the detection result storage unit 122. These processes are performed for each of the frame images extracted from the moving image file.

In S32 (first detection step), the first detection unit 302 detects an object for all the detection objects from the frame image extracted in S31. Specifically, the first detection unit 302 uses the frame image stored in "thrash_res" of the detection result storage unit 122 as input data, and inputs the frame image into the trained model of the script name "all". Output object information. Then, the first detection unit 302 associates the frame image determined that the object has been detected based on the object information with the object information to obtain a detection result, and records it in "all_res" of the detection result storage unit 122. These processes are performed for each of the frame images stored in "trash_res".

In S33 (second detection step), the second detection unit 303 detects the garbage bag and the slope 600 from the image extracted in S31 by the third learned model. Specifically, the second detection unit 303 uses the frame image recorded in "trash_res" of the detection result storage unit 122 as input data, and inputs the frame image into the trained model of the script name "bag". Output object information. Then, the second detection unit 303 determines whether or not an object, that is, a garbage bag or a slope is detected based on the object information, and if it is determined that the object is detected, associates the frame image with the object information. The detection result is recorded in "all_res" of the detection result storage unit 122. These processes are performed for each of the frame images stored in "trash_res".

The garbage bag and the slope 600 may be detected by using different trained models. Further, the detection target of the second detection unit 303 may be an object different from the detection target of the first detection unit 302, and is not limited to the garbage bag or the slope 600. However, the detection target of the second detection unit 303 is preferably an object having a similar appearance to the detection target of the first detection unit 302. For example, the detection target of the second detection unit 303 may be an object (for example, corrugated cardboard) that is similar in appearance to an unsuitable object but is not an unsuitable object.

In S34 (confirmation step), the detection result integration unit 304 confirms the final detection result based on the detection results of S32 and S33. More specifically, the detection result integration unit 304 detects the rest of the detection object detected by the first detection unit 302 in S32, excluding the one detected by the second detection unit 303 as a garbage bag or a slope in S33. The final detection result of the object.

Specifically, the detection result integration unit 304 specifies the range occupied by the detection target on the image from the object information of each detection target stored in "all_res" by the first detection unit 302. Next, the detection result integration unit 304 determines whether or not the garbage bag or the slope 600 is detected in the above range based on the object information stored in "all_res" by the second detection unit 303. Here, when the detection result integration unit 304 determines that the garbage bag or the slope 600 is not detected in the above range, the object information of the detection target and the frame image are associated with each other to obtain the final detection result. , Record in "final_res" of the detection result storage unit 122. On the other hand, when the detection result integration unit 304 determines that the garbage bag or the slope 600 is detected in the above range, the detection result integration unit 304 does not record the object information and the frame image of the detection target object. That is, the detection result of this detection object is invalidated as an erroneous detection.

In S35, the detection result is output in the same manner as in S13 of FIG. 5, and the process ends. The output detection result may be read from "final_res" in the detection result storage unit 122.

[Modification example]
Artificial intelligence / machine learning algorithms other than machine-learned neural networks (including deep-learned ones) can also be used for object detection, object classification, and the like in each of the above-described embodiments.

The execution subject of each process described in each of the above embodiments can be changed as appropriate. For example, at least one of the blocks shown in FIGS. 1, 8 or 10 may be omitted, and the omitted processing unit may be provided in the other device or a plurality of devices. In this case, the processing of each of the above-described embodiments is executed by one or more information processing devices.

Further, in each of the above embodiments, an example of detecting an unsuitable substance or the like from a garbage image has been described, but the object to be detected is arbitrary and is not limited to the unsuitable object or the like. Further, the input data for the trained model used by the information processing device 1 is not limited to image data, and may be, for example, audio data. In this case, the information processing device 1 may be configured to detect a predetermined sound component included in the input voice data as a detection target.

The present invention is not limited to the above-described embodiments, and various modifications can be made within the scope of the claims, and the embodiments obtained by appropriately combining the technical means disclosed in the different embodiments. Is also included in the technical scope of the present invention.

1 Information processing device 101, 201, 302 First detection unit 102, 202A, 202B, 303 Second detection unit 304 Detection result integration unit

Claims (7)

A first detection unit that detects the first detection target by inputting input data into the first trained model that has been machine-learned so that a plurality of types of first detection targets can be detected.
The input data is input to the second trained model machine-learned so that the second detection target, which is at least a part of the first detection target of the plurality of types, can be detected, and the second detection target is set. The input data is input to the third trained model that has been machine-learned so that it can be detected or a third detection target different from the first detection target can be detected, and the third detection target is set. It is equipped with a second detection unit for detection.
An information processing apparatus characterized in that the final detection result is determined based on the detection result of the first detection unit and the detection result of the second detection unit. The second detection unit inputs the input data to the second trained model to detect the second detection target, and then detects the second detection target.
The first detection unit and the second detection unit output the detection result to a common output destination.
The information processing apparatus according to claim 1, wherein the detection result of the first detection unit and the detection result of the second detection unit, which are output to the common output destination, are the final detection results. .. The first detection unit inputs a plurality of input data to the first trained model and detects the first detection target from each input data.
The second detection unit inputs the input data in which the first detection target is detected by the first detection unit to the second trained model among the plurality of input data, and the second detection target Detected and
The information processing apparatus according to claim 1, wherein the detection result of the second detection unit is used as the final detection result. The second detection unit inputs the input data to the third trained model to detect the third detection target, and then detects the third detection target.
The remainder of the detection target detected by the first detection unit as the first detection target, excluding the detection target detected by the second detection unit as the third detection target, is used as the detection result of the first detection target. The information processing apparatus according to claim 1, further comprising a detection result integration unit. The above input data is image data,
The first detection unit inputs data having a reduced resolution of the image data into the first trained model, or
Any of claims 1 to 4, wherein the second detection unit inputs data having a reduced resolution of the image data to the second trained model or the third trained model. The information processing apparatus according to item 1. An information processing method executed by one or more information processing devices.
A first detection step in which input data is input to a first trained model machine-learned so that a plurality of types of detection targets can be detected, and the detection target is detected from the input data.
A second trained model machine-learned to detect at least a part of the plurality of types of detection targets, or machine-learned to detect a detection target different from the first trained model. A second detection step of inputting the above input data into the third trained model and detecting a detection target from the input data is included.
An information processing method comprising: a confirmation step of determining a final detection result based on a detection result of the first detection step and a detection result of the second detection step. The information processing program for operating a computer as the information processing device according to claim 1, wherein the computer functions as the first detection unit and the second detection unit.

Images