JP7436801B2 - Information output program, information output device, and information output method - Google Patents

Description

The present invention relates to an information output program, an information output device, and an information output method.

For example, in fields such as maritime patrol and coastal monitoring, workers can detect objects (e.g., suspicious ships, etc.) and identify the type of objects shown in the video by checking the content of video captured in the surveillance area. etc. Specifically, for example, when a plurality of objects are detected in a captured moving image, the operator visually identifies the type of each object (see, for example, Patent Document 1).

International Publication No. 2018/173800

However, if all detected objects are output as monitoring objects, there is a problem that objects appearing in moving images cannot be efficiently monitored.

Therefore, in one aspect, an object of the present invention is to provide an information output program, an information output device, and an information output method that make it possible to efficiently monitor objects shown in moving images.

In one aspect of the embodiment, by using a first learning model that acquires a moving image and learns a candidate type of an object included in the image and a probability that the type of the object is the candidate type, A classification result for each type of object included in a plurality of images constituting a moving image and a first accuracy indicating the accuracy of the classification result are obtained, and for each object included in the plurality of images, Based on the classification result and the first accuracy corresponding to each object, it is determined whether each object satisfies a predetermined condition, and among the objects included in the plurality of images, the predetermined Outputs information indicating that the object determined to satisfy the condition is to be monitored.

According to one aspect, it is possible to efficiently monitor objects shown in videos.

FIG. 1 is a diagram illustrating the configuration of an information processing system 10. FIG. 2 is a diagram illustrating the hardware configuration of the information processing device 1. FIG. 3 is a block diagram of the functions of the information processing device 1. FIG. 4 is a flowchart illustrating an outline of information output processing in the first embodiment. FIG. 5 is a flowchart illustrating details of information output processing in the first embodiment. FIG. 6 is a flowchart illustrating details of information output processing in the first embodiment. FIG. 7 is a flowchart illustrating details of information output processing in the first embodiment. FIG. 8 is a flowchart illustrating details of information output processing in the first embodiment. FIG. 9 is a flowchart illustrating details of information output processing in the first embodiment. FIG. 10 is a flowchart illustrating details of information output processing in the first embodiment. FIG. 11 is a flowchart illustrating details of information output processing in the first embodiment. FIG. 12 is a diagram illustrating a specific example of the first learning data 131. FIG. 13 describes a specific example of the process of S22. FIG. 14 is a diagram illustrating a specific example of the second learning data 132. FIG. 15 is a diagram illustrating a specific example of the process of S42. FIG. 16 is a diagram illustrating a specific example of the process of S53. FIG. 17 is a diagram illustrating a specific example of the detection result information 135. FIG. 18 is a diagram illustrating a specific example of the process of S61. FIG. 19 is a diagram illustrating a specific example of the classification result information 134. FIG. 20 is a diagram illustrating a specific example of the process of S64. FIG. 21 is a diagram illustrating a specific example of the process of S72.

[Information processing system configuration]
First, the configuration of the information processing system 10 will be explained. FIG. 1 is a diagram illustrating the configuration of an information processing system 10.

The information processing system 10 shown in FIG. 1 includes an information processing device 1 and an operation terminal 3. The information processing system 10 shown in FIG. The operating terminal 3 is, for example, a PC (Personal Computer) on which a worker inputs necessary information, and is accessible to the information processing device 1 via a network NW such as the Internet.

For example, the information processing device 1 selects candidate types of objects (hereinafter referred to as types) included in each image (hereinafter also referred to as image data) constituting a moving image photographed in a monitoring area (hereinafter also referred to as video data). By using a learning model (hereinafter also referred to as the first learning model) that has learned the candidate type (also referred to as a candidate) and the probability (for example, probability) that the candidate type is correct as the type of the object, Among the objects to be inspected, identify those that require visual confirmation by the worker.

Specifically, the information processing device 1 obtains classification results for each type of object included in the plurality of image data and the accuracy of the classification results, for example, by using the first learning model. The information processing device 1 then determines whether or not each object satisfies a predetermined condition based on the classification result and accuracy corresponding to each object included in the plurality of image data. conduct. Thereafter, the information processing device 1 outputs information indicating that the object that is determined to satisfy a predetermined condition among the objects included in the plurality of image data is the object to be monitored.

That is, conventionally, for example, when moving images that require detection of a target object are frequently generated, there is a possibility that the burden on the operator due to the detection of the target object increases. Further, for example, if the number of objects that need to be checked further increases in the future, it may become difficult to secure the necessary number of workers in the future.

Therefore, the information processing device 1 according to the present embodiment is configured to, for example, detect objects whose type corresponding to the classification result output by the first learning model is not included in the types of monitoring targets, and If there is an object for which the accuracy of the output classification result is higher than a predetermined threshold, it is determined that there is no need to monitor the existing object. Also, for example, if the type corresponding to the classification result output by the first learning model is included in the type of the monitored object, or if the accuracy of the classification result output by the first learning model is higher than a predetermined threshold value, If a low target object exists, the information processing device 1 determines that it is necessary to monitor the existing target object.

Then, the information processing device 1 outputs, to the operation terminal 3, only information about the object that is determined to need to be monitored, for example. Thereafter, the operator visually identifies, for example, the type of the object whose information has been output to the operating terminal 3.

Thereby, the information processing device 1 can suppress the burden on the worker associated with monitoring the target object.

[Hardware configuration of information processing system]
Next, the hardware configuration of the information processing system 10 will be explained. FIG. 2 is a diagram illustrating the hardware configuration of the information processing device 1.

As shown in FIG. 2, the information processing device 1 includes a CPU 101 that is a processor, a memory 102, a communication device 103, and a storage medium 104. Each part is connected to each other via a bus 105.

The storage medium 104 is, for example, a program storage area (not shown) that stores a program 110 for performing a process (hereinafter also referred to as information output process) of outputting information indicating an object that a worker needs to monitor. has. Furthermore, the storage medium 104 includes, for example, a storage unit 130 (hereinafter also referred to as an information storage area 130) that stores information used when performing information output processing. Note that the storage medium 104 may be, for example, an HDD (Hard Disk Drive) or an SSD (Solid State Drive).

The CPU 101 executes the program 110 loaded into the memory 102 from the storage medium 104 to perform information output processing.

Furthermore, the communication device 103 communicates with the operation terminal 3 via the network NW, for example.

[Functions of information processing system]
Next, the functions of the information processing system 10 will be explained. FIG. 3 is a block diagram of the functions of the information processing device 1.

As shown in FIG. 3, the information processing device 1 includes an information receiving unit 111 (hereinafter also referred to as an information acquisition unit 111) through organic cooperation between hardware such as a CPU 101 and a memory 102 and a program 110. ), the information management unit 112, the first model generation unit 113, the second model generation unit 114, the result acquisition unit 115, the first condition determination unit 116, the second condition determination unit 117, and the result output unit 118, and various other functions are realized.

Further, the information processing device 1 stores first learning data 131, second learning data 132, video data 133, classification result information 134, and detection result information 135, for example, as shown in FIG. It is stored in area 130.

The information receiving unit 111 receives, for example, various information input by a worker via the operating terminal 3. Specifically, the information receiving unit 111 receives, for example, first learning data 131 used for generating the first learning model. The first learning data 131 is, for example, learning data that each includes a learning image including a target object and information indicating the type of target object included in the learning image.

Further, the information receiving unit 111 receives, for example, second learning data 132 used for generating the second learning model. The second learning data 132 is, for example, learning data that each includes a learning image including a target object and position information of the target included in the learning image.

Further, the information receiving unit 111 receives, for example, video data 133 acquired by a photographing device (not shown) such as a camera that photographs objects on the sea. Then, the information management unit 112 stores, for example, the first learning data 131, the second learning data 132, and the video data 133 received by the information receiving unit 111 in the information storage area 130.

Note that the information receiving unit 111 may acquire various information stored in an information storage area (not shown) of the operating terminal 3 by voluntarily accessing the operating terminal 3, for example. good. Further, the information receiving unit 111 may acquire the video data 133 stored in an information storage area (not shown) of the imaging device by, for example, spontaneously accessing the imaging device. .

The first model generation unit 113 generates a first learning model by learning the first learning data 131 stored in the information storage area 130.

The second model generation unit 114 generates a second learning model by learning the second learning data 132 stored in the information storage area 130.

The result acquisition unit 115 acquires the video data 133 stored in the information storage area 130. Then, by using the second learning model generated by the second model generation unit 114, the result acquisition unit 115 obtains a probability (hereinafter referred to as , also called second accuracy). Specifically, the result acquisition unit 115 acquires the second accuracy that is output when a plurality of image data are input to the second learning model. After that, the information management unit 112 generates, for example, detection result information 135 indicating the second accuracy acquired by the result acquisition unit 115, and stores it in the information storage area 130.

Furthermore, the result acquisition unit 115 acquires a plurality of partial image data for each of the objects included in the plurality of image data forming the video data 133. Specifically, the result acquisition unit 115 acquires (extracts) a plurality of partial image data from a plurality of image data forming the video data 133 by referring to the detection result information 135 stored in the information storage area 130, for example. )do. Then, the result acquisition unit 115 uses the first learning model generated by the first model generation unit 113 to obtain the classification results for each type of object included in the plurality of partial image data and the classification results. Accuracy (hereinafter also referred to as first accuracy) is obtained. Specifically, the result acquisition unit 115 acquires the classification result and first accuracy that are output when a plurality of image data are input to the first learning model. Thereafter, the information management unit 112 generates, for example, classification result information 134 indicating the classification result and the first accuracy acquired by the result acquisition unit 115, and stores it in the information storage area 130.

For example, the first condition determination unit 116 refers to the classification result information 134 stored in the information storage area 130, and determines whether the classification result corresponding to each object satisfies the first condition for each of the plurality of partial image data. and whether the first accuracy corresponding to each object satisfies the second condition. Specifically, for example, if the classification result indicates that each object has a predetermined attribute (for example, a warship), the first condition determination unit 116 determines the classification result corresponding to each object. It is determined that the first condition is satisfied. Further, the first condition determination unit 116 determines that, for example, when the first accuracy corresponding to each target object is larger than a threshold value (hereinafter also referred to as the first threshold value), the first accuracy corresponding to each target object is set to the second condition. It is determined that the conditions are met.

For example, the second condition determination unit 117 refers to the classification result information 134 and the detection result information 135 stored in the information storage area 130, and determines whether the first accuracy corresponding to each target object is the first accuracy for each of the plurality of partial image data. It is determined whether the two conditions are satisfied, and it is determined whether the second accuracy corresponding to each object satisfies the third condition. Specifically, the second condition determining unit 117 determines that the second accuracy corresponding to each object is larger than a threshold (hereinafter also referred to as a second threshold), for example. It is determined that three conditions are met.

The result output unit 118 selects, among the objects included in the plurality of partial image data, objects for which the classification result has determined to satisfy the first condition and objects for which the first accuracy has determined to satisfy the second condition. output information (for example, an alert) indicating the

In addition, the result output unit 118 determines that among the objects included in the plurality of partial image data, the object whose first accuracy satisfies the second condition and whose second accuracy satisfies the third condition has not been learned. Outputs information indicating that.

[Outline of first embodiment]
Next, an outline of the first embodiment will be described. FIG. 4 is a flowchart illustrating an outline of information output processing in the first embodiment.

As shown in FIG. 4, the information processing device 1 waits until the information output timing comes (NO in S1). The information output timing may be, for example, the timing at which the operator inputs information indicating that monitoring of objects on the sea is to be started.

Then, when the information output timing has come (YES in S1), the information processing device 1 acquires the video data 133 (S2). Specifically, the information processing device 1 acquires the video data 133 stored in the information storage area 130, for example. Further, the information processing device 1 obtains, for example, video data 133 stored in an external storage device (not shown).

Next, the information processing device 1 performs the processing in S2 by using the first learning model that has learned the object type candidates included in the image data and the probability that the type candidates are correct as the object type. A classification result for each type of object included in the plurality of image data constituting the video data acquired in (S3) is obtained.

Next, the information processing device 1 calculates each target object included in the plurality of image data constituting the video data obtained in the process of S2 based on the classification result and the first accuracy obtained in the process of S3. It is determined whether the object satisfies a predetermined condition (S4).

After that, the information processing device 1 determines that the object that is determined to satisfy a predetermined condition in the process of S4, among the objects included in the plurality of image data forming the video data acquired in the process of S2, is to be monitored. Information indicating this is output (S5).

Thereby, by using the first learning model, the information processing device 1 is able to specify an object that requires visual confirmation by the operator. Therefore, the information processing device 1 can suppress the burden on the worker associated with monitoring the object.

[Details of the first embodiment]
Next, details of the first embodiment will be described. 5 to 11 are flowcharts illustrating details of information output processing in the first embodiment. Further, FIGS. 12 to 21 are diagrams illustrating details of the information output processing in the first embodiment.

[First learning data storage process]
First, among the information output processing, the processing for storing the first learning data 131 (hereinafter also referred to as the first learning data storage processing) will be explained. FIG. 5 is a flowchart illustrating the first learning data storage process.

As shown in FIG. 5, the information receiving unit 111 of the information processing device 1 waits, for example, until it receives the plurality of first learning data 131 transmitted from the operating terminal 3 (NO in S11). That is, the information receiving unit 111 waits until the first learning data 131 generated by the worker is transmitted via the operation terminal 3, for example.

Then, when the plurality of first learning data 131 is received (YES in S11), the information management unit 112 of the information processing device 1 stores the plurality of first learning data 131 received in the process in S11 in the information storage area 130. (S12). A specific example of the first learning data 131 will be described below.

[Specific example of first learning data]
FIG. 12 is a diagram illustrating a specific example of the first learning data 131.

The first learning data 131 shown in FIG. 12 includes an "item number" indicating identification information of each of the first learning data 131, "image data" indicating each learning image data itself, and each learning image data. ``type'' indicating the type of object included (type of correct answer). Note that the following description assumes that the "type" is set to one of "AAA", "BBB", and "CCC".

Specifically, the first learning data 131 whose "item number" is "1" in FIG. 12 includes learning image data whose name is "IMAGEa1" and whose "type" is "AAA". It is data.

In addition, the first learning data 131 whose "item number" is "2" in FIG. 12 is learning data which includes learning image data whose name is "IMAGEa2" and whose "type" is "BBB". be. Description of other first learning data 131 included in FIG. 12 will be omitted.

[First model generation process]
Next, of the information output processing, the processing for generating the first model (hereinafter also referred to as the first model generation processing) will be explained. FIG. 6 is a flowchart diagram illustrating the first model generation process.

As shown in FIG. 6, the first model generation unit 113 of the information processing device 1 waits, for example, until the first model generation timing comes (NO in S21). The first model generation timing may be, for example, the timing at which the operator inputs information to the information processing device 1 to the effect that the first model is to be generated.

Then, when the first model generation timing has come (YES in S21), the first model generation unit 113 performs the first learning by learning the plurality of first learning data 131 stored in the information storage area 130. A model is generated (S22). A specific example of the process of S22 will be described below.

[Specific example of processing in S22]
FIG. 13 describes a specific example of the process of S22.

As shown in FIG. 13, the first model generation unit 113 includes, for example, a plurality of output neurons (hereinafter also referred to as first output neurons) NE11 corresponding to each of the object type candidates, and a plurality of output neurons (hereinafter also referred to as first output neurons) NE11 corresponding to the first accuracy. A first model MD01 having one output neuron (hereinafter also referred to as a second output neuron) NE12 is generated. In the example shown in FIG. 13, the first output neuron NE11 includes first output neurons NE11a, NE11b, and NE11c corresponding to the type candidates "AAA", "BBB", and "CCC", respectively.

For example, for the information whose "item number" is "1" in the first learning data 131 explained in FIG. 12, "IMAGEa1" is set as "image data" and "AAA" is set as "type". has been done. Therefore, for example, when the feature amount of the image data whose name is "IMAGEa1" is input from the input neuron NE13, the first model generation unit 113 generates a value output from each of the first output neurons NE11 and "AAA". The value corresponding to each weight in the first learning model is updated so that the difference from the correct value ("1", "0", and "0") corresponding to "" is reduced. Specifically, in this case, the first model generation unit 113 calculates the difference between the value output from the first output neuron NE11a and "1", and the difference between the value output from the first output neuron NE11b and "0". , and the value corresponding to each weight in the first learning model is updated so that the difference between the value output from the first output neuron NE11c and "0" becomes smaller.

For example, for the information whose "item number" is "2" in the first learning data 131 explained in FIG. 12, "IMAGEa2" is set as "image data" and "BBB" is set as "type". has been done. Therefore, for example, when image data whose name is "IMAGEa2" or its feature amount is input from the input neuron NE13, the first model generation unit 113 generates the values output from each of the first output neurons NE11 and "IMAGEa2". The value corresponding to each weight in the first learning model is updated so that the difference from the correct value ("0", "1", and "0") corresponding to "BBB" becomes smaller. Specifically, in this case, the first model generation unit 113 calculates the difference between the value output from the first output neuron NE11a and "0" and the difference between the value output from the first output neuron NE11b and "1". , and the value corresponding to each weight in the first learning model is updated so that the difference between the value output from the first output neuron NE11c and "0" becomes smaller.

Note that, before comparing the output value from the first output neuron NE11 with the correct value, the first model generation unit 113 converts the output value from the first output neuron NE11 into the output value from the second output neuron NE12. It may also be one that performs correction.

Specifically, in this case, the first model generation unit 113 corrects the output value from the first output neuron NE11 so that the closer the output value from the second output neuron NE12 is to "0", the closer the output value from the first output neuron NE11 is to the correct value. conduct. On the other hand, the first model generation unit 113 decreases the correction width of the output value from the first output neuron NE11 as the output value from the second output neuron NE12 approaches "1". Furthermore, in this case, the first model generation unit 113 sets a penalty value that is larger as the correction width of the output value from the first output neuron NE11 is larger. Then, the first model generation unit 113 adjusts each weight in the first learning model so that both the difference between the output value (value after correction) from the first output neuron NE11 and the correct value and the penalty value are small. Update the corresponding value.

[Second learning data storage process]
Next, of the information output processing, the processing for storing the second learning data 132 (hereinafter also referred to as the second learning data storage processing) will be explained. FIG. 7 is a flowchart illustrating the second learning data storage process.

As shown in FIG. 7, the information receiving unit 111 waits, for example, until receiving the plurality of second learning data 132 transmitted from the operation terminal 3 (NO in S31). That is, the information receiving unit 111 waits until the second learning data 132 generated by the worker is transmitted via the operation terminal 3, for example.

When the plurality of first learning data 131 is received (YES in S31), the information management unit 112 stores the plurality of second learning data 132 received in the process of S31 in the information storage area 130 (S32). A specific example of the second learning data 132 will be described below.

[Specific example of second learning data]
FIG. 14 is a diagram illustrating a specific example of the second learning data 132.

The second learning data 132 shown in FIG. 14 includes "item number" indicating the identification information of each of the second learning data 132, "image data" indicating each learning image data itself, and each learning image data. "Position" indicating the upper left position of the included object (frame containing the object), "Width" indicating the width of the object (frame containing the object) included in each learning image data, and each learning The item "height" indicates the height of the object (the frame containing the object) included in the image data for use.

Specifically, the second learning data 132 whose “item number” is “1” in FIG. 14 includes learning image data whose name is “IMAGEb1” and whose “position” is “10, 32”. information about an object whose "width" is "10" and "height" is "8", and information about an object whose "position" is "24,30" and whose "width" is "4". information about an object whose "height" is "2", its "position" is "28, 20", its "width" is "8", and its "height" is "6". This is learning data that includes information about the target object. That is, the second learning data 132 whose "item number" is "1" in FIG. 14 includes information about three objects included in the learning image data whose name is "IMAGEb1".

Further, the second learning data 132 whose “item number” is “2” in FIG. 14 includes learning image data whose name is “IMAGEb2”, and whose “position” is “34, 36”. Information about an object whose "width" is "4" and "height" is "2" and whose "position" is "10,12" and whose "width" is "12" and " This learning data includes information about an object whose height is "6". That is, the second learning data 132 whose "item number" in FIG. 14 is "2" includes information about two objects included in the learning image data whose name is "IMAGEb2". A description of the other second learning data 132 included in FIG. 14 will be omitted.

[Second model generation process]
Next, of the information output processing, the processing for generating the second model (hereinafter also referred to as the second model generation processing) will be explained. FIG. 8 is a flowchart illustrating the second model generation process.

As shown in FIG. 8, the first model generation unit 113 of the information processing device 1 waits, for example, until the second model generation timing comes (NO in S41). The second model generation timing may be, for example, the timing at which the operator inputs information to the information processing device 1 to the effect that the second model is to be generated. Further, the second model generation timing may be, for example, the timing immediately after the first model is generated.

Then, when the second model generation timing has come (YES in S41), the second model generation unit 114 performs the second learning by learning the plurality of second learning data 132 stored in the information storage area 130. A model is generated (S42). A specific example of the process of S42 will be described below.

[Specific example of processing in S42]
FIG. 15 is a diagram illustrating a specific example of the process of S42.

As shown in FIG. 15, the second model generation unit 114 generates, for example, a second model MD02 having a plurality of output neurons NE21 corresponding to each of a plurality of divided image data obtained by dividing the learning image data. In the example shown in FIG. 15, the output neuron NE21 includes output neurons NE21a, NE21b, NE21c, NE21d, NE21e, NE21f, NE21g, NE21h, NE21i corresponding to each of the 16 divided image data obtained by dividing the learning image data. , NE21j, NE21k, NE21l, NE21m, NE21n, NE21о, and NE21p.

For example, for the information whose "item number" is "3" in the second learning data 132 explained in FIG. 14, "IMAGEb3" is set as "image data", and "position", "width" and " Information about one object is set as "Height". Therefore, for example, when the divided image data corresponding to the output neuron NE21d includes an object among the divided image data of the image data whose name is "IMAGEb3", the second model generation unit 114 is configured to When the feature quantity of the image data "IMAGEb3" is input from the input neuron NE23, the difference between the output value from the output neuron NE21d and "1" and the output from each of the output neurons NE21 other than the output neuron NE21d The values corresponding to each weight in the second learning model are updated so that the difference between the value and "0" becomes smaller.

Further, for example, for the information whose "item number" is "2" in the second learning data 132 explained in FIG. 14, "IMAGEb2" is set as "image data", and "position", "width" and " Information about the two objects is set as "height". Therefore, for example, if the second model generation unit 114 includes an object in each of the divided image data corresponding to the output neurons NE21f and NE21m among the divided image data of the image data whose name is "IMAGEb2" In, when the feature amount of the image data whose name is "IMAGEb2" is input from the input neuron NE23, the difference between the output value from each of the output neurons NE21f and NE21m and "1", and the output neurons other than these The values corresponding to each weight in the second learning model are updated so that the difference between the output value from each NE 21 and "0" becomes smaller.

[Main processing of information output processing]
Next, the main processing of the information output processing will be explained. 9 to 11 are flowcharts illustrating the main processing of the information output processing.

As shown in FIG. 9, the result acquisition unit 115 of the information processing device 1 waits, for example, until the information output timing comes (NO in S51). The information output timing may be, for example, the timing at which the operator inputs information indicating that monitoring of objects on the sea is to be started.

Then, when the information output timing has come (S51: YES), the result acquisition unit 115 acquires the video data 133 (S52).

Subsequently, by using the second learning model generated in the process of S42, the result acquisition unit 115 determines whether the target object is included in each image data for each of the plurality of image data included in the video data 133 acquired in the process of S52. A second accuracy indicating the existing accuracy is acquired (S53). A specific example of the process of S53 will be described below.

[Specific example of processing in S53]
FIG. 16 is a diagram illustrating a specific example of the process of S53.

As shown in FIG. 16, the result acquisition unit 115 inputs, for example, image data IMAGEc1 included in the video data 133 to the second learning model MD02. Then, the result acquisition unit 115 acquires, for example, the values output from each of the output neurons NE21 of the second learning model MD02 in response to the input of the image data IMAGEc1.

After that, the result acquisition unit 115 identifies the position of the object included in the image data IMAGEc1, for example, from each of the values acquired from the second learning model MD02. Specifically, as shown in FIG. 16, the result acquisition unit 115, for example, obtains each of partial image data IMAGEd1, partial image data IMAGEd2, and partial image data IMAGEd3 as partial image data including the object included in the image data IMAGEc1. Identify the location of.

Further, the result acquisition unit 115 may, for example, calculate the maximum value of the values corresponding to the position of each object among the values output from each of the output neurons NE2 of the second learning model MD02 for each specified object. Obtained as 2 accuracy.

Note that the position of the object and the second accuracy included in the image data IMAGEc1 may be output from the second model MD02. In this case, the result acquisition unit 115 may acquire each piece of information output from the second model MD02.

Returning to FIG. 9, the information management unit 112 generates, for example, detection result information 135 including the second accuracy obtained in the process of S53 for each image data included in the video data 133 obtained in the process of S52 (S54 ). The information management unit 112 then stores the generated detection result information 135 in the information storage area 130. A specific example of the detection result information 135 will be described below.

[Specific example of detection result information]
FIG. 17 is a diagram illustrating a specific example of the detection result information 135.

The detection result information 135 shown in FIG. 17 includes an "item number" indicating each piece of information included in the detection result information 135, "partial image data" indicating identification information of partial image data including each target object, and each target object. It has an item "second accuracy" indicating a second accuracy corresponding to .

Specifically, in the detection result information 135 shown in FIG. 17, for information whose "item number" is "1", "IMAGEd1" is set as "partial image data", and "0. 9" is set.

In addition, in the detection result information 135 shown in FIG. 17, for information whose "item number" is "2", "IMAGEd2" is set as "partial image data", and "0.7" is set as "second accuracy". is set. Description of other information included in FIG. 17 will be omitted.

Note that the detection result information 135 may include information indicating the position of the partial image data identified in the process of S53.

Returning to FIG. 9, the information management unit 112 acquires partial image data including the object included in each image data from the image data included in the video data 133 acquired in the process of S52 (S55).

Specifically, the information management unit 112 acquires partial image data from each image data included in the video data 133 acquired in the process of S52, for example, by referring to the position of the partial image data identified in the process of S53. I do.

Note that when the detection result information 135 includes information indicating the position of the partial image data identified in the process of S53, the information management unit 112 refers to the detection result information 135 stored in the information storage area 130. It is also possible to acquire partial image data.

Then, as shown in FIG. 10, by using the first learning model generated in the process of S22, the result acquisition unit 115 analyzes the target included in each partial image data for each partial image data acquired in the process of S55. A classification result of the type of object and a first accuracy indicating the accuracy of the classification result are obtained (S61). A specific example of the process in S61 will be described below.

[Specific example of processing in S61]
FIG. 18 is a diagram illustrating a specific example of the process of S61.

As shown in FIG. 18, the result acquisition unit 115 inputs, for example, the partial image data IMAGEd1 acquired in the process of S55 to the first learning model MD01. Then, the result acquisition unit 115 acquires, for example, the values output from each of the output neuron NE11 and the output neuron NE12 of the first learning model MD01 in response to input of the partial image data IMAGEd1.

Specifically, as shown in FIG. 18, the result acquisition unit 115 determines that, for example, the probability that the type of the object included in the partial image data IMAGEd1 is "AAA" is "0.6", and the probability that the type of the object included in the partial image data IMAGEd1 is "BBB" is "0.6". Information indicating that a certain accuracy is "0.3" and that the accuracy of "CCC" is "0.1" is acquired as a classification result.

Further, as shown in FIG. 18, the result acquisition unit 115 acquires, for example, "0.9" as the first accuracy of the classification result for the type of object included in the partial image data IMAGEd1.

Returning to FIG. 10, the information management unit 112 generates, for example, classification result information 134 including the classification result obtained in the process of S61 and the first accuracy for each partial image data obtained in the process of S55 (S62). The information management unit 112 then stores the generated classification result information 134 in the information storage area 130. A specific example of the classification result information 134 will be described below.

[Specific example of classification result information]
FIG. 19 is a diagram illustrating a specific example of the classification result information 134.

The classification result information 134 shown in FIG. 19 includes "item number" indicating each piece of information included in the classification result information 134, "partial image data" indicating identification information of partial image data including each object, and each object. The items include "classification result" indicating the classification result of the type of object, and "first accuracy" indicating the first accuracy of the classification result for each type of object.

Specifically, in the classification result information 134 shown in FIG. 19, "IMAGEd1" is set as "partial image data" for information whose "item number" is "1". In the classification result information 134 shown in FIG. 19, the information whose "item number" is "1" also includes the types of objects included in the partial image data IMAGEd1 as "AAA" and "BBB". ” and “CCC” with respective accuracies of “0.6,” “0.3,” and “0.1” are set. Furthermore, in the classification result information 134 shown in FIG. 19, "0.8" is set as the "first accuracy" for information whose "item number" is "1".

Further, in the classification result information 134 shown in FIG. 19, "IMAGEd2" is set as "partial image data" for information whose "item number" is "2". In addition, in the classification result information 134 shown in FIG. 19, the information whose "item number" is "2" includes the types of objects included in the partial image data IMAGEd2 as "AAA" and "BBB". ” and “CCC” with respective accuracies of “0.2,” “0.7,” and “0.1” are set. Furthermore, in the classification result information 134 shown in FIG. 19, "0.9" is set as the "first accuracy" for information whose "item number" is "2". Description of other information included in FIG. 19 will be omitted.

Returning to FIG. 10, the second condition determination unit 117 of the information processing device 1 determines, for each partial image data acquired in the process of S55, whether or not the classification result corresponding to each partial image data satisfies the first condition; , it is determined whether the first accuracy corresponding to each partial image data satisfies the second condition (S63).

Specifically, for example, if the classification result indicates that each object has a predetermined attribute (for example, a warship), the first condition determination unit 116 determines the classification result corresponding to each object. It is determined that the first condition is satisfied. Further, the first condition determination unit 116 determines that the first accuracy corresponding to each target object satisfies the second condition, for example, when the first accuracy corresponding to each target object is larger than the first threshold value. .

Then, the result output unit 118 of the information processing device 1 selects the classification result obtained in the process of S61, which is included in each of the partial image data determined to satisfy the first condition, among the partial image data obtained in the process of S55. An alert is output to the operation terminal 3 regarding the target object and the target object included in each of the partial image data for which it is determined that the first accuracy obtained in the process of S61 satisfies the second condition (S64). A specific example of the process of S64 will be described below.

[Specific example of processing in S64]
FIG. 20 is a diagram illustrating a specific example of the process of S64.

Among the information shown in FIG. 20, for the information whose "item number" is "1", if the first accuracy is higher than the first threshold and the object requires monitoring, an alert is output. It is shown that. In addition, for the information whose "item number" is "3" among the information shown in FIG. 20, if the first accuracy is lower than the first threshold and the object requires monitoring, the alert output is indicates that it will be done. Furthermore, among the information shown in FIG. 20, the information whose "item number" is "4" does not output an alert if the first accuracy is lower than the first threshold and the object does not require monitoring. This indicates that the

On the other hand, for the information whose "item number" is "2" among the information shown in FIG. 20, if the first accuracy is higher than the first threshold and the object does not require monitoring, an alert will not be output. It shows that it will not happen.

In other words, there is a strong possibility that a type of object that has a high first accuracy and does not require monitoring is an object that does not require a worker to detect (an object that does not require a worker to take action). It can be judged that it is high. Therefore, when the result output unit 118 detects an object of a type that has a high first accuracy and does not require monitoring, it does not notify the operator (output an alert).

This makes it possible for the information processing device 1 to reduce the workload associated with monitoring objects at sea, for example.

Further, as shown in FIG. 11, the first condition determination unit 116 of the information processing device 1 determines that the first accuracy corresponding to each partial image data satisfies the second condition for each partial image data acquired in the process of S55. It is determined whether or not the second accuracy corresponding to each partial image data satisfies the third condition (S71).

Specifically, the second condition determination unit 117 determines that, for example, when the second accuracy corresponding to each target object is larger than the second threshold value, the second accuracy corresponding to each target object satisfies the third condition. judge.

Then, the result output unit 118 determines that among the partial image data acquired in the process of S55, the first accuracy acquired in the process of S61 does not satisfy the second condition, and that the second accuracy acquired in the process of S53 is the second accuracy. Information indicating that the object included in the partial image data that satisfies the three conditions has not been learned is output (S72). A specific example of the process of S72 will be described below.

[Specific example of processing in S72]
FIG. 21 is a diagram illustrating a specific example of the process of S72. In the graph shown in FIG. 21, the horizontal axis corresponds to the first accuracy, and the vertical axis corresponds to the second accuracy.

The graph shown in FIG. 21 shows that, for example, an object whose first accuracy exceeds the first threshold is determined to be an object that has already been learned in the first learning model and the second learning model.

Further, the graph shown in FIG. 21 shows that, for example, an object whose first accuracy is less than the first threshold is determined to be an object that may not have been learned in the first learning model and the second learning model. It shows.

Furthermore, the graph shown in FIG. 21 shows that an object whose first accuracy is less than the first threshold and whose second threshold is less than the second threshold is a learned object, but noise contained in the video data 133, etc. This indicates that the object is judged to be difficult to determine its type due to the influence of Furthermore, the graph shown in FIG. 21 indicates that an object whose first accuracy is less than the first threshold and whose second threshold is greater than the second threshold is determined to be an unlearned object. .

In other words, it can be determined that an object for which the first accuracy is low and the second accuracy is high is extremely likely to be an object that has not been learned in the first learning model and the second learning model. Therefore, when there is an object for which the first accuracy is low and the second accuracy is high, the result output unit 118 identifies the object as an unlearned object.

This makes it possible for the information processing device 1 to classify each object with a low first accuracy into either a learned object or an unlearned object. Then, for example, the operator can increase the accuracy of each learning model by having the first learning model and the second learning model learn only the object specified as an unlearned object.

In this way, the information processing device 1 according to the present embodiment uses the first learning model to obtain classification results for each type of object included in the plurality of image data constituting the video data 133, and and the first accuracy of the classification result. Then, the information processing device 1 determines, for each object included in the plurality of image data, whether each object satisfies a predetermined condition based on the classification result and the first accuracy corresponding to each object. Make a judgment. Thereafter, the information processing device 1 outputs information indicating that the object that is determined to satisfy a predetermined condition among the objects included in the plurality of image data is the object to be monitored.

That is, for example, if the type corresponding to the classification result output by the first learning model is not included in the types of monitoring targets (for example, a predetermined type of monitoring target), the first learning model If there is an object for which the accuracy of the classification result outputted by is higher than a predetermined threshold (for example, a predetermined threshold), the information processing device 1 determines that there is no need to monitor the existing object. to decide. On the other hand, for example, if the type corresponding to the classification result output by the first learning model is included in the types of the monitored object, or if the accuracy of the classification result output by the first learning model is lower than a predetermined threshold value, If a low target object exists, the information processing device 1 determines that it is necessary to monitor the existing target object.

As a result, the information processing device 1 can use the first learning model to identify objects that need to be visually monitored by the operator. Therefore, the information processing device 1 can suppress the burden on the worker associated with monitoring the object.

The above embodiments can be summarized as follows.

(Additional note 1)
Obtain video images,
By using a first learning model that has learned candidate types of objects included in an image and the probability that the type of the object is the candidate type, Obtaining a classification result for each type and a first accuracy indicating the accuracy of the classification result,
For each object included in the plurality of images, it is determined whether each object satisfies a predetermined condition based on the classification result and the first accuracy corresponding to each object,
Outputting information indicating that the object that is determined to satisfy the predetermined condition among the objects included in the plurality of images is a monitoring target;
An information output program characterized by causing a computer to execute processing.

(Additional note 2)
In Appendix 1, further,
Before the process of acquiring the classification result and the first accuracy, a plurality of first learning images each including a learning image including a target object and information indicating a candidate type of the target object included in the learning image are performed. generating the first learning model by performing data learning;
An information output program characterized by causing a computer to execute processing.

(Additional note 3)
In Appendix 1,
In the process of performing the determination, for each target object included in the plurality of images, it is determined whether the classification result corresponding to each target object satisfies a first condition, and the first accuracy corresponding to each target object. Determine whether or not satisfies the second condition,
In the outputting process, among the objects included in the plurality of images, the classification result determines that the classification result satisfies the first condition, and the first accuracy determines that the object satisfies the second condition. Output alerts about and,
An information output program characterized by:

(Additional note 4)
In Appendix 3,
In the determination process, if the classification result indicates that each object has a predetermined attribute, the classification result corresponding to each object satisfies the first condition. judge,
An information output program characterized by:

(Appendix 5)
In Appendix 3,
In the process of performing the determination, if the first accuracy corresponding to each target object is larger than a first threshold value, it is determined that the first accuracy corresponding to each target object satisfies the second condition.
An information output program characterized by:

(Appendix 6)
In Appendix 3, further,
obtaining a second accuracy indicating the probability that the object exists in each of the plurality of images by using a second learning model that has learned position information of the object in the image;
Let the computer carry out the process,
In the process of performing the determination, it is determined for each target object included in the plurality of images whether the first accuracy corresponding to each target object satisfies the second condition, and determining whether the second accuracy satisfies a third condition;
In the outputting process, among the objects included in the plurality of images, an object whose first accuracy does not satisfy the second condition and whose second accuracy satisfies the third condition is unlearned. Output information indicating that there is a
An information output program characterized by:

(Appendix 7)
In Appendix 6, further,
Before the process of acquiring the second accuracy, learning a plurality of second learning data each including a learning image including a target object and position information of the target in the learning image, thereby obtaining the second accuracy. 2 Generate a learning model,
An information output program characterized by causing a computer to execute processing.

(Appendix 8)
In Appendix 6,
In the process of performing the determination, if the second accuracy corresponding to each target object is larger than a second threshold, it is determined that the second accuracy corresponding to each target object satisfies the third condition.
An information output program characterized by:

(Appendix 9)
Obtain video images,
By using a first learning model that has learned candidate types of objects included in an image and the probability that the type of the object is the candidate type, Obtaining a classification result for each type and a first accuracy indicating the accuracy of the classification result,
By using a second learning model that has learned position information of the target object in the image, a second accuracy indicating the probability that the target object exists in each of the plurality of images is obtained;
For each object included in the plurality of images, it is determined whether the first accuracy corresponding to each object satisfies the condition, and the second accuracy corresponding to each object satisfies the other conditions. Determine whether it satisfies or not,
Among the objects included in the plurality of images, information indicating that an object for which the first accuracy does not satisfy the condition and whose second accuracy satisfies the other condition is unlearned. ,
An information output program characterized by causing a computer to execute processing.

(Appendix 10)
an information acquisition unit that acquires moving images;
By using a first learning model that has learned candidate types of objects included in an image and the probability that the type of the object is the candidate type, it is possible to determine the type of objects included in the plurality of images constituting the moving image. a result acquisition unit that acquires a classification result for each type and a first accuracy indicating the accuracy of the classification result;
a first condition for determining whether each object satisfies a predetermined condition based on the classification result and the first accuracy corresponding to each object for each object included in the plurality of images; A determination section;
a result output unit that outputs information indicating that an object that is determined to satisfy the predetermined condition among the objects included in the plurality of images is a monitoring target;
An information output device characterized by:

(Appendix 11)
In Appendix 10,
The first condition determining unit determines, for each object included in the plurality of images, whether or not the classification result corresponding to each object satisfies a first condition, and whether or not the classification result corresponding to each object satisfies the first condition. Determine whether the accuracy satisfies the second condition,
The result output unit selects, among the objects included in the plurality of images, an object for which the classification result has determined to satisfy the first condition and an object for which the first accuracy has determined to satisfy the second condition. Output alerts about and,
An information output device characterized by:

(Appendix 12)
In Appendix 11,
The result acquisition unit acquires a second probability indicating the probability that the target object exists in each of the plurality of images by using a second learning model that has learned position information of the target object in the image, and further,
For each object included in the plurality of images, it is determined whether the first accuracy corresponding to each object satisfies the second condition, and the second accuracy corresponding to each object is determined to be a second condition determination unit that determines whether the three conditions are satisfied;
The result output unit determines that, among the objects included in the plurality of images, an object for which the first accuracy does not satisfy the second condition and whose second accuracy satisfies the third condition is unlearned. Output information indicating that there is a
An information output device characterized by:

(Appendix 13)
an information acquisition unit that acquires moving images;
By using a first learning model that has learned candidate types of objects included in an image and the probability that the type of the object is the candidate type, By obtaining the classification results for each type and the first accuracy indicating the accuracy of the classification results, and further using a second learning model that has learned the position information of the object in the images, a result acquisition unit that acquires a second accuracy indicating the accuracy that the target object exists in each case;
For each object included in the plurality of images, it is determined whether the first accuracy corresponding to each object satisfies the condition, and the second accuracy corresponding to each object satisfies the other conditions. a second condition determination unit that determines whether or not the condition is satisfied;
Among the objects included in the plurality of images, information indicating that an object for which the first accuracy does not satisfy the condition and whose second accuracy satisfies the other condition is unlearned. having a result acquisition section;
An information output device characterized by:

(Appendix 14)
Obtain video images,
By using a first learning model that has learned candidate types of objects included in an image and the probability that the type of the object is the candidate type, it is possible to determine the type of objects included in the plurality of images constituting the moving image. Obtaining a classification result for each type and a first accuracy indicating the accuracy of the classification result,
For each object included in the plurality of images, it is determined whether each object satisfies a predetermined condition based on the classification result and the first accuracy corresponding to each object,
Outputting information indicating that the object that is determined to satisfy the predetermined condition among the objects included in the plurality of images is a monitoring target;
An information output method characterized by causing a computer to perform processing.

(Appendix 15)
In Appendix 14,
In the process of performing the determination, for each target object included in the plurality of images, it is determined whether the classification result corresponding to each target object satisfies a first condition, and the first accuracy corresponding to each target object. Determine whether or not satisfies the second condition,
In the outputting process, among the objects included in the plurality of images, the classification result determines that the classification result satisfies the first condition, and the first accuracy determines that the object satisfies the second condition. Output alerts about and,
An information output method characterized by:

(Appendix 16)
In addition 15, further,
obtaining a second accuracy indicating the probability that the object exists in each of the plurality of images by using a second learning model that has learned position information of the object in the image;
Let the computer carry out the process,
In the process of performing the determination, it is determined for each target object included in the plurality of images whether the first accuracy corresponding to each target object satisfies the second condition, and determining whether the second accuracy satisfies a third condition;
In the outputting process, among the objects included in the plurality of images, objects for which the first accuracy does not satisfy the second condition and whose second accuracy satisfies the third condition are unlearned. Output information indicating that there is a
An information output method characterized by:

(Appendix 17)
Obtain video images,
By using a first learning model that has learned candidate types of objects included in an image and the probability that the type of the object is the candidate type, it is possible to determine the type of objects included in the plurality of images constituting the moving image. Obtaining a classification result for each type and a first accuracy indicating the accuracy of the classification result,
By using a second learning model that has learned position information of the target object in the image, a second accuracy indicating the probability that the target object exists in each of the plurality of images is obtained;
For each object included in the plurality of images, it is determined whether the first accuracy corresponding to each object satisfies the condition, and the second accuracy corresponding to each object satisfies the other conditions. Determine whether it satisfies or not,
Outputting information indicating that, among the objects included in the plurality of images, an object whose first accuracy does not satisfy the condition and whose second accuracy satisfies the other condition has not been learned. ,
An information output method characterized by causing a computer to perform processing.

1: Information processing device 3: Operation terminal 10: Information processing system NW: Network

Claims (10)

Obtain video images,
By using a first learning model that has learned candidate types of objects included in an image and the probability that the type of the object is the candidate type, it is possible to determine the type of objects included in the plurality of images constituting the moving image. Obtaining a classification result for each type and a first accuracy indicating the accuracy of the classification result,
By using a second learning model that has learned position information of the target object in the image, a second accuracy indicating the probability that the target object exists in each of the plurality of images is obtained;
For each object included in the plurality of images, it is determined whether the first accuracy corresponding to each object satisfies a second condition, and the second accuracy corresponding to each object satisfies a third condition. Determine whether the conditions are met,
Information indicating that among the objects included in the plurality of images, an object for which the first accuracy does not satisfy the second condition and for which the second accuracy satisfies the third condition has not been learned. Output,
An information output program characterized by causing a computer to execute processing. In claim 1,
For each object included in the plurality of images, it is determined whether each object satisfies a predetermined condition based on the classification result and the first accuracy corresponding to each object,
Outputting information indicating that the object that is determined to satisfy the predetermined condition among the objects included in the plurality of images is a monitoring target;
An information output program characterized by causing a computer to execute processing. In claim 2 , further:
Before the process of acquiring the classification result and the first accuracy, a plurality of first learning images each including a learning image including a target object and information indicating a candidate type of the target object included in the learning image are performed. generating the first learning model by performing data learning;
An information output program characterized by causing a computer to execute processing. In claim 2 ,
In the process of performing the determination, for each target object included in the plurality of images, it is determined whether the classification result corresponding to each target object satisfies a first condition, and the first accuracy corresponding to each target object. Determine whether or not satisfies the second condition,
In the outputting process, among the objects included in the plurality of images, the classification result determines that the classification result satisfies the first condition, and the first accuracy determines that the object satisfies the second condition. Output alerts about and,
An information output program characterized by: In claim 4 ,
In the determination process, if the classification result indicates that each object has a predetermined attribute, the classification result corresponding to each object satisfies the first condition. judge,
An information output program characterized by: In claim 4 ,
In the process of performing the determination, if the first accuracy corresponding to each target object is larger than a first threshold value, it is determined that the first accuracy corresponding to each target object satisfies the second condition.
An information output program characterized by: In claim 1 , further:
Before the process of acquiring the second accuracy, learning a plurality of second learning data each including a learning image including a target object and position information of the target in the learning image, thereby obtaining the second accuracy. 2 Generate a learning model,
An information output program characterized by causing a computer to execute processing. In claim 1 ,
In the process of performing the determination, if the second accuracy corresponding to each target object is larger than a second threshold, it is determined that the second accuracy corresponding to each target object satisfies the third condition.
An information output program characterized by: an information acquisition unit that acquires moving images;
By using a first learning model that has learned candidate types of objects included in an image and the probability that the type of the object is the candidate type, it is possible to determine the type of objects included in the plurality of images constituting the moving image. By acquiring the classification results for each type and the first accuracy indicating the accuracy of the classification results, and further using a second learning model that has learned the position information of the object in the images, a result acquisition unit that acquires a second accuracy indicating the accuracy that the target object exists in each case;
For each object included in the plurality of images, it is determined whether the first accuracy corresponding to each object satisfies a second condition, and the second accuracy corresponding to each object satisfies a third condition. a second condition determination unit that determines whether the condition is satisfied;
Information indicating that among the objects included in the plurality of images, an object for which the first accuracy does not satisfy the second condition and for which the second accuracy satisfies the third condition has not been learned. It has a result acquisition unit that outputs.
An information output device characterized by: Obtain video images,
By using a first learning model that has learned candidate types of objects included in an image and the probability that the type of the object is the candidate type, it is possible to determine the type of objects included in the plurality of images constituting the moving image. Obtaining a classification result for each type and a first accuracy indicating the accuracy of the classification result,
By using a second learning model that has learned position information of the target object in the image, a second accuracy indicating the probability that the target object exists in each of the plurality of images is obtained;
For each object included in the plurality of images, it is determined whether the first accuracy corresponding to each object satisfies a second condition, and the second accuracy corresponding to each object satisfies a third condition. Determine whether the conditions are met,
Information indicating that among the objects included in the plurality of images, an object for which the first accuracy does not satisfy the second condition and for which the second accuracy satisfies the third condition has not been learned. Output,
An information output method characterized by causing a computer to perform processing.

Images