JP2022056744A - Annotation device, annotation method and annotation program - Google Patents

Abstract

To provide an annotation device, an annotation method and an annotation program with which it is possible to create training data appropriately.SOLUTION: The annotation device, the annotation method and the annotation program are designed to add annotation information to moving-image data in accordance with operation and create training data that is used for the machine learning of a trained model. In the process of creating training data, the position of an object included in the moving-image is specified and an object label that represents the type of the object is added as annotation information to the moving-image data in accordance with operation, a relationship label that represents the type of an event that the plurality of objects included in the moving-image are correlated is added as annotation information to the moving-image data in accordance with operation, and the object label of the object related to the event which the relationship label is concerned with is specified from the added object labels and added as annotation information to the moving-image data in accordance with operation.SELECTED DRAWING: Figure 25

Description

The present invention relates to an annotation device, an annotation method, and an annotation program.

As a conventional technique for performing various detections from moving images such as a drive recorder using artificial intelligence and deep learning, for example, Patent Document 1 discloses an information processing apparatus. .. This information processing device includes an acquisition unit, a signal area recognition unit, a speed information acquisition unit, an acceleration information acquisition unit, and a determination unit. The acquisition unit acquires a photographed image taken by a photographing device mounted on the vehicle. The signal area recognition unit recognizes a red signal area indicating a red signal of a traffic light from the captured images acquired by the acquisition unit. The speed information acquisition unit acquires speed information indicating the speed of the vehicle. The acceleration information acquisition unit acquires acceleration information indicating the acceleration of the vehicle. The determination unit is based on speed information or acceleration information, a red light region, and predetermined red light ignoring identification information for identifying that the vehicle is driving ignoring the red light. Determines whether or not the vehicle is driving ignoring the red light. In this case, the red light ignoring identification information is created in advance by using a machine learning method using SVM (Support Vector Machine).

Japanese Unexamined Patent Publication No. 2018-072940

By the way, when the information processing apparatus described in Patent Document 1 described above uses a trained model for detecting an event caused by the relationship of a plurality of objects, it is appropriate for the trained model to be machine-learned. Teacher data is required.

The present invention has been made in view of the above circumstances, and an object of the present invention is to provide an annotation device, an annotation method, and an annotation program capable of appropriately creating teacher data.

In order to achieve the above object, the annotation device according to the present invention has a display unit capable of displaying a moving image represented by moving image data, an operation unit that accepts an operation, and the moving image according to an operation on the operation unit. It is provided with a processing unit capable of executing annotation processing for adding annotation information to image data and creating teacher data used for machine learning of a trained model, and the processing unit transfers to the operation unit in the annotation processing. The process of specifying the position of the object included in the moving image according to the operation and adding the object label indicating the type of the object to the moving image data as the annotation information, and the moving image according to the operation to the operation unit. The process of adding a relational label indicating the type of an event in which a plurality of objects included in the above correlates to the moving image data as the annotation information, and the relational label from the object label added according to the operation to the operation unit. It is characterized in that a process of designating the object label of the object related to the target of the event and adding it to the moving image data as the annotation information is executed.

Further, in the annotation device, the processing unit can handle a plurality of predetermined moving image data files as the moving image data representing a series of the moving images in the annotation processing.

In order to achieve the above object, the annotation method according to the present invention includes a step of displaying a moving image represented by moving image data, a step of accepting an operation, and adding annotation information to the moving image data according to the operation. Including the step of creating the teacher data used for machine learning of the trained model, in the step of creating the teacher data, the position of the object included in the moving image is specified according to the operation, and the object representing the type of the object is specified. A label is added to the moving image data as the annotation information, and a relational label indicating the type of an event in which a plurality of objects included in the moving image correlate according to an operation is added to the moving image data as the annotation information. It is characterized in that the object label of the object related to the event targeted by the relational label is designated from the object label added according to the operation and added to the moving image data as the annotation information.

In order to achieve the above object, the annotation program according to the present invention displays the moving image represented by the moving image data, accepts an operation, and adds annotation information to the moving image data according to the operation to obtain a trained model. An object that specifies the position of an object included in the moving image according to the operation and represents the type of the object in the process of creating the teacher data used for machine learning, causing the computer to execute each process and creating the teacher data. A label is added to the moving image data as the annotation information, and a relational label indicating the type of an event in which a plurality of objects included in the moving image correlate according to an operation is added to the moving image data as the annotation information. Each process of designating the object label of the object related to the event targeted by the relation label from the object label added according to the operation and adding it to the moving image data as the annotation information is performed by the computer. It is characterized by having it executed.

The annotation device, annotation method, and annotation program according to the present invention have an effect that teacher data can be appropriately created.

FIG. 1 is a block diagram showing a schematic configuration of an annotation device according to an embodiment. FIG. 2 is a schematic diagram illustrating object detection. FIG. 3 is a schematic diagram illustrating object detection. FIG. 4 is a schematic diagram illustrating behavior detection. FIG. 5 is a schematic diagram illustrating the relationship behavior detection. FIG. 6 is a schematic diagram illustrating relationship behavior detection. FIG. 7 is a schematic diagram illustrating the relationship behavior detection. FIG. 8 is a schematic diagram showing the processing of the learning phase and the use phase. FIG. 9 is a diagram showing an example of a screen displayed on the display device of the annotation device according to the embodiment. FIG. 10 is a diagram showing an example of a screen displayed on the display device of the annotation device according to the embodiment. FIG. 11 is a diagram showing an example of a screen displayed on the display device of the annotation device according to the embodiment. FIG. 12 is a diagram showing an example of a screen displayed on the display device of the annotation device according to the embodiment. FIG. 13 is a diagram showing an example of a screen displayed on the display device of the annotation device according to the embodiment. FIG. 14 is a diagram showing an example of a screen displayed on the display device of the annotation device according to the embodiment. FIG. 15 is a schematic diagram illustrating a moving image data file in the annotation device according to the embodiment. FIG. 16 is a diagram showing an example of a screen displayed on the display device of the annotation device according to the embodiment. FIG. 17 is a diagram showing an example of a screen displayed on the display device of the annotation device according to the embodiment. FIG. 18 is a diagram showing an example of a screen displayed on the display device of the annotation device according to the embodiment. FIG. 19 is a diagram showing an example of a screen displayed on the display device of the annotation device according to the embodiment. FIG. 20 is a diagram showing an example of a screen displayed on the display device of the annotation device according to the embodiment. FIG. 21 is a schematic diagram illustrating a file format in the annotation device according to the embodiment. FIG. 22 is a schematic diagram illustrating a file format in the annotation device according to the embodiment. FIG. 23 is a flowchart illustrating an example of processing in the annotation device according to the embodiment. FIG. 24 is a flowchart illustrating an example of processing in the annotation device according to the embodiment. FIG. 25 is a flowchart illustrating an example of processing in the annotation device according to the embodiment.

Hereinafter, embodiments according to the present invention will be described in detail with reference to the drawings. The present invention is not limited to this embodiment. In addition, the components in the following embodiments include those that can be easily replaced by those skilled in the art, or those that are substantially the same.

[Embodiment]
The annotation device 1 of the present embodiment shown in FIG. 1 constitutes an annotation tool for performing annotation processing on moving image data. Here, the annotation process is a process of adding annotation information to the moving image data to create teacher data D2 (see FIG. 8) used for machine learning of the trained model M (see FIG. 8). The annotation information here is metadata added to the moving image data in order to establish the moving image data as the teacher data D2 in the machine learning of the trained model M.

In the annotation device 1 of the present embodiment, the teacher data D2 created by the annotation process is typically used when a trained model M for relational action detection is generated by machine learning. Here, the relational behavior detection is a detection technique using an image, and detects an event caused by the relation of a plurality of objects.

Here, as the detection technique using an image, in addition to the relational action detection, for example, there are object detection (Object Detection), action detection (Action Detection) and the like.

As shown in FIG. 2, the object detection is to detect the type and position of an object from the still image constituting the moving image (video). Furthermore, object detection is to detect the static position and type of an object at a certain moment from a still image. Objects detected by object detection are, for example, vehicles, passers-by, bicycles, obstacles, street lights, signboards, utility poles, signs, stop lines, and the like. FIG. 2 shows, as an example, a case where a "bicycle" and a "stop line" are detected as an object by detecting an object in a still image, and the positions of the detected "bicycle" and the "stop line" are It is indicated by a rectangular frame. Further, in this object detection, as shown in FIG. 3, it is also possible to detect a change in the position of an object by comparing a plurality of still images constituting a moving image. In FIG. 3, as an example, "bicycle" and "stop line" are detected as objects by object detection in a plurality of still images constituting a moving image, and the position of "bicycle" changes from right to left. Represents the case where it is detected.

On the other hand, as shown in FIG. 4, the behavior detection is the movement (behavior) of an object alone from the temporal change of the position, state, etc. of the object detected by the object detection from a plurality of still images constituting the moving image. It is to detect. In other words, behavior detection is to detect the movement (behavior) of an object with time change. In this case, for example, the fact that the detected object is stationary at that position (in other words, that it is not moving), such as a "stop line", is also an example of the movement (behavior) of the object. It is included in the category of the concept of movement. In the behavior detection, for example, even when a plurality of objects are detected from the still images constituting the moving image, the motion of each object alone is detected. In FIG. 4, as an example, two "bicycles" are detected by object detection in a plurality of still images constituting a moving image, and then the two "bicycles" are "staggered and run" by behavior detection. It represents the case where the movement of the object alone is detected.

In this behavior detection, it is possible to detect the state or change of an object alone (for example, the movement of a "bicycle" swaying and running), and for example, it is possible to predict danger to the driver of a vehicle, but it is detected. It is not possible to detect the meaning of behavior by associating a plurality of objects with each other. Therefore, for example, in the example of FIG. 4, it is determined that the two "bicycles" that are staggering and traveling are both dangerous regardless of their relationship with other objects.

On the other hand, in the above-mentioned relationship behavior detection, the relationship between the objects detected from the moving image is detected, and these movements are related to detect the meaning (type) of the mutual behavior. More specifically, as shown in FIG. 5, the relational behavior detection is a behavior caused by the relation of a plurality of objects from the movement of a single object detected by the behavior detection from a plurality of still images constituting a moving image. (Behavior), it detects the relationship between objects. In other words, the relational behavior detection is to detect the behavior caused by the movement of a plurality of objects with time change. The behavior that occurs in relation to the movement of a plurality of objects can be rephrased as, for example, "an event that occurs in response to a change in the relationship between a plurality of objects". In FIG. 5, as an example, two "bicycles" are detected by object detection in a plurality of still images constituting a moving image, and the two "bicycles" are "staggering and running" by behavior detection. It represents the case where the movement of a single object is detected. Then, in FIG. 5, the "bicycle" on the front side, which is swaying and traveling, is heading toward the traveling direction of the vehicle in response to the movement of the vehicle trying to turn left by the related behavior detection, while the "bicycle" on the back side. Represents the case where the behavior (event) that "is heading in a direction different from the traveling direction of the vehicle" is detected. In the relationship behavior detection, it is possible to judge that it is dangerous for the "bicycle" on the front side that is heading in the direction of travel of the vehicle, but it is swaying but heading in a different direction from the vehicle. It is also possible to judge that it is not dangerous for the "bicycle" on the back side.

As another example, FIG. 6 shows, as a type of behavior (event) caused by the movement of a plurality of objects, for example, “red” from the relationship of movement of “red light”, “stop line”, and “vehicle”. It represents the case where "safe behavior (event) that is not a traffic violation that the vehicle stopped on the stop line" is detected at the traffic light. On the other hand, FIG. 7 shows the types of behaviors (events) that occur in relation to the movements of a plurality of objects. Represents the case where "dangerous behavior (event)", which is a traffic violation that does not stop on the stop line (ignoring the signal), is detected.

The relationship behavior detection based on the above image is performed, for example, by using the trained model M by the detection system Sys1 as shown in FIG. In this detection system Sys1, as shown in FIG. 8, there is a learning phase in which a process of generating a trained model M is performed, and a usage phase in which the above-mentioned relationship behavior detection is actually performed using the trained model M. .. Hereinafter, the outline of the detection system Sys1 will be briefly described.

In the learning phase, the detection system Sys1 acquires a learning data set D3 including input data D1 and teacher data D2 in order to train the trained model M. The training data set D3 is a teacher data set used when the trained model M is generated by machine learning. The learning data set D3 is configured by associating the input data D1 as an explanatory variable and the teacher data D2 as an objective variable corresponding to the input data D1 as a set. Furthermore, the training data set D3 is composed of the input data D1 quantified as an explanatory variable and the teacher data D2 quantified as an objective variable. In the present embodiment, the input data D1 constituting the learning data set D3 is, for example, moving image data representing a moving image around the vehicle taken by a camera mounted on the vehicle. On the other hand, the teacher data D2 constituting the learning data set D3 is data representing the type of behavior (event) caused by the involvement of a plurality of objects in the moving image represented by the moving image data represented by the input data D1. , Created by the annotation device 1 of the present embodiment. The detection system Sys1 can acquire the learning data set D3 from, for example, the learning data providing system (server) Sv (see FIG. 1) described later.

Then, in the learning phase, the detection system Sys1 generates a trained model M by performing machine learning based on various machine learning algorithms AL using a plurality of learning data sets D3 as teacher data sets. The machine learning algorithm AL to be used includes, for example, deep learning, neural network, logistic regression, ensemble learning, support vector machine, and random machine. Known algorithms such as Random Forest) and Naive Bays can be mentioned. The detection system Sys1 performs machine learning of the trained model M with the input data D1 as the explanatory variable and the teacher data D2 as the objective variable in the training data set D3.

In the present embodiment, the detection system Sys1 generates a trained model M used for the relationship behavior detection based on the moving image as described above as a result of the above machine learning. The trained model M is realized by, for example, a neural network. As the neural network, for example, a neural network having an arbitrary configuration such as a CNN (Convolutional Neural Network), an RNN (Recurrent Neural Network), or an LSTM (Long short-term memory) network can be adopted. The detection system Sys1 learns the learning weighting coefficient and the like used as weighting in the neural network by performing machine learning using a plurality of learning data sets D3, and generates the trained model M. The trained model M is not limited to a single model, and may be configured by combining a plurality of models.

In the present embodiment, the trained model M generated by machine learning in the learning phase specifies the type of behavior (event) caused by the involvement of a plurality of objects from the moving image data representing the moving image as described above. Is the model used for. That is, this trained model M is a model in which the input is "moving image data" and the output is "type of behavior (event)". That is, the trained model M is functioned to accept the input of the moving image data and output the type of the behavior (event) caused by the involvement of a plurality of objects from the moving image data.

In the use phase, the detection system Sys1 uses the trained model M generated in the learning phase to specify the type of behavior (event) caused by the involvement of a plurality of objects. In the use phase, the process of specifying the type of the behavior (event) corresponds to the process related to the above-mentioned related behavior detection. In the use phase, the detection system Sys1 inputs the moving image data to be detected as the detection target data (input data) D4 into the trained model M, and the behavior caused by the involvement of a plurality of objects from the moving image data. Output the type of (event). The detection system Sys1 outputs the output value quantifying the type of the above behavior as the type specification result data (output data) D5.

Then, the annotation device 1 of the present embodiment creates the teacher data D2 of the learning data set D3 used for machine learning of the trained model M for detecting the relational behavior in the detection system Sys1 configured as described above. It is a thing. The teacher data D2 created by the annotation device 1 is generated when a plurality of objects included in the moving image are involved as annotation information for being established as an objective variable in machine learning of the trained model M as described above. Data representing the type of behavior (event) is added to the moving image data. In other words, the teacher data D2 can also be said to be relationship tracking data that tracks the relationship between a plurality of objects in an event reflected in a moving image. It can also be said that the annotation device 1 constitutes a relational tracking annotation tool that tracks behaviors (events) caused by the involvement of a plurality of objects in moving image data. Here, the annotation device 1 is realized by various computer devices such as a personal computer, a workstation, and a tablet terminal. Further, the annotation device 1 may be realized by a single computer device or may be realized by a plurality of computer devices. Hereinafter, each configuration of the annotation device 1 will be described in detail with reference to FIGS. 1 and 9 to 14.

Specifically, the annotation device 1 includes a display device 10, an operation device 20, a data input / output device 30, a storage circuit 40, and a processing circuit 50. The display device 10, the operation device 20, the data input / output device 30, the storage circuit 40, and the processing circuit 50 are connected to each other so as to be communicable with each other via a network.

The display device 10 is a display unit capable of displaying a moving image represented by moving image data. The display device 10 is composed of an image display device that outputs and displays various image information such as a liquid crystal display, a plasma display, and an organic EL display.

The operation device 20 is an operation unit that receives various operations by an operator or the like on the annotation device 1. The display device 10 is an operation input device that receives various operation inputs from an operator or the like, such as a mouse, a keyboard, a trackball, a switch, a button, a joystick, a touch pad, a touch screen, a non-contact input circuit, and a voice input circuit. Consists of.

The data input / output device 30 is a data input / output unit that inputs / outputs data (information) to the annotation device 1. The data input / output device 30 accepts data (information) input from another device other than the annotation device 1 and outputs data (information) to the other device. The data input / output device 30 is composed of, for example, a communication interface, a recording medium interface, and the like. The communication interface transmits / receives various data between the annotation device 1 and other devices via communication regardless of whether it is wired or wireless. The recording medium interface includes, for example, a hard disk drive (HDD), a solid state drive (SSD), a flexible disk (FD), a magneto-optical disk (Magnet-Optical disk), a CD-ROM, a DVD, a USB memory, an SD card memory, and a Flash. Reads and writes various data to a recording medium such as a memory.

The data input / output device 30 of the present embodiment can transmit / receive data to / from at least the learning data providing system Sv. The learning data providing system Sv stores, for example, a large number of input data D1, teacher data D2, learning data set D3, etc. in a database, searches for these data as necessary, and stores them in another device. It is a system to provide. Typically, the annotation device 1 inputs moving image data to be annotated from the learning data providing system Sv via the data input / output device 30. Further, the teacher data D2 created by the annotation device 1 is registered, stored and managed in the learning data providing system Sv via the data input / output device 30.

The storage circuit 40 is a circuit for storing various data. The storage circuit 40 is realized by, for example, a RAM (Random Access Memory), a semiconductor memory element such as a flash memory, a hard disk, an optical disk, or the like. The storage circuit 40 stores, for example, a program for the annotation device 1 to realize various functions. The program stored in the storage circuit 40 includes a program for functioning the display device 10, a program for functioning the operating device 20, a program for functioning the data input / output device 30, a program for functioning the processing circuit 50, and the like. Further, the storage circuit 40 stores various data such as moving image data input via the data input / output device 30, data necessary for various processing in the processing circuit 50, and teacher data D2 created in the annotation device 1. do. In the storage circuit 40, these various data are read out as needed by the processing circuit 50 and the like. The storage circuit 40 may be realized by a cloud server or the like connected to the annotation device 1 via a network.

The processing circuit 50 is a processing unit that constitutes a circuit that realizes various processing functions in the annotation device 1. The processing circuit 50 is realized by, for example, a processor. The processor is, for example, a CPU (Central Processing Unit), an MPU (Micro Processing Unit), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Circuit), or the like. The processing circuit 50 realizes each processing function by, for example, executing a program read from the storage circuit 40.

The outline of the overall configuration of the annotation device 1 according to the present embodiment has been described above. Under such a configuration, the processing circuit 50 according to the present embodiment adds annotation information to the moving image data according to the operation to the operating device 20, and uses the teacher data D2 for machine learning of the trained model M. It has a function to enable the annotation processing to be created.

Specifically, the processing circuit 50 of the present embodiment has a display processing unit 51, an operation processing unit 52, a task creation processing unit 54, and an annotation processing unit in terms of functional concepts in order to realize the various processing functions. Consists of 55. The processing circuit 50 may execute, for example, a program read from the storage circuit 40 to execute the display processing unit 51, the operation processing unit 52, the data input / output processing unit 53, the task creation processing unit 54, and the annotation processing unit 55. Realize each processing function of.

The display processing unit 51 is a part having a function of controlling the display device 10 and executing various processes for displaying a moving image or the like by the display device 10.

The operation processing unit 52 is a part having a function of controlling the operation device 20 and executing various processes for receiving the operation by the operation device 20.

The data input / output processing unit 53 has a function of controlling the data input / output device 30 and executing various processes for transmitting / receiving data between the annotation device 1 and another device by the data input / output device 30. It is a part.

The task creation processing unit 54 is a part having a function capable of executing various processes for creating a series of tasks to be processed by the annotation processing unit 55.

The annotation processing unit 55 is a part having a function capable of executing various processes for annotation processing for creating teacher data D2 by adding annotation information to moving image data. The annotation processing unit 55 executes, for example, annotation processing corresponding to a series of tasks created by the task creation processing unit 54.

More specifically, as shown in FIGS. 9 and 10, the annotation processing unit 55 may operate the operation device 20 by an administrator, a worker, or the like (hereinafter, simply abbreviated as “operation”) in the annotation processing. There is.) It is possible to execute the process of adding the object label to the moving image data as annotation information. The object label added to the moving image data as annotation information is metadata indicating the type of the object by specifying the position of the object included in the moving image represented by the moving image data.

Further, as shown in FIGS. 11 and 12, the annotation processing unit 55 can execute a process of adding a relational label as annotation information to the moving image data according to the operation in the annotation process. The relationship label added to the moving image data as annotation information is metadata representing the type of event in which a plurality of objects included in the moving image represented by the moving image data correlate.

Then, as shown in FIGS. 13 and 14, the annotation processing unit 55 is an object related to the event targeted by the relation label from the object label added as described above according to the operation in the annotation processing. It is possible to specify the object label of the above and add it to the moving image data as annotation information. In other words, the annotation processing unit 55 can execute a process of designating an object related to the event specified by the relation label from the object specified by the object label and adding it to the moving image data as annotation information.

As a result of these processes, the annotation processing unit 55 can add annotation information to the moving image data to create the teacher data D2. As described above, the object label and the relationship label added as the annotation information can also be said to record the relationship tracking information that traces the relationship of each object reflected in the moving image represented by the moving image data. The annotation processing unit 55 stores the created teacher data D2 in the storage circuit 40. Then, the teacher data D2 created by the annotation processing unit 55 is transmitted to the learning data providing system Sv via the data input / output device 30 by the processing of the data input / output processing unit 53, and is registered, stored, and managed. Hereinafter, an example of each of the above processes in the annotation process will be described more specifically with reference to each figure.

9 and 10 schematically show an example of the annotation screen 100A displayed on the display device 10 when an object label is added to the moving image data. Here, FIG. 9 shows the entire annotation screen 100A, and FIG. 10 shows the transition of the display in a part of the annotation screen 100A.

The display processing unit 51 causes the display device 10 to display the annotation screen 100A as shown in FIG. 9 when performing an operation of adding an object label to the moving image data. The annotation processing unit 55 executes a process of adding an object label to the moving image data according to an operation via the annotation screen 100A.

On the annotation screen 100A exemplified in FIG. 9, a moving image display area 101, a label addition operation area 102, a label selection area 103, and a task end operation area 104 are displayed together with various information.

The moving image display area 101 is an area for displaying a moving image of moving image data to be annotated. The moving image display area 101 typically displays a moving image of moving image data corresponding to a series of tasks created by the task creation processing unit 54. The moving image display area 101 can typically display each frame constituting the moving image as a still image. The moving image display area 101 may reproduce the moving image to be displayed in response to the operation of the moving image operation area 101a displayed on the annotation screen 100A, and sequentially transition the frames displayed in the moving image display area 101. can. The annotation processing unit 55 can sequentially annotate each frame according to the operation by sequentially transitioning the frames displayed in the moving image display area 101.

The label addition operation area 102 is an area operated when a label is added to the moving image displayed in the moving image display area 101.

The label selection area 103 is an area operated when selecting the type of label to be added to the moving image. In other words, the label displayed in the label selection area 103 corresponds to a label candidate that can be selected in the annotation process. The label that can be selected in the annotation process may be defined by, for example, a label definition file read into the annotation device 1, or may be appropriately set according to the operation. Typically, the labels that can be selected in the annotation process are fixed until the end of a series of tasks for adding labels to the moving image data.

The task end operation area 104 is an area operated when a series of tasks for adding a label to moving image data is completed.

When the label addition operation area 102 is selected in response to the operation on the annotation screen 100A, the annotation processing unit 55 has a rectangular frame (circular frame, multiple) for designating a position on the moving image displayed in the moving image display area 101. (It may be a rectangular frame, etc.) is added. Then, the annotation processing unit 55 specifies the position of the object by designating the position of the object included in the moving image by the rectangular frame according to the operation.

Then, the annotation processing unit 55 selects one of the labels from the label list displayed in the label selection area 103 according to the operation, so that the object representing the type of the object whose position is specified above is represented. Determine the label. Then, the annotation processing unit 55 adds an object label representing the determined type of the object to the moving image data as annotation information.

The label selection area 103 illustrated in FIG. 9 displays the main list 103A representing selectable labels (candidate labels). As an example, the main list 103A illustrated in FIG. 9 displays a list of labels representing "relationship", "traffic light", "speed sign", "pause sign", and the like. The "traffic light", "speed sign", and "pause sign" in the main list 103A represent the object label, while the "relationship" represents the relationship label described later. When adding an object label to moving image data, the annotation device 1 typically includes one of object labels such as a "traffic light", a "speed sign", and a "stop sign" depending on the operation. One label is selected.

Here, as illustrated in FIG. 10, the display processing unit 51 is selected when any one object label is selected from the main list 103A displayed in the label selection area 103 according to the operation. The sublist 103B is displayed in the label selection area 103 according to the type of the object label. Sublist 103B is a label list for selecting more detailed attributes that further subdivide the selected object label type. In the example of FIG. 10, when the object label of the "traffic light" is selected according to the operation, the display processing unit 51 selects the label of the sublist 103B representing the attributes such as "blue", "yellow", and "red". It is displayed in the area 103.

The annotation processing unit 55 selects one of the attributes from the sublist 103B displayed in the label selection area 103 according to the operation, so that the type (including the attribute) of the object whose position is specified above is selected. ) Is determined. At this time, the annotation processing unit 55 stores the frame in which the object label is determined as described above as a start key frame (first frame) in which the object targeted by the object label is displayed. To memorize. Here, the object targeted by the object label determined as described above corresponds to, in other words, the object specified by the object label.

If a more detailed attribute can be selected according to the type of the attribute selected above, the display processing unit 51 displays a sublist for selecting the detailed attribute in the label selection area 103. It is also possible.

Then, when the object label is determined as described above, the display processing unit 51 annotates the erase operation area 106 and the end operation area 107 together with the determination label display image 105, for example, as illustrated in FIG. Display on screen 100A.

The determination label display image 105 is an image showing the type of the object label determined as described above. For example, the annotation processing unit 55 may be able to reselect the object label when the determination label display image 105 is selected according to the operation.

The erasing operation area 106 is an area operated when the object label determined as described above is once deleted. When the erasing operation area 106 is selected according to the operation, the display processing unit 51 displays the erasing decision image 108 on the annotation screen 100A. When "OK" is selected in response to the operation in the deletion decision image 108, the annotation processing unit 55 temporarily erases the object label determined as described above. On the other hand, when "Cancel" is selected in response to the operation in the erasure decision image 108, the display processing unit 51 hides the erasure decision image 108 and returns it to the original display screen.

The end operation area 107 is an area operated when determining the final frame in which the object targeted by the object label determined as described above appears. In other words, the end operation area 107 is an area for performing an operation for specifying the final frame of the object label by being selected by the end frame in which the object of the object label is reflected.

For example, the display processing unit 51 sequentially shifts the frames of the moving image displayed in the moving image display area 101 according to the operation, and sets the final frame in which the object targeted by the label is displayed in the moving image display area 101. To display. During this time, if the position of the object targeted by the label changes, the annotation processing unit 55 corrects the position of the object specified by the rectangular frame in response to the change in the position of the object according to the operation. .. At this time, the annotation processing unit 55 sets the position of the object in each frame between the frame in which the position of the object is previously specified and the frame in which the position of the object is specified again, and the position of the object in the previous frame. Calculated and specified by linear interpolation based on the position and the position of the object in the subsequent frame. That is, the annotation processing unit 55 calculates the position of the object in each frame between them based on the position of the object specified in the previous frame and the position of the object redesignated in this frame by linear interpolation or the like. Identify. As a result, the annotation processing unit 55 tracks the position change of the object targeted by the object label.

Then, when the end operation area 107 is selected according to the operation, the annotation processing unit 55 stores the frame as an end key frame (final frame) in which the object specified by the object label is reflected. Store in circuit 40. As a result, the annotation processing unit 55 can determine the time range of the object label.

As described above, the annotation processing unit 55 determines the position of the object and the object label according to the type of the object for the object to be annotated according to the operation, and determines the object label according to the type of the object. By specifying the start key frame and the end key frame of, the time range of the object label in this moving image can be determined.

At this time, as illustrated in FIG. 10, the display processing unit 51 changes the display mode of the end operation area 107 so as to indicate that the end operation area 107 has been selected (for example, “End?” Is changed to “End?”. It is set to "End" and grayed out), and the erase operation area 106 is hidden. Here, the annotation processing unit 55 can temporarily cancel the above-mentioned processing for determining the end key frame by, for example, reselecting the end operation area 107 after changing the display mode according to the operation. When the end key frame is confirmed, the display processing unit 51 hides the decision label display image 105, the end operation area 107, and the like, and the annotation processing unit 55 shifts to the annotation processing of the next frame.

The annotation processing unit 55 can add the object label determined for each object reflected in the moving image displayed in the moving image display area 101 to the moving image data as annotation information as described above. At this time, the annotation processing unit 55 also adds a coordinate value or the like representing the position of the object specified by the rectangular frame as the information of the object label. An unlimited number of object labels can be added instead of one.

Next, FIGS. 11 and 12 schematically show an example of the annotation screen 100B displayed on the display device 10 when a related label is added to the moving image data. Again, FIG. 11 shows the entire annotation screen 100B, and FIG. 12 shows the transition of the display in a part of the annotation screen 100A.

The display processing unit 51 causes the display device 10 to display the annotation screen 100B as shown in FIG. 11 when performing an operation of adding the related label to the moving image data. The annotation processing unit 55 executes a process of adding a relational label to the moving image data according to an operation via the annotation screen 100B.

Similar to the annotation screen 100A, the annotation screen 100B illustrated in FIG. 11 displays a moving image display area 101, a label addition operation area 102, a label selection area 103, a task end operation area 104, and the like. Then, in addition to these, the added label display area 109 is displayed on the annotation screen 100B.

The added label display area 109 is an area for displaying a list of the types of object labels already added as described above in the moving image frame displayed in the moving image display area 101.

In this case, the display processing unit 51 causes the moving image display area 101 to display a frame at which an event (behavior) in which the target object of the object label added as described above starts to correlate with each other according to the operation starts. Then, the annotation processing unit 55 starts creating the relation label from the frame at which the event starts. In this state, when the label addition operation area 102 is selected in response to the operation on the annotation screen 100B, the annotation processing unit 55 adds the display of the rectangular frame to the moving image displayed in the moving image display area 101. When the annotation processing unit 55 adds a relational label to the moving image data, the entire frame of the moving image is designated by the rectangular frame according to the operation.

Then, the annotation processing unit 55 shifts to the selection of the type of the relation label by selecting the relation label representing the "relationship" from the label list displayed in the label selection area 103 according to the operation.

As illustrated in FIG. 12, when the relationship label representing "relationship" is selected from the main list 103A displayed in the label selection area 103 according to the operation, the display processing unit 51 responds to the type of the relationship label. The sublist 103C is displayed in the label selection area 103. Sublist 103C is a label list for selecting the type of relational label, in other words, the type of event (behavior) in which a plurality of objects correlate. In the example of FIG. 12, when the relationship label of "relationship" is selected according to the operation, the display processing unit 51 represents the type of the relationship label such as "safety", "caution", and "violation". Is displayed in the label selection area 103.

Further, here, as illustrated in FIG. 12, the display processing unit 51 selects one of the related label types from the sublist 103C displayed in the label selection area 103 according to the operation. The sublist 103D is displayed in the label selection area 103 according to the type of the selected relational label. The sublist 103D is a label list for selecting more detailed attributes by further subdividing the selected relation label type. In the example of FIG. 12, when the relational label of "violation" is selected according to the operation, the display processing unit 51 selects the label of the sublist 103D representing the attributes such as "ignore traffic light" and "ignore stop line". It is displayed in the area 103.

The annotation processing unit 55 selects one of the attributes from the sublist 103D displayed in the label selection area 103 according to the operation, so that the event (multiple objects) that occur in the current frame is recognized. Determine the relationship label that represents the type (including attributes) of the correlated event). At this time, the annotation processing unit 55 stores the current frame in which the relational label is determined as described above as a start key frame (first frame) in which the event targeted by the relational label is displayed. To memorize. Here, the event targeted by the relationship label determined as described above corresponds to, in other words, the event specified by the relationship label.

Then, when the relational label is determined as described above, the display processing unit 51 annotates the erase operation area 106 and the end operation area 107, for example, as illustrated in FIG. 12, as in FIG. Display on screen 100B.

The erase operation area 106 is an area operated when the relational label determined as described above is temporarily deleted, as described above. When the erasing operation area 106 is selected according to the operation, the display processing unit 51 displays the erasing decision image 108 on the annotation screen 100B. When "OK" is selected in response to the operation in the deletion decision image 108, the annotation processing unit 55 temporarily erases the relation label determined as described above. On the other hand, when "Cancel" is selected in response to the operation in the erasure decision image 108, the display processing unit 51 hides the erasure decision image 108 and returns it to the original display screen.

Similar to the above, the end operation area 107 is an area operated when determining the final frame in which the event targeted by the relationship label determined as described above appears. In other words, the end operation area 107 is an area for performing an operation for specifying the final frame of the relation label by being selected in the end frame in which the target of the relation label is reflected.

For example, the display processing unit 51 sequentially shifts the frames of the moving image displayed in the moving image display area 101 according to the operation, and sets the final frame in which the event targeted by the label is displayed in the moving image display area 101. To display.

Then, when the end operation area 107 is selected according to the operation, the annotation processing unit 55 stores the frame as an end key frame (final frame) in which the event specified by the relational label is reflected. Store in circuit 40. As a result, the annotation processing unit 55 can determine the time range of the relation label.

As described above, the annotation processing unit 55 determines the relational label according to the type of the event for the event to be annotated according to the operation, and the start key frame and the end key of the relational label. By specifying the frame, the time range of the relation label in this moving image can be determined.

At this time, similarly to the above, the display processing unit 51 changes the display mode of the end operation area 107 so as to indicate that the end operation area 107 has been selected, and the erase operation area 107, as illustrated in FIG. 106 is hidden. Here, too, the annotation processing unit 55 can temporarily cancel the above-mentioned processing for determining the end key frame by, for example, reselecting the end operation area 107 after the display mode is changed according to the operation. When the end key frame is confirmed, the display processing unit 51 hides the end operation area 107 and the like, and the annotation processing unit 55 shifts to the annotation processing of the next frame.

The annotation processing unit 55 can add the relational label determined for each event of the moving image displayed in the moving image display area 101 as the annotation information to the moving image data as described above. The number of relation labels may be limited to one and not one.

Next, FIGS. 13 and 14 schematically show an example of the annotation screen 100C displayed on the display device 10 when designating the object label of the object related to the event targeted by the relation label from the added object label. It is represented by. Again, FIG. 13 shows the entire annotation screen 100C, and FIG. 14 shows the transition of the display in a part of the annotation screen 100C.

The display processing unit 51 displays the annotation screen 100C as shown in FIG. 13 when performing an operation of adding annotation information for designating an object label of an object related to an event targeted by the relation label to the moving image data. Display on 10. The annotation processing unit 55 executes a process of adding annotation information that specifies an object related to an event to the moving image data in response to an operation via the annotation screen 100C.

Similar to the annotation screens 100A and 100B, the annotation screen 100C illustrated in FIG. 13 displays a moving image display area 101, a label addition operation area 102, a label selection area 103, a task end operation area 104, and the like. Then, in addition to these, the annotation screen 100B displays the slide bar display area 110.

The slide bar display area 110 is an area in which the time range in the moving image of each of the object label and the relational label added and determined as described above is displayed by the slide bar. In addition, the annotation processing unit 55 can modify the start key frame and the end key frame of the time range in each label after the fact by changing the start end position and the end position of the slide bar according to the operation. ..

When the start key frame and the end key frame of the relation label are determined as described above and the time range of the relation label in the target moving image is determined, the annotation processing unit 55 includes even one frame in the time range. Search and extract object labels. Then, the display processing unit 51 displays the time range of all the object labels extracted by the annotation processing unit 55 and the time range of the relational labels in the slide bar display area 110 with different slide bars.

In the example of FIG. 13, the display processing unit 51 displays the type (name) of the object corresponding to each object label at the left end of each slide bar in the slide bar display area 110. Here, when there are a plurality of object labels of the same type, the display processing unit 51 adds information for distinguishing them from each other, such as adding a serial number to the display name, and displays the label. You may. Further, the display processing unit 51 may display each slide bar in the slide bar display area 110 and the rectangular frame on the moving image in the moving image display area 101 by color coding according to each object label. Further, the display processing unit 51 may display information about the corresponding object label in a pop-up by moving the pointer to the slide bar or the rectangular frame according to the operation.

Then, the annotation processing unit 55 selects a relational label representing "relationship" from the label list displayed in the label selection area 103 according to the operation, so that the object related to the event targeted by the relational label is selected. Move to the selection of.

As illustrated in FIG. 14, the display processing unit 51 selects the sublist 103E as a label when the relation label representing the "relationship" is selected from the main list 103A displayed in the label selection area 103 according to the operation. It is displayed in the area 103. Sublist 103E is a label list of object labels that are candidates for objects related to the event targeted by the relation label. Here, the sublist 103E is a label list of object labels (candidate labels) extracted as labels included in the time range of the relational label by the annotation processing unit 55 as described above. In other words, the sublist 103E is a label list of object labels that are included in at least one frame in the time range of the relational label. In the example of FIG. 14, when the relationship label of "relationship" is selected according to the operation, the display processing unit 51 displays an object label such as "traffic light", "pause line 1", and "pause line 2". The sublist 103E to be represented is displayed in the label selection area 103.

The annotation processing unit 55 selects any object label from the sublist 103E displayed in the label selection area 103 according to the operation, so that the object label already added is targeted by the related label. Determine the object label of the object involved in the event. For example, for the event of "ignoring the traffic light", the annotation processing unit 55 selects an object label representing the "traffic light" and an object label representing the "stop line 1" in the lane of the target vehicle according to the operation. On the other hand, the object label representing "stop line 2" in the opposite lane of the target vehicle's lane is not selected. The annotation processing unit 55 can determine a plurality of object labels of the object related to the event specified by the relation label according to the operation.

Then, when the object label of the object related to the event targeted by the relation label is determined as described above, the display processing unit 51, for example, as illustrated in FIG. 14, in the same manner as above. The erase operation area 106 and the end operation area 107 are displayed on the annotation screen 100C.

Similarly to the above, the erasing operation area 106 is an area operated when the object label determined as described above is temporarily deleted. When the erasing operation area 106 is selected according to the operation, the display processing unit 51 displays the erasing decision image 108 on the annotation screen 100C. When "OK" is selected in response to the operation in the deletion decision image 108, the annotation processing unit 55 temporarily erases the object label determined as described above. On the other hand, when "Cancel" is selected in response to the operation in the erasure decision image 108, the display processing unit 51 hides the erasure decision image 108 and returns it to the original display screen.

The end operation area 107 is an area operated when the determination of the object label of the object related to the event specified by the relation label as described above is completed. When the end operation area 107 is selected according to the operation, the annotation processing unit 55 ends the determination of the object label of the object related to the event specified by the relation label.

At this time, similarly to the above, the display processing unit 51 changes the display mode of the end operation area 107 so as to indicate that the end operation area 107 has been selected, and the erase operation area 107, as illustrated in FIG. 106 is hidden.

The annotation processing unit 55 can specify the object label of the object related to the event targeted by the relation label from the object label added as described above and add it to the moving image data as annotation information.

As described above, the annotation processing unit 55 adds various annotation information to the moving image data to create the teacher data D2, and stores the created teacher data D2 in the storage circuit 40. The teacher data D2 is, for example, a "path that can identify a moving image data file" corresponding to each label, an object to which a label is attached, and a "start key frame (number of the first confirmed frame)" that shows an event. It contains information such as "end key frame (the number of the last confirmed frame)". Then, when the series of tasks for adding a label to the moving image data is completed, the annotation processing unit 55 ends the task by selecting the task end operation area 104 according to the operation.

Here, the size (data amount) of the moving image data file (moving image data file) that is the target of the annotation processing as described above tends to be relatively large. For this reason, as a moving image data file, it is generally performed to divide a series of moving images into a file for a certain period of time or at a certain size and save the file. In this case, for example, in an existing annotation device, a label is created by annotation processing for one divided file, and one annotation file (teacher data) is saved as a work result for the one divided file. I have something to do.

Here, for example, when an existing annotation device performs a process of adding an object label to an object appearing in a moving image of a certain divided file in a divided divided file group, it is a target of processing. The object may not be completely reflected in the moving image of the divided file, and the object may be continuously reflected in the moving image of the next divided file in chronological order.

At this time, for example, the worker closes the split file currently being processed on the existing annotation device, opens the next split file, and continues the work, but the object label added earlier is the moving image of the next split file. However, it may be necessary to reset it so that it will be handled continuously from the moving image of the previous split file. In this case, for example, if there is even a slight difference in the settings, an error may occur in the teacher data, and as a result, the quality of the teacher data may be deteriorated.

In addition, when a series of moving images is made into a plurality of divided files as described above, a plurality of annotation files (teacher data) are saved for a plurality of divided files even though the series of moving images is a series of moving images. There is. Some existing annotation devices have a function of inputting arbitrary text such as remarks to an annotation file (teacher data), and an ID number or the like can be input to the arbitrary item. There are also things. In the existing annotation device having such a function, for example, the same ID number may be set for the same object divided into different divided files. Then, for example, by using a conversion tool having a function of extracting a series of labels from a plurality of annotation files based on the ID number and concatenating and saving the series of labels as one annotation file, a series of moving images can be obtained. Even if the image is divided into a plurality of divided files, there are some that can be converted so that the label for the same object becomes one annotation file (teacher data).

However, creating such a conversion tool is cumbersome. Further, the ID number indicating that the label is a series of labels needs to be separately recorded by an operator or the like. The total number of labels often exceeds, for example, thousands or tens of thousands, and such work is very cumbersome, error-prone, and can also reduce the quality of teacher data. There is.

Therefore, in the annotation processing as described above, the annotation processing unit 55 of the present embodiment handles a plurality of predetermined moving image data files as moving image data representing a series of moving images. Then, the annotation processing unit 55 typically creates one set of teacher data D2 (annotation file) for a plurality of moving image data files handled as moving image data representing a series of moving images in this way. Can be configured to. Here, the task creation processing unit 54 creates a series of tasks to be processed by the annotation processing unit 55 so that a plurality of predetermined moving image data files are treated as moving image data representing a series of moving images. Hereinafter, a specific description will be given.

First, as illustrated in FIG. 15, the task creation processing unit 54 of the present embodiment stores a plurality of moving image data files that are a series of moving images in the designated file storage area 40a of the storage circuit 40. Create a list file that records the data in chronological order. Here, the creation of this list file matches the search conditions when, for example, a search command in which search conditions are set is sent to a data infrastructure (for example, a learning data providing system Sv or the like) in which big data is stored. A data infrastructure that automatically outputs a list file can be used, but the present invention is not limited to this, and the list file may be manually created. Further, the task creation processing unit 54 may set file names that are serial numbers in chronological order for a plurality of moving image data files that are a series of moving images.

The annotation device 1 of the present embodiment implements a management mode using the management mode screen 200 as illustrated in FIG. 16 in addition to the work mode using the annotation screens 100A, 100B, 100C and the like described above. There is. In addition, this annotation device 1 can set user authority such as "administrator" who can execute both the work mode and the management mode and "worker" who can execute only the work mode. It may have a function. Further, in the annotation device 1, the device for the "administrator" and the device for the "worker" may be configured separately.

The management mode screen 200 illustrated in FIG. 16 is a screen displayed on the display device 10 in the management mode, and the task creation operation area 201 is displayed. The task creation operation area 201 is an area operated when creating a series of tasks for adding a label to moving image data. Here, the management mode screen 200 shows only the task creation operation area 201, and the illustration of other management operation areas is omitted.

When the task creation operation area 201 is selected according to the operation, the display processing unit 51 causes the display device 10 to display the task creation screen 300 as illustrated in FIG. The task creation screen 300 illustrated in FIG. 17 is a screen displayed on the display device 10 when creating a series of tasks for adding a label to moving image data. Furthermore, the task creation screen 300 is a screen displayed when a plurality of moving image data files to be handled as moving image data representing a series of moving images are specified in the annotation process. The display processing unit 51 displays a file dialog on the task creation screen 300 to display a moving image data file to be handled as a series of tasks, in other words, a moving image data file to be handled as moving image data representing a series of moving images. Let me select.

The task creation processing unit 54 specifies, for example, a plurality of moving image data files to be handled as moving image data representing a series of moving images by selecting the above list file according to an operation via this file dialog. do. Alternatively, the task creation processing unit 54 may, for example, select a plurality of moving image data files in a series of serial number file names when "Open" in the file dialog is selected in response to an operation when no file is selected. Recognizes that is set to be the saved directory.

In this way, the task creation processing unit 54 creates a series of tasks so that the plurality of moving image data files specified here are treated as moving image data representing a series of moving images. As a result, in the annotation processing, the annotation processing unit 55 handles the plurality of moving image data files specified here as moving image data representing a series of moving images. As a result, the annotation device 1 sets a series of the plurality of moving image data files specified here in the moving image display area 101 of the annotation screens 100A, 100B, 100C even if the plurality of divided moving image data files are divided. It can be continuously reproduced like the moving image of the moving image data of.

Here, the annotation device 1 only handles a plurality of moving image data files designated as described above as a series of moving image data, and the plurality of moving image data files are combined into one moving image data file. It does not convert. That is, the annotation device 1 stores a series of a plurality of moving image data files as a list.

Further, on the annotation screens 100A, 100B, 100C described in FIGS. 9, 11, and 13 described above, the file name display area 111, the file selection area 112, and the like are displayed together with the moving image display area 101 and the like. The file name display area 111 is an area for displaying the file name of the moving image data file corresponding to the moving image displayed in the moving image display area 101. The file selection area 112 is an area operated when selecting a moving image data file corresponding to the moving image displayed in the moving image display area 101.

For example, when the display processing unit 51 selects "Next" in the file selection area 112 according to the operation, the display processing unit 51 is the next moving image of the moving image data file whose file name is currently displayed in the file name display area 111. The moving image of the data file is displayed in the moving image display area 101 from the beginning. Similarly, when the display processing unit 51 selects "Prev" in the file selection area 112 according to the operation, the display processing unit 51 currently displays the video in front of the moving image data file whose file name is displayed in the file name display area 111. The moving image of the image data file is displayed in the moving image display area 101 from the beginning.

Further, for example, when the file name display area 111 is selected according to the operation, the display processing unit 51 can display the file list 111A in the file name display area 111 as illustrated in FIG. The file list 111A is a file name list of a plurality of moving image data files designated as a series of tasks as described above. For example, in the file list 111A displayed in the file name display area 111, the display processing unit 51 sets the character color of the file name of the moving image data file that has already been played back to gray, and the file name of the moving image data file that is currently being played back. The text color of the file name of the moving image data file that has not been played back can be displayed differently, such as black and red. Further, when another moving image data file is selected according to the operation, the display processing unit 51 can display the moving image of the selected moving image data file in the moving image display area 101 from the beginning.

In this way, the annotation device 1 has a function of arbitrarily designating a moving image data file such as a worker or an administrator and reproducing the moving image. For example, at the final time of the moving image data file currently being worked on. If there is a work continuation state label for which an end key frame has not been set, it has a function to remember that the work of adding the label has not been completed. For example, when the display processing unit 51 has a label in the work continuation state as described above, when the moving image of the moving image data file after the moving image data file is being reproduced, the display processing unit 51 may use the label selection area 103 or the like. The label of the work continuation state may be displayed in the added label display area 109. Further, for example, the display processing unit 51 displays the points at the coordinates using the coordinate values of the rectangular frame representing the position of the object specified by the object label, and connects the points with a line. It may be displayed in. In this way, the annotation device 1 can also store that the object label is an object label for the same object even if the moving images of a plurality of different moving image data files are reproduced.

Then, when the series of tasks by the plurality of moving image data files designated as described above is completed, the annotation processing unit 55 responds to the operation by the task end operation area 104 (see FIGS. 9, 11, and 13). ) Is selected to end the task. On the other hand, the annotation processing unit 55 annotates when the task end operation area 104 or the screen close button is selected according to the operation while the series of tasks has not been completed and the label of the work continuation state is present. Start the work end process. At this time, if there is a label in the work continuation state, the display processing unit 51 may display the work end confirmation image 113 as illustrated in FIG. 19 on the annotation screens 100A, 100B, 100C. The work end confirmation image 113 is an image for confirming the end of work, and displays, for example, a character image such as "When the work is completed, the work data of the label in the continuation state is erased, but is it still completed?" There is. When "OK" is selected in response to the operation in the work completion confirmation image 113, the annotation processing unit 55 can perform processing so as to erase the work data of the label in the work continuation state and end the work. On the other hand, when "Cancel" is selected in response to the operation in the work completion confirmation image 113, the display processing unit 51 hides the work completion confirmation image 113 and returns it to the original display screen.

Further, even if there is no label for the work continuation state, the display processing unit 51 once selects the task end operation area 104 according to the operation, and the work end determination image as illustrated in FIG. 20 once. 114 may be displayed on the annotation screens 100A, 100B, 100C. The work end determination image 114 is an image for determining the end of work, for example, is the task completed? A character image such as "" is displayed. The annotation processing unit 55 ends the task when "OK" is selected according to the operation in the work end determination image 114. On the other hand, when "Cancel" is selected in response to the operation in the work end determination image 114, the display processing unit 51 hides the work end determination image 114 and returns it to the original display screen.

Here, in the plurality of moving image data files handled as moving image data representing a series of moving images as described above, the object reflected in a certain moving image data file is the next object in the annotation processing unit 55 of the present embodiment. Even when it is continuously reflected in the moving image data file, for example, as illustrated in FIG. 21, it has a function of saving as one teacher data D2 (annotation file) in which information about one object label is recorded.

Further, in the relational label, one object label is linked with another object label. Regarding the relational label, for example, as illustrated in FIG. 22, the annotation processing unit 55 of the present embodiment provides a key for recording the ID of the related label in one of the header information, and sets the value of the key as the value of the key. It may have a function of saving as teacher data D2 (annotation file) by a format for recording the ID number of the object label related to the relation label.

The teacher data D2 (annotation file) illustrated in FIGS. 21 and 22 is exemplified as a text file format, but is like JSON (JavaScript (registered trademark) Object Notification) or XML (Extensible Markup Language). It may be semi-structured data whose format is defined by a data description language. In this case, the annotation processing unit 55, for example, in the teacher data D2 (annotation file) of the relation label, the information of the object label related to the relation label is not only the ID number but also the time when the object label is related to the relation label. The range etc. can also be recorded. Here, JSON is not only used only in the programming language Javascript (registered trademark), but is also used as one of the general-purpose data exchange formats like XML.

Next, a series of processing procedures from task creation to annotation work will be described with reference to the flowcharts of FIGS. 23, 24, and 25. In the method described below, various programs are executed by the processing circuit 50 of the annotation device 1 according to the operation, so that the processing related to each step is executed.

First, with reference to FIG. 23, each process in the method of creating the task provided for the annotation work will be described. The method of creating this task is typically performed mainly by the annotation device 1 that constitutes the device for the administrator.

First, the data input / output processing unit 53 of the processing circuit 50 makes a communication connection to the annotation server according to, for example, an operation of the administrator (step S1). The annotation server is configured by, for example, the above-mentioned learning data providing system Sv or the like. Then, the task creation processing unit 54 of the processing circuit 50 inputs the search condition of the moving image data for which the teacher data is to be created according to the operation of the administrator (step S2), and the data input / output processing unit 53 sets the data input / output processing unit 53. The input search condition is transmitted to the annotation server (step S3). As the search condition, the task creation processing unit 54 can specify, for example, the date and time of the target moving image data, the vehicle, and the like.

Then, the data input / output processing unit 53 receives the search result searched according to the search conditions on the annotation server (step S4), and the task creation processing unit 54 determines whether or not data matching the search conditions exists. Is determined (step S5). When it is determined that the data matching the search condition does not exist (step S5: Yes), the task creation processing unit 54 returns to the processing of step S1 and repeatedly executes the subsequent processing. When the task creation processing unit 54 determines that the data matching the search conditions exists (step S5: No), the display processing unit 51 of the processing circuit 50 displays, for example, the search result on the display device 10. The search result is output to the administrator (step S6).

Next, the task creation processing unit 54 sets data conditions for allocating moving image data files corresponding to the search results to form a series of work tasks according to the operation of the administrator (step S7), and the data conditions. Generate a work task according to (step S8). Then, the display processing unit 51 outputs the information of the work task to the administrator by displaying the generated work task information on the display device 10, for example (step S9).

Next, the task creation processing unit 54 assigns each work task generated in response to the operation of the administrator to each worker (step S10), and then, for example, the work assigned to the device or the like for each worker. The task information is notified (step S11), and the process according to this flowchart is terminated.

Next, with reference to FIG. 24, each process in the execution method of the task provided as the annotation work will be described. The method of executing this task is typically performed mainly by the annotation device 1 that constitutes the device for the worker.

The annotation processing unit 55 starts, for example, a work task assigned to the worker according to the operation of the worker (step S101), and the data input / output processing unit 53 uses data conditions attached to the assigned work task. To the annotation server (step S102).

Then, the data input / output processing unit 53 receives the moving image data file matching the data condition from the annotation server (step S103), and stores it in the designated file storage area 40a or the like of the storage circuit 40.

Then, the annotation processing unit 55 handles the moving image data file received in response to the operator's operation as moving image data representing a series of moving images, and executes the annotation work (step S104).

Here, the display processing unit 51 displays a moving image on the display device 10 according to the operation of the worker and reproduces it (step S105), and the annotation processing unit 55 is the target of the annotation processing according to the operation of the worker. It is determined whether or not an object or an event is detected (step S106). The annotation processing unit 55 can determine, for example, whether or not an object or event to be annotated is detected depending on whether or not there is an operation on the label addition operation area 102. When it is determined that the object or event to be annotated is not detected (step S106: No), the annotation processing unit 55 returns to the process of step S105 and repeatedly executes the subsequent processes.

When the annotation processing unit 55 determines that an object or event to be annotated is detected (step S106: Yes), when annotation information such as an object label or a related label is added according to the operation of the operator. (Step S107), the information of the corresponding moving image data is added to the teacher data (annotation file) together with the annotation information (step S108).

Then, the annotation processing unit 55 determines whether or not the data condition includes a condition other than the moving image data (step S109). When the annotation processing unit 55 determines that the data condition includes a condition other than the moving image data (step S109: Yes), the annotation processing unit 55 also adds information of other data corresponding to the annotation information to the teacher data (annotation file) (step S109: Yes). S110).

After that, the annotation processing unit 55 determines whether or not the end of the annotation work has been input according to the operation of the worker (in other words, whether or not the task end operation area 104 has been selected) (step S111), and annotates. When it is determined that the end of the work has been input (step S111: Yes), the process according to this flowchart is terminated. When it is determined that the end of the annotation work has not been input (step S111: No), the annotation processing unit 55 returns to the processing of step S105 and repeatedly executes the subsequent processing.

When the annotation processing unit 55 determines in the process of step S109 that the data condition does not include a condition other than moving image data (step S109: No), the process of step S110 is skipped and the process proceeds to the process of step S111.

Next, with reference to FIG. 25, each process in a more specific execution method of the annotation work will be described. The method of executing this annotation work is typically performed mainly by the annotation device 1 that constitutes the device for the worker.

The display processing unit 51 displays a moving image on the display device 10 and reproduces it according to the operation of the operator (step S201), and the annotation processing unit 55 is an object to be annotated according to the operation of the operator. Is determined (step S202). The annotation processing unit 55 can determine, for example, whether or not an object to be annotated is detected depending on whether or not there is an operation on the label addition operation area 102. When it is determined that the object to be annotated is not detected (step S202: No), the annotation processing unit 55 returns to the process of step S201 and repeatedly executes the subsequent processes.

When the display processing unit 51 determines that the object to be annotated is detected by the annotation processing unit 55 (step S202: Yes), the display processing unit 51 displays the moving image reproduced on the display device 10 in response to the operator's operation. Stop (step S203).

Then, the annotation processing unit 55 newly creates an object label for the object detected above (step S204) according to the operation of the operator, and annotates information such as the type and position of the object corresponding to this object label. (Step S205). At this time, the annotation processing unit 55 stores the current frame in the storage circuit 40 as the start key frame of the object label.

Next, the display processing unit 51 displays a moving image on the display device 10 and reproduces it according to the operation of the operator (step S206), and the annotation processing unit 55 displays the moving image on the object label according to the operation of the operator. It is determined whether or not the modification of the position designation of the corresponding object is determined (step S207). When it is determined by the annotation processing unit 55 that the annotation processing unit 55 has decided to correct the position designation of the object (step S207: Yes), the display processing unit 51 displays the moving image reproduced on the display device 10 according to the operation of the operator. Stop (step S208).

Then, the annotation processing unit 55 corrects the position designation of the object targeted by the object label according to the operation of the operator (step S209), and the position of the object designated in the previous frame (for example, the start key frame). And, based on the position of the object redesignated in this frame, the position of the object in each frame between them is calculated and specified by linear interpolation or the like (step S210). Then, the display processing unit 51 displays the moving image on the display device 10 and reproduces it according to the operation of the operator (step S211).

Next, the annotation processing unit 55 determines whether or not the frame in which the object of the object of the object label determined as described above is reflected has been completed according to the operation of the operator (step S212). The annotation processing unit 55 can determine, for example, whether or not the frame in which the object targeted by the object label is reflected has ended, depending on whether or not there is an operation on the end operation area 107. Further, when the annotation processing unit 55 determines in the process of step S207 described above that the correction of the position designation of the object targeted by the object label has not been determined (step S207: No), steps S208 to S208 to The process of step S211 is skipped and the process proceeds to the process of step S212.

When it is determined that the frame in which the object to be detected is reflected has not been completed (step S212: No), the annotation processing unit 55 returns to the processing of step S206 and repeatedly executes the subsequent processing. When the annotation processing unit 55 determines that the frame in which the object to be detected is reflected has been completed (step S212: Yes), the annotation processing unit 55 executes the end of this object label work according to the operation of the operator (step S213), and is present. The frame is stored in the storage circuit 40 as the end key frame of the object label, and the time range of the object label in this moving image is determined.

Next, the annotation processing unit 55 determines whether or not another object to be annotated is detected in the same time zone according to the operation of the operator (step S214). When the display processing unit 51 determines that another object to be annotated is detected by the annotation processing unit 55 (step S214: Yes), the display processing unit 51 searches for and displays the corresponding playback frame according to the operation of the operator. (Step S215), the process returns to the process of step S204, and the subsequent processes are repeatedly executed.

When the annotation processing unit 55 determines that another object to be annotated has not been detected (step S214: No), whether or not an event to be annotated has been detected according to the operation of the operator. (Step S216). The annotation processing unit 55 can determine, for example, whether or not an event to be annotated is detected depending on whether or not there is an operation on the label addition operation area 102. When the annotation processing unit 55 determines that the event to be annotated is not detected (step S216: No), the annotation processing unit 55 ends the processing according to this flowchart.

When the display processing unit 51 determines that the event to be annotated is detected by the annotation processing unit 55 (step S216: Yes), the display processing unit 51 searches for and displays the corresponding playback frame according to the operation of the operator (step S216: Yes). Step S217).

Then, the annotation processing unit 55 newly creates a relational label for the event detected above according to the operation of the operator (step S218), and adds the type of the event corresponding to this relational label as annotation information. (Step S219). At this time, the annotation processing unit 55 stores the current frame in the storage circuit 40 as the start key frame of the relation label.

Then, the annotation processing unit 55 designates the object label of the object related to the event targeted by the relation label according to the operation of the worker, and adds it as the annotation information of the relation label (step S220). Then, the display processing unit 51 displays the moving image on the display device 10 and reproduces it according to the operation of the operator (step S221).

Next, the annotation processing unit 55 determines whether or not the frame in which the event targeted by the relation label determined as described above is reflected has ended according to the operation of the operator (step S222). The annotation processing unit 55 can determine, for example, whether or not the frame in which the event targeted by the relation label is displayed has ended depending on whether or not there is an operation on the end operation area 107.

When it is determined that the frame in which the event targeted by the relation label is displayed has not been completed (step S222: No), the annotation processing unit 55 returns to the processing of step S221 and repeatedly executes the subsequent processing. When it is determined by the annotation processing unit 55 that the frame in which the event targeted by the relation label is displayed has ended (step S222: Yes), the display processing unit 51 is played back on the display device 10 according to the operation of the operator. The moving image is stopped (step S223).

Then, the annotation processing unit 55 executes the end of the relational label work according to the operation of the operator (step S224), stores the current frame as the end key frame of the relational label in the storage circuit 40, and makes this moving image. The time range of the relation label in the image is determined, and the process according to this flowchart is terminated.

The method related to the annotation work of FIG. 25 described above is "a step of displaying the moving image represented by the moving image data, a step of accepting an operation, and a trained model in which annotation information is added to the moving image data according to the operation. In the step of creating teacher data, including the step of creating teacher data used for machine learning, the position of the object included in the moving image is specified according to the operation, and the object label indicating the type of the object is used as annotation information. A relational label that is added to the moving image data and indicates the type of event in which multiple objects included in the moving image correlate according to the operation is added to the moving image data as annotation information, and from the above-added object label according to the operation. A step corresponding to each step of "annotation method characterized by designating an object label of an object related to an event targeted by a relational label and adding it to moving image data as annotation information" is included. Further, this "annotation method" can be realized by executing a "annotation program" prepared in advance on a computer such as a personal computer or a workstation. This "annotation program" causes the computer to execute each process of the above-mentioned steps.

The annotation device 1, the annotation method, and the annotation program described above can create teacher data used for machine learning of the trained model M for detecting an event that occurs when a plurality of objects are involved. In this case, the annotation device 1, the annotation method, and the annotation program provide annotation information for the moving image data, an object label indicating the type of the object reflected in the moving image, and the type of the event in which a plurality of objects are correlated. A relational label can be added to represent it. Then, the annotation device 1, the annotation method, and the annotation program also specify and add the object label of the object that correlates with the event targeted by the relation label as the annotation information for the moving image data. can do. As a result, the annotation device 1, the annotation method, and the annotation program can appropriately create teacher data. As a result, for example, the detection system Sys1 can generate a trained model M for detecting the relational behavior by machine learning using the teacher data, and properly performs the relational behavior detection using the trained model M. be able to.

For example, the annotation device 1, the annotation method, and the annotation program described above have an object label representing "a pedestrian traffic light that is a red light" and "walking on a pedestrian crossing" for an object reflected in a certain moving image. When the object label representing "pedestrian" is added and the relation label "ignoring the pedestrian signal" is added as the type of the pedestrian traffic light and the event that the pedestrian correlates with, the object related to the relation label. As the object label of, add annotation information associated with the object label representing "pedestrian traffic light which is a red light" and the object label representing "pedestrian walking on a pedestrian crossing", and use it as teacher data. Can be done. Then, the detection system Sys1 is, for example, from the moving image data of the moving image showing the "pedestrian traffic light which is a red light" and the "pedestrian walking on the pedestrian crossing" by machine learning using the teacher data. A trained model M capable of detecting "pedestrian signal ignorance" can be generated, and the trained model M is used to detect an event such as "pedestrian signal ignorance" (related behavior detection). Can be done.

Further, the annotation device 1, the annotation method, and the annotation program described above handle a plurality of predetermined moving image data files as moving image data representing a series of moving images in the annotation processing. As a result, the annotation device 1, the annotation method, and the annotation program can continuously reproduce moving images and perform annotation processing without making the operator aware of file switching, thus suppressing complicated work. And workability can be improved. In addition, the annotation device 1, the annotation method, and the annotation program can seamlessly perform annotation processing even for objects and events that span different files without the labels being separated, so that the quality of the teacher data can be improved. It is possible to suppress the decrease. Further, since the annotation device 1, the annotation method, and the annotation program can create one set of teacher data in units of moving image data handled as a series of moving images, not in units of a plurality of moving image data files. In this respect as well, it is possible to suppress complicated work such as collecting the divided teacher data and prevent deterioration of the quality of the teacher data.

The annotation device, annotation method, and annotation program according to the above-described embodiment of the present invention are not limited to the above-mentioned embodiments, and various modifications can be made within the scope of the claims.

The annotation device 1 described above has a sound reproduction function in addition to the moving image reproduction function, and may apply annotation processing to, for example, sound data in addition to the moving image data. .. Further, the annotation device 1 has a function of two-dimensionally plotting time-series data such as IoT (Internet of Things) data, and even if the annotation process is applied to the time-series data as well. good. Taking sound data as an example of data having a structure different from that of moving image data, for example, the annotation device 1 sets an event of "surprise" as a type of event in which an "alarm sound" and a voice of "wow" correlate with each other. In addition to adding a relational label to represent, as a sound label related to the event targeted by the relational label, a sound label for the voice "wow" and a sound label for the "alarm sound" are specified and added to the sound data as annotation information. Can be added. Further, in this case, the annotation device 1 may apply a function of handling a plurality of predetermined moving image data files as moving image data representing a series of moving images to a sound data file or an IoT data file.

The processing circuit 50 described above has been described as assuming that each processing function is realized by a single processor, but the present invention is not limited to this. The processing circuit 50 may realize each processing function by combining a plurality of independent processors and executing a program by each processor. Further, the processing function of the processing circuit 50 may be appropriately distributed or integrated into a single or a plurality of processing circuits. Further, the processing function of the processing circuit 50 may be realized by a program in whole or in any part thereof, or may be realized as hardware by wired logic or the like.

The program executed by the processor described above is provided by being incorporated in the storage circuit 40 or the like in advance. It should be noted that this program may be provided as a file in a format that can be installed on these devices or in an executable format, recorded on a storage medium that can be read by a computer. Further, this program may be stored on a computer connected to a network such as the Internet, and may be provided or distributed by being downloaded via the network.

The annotation device, annotation method, and annotation program according to the present embodiment may be configured by appropriately combining the components of the embodiments and modifications described above.

1 Annotation device 10 Display device (display unit)
20 Operation equipment (operation unit)
30 Data input / output device 40 Storage circuit 40a Designated file storage area 50 Processing circuit (processing unit)
51 Display processing unit 52 Operation processing unit 53 Data input / output processing unit 54 Task creation processing unit 55 Annotation processing unit 100A, 100B, 100C Annotation screen 101 Video display area 101a Video operation area 102 Label addition operation area 103 Label selection area 103A Main list 103B, 103C, 103D, 103E Sublist 104 Task end operation area 105 Decision label display image 106 Erase operation area 107 End operation area 108 Erase decision image 109 Added label display area 110 Slide bar display area 111 File name display area 111A File list 112 File selection area 113 Work end confirmation image 114 Work end decision image 200 Management mode screen 201 Task creation operation area 300 Task creation screen AL Machine learning algorithm D1 Input data D2 Teacher data D3 Learning data set D4 Detection target data D5 Type Specific result data M Trained model Sv Training data provision system Sys1 detection system

Claims (4)

A display unit that can display the moving image represented by the moving image data,
The operation unit that accepts operations and
It is provided with a processing unit capable of executing annotation processing for adding annotation information to the moving image data in response to an operation on the operation unit and creating teacher data used for machine learning of the trained model.
In the annotation process, the processing unit specifies the position of an object included in the moving image in response to an operation on the operating unit, and adds an object label indicating the type of the object to the moving image data as the annotation information. Processing, processing to add a relational label indicating the type of an event in which a plurality of objects included in the moving image correlate to the moving image data as the annotation information, and the operation unit. From the object label added in response to the operation to, the object label of the object related to the event targeted by the relational label is designated, and a process of adding to the moving image data as the annotation information is executed. Characterized by that,
Annotation device. In the annotation processing, the processing unit handles a plurality of predetermined moving image data files as the moving image data representing the series of the moving images.
The annotation device according to claim 1. Steps to display the moving image represented by the moving image data,
Steps to accept operations and
It includes a step of adding annotation information to the moving image data according to an operation to create teacher data used for machine learning of a trained model.
In the step of creating the teacher data, the position of the object included in the moving image is specified according to the operation, and the object label indicating the type of the object is added to the moving image data as the annotation information, and the operation is performed. A relational label indicating the type of an event in which a plurality of objects included in the moving image correlate is added to the moving image data as the annotation information, and the object label added according to the operation is the target of the relational label. It is characterized in that the object label of the object related to the event is designated and added to the moving image data as the annotation information.
Annotation method. The moving image represented by the moving image data is displayed, and the moving image is displayed.
Accept the operation,
Annotation information is added to the moving image data according to the operation to create teacher data used for machine learning of the trained model.
Let the computer perform each process
In the process of creating the teacher data, the position of the object included in the moving image is specified according to the operation, and the object label indicating the type of the object is added to the moving image data as the annotation information, and the operation is performed. A relationship label indicating the type of event in which a plurality of objects included in the moving image correlate is added to the moving image data as the annotation information, and the object label added according to the operation is the target of the relationship label. The object label of the object related to the event is designated and added to the moving image data as the annotation information.
It is characterized in that each process is executed by the computer.
Annotation program.

Images