JP6544112B2 - Confirmation support device - Google Patents

Description

  The present invention relates to a confirmation support apparatus that supports confirmation of a subject's behavior.

  In the process of developing a driving support system for supporting the driving of a vehicle, it is known to quantitatively grasp the behavior of a driver as a subject and reflect it on control parameters of the system. Here, when the subject's driving operation, auxiliary equipment operation, confirmation action, and other preparatory actions are also comprehended, the subject can not be comprehended based on only the vehicle information acquired via CAN (Control Area Network). It is performed to analyze the image taken and analyze the behavior of the subject.

  Then, in order to streamline the task of analyzing the behavior of the subject, the analyst operates the keyboard while visually recognizing the image obtained by photographing the subject, thereby relating to the playback time of the image and showing the behavior of the subject. It is done to tag. For example, in Patent Document 1, the gaze direction of the subject in the image is associated with the key on the keyboard, and the analyst presses the key corresponding to the gaze direction of the subject while visually recognizing the image. There is disclosed an apparatus for associating and recording a reproduction time and an action tag as information indicating a gaze direction.

JP, 2014-202636, A

  Although the device described in Patent Document 1 can efficiently perform the task of recording the action tag in association with the playback time of the video, there is no disclosure about the method by which the analyst confirms the recorded action tag. . As a method for the analyst to confirm the recorded action tag, it is conceivable to display a reproduction time to which the action tag is attached and information indicating the action tag together with a video of the subject.

  However, in such a method, it is difficult for the analyst to simultaneously check the video of the subject and the information indicating the action tag, and as a result, is the recorded action tag appropriate? There is a problem that it takes time to check whether or not it is.

  Therefore, the present invention has been made in view of these points, and it is an object of the present invention to provide a confirmation support device capable of efficiently confirming a subject's action tag associated with a picture showing the subject's action. .

  In the confirmation support apparatus of the present invention, a display control means for displaying on a display means an image obtained by photographing a subject, a receiving means for receiving a tag type indicating the type of the action of the subject, and the receiving means receiving the tag type Storage control means for associating the information indicating the tag type with the reproduction time of the video and storing the information in the storage means, and referring to the storage means at the time of reproduction of the video, at reproduction time corresponding to the tag type And output control means for outputting information corresponding to the tag type.

  By doing this, the analyst can recognize the information corresponding to the tag type output at the playback time of the video while visually recognizing the video taken of the subject, so the analyst can keep an eye on the video. It is possible to confirm the type of tag associated with the video, that is, whether or not the action tag is appropriate. Therefore, the confirmation support apparatus can efficiently confirm the subject's action tag associated with the video indicating the subject's action.

The output control means may output a sound corresponding to the tag type at a reproduction time corresponding to the tag type.
By doing this, the analyst can recognize the tag type associated with the video without listening to the video by listening to the sound corresponding to the tag type.

The output control means may be configured to set the first reproduction time when the time between a first reproduction time corresponding to one tag type and a second reproduction time corresponding to a tag type received next to the tag type is equal to or longer than a predetermined time. Even after the information corresponding to the one tag type is output at one reproduction time, the reproduction speed of the video is increased to a third reproduction time later than the first reproduction time and earlier than the second reproduction time. Good.
By doing this, the analyst can quickly confirm the portion of the image to which the action is not attached, as compared with the case where the normal reproduction speed is checked.

The output control means may be configured to set the first reproduction time when the time between a first reproduction time corresponding to one tag type and a second reproduction time corresponding to a tag type received next to the tag type is equal to or longer than a predetermined time. Even after the information corresponding to the one tag type is output at one reproduction time, the reproduction position of the video is moved to a third reproduction time that is later than the first reproduction time and earlier than the second reproduction time. Good.
By doing this, the analyst can omit the reproduction of the part without the tag indicating the action in the video, and can perform the confirmation work limited to the part with the tag.

  The storage control means receives an offset time for changing the reproduction time corresponding to the tag type, and the reproduction time corresponding to each of all the tag types stored in the storage means from the reproduction time Only the offset time may be changed.

  Since the analyst specifies the tag type after confirming the action of the subject indicated by the video, the reproduction time of the picture associated with the tag type has a time lag with respect to the reproduction time corresponding to the action of the subject. On the other hand, since the storage control unit changes the reproduction time corresponding to each of all the tag types from the reproduction time by the offset time, the reproduction time corresponding to each of all the tag types is the action of the subject. It can be corrected to the playback time corresponding to.

  According to the present invention, it is possible to efficiently check the subject's action tag associated with the image showing the subject's action.

It is a figure which shows the structure of a confirmation assistance apparatus. It is a figure which shows the example of a confirmation screen. It is a figure which shows the correspondence of a key and tag type. It is a flowchart concerning attachment of an action tag. It is a flowchart which concerns on confirmation of the action tag provided to the image | video.

[Configuration of Confirmation Support Device 10]
Hereinafter, the confirmation support apparatus 10 according to an embodiment of the present invention will be described. FIG. 1 is a diagram showing the configuration of the confirmation support apparatus 10. As shown in FIG. The confirmation support device 10 is, for example, a computer that supports confirmation of the action of a subject driving the vehicle 1.
The confirmation support device 10 includes a display unit 11, an input unit 12, a sound output unit 13, a storage unit 14, and a control unit 15.

  The display unit 11 is, for example, a liquid crystal display. The display unit 11 displays an image obtained by photographing a subject driving the vehicle 1 according to the control of the control unit 15.

The input unit 12 is, for example, a keyboard or a mouse. The input unit 12 receives an operation from the user of the confirmation support apparatus 10. The user of the confirmation support apparatus 10 is, for example, a video analyst. Hereinafter, the user of the confirmation support apparatus 10 is referred to as an “analyzer”.
The sound output unit 13 is, for example, a speaker. The sound output unit 13 outputs a sound corresponding to the type (tag type) of the action tag attached to the video according to the control of the control unit 15.

The storage unit 14 is, for example, a ROM, a RAM, or a hard disk. The storage unit 14 stores various programs for causing the control unit 15 to function. In addition, the storage unit 14 stores a video file indicating a video obtained by shooting a subject driving the vehicle 1. Here, the video is photographed by a photographing device 2 such as a camera attached to the vehicle 1, for example. The photographing device 2 stores a video file indicating a photographed video in a storage medium attached to itself. The analyst mounts the storage medium in which the video file is stored in the confirmation support apparatus 10, and stores the video file stored in the storage medium in the storage unit 14. The video file may be stored in the storage unit 14 by being acquired via a communication network such as a LAN or the Internet.
In addition, the storage unit 14 stores action tag information in which information indicating a tag type indicating the type of the subject's action is associated with the reproduction time of the video.

The control unit 15 is, for example, a CPU. As illustrated in FIG. 1, the control unit 15 includes a display control unit 151, a reception unit 152, a storage control unit 153, and an output control unit 154.
The display control unit 151 causes the display unit 11 to display an image obtained by photographing the subject. Specifically, first, in response to the input unit 12 receiving a predetermined operation from the analyst, the display control unit 151 causes the display unit 11 to display an analysis screen 100 including a video display area for displaying a video. .

  FIG. 2 is a diagram showing an example of the analysis screen 100. As shown in FIG. FIG. 2A shows a state before an image is displayed on the analysis screen 100. FIG. The analysis screen 100 shown in FIG. 2A includes an image display area 101, action tag information 102 regarding an action tag attached to the picture, a reproduction button 103, an offset button 104, and a reproduction speed setting button 105. It contains.

  When the analyst selects a video file indicating a video of the subject taken and stored in the storage unit 14 and presses the play button 103, the display control unit 151 plays back the video corresponding to the video file. , And display on the video display area 101. Note that, in response to the selection of the video file stored in the external device connected to the confirmation support device via the communication network, the display control unit 151 displays the video corresponding to the video file in the video display area. It may be displayed on 101.

  Further, in response to pressing of the playback speed setting button 105, the display control unit 151 causes the display unit 11 to display a setting screen for receiving the setting of the playback speed. Then, the display control unit 151 reproduces the video based on the reproduction speed received on the setting screen, and causes the video display area 101 to display the video.

  The receiving unit 152 receives a tag type indicating the type of the subject's action. Specifically, the receiving unit 152 receives the tag type via the keyboard constituting the input unit 12. The tag type is associated with a part of keys constituting the keyboard. FIG. 3 is a diagram showing the relationship between keys and tag types.

  The tag type corresponds to, for example, the direction of the face of the subject, and is associated with a part of keys constituting the ten key. As shown in FIG. 3, the “8” key is associated with a tag type indicating that the subject's face is facing upward. Similarly, the "6" key is a tag type indicating that the subject's face is facing the right direction, the "5" key is a tag type indicating that the subject's face is facing the front direction, "4" The key is associated with a tag type indicating that the subject's face is facing left, and the “2” key is associated with the tag type indicating that the subject's face is facing downward.

In the present embodiment, the information indicating the tag type is assumed to be numbers, alphabets, symbols, etc. of the associated key. For example, when the subject shown in the image turns to the right, the analyst presses the "6" key. Accordingly, the receiving unit 152 receives a tag type “6” indicating that the subject is facing the right as a tag type indicating the type of the subject's action.
In the present embodiment, an example in which the tag type corresponds to the direction of the face of the subject is described. However, the present invention is not limited thereto. The tag type may correspond to the action of the subject or the state of the subject.

  When the receiving unit 152 receives the tag type, the display control unit 151 sets, as action tag information, a time interval between the information indicating the tag type and the tag type received immediately before as shown in FIG. The analysis screen 100 displays a table in which “lap” indicating time), “time” indicating playback time of the video when the tag type is accepted, and remarks are associated with each other. Here, it is assumed that the “time” is specified by, for example, the analyst inputting in advance time information indicating the time of the start point of the video. When a cell corresponding to the remarks is selected through the input unit 12, text information can be input to the cell.

  When the receiving unit 152 receives the tag type, the storage control unit 153 associates an action tag indicating the action of the subject with the image. Specifically, when the receiving unit 152 receives a tag type, the storage control unit 153 associates information indicating the tag type, the reproduction time of the video, the lap time, and the remarks, and stores the information as action tag information. Make it memorize in 14. For example, the storage control unit 153 generates a CSV file including action tag information in which information indicating a tag type, video playback time, lap time, and remarks are associated, and stores the CSV file in the storage unit 14 Let The file name of the CSV file corresponds to, for example, the file name of a video file indicating a video. The storage control unit 153 may associate the information indicating the tag type, the lap time, and the remarks with each other without storing the reproduction time of the video in the CSV file.

  In addition, although the storage control unit 153 generates the CSV file in response to the reception unit 152 receiving the tag type, the present invention is not limited to this. When the action tag information 102 is displayed on the analysis screen as illustrated in FIG. 2B, the storage control unit 153 receives the operation of storing the action tag information by the input unit 12. A CSV file including the action tag information 102 displayed on the analysis screen may be generated, and the CSV file may be stored in the storage unit 14.

Also, the analyst can select one row of the table showing the action tag information 102 through the input unit 12. When the play button 103 is pressed in a state where one row is selected, the display control unit 151 starts playback of the video from the playback time corresponding to the one row. In this state, when the reception unit 152 receives the tag type, the storage control unit 153 adds the received tag type and the reproduction time of the video to the storage unit 14 and stores the same.
In addition, the storage control unit 153 may delete the action tag information corresponding to the one row, for example, in response to pressing the Delete button of the keyboard in the state where the one row is selected.

  In addition, since the analyst specifies the tag type after confirming the subject's action shown by the video, the playback time of the video associated with the tag type has a time lag with respect to the playback time corresponding to the subject's action. It occurs.

  Therefore, the storage control unit 153 receives an offset time for changing the reproduction time corresponding to the tag type. For example, as shown in FIG. 2B, when the video and action tag information 102 are displayed, when the offset button 104 provided on the analysis screen 100 is pressed, the display control unit 151 performs an offset. Display the acceptance screen for accepting time.

When the storage control unit 153 receives the offset time from the analyst via the reception screen, the storage control unit 153 refers to the CSV file stored in the storage unit 14 corresponding to the video, and copes with all the tag types. The playback time to be changed is changed from the playback time by the offset time.
Further, the display control unit 151 changes the reproduction time corresponding to each of all the tag types displayed on the display unit from the reproduction time by the offset time and displays the same.
By doing this, the confirmation support device 10 can correct the reproduction time corresponding to each of all the tag types to the reproduction time corresponding to the action of the subject.

  The output control unit 154 refers to the storage unit 14 at the time of video reproduction, and outputs information corresponding to the tag type at the reproduction time corresponding to the tag type. Specifically, when the output control unit 154 starts the reproduction of the video, the storage unit 14 stores the CSV file storing the action tag information in correspondence with the file name of the video file indicating the video. Information indicating the tag type stored in the CSV file and the playback time of the video are acquired. Then, the output control unit 154 causes the sound output unit 13 to output the sound corresponding to the tag type when the reproduction time corresponding to the acquired tag type arrives during the reproduction of the video.

  For example, the output control unit 154 may select a tag type “8” indicating upward, a tag type “6” indicating right, a tag type “5” indicating forward, a tag type “4” indicating left, In accordance with the tag type “2” indicating “,” the notification sound of sine waves with frequencies of 1300 Hz, 1500 Hz, 1200 Hz, 900 Hz, and 1800 Hz is output from the sound output unit 13. Here, the length of the notification sound is, for example, 0.3 seconds to 0.8 seconds. The notification sound may be voice indicating a direction such as “upper” or “right”.

By doing this, the analyst can recognize the type of action attached to the video by the sound corresponding to the tag type without taking a look at the video.
In the present embodiment, the sound corresponding to the tag type is output from the sound output unit 13 as the information corresponding to the tag type. However, the present invention is not limited to this. For example, the confirmation support apparatus 10 is provided with a vibration unit that generates vibration, and generates vibration corresponding to the tag type, or the background color of the analysis screen 100 displayed on the display unit 11 corresponds to the tag type. It may be changed to the color attached.

  Also, at the time of video playback, the output control unit 154 has a predetermined time between the first playback time corresponding to one tag type and the second playback time corresponding to the tag type received next to the tag type. In the above case, after outputting information corresponding to one tag type at the first reproduction time, the video reproduction speed is increased to the third reproduction time that is later than the first reproduction time and earlier than the second reproduction time. .

For example, when the time between the first reproduction time and the second reproduction time is 10 seconds or more, the output control unit 154 increases the reproduction speed of the video to 3 seconds before the second reproduction time. In the example shown in FIG. 2B, the tag type "4" is received at the reproduction time "24.15 seconds", and then the tag type "6" is received at the reproduction time "45.50 seconds". Therefore, the output control unit 154 outputs the sound corresponding to the tag type “4” at the reproduction time “24.15 seconds” to the sound output unit 13 and then three seconds before the reproduction time “45.50 seconds” ( 42. Increase the video playback speed (for example, double speed) up to 42.50 seconds.
By doing this, the analyst can quickly confirm the portion of the image to which the action is not attached, as compared with the case where the normal reproduction speed is checked.

Although the output control unit 154 increases the video playback speed when the time between the first playback time and the second playback time is equal to or longer than a predetermined time, the present invention is not limited to this. The output control unit 154 may move the playback position of the video to the third playback time when the time between the first playback time and the second playback time is equal to or longer than a predetermined time.
By doing this, the analyst can omit the reproduction of the part without the tag indicating the action in the video, and can perform the confirmation work limited to the part with the tag.

[Flowchart for Assigning Action Tag]
FIG. 4 is a flowchart showing a procedure of processing when the confirmation support device 10 attaches an action tag to a video.

First, the display control unit 151 displays the analysis screen, starts reproduction of the video selected by the analyst, and causes the video display area 101 to display the video (S10).
Subsequently, the receiving unit 152 determines whether a tag type has been received from the analyst via the input unit 12 (S20). If the receiving unit 152 determines that the tag type is received, the process proceeds to S30, and if it is determined that the tag type is not received, the process proceeds to S40.

  In S30, the storage control unit 153 associates the information indicating the tag type received by the receiving unit 152 with the reproduction time of the video when the receiving unit 152 receives the tag type, and stores the information in the storage unit 14.

  In S40, the display control unit 151 determines whether the reproduction of the video is completed. If the display control unit 151 determines that the reproduction of the video is ended, the processing according to the flowchart is ended, and if it is determined that the reproduction of the video is not ended, the processing proceeds to S20.

[Flowchart related to confirmation of action tag attached to video]
FIG. 5 is a flowchart showing a procedure of processing when the confirmation support device 10 outputs information related to an action tag attached to a video. In the flowchart, it is assumed that a CSV file including action tag information corresponding to the video selected by the analyst is present at the start of the process.

First, the display control unit 151 acquires action tag information included in a CSV file corresponding to a video selected by the analyst (S110), and causes the acquired action tag information to be displayed on the analysis screen.
Subsequently, the display control unit 151 starts reproduction of the video selected by the analyst and causes the video display area 101 to display the video (S120).

Subsequently, the output control unit 154 detects the first action tag with reference to the acquired action tag information (S130).
Subsequently, the output control unit 154 determines whether the time until the reproduction time corresponding to the next action tag is equal to or longer than a predetermined time (S140). If the output control unit 154 determines that the time until the reproduction time corresponding to the next action tag is a predetermined time or more, the process proceeds to S150, and the reproduction time immediately before the reproduction time corresponding to the next action tag (for example, The playback speed is raised to 3 seconds before the playback time) or the playback is skipped to the playback time immediately before the playback time corresponding to the next action tag.

If the output control unit 154 determines that the time until the reproduction time corresponding to the next action tag is less than the predetermined time, the process proceeds to S160. The output control unit 154 also shifts the process to S160 even when there is no next action tag.
In S160, the output control unit 154 determines whether the reproduction time of the image corresponding to the action tag has come. When determining that the reproduction time corresponding to the action tag has arrived, the output control unit 154 shifts the processing to S170 and causes the sound output unit 13 to output a sound corresponding to the tag type of the action tag.
Further, when determining that the reproduction time corresponding to the action tag has not arrived, the output control unit 154 repeats the process of S160, and waits until the reproduction time corresponding to the action tag arrives.

In S180, the display control unit 151 determines whether the reproduction of the video is completed. If the display control unit 151 determines that the reproduction of the video is ended, the processing according to the flowchart is ended, and if it is determined that the reproduction of the video is not ended, the processing proceeds to S190.
In S190, the output control unit 154 determines whether there is a next action tag. When it is determined that the next action tag is present, the output control unit 154 moves the process to S140, and when it is determined that the next action tag is not present, the process according to the present flowchart ends.

  Although the sound corresponding to the tag type of the action tag is output at the reproduction time corresponding to the action tag in the flowchart shown in FIG. 5, the present invention is not limited to this, and the tag type can be received as shown in FIG. You may do so.

[Effect in the present embodiment]
As described above, the confirmation support device 10 causes the display control unit 151 to display an image obtained by photographing the subject, the receiving unit 152 receiving the tag type indicating the type of the subject's action, and the receiving unit 152 When the tag type is accepted, the storage control unit 153 associates the information indicating the tag type with the reproduction time of the video and stores the information in the storage unit 14, and when reproducing the video, refers to the storage unit 14 and supports the tag type. And an output control unit 154 that outputs information corresponding to the tag type at the playback time.

  By doing this, the analyst can recognize the information corresponding to the tag type output at the playback time of the video while visually recognizing the video taken of the subject, so the analyst can keep an eye on the video. , It can be confirmed whether the tag type associated with the video is appropriate. Therefore, the confirmation support apparatus 10 can efficiently confirm the subject's action tag associated with the video indicating the subject's action.

  As mentioned above, although this invention was demonstrated using embodiment, the technical scope of this invention is not limited to the range as described in the said embodiment. It is apparent to those skilled in the art that various changes or modifications can be added to the above embodiment. It is also apparent from the scope of the claims that the embodiments added with such alterations or improvements can be included in the technical scope of the present invention.

  For example, in the above-described embodiment, an example has been described in which the confirmation support device 10 confirms whether the action tag associated with the video by the analyst is appropriate. However, in the face direction detection system that detects the face direction of the subject You may use for confirmation of a detection condition. For example, when the face direction angle indicating the face direction of the subject exceeds the predetermined angle threshold by analyzing the image, the confirmation support device 10 determines that the face direction of the subject is left, right, up, or down. A flag of the face orientation system to be acquired may be acquired as the tag type, and the flag may be stored in association with the reproduction time of the video. Then, when reproducing the video, the confirmation support apparatus 10 may output an alarm sound associated with the flag at the playback time associated with the flag. By doing this, the analyst can easily judge whether the detection timing of the face direction detection system is appropriate and false detection and non-detection while listening to the notification sound.

Reference Signs List 1 vehicle 2 imaging apparatus 10 confirmation support device 11 display unit 12 input unit 13 sound output unit 14 storage unit 15 control unit 151 display control unit 152 reception unit 153 storage control unit 154 output control unit

Claims (4)

Display control means for displaying on a display means an image obtained by photographing a subject;
Accepting means for accepting a tag type indicating the type of action of the subject;
Storage control means for associating the information indicating the tag type with the reproduction time of the video and storing the information in the storage means when the receiving means receives the tag type;
An output control unit that outputs information corresponding to the tag type at a reproduction time corresponding to the tag type by referring to the storage unit when the video is reproduced;
Equipped with
The output control means may be configured to set the first reproduction time when the time between a first reproduction time corresponding to one tag type and a second reproduction time corresponding to a tag type received next to the tag type is equal to or longer than a predetermined time. After outputting the information corresponding to the one tag type at one reproduction time, increasing the reproduction speed of the video to a third reproduction time which is later than the first reproduction time and earlier than the second reproduction time Feature
Check support device.   Display control means for displaying on a display means an image obtained by photographing a subject;
  Accepting means for accepting a tag type indicating the type of action of the subject;
  Storage control means for associating the information indicating the tag type with the reproduction time of the video and storing the information in the storage means when the receiving means receives the tag type;
  An output control unit that outputs information corresponding to the tag type at a reproduction time corresponding to the tag type by referring to the storage unit when the video is reproduced;
  Equipped with
  The output control means may be configured to set the first reproduction time when the time between the first reproduction time corresponding to one tag type and the second reproduction time corresponding to the tag type received next to the tag type is equal to or longer than a predetermined time After the information corresponding to the one tag type is output at one reproduction time, moving the reproduction position of the video to a third reproduction time earlier than the first reproduction time and earlier than the second reproduction time Feature
  Confirmation support device. The output control means outputs a sound corresponding to the tag type at a reproduction time corresponding to the tag type.
The confirmation support apparatus according to claim 1 . The storage control means receives an offset time for changing the reproduction time corresponding to the tag type, and the reproduction time corresponding to each of all the tag types stored in the storage means from the reproduction time It is characterized by changing only the offset time,
The confirmation assistance apparatus of any one of Claim 1 to 3 .

Images