JP2023154608A - Video analysis device, video analysis method, and video analysis program - Google Patents

Abstract

To provide a video analysis device capable of improving searchability for finding a target scene intended by a user.SOLUTION: A video analysis device includes an acquisition unit for acquiring video identification information and a video file, a camera work detector for detecting a predetermined camera work included in the video file based on the movement of an image, and extracting a start time and an end time of a scene including the camera work, a tagging unit for referring to a database that associates types of camera works with nuances corresponding to the camera works to add a tag representing the nuance corresponding to the detected camera work every scene, a video analysis data creation unit for creating video analysis data that includes video identification information and associates a camera work, a start time and an end time, and tag information for each scene, and an output unit for outputting the video analysis data.SELECTED DRAWING: Figure 1

Description

The present invention relates to a video analysis device, a video analysis method, and a video analysis program.

Video platform (PF) services that allow users to create and post videos themselves are becoming widespread, making video production more accessible and improving its quality. With the increase in images and videos (hereinafter also referred to as "content"), the development of PFs to manage them is accelerating. Among them, methods have been reported that utilize analysis of videos, images, audio, and language by artificial intelligence (AI) to improve searchability by tagging (marking) content (for example, patent documents 1).

The tagging method described in Patent Document 1 involves voice recognition of a video file, converting it into text information, image analysis of the video file to determine scene changes in the video file, and temporal breaks and content in the text information. The video file is divided into multiple scenes based on breaks and scene changes, tags are extracted from the text information according to predetermined rules, and the extracted tags are divided into multiple scenes from among the multiple scenes. It is added to the corresponding scene.

However, there is a problem in that it is difficult to tag images that do not include audio by simply analyzing the audio included in the video file.

Furthermore, a method of detecting camera work from a video file is known (for example, Non-Patent Document 1). However, although camera work is one of the important elements in content production, it is difficult for general users who are not professionals to associate the types of camera work (phrases) and the nuances that accompany them. Therefore, the problem is that it is difficult for users to search for the video they want to watch from among multiple video files by adding tags to search for what kind of scenes are included in the video. there were.

JP2020-79982A

Atsuo Yoshitaka, Ryoji Matsui, So Hiramatsu, "Extraction of emotional information using camera work," Journal of the Information Processing Society of Japan, Vol. 47, No. 6, p. 1696-1707

SUMMARY OF THE INVENTION An object of the present invention is to provide a video analysis device that can improve searchability for finding a target scene intended by a user from among a plurality of video files.

A video analysis device according to an embodiment of the present disclosure includes an acquisition unit that acquires video identification information and a video file, and detects predetermined camera work included in the video file based on the movement of the image, and performs the camera work. A camera work detection unit that extracts the start time and end time of the included scene, and a database that associates the type of camera work with the nuances corresponding to the camera work, and corresponds to the detected camera work for each scene. Video analysis data creation that creates video analysis data that includes a tagging section that adds tags that represent nuances, video identification information, and associates information about camera work, start time and end time, and tags for each scene. and an output unit that outputs video analysis data.

The video analysis device according to an embodiment of the present disclosure further includes an object recognition unit that recognizes an object in an image included in the video file, and the tagging unit adds a tag based on the object to the video analysis data. good.

The video analysis device according to an embodiment of the present disclosure may further include a voice recognition unit that recognizes audio included in the video file, and the tagging unit may add a tag based on the audio to the video analysis data.

The video analysis device according to an embodiment of the present disclosure further includes a character recognition unit that recognizes characters in images included in the video file, and the tagging unit adds tags based on the characters to the video analysis data. good.

A video analysis device according to an embodiment of the present disclosure includes a search information acquisition unit that acquires information regarding a search tag for searching a desired video, and a search information acquisition unit that searches for a video corresponding to the search tag by referring to video analysis data. The video search unit may further include a video search unit for searching.

The video analysis device according to an embodiment of the present disclosure may further include a display unit that refers to the video analysis data and displays an image at the start time of the scene.

In the video analysis device according to an embodiment of the present disclosure, camera work may include at least one of panning, tilting, tracking, zooming in, character dolly, zooming out, and pullback.

In a video analysis method according to an embodiment of the present disclosure, an acquisition unit acquires video identification information and a video file, and a camerawork detection unit detects a predetermined camerawork included in the video file based on the movement of the image. The tagging unit detects and extracts the start time and end time of the scene including the camerawork, and the tagging unit refers to a database that associates the type of camerawork with the nuance corresponding to the camerawork, and then, for each scene, A tag representing the nuance corresponding to the detected camera work was added, and the video analysis data creation unit included video identification information and associated information about the camera work, start time and end time, and tag for each scene. The present invention is characterized in that the video analysis data is created and the output unit outputs the video analysis data.

A video analysis program according to an embodiment of the present disclosure causes a processor to acquire video identification information and a video file, detect predetermined camera work included in the video file based on the movement of the image, and perform the camera work. The start time and end time of the included scene are extracted, and a tag representing the nuance corresponding to the detected camera work is created for each scene by referring to a database that associates the type of camera work with the nuance corresponding to the camera work. create video analysis data that includes video identification information and associate information about camera work, start time and end time, and tags for each scene, and output the video analysis data. It is characterized by

According to the video analysis device according to an embodiment of the present disclosure, it is possible to improve search performance for finding a target scene intended by a user from among a plurality of video files.

FIG. 1 is a configuration block diagram of a video analysis device according to an embodiment of the present disclosure. 1 is a flowchart for explaining the procedure of a video analysis method according to an embodiment of the present disclosure. It is an example of a terminal display screen when uploading a video file from a terminal to a video analysis device according to an embodiment of the present disclosure. It is an example of a tagging result screen based on camera work detected using the video analysis device according to an embodiment of the present disclosure. 1 is an example of video analysis data created using a video analysis device according to an embodiment of the present disclosure. It is a flowchart for explaining the procedure of detecting camera work from a video file using a video analysis device according to an embodiment of the present disclosure. This is an example of a database showing the correspondence between camera work and nuances (impression tags) obtained from the camera work. It is an example of a terminal display screen when searching for a video file from a terminal in a video analysis device according to an embodiment of the present disclosure. FIG. 2 is a configuration block diagram of a video analysis device according to a modification of an embodiment of the present disclosure. It is an example of an additional information extraction result screen extracted from a scene including camera work detected using a video analysis device according to a modification of an embodiment of the present disclosure. It is an example of additional information analysis data created using a video analysis device according to a modification of an embodiment of the present disclosure. It is an example of video analysis data including additional information created using a video analysis device according to a modification of an embodiment of the present disclosure.

Hereinafter, a video analysis device, a video analysis method, and a video analysis program according to the present invention will be described with reference to the drawings. However, it should be noted that the technical scope of the present invention is not limited to these embodiments, but extends to the invention described in the claims and equivalents thereof.

(Overview of video analysis system)
First, an overview of a video analysis system including a video analysis device according to an embodiment of the present disclosure will be described. FIG. 1 shows a configuration diagram of a video analysis system 101 according to an embodiment of the present disclosure. The video analysis system 101 includes a video analysis device 10 and a terminal 20, which are connected via a communication network 30 so as to be able to transmit and receive data by wire or wirelessly. Although only one terminal 20 is shown in FIG. 1, there may be a plurality of terminals 20.

A user uses the terminal 20 to upload a video file to the video analysis device 10. The video file may be one created by the user, or may be part or all of an existing video file.

The video analysis device 10 may be, for example, a server. The video analysis device 10 performs image analysis on the video file received from the terminal 20 and detects camera work included in the video file. Here, "camera work" refers to how to move the camera when shooting a video. Camera work includes, for example, ``pan'', which is an operation to rotate the camera horizontally, ``tilt'', which is an operation to rotate the camera vertically, and ``pan,'' which is an operation to move the camera horizontally or vertically. There are "trucks" etc. However, the examples are not limited to these examples.

By changing the camera work, you can change the expression and psychological impression (nuance). For example, if a video includes a scene where the camera work is "panning", that scene is considered to express "impression" or "dynamics". If you prepare a database that associates various types of camera work with the nuances expressed by the camera work, you can refer to the database based on the detected camera work and extract the nuances contained in that video. can do.

By tagging the video file with the nuances extracted in this way, it is possible to objectively identify what kinds of nuances scenes are included in the video file. By preparing video analysis data tagged with nuances extracted from video files, video files containing desired nuances can be searched based on the tags.

For example, a case will be described in which a user who is about to create a video searches for a video containing a desired scene from among a plurality of videos currently uploaded for reference. In this case, it takes a lot of effort to play each of the multiple videos from beginning to end to find the desired scene. Therefore, for example, when a user searches for a video that includes a moving scene, it would be efficient if the user could search for which video and in which time period the moving scene is included.

According to the video analysis system according to an embodiment of the present disclosure, by detecting camerawork included in a video file and tagging the video file with nuances expressed by the detected camerawork, searchability of a desired video can be improved. can be improved.

(Configuration of video analysis device)
Next, a video analysis device 10 according to an embodiment of the present disclosure will be described. As shown in FIG. 1, the video analysis device 10 includes a control section 11, a transmitter/receiver section 12, a storage section 13, an output section 14, a timer section 15, and a display section 16. They are connected by a bus 17.

The transmitting/receiving unit 12 transmits and receives data to and from the terminal 20 via the communication network 30. In particular, the transmitter/receiver 12 acquires a video file and information for searching for a video file from the terminal 20.

The storage unit 13 is a storage device such as a semiconductor memory or a hard disk. The storage unit 13 stores video files acquired from the terminal 20 and a database. Furthermore, the storage unit 13 may store a program for controlling the video analysis device 10.

The control unit 11 includes an acquisition unit 1, a camerawork detection unit 2, a tagging unit 3, a video analysis data creation unit 4, a search information acquisition unit 5, and a video search unit 6. , is realized by executing a program stored in the storage unit 13 by a processor such as a CPU provided in the video analysis device 10.

The acquisition unit 1 acquires video identification information and a video file from the terminal 20 via the communication network 30. The video identification information is information for identifying a video file, and may be, for example, numbers, letters, symbols, or a combination thereof. A video file is a file in which video and audio are stored. The format of the video file is, for example, MP4, MOV, WMV, AVI, etc., but is not limited to these examples. The acquisition unit 1 outputs the acquired video file to the camera work detection unit 2.

The camerawork detection unit 2 detects a predetermined camerawork included in the video file based on the movement of the image, and extracts the start time and end time of the scene including the camerawork. A method for detecting camera work from a video file will be described later.

The tagging unit 3 refers to a database that associates types of camerawork with nuances corresponding to the camerawork, and adds a tag representing the nuance corresponding to the detected camerawork to each scene. The database may be stored in the storage unit 13 in advance. One or more nuances may be expressed by one camera work.

Note that instead of obtaining information relating the type of camera work and the nuance corresponding to the camera work by referring to a database, the relationship between the camera work and the nuance corresponding to the camera work may be incorporated into the program.

The video analysis data creation unit 4 creates video analysis data that includes video identification information and associates camera work, start time, end time, and tag information for each scene.

The output unit 14 outputs video analysis data. For example, the output unit 14 may output the video analysis data to the terminal 20 via the communication network 30.

The clock unit 15 is a module that outputs the current time. The timer 15 may measure the start time and end time of the scene where the camerawork detecting unit 2 changes the camerawork from the video file as the elapsed time from the start time of the beginning of the video.

The display unit 16 is a display device such as a liquid crystal display device or an organic EL display device. The display unit 16 may display the video analysis data, and may refer to the video analysis data to display an image at the start time of the scene.

The search information acquisition unit 5 acquires information regarding a search tag for searching for a desired video. The search tag may be input to the input unit 21 of the terminal 20 and acquired by the search information acquisition unit 5 via the communication network 30.

The video search unit 6 refers to the video analysis data and searches for a video corresponding to the search tag. Video analysis data includes video identification information, and information about camera work, start time and end time, and tags are associated with each scene, so based on the search tag, videos containing the desired scene can be searched. Files can be searched.

(Terminal configuration)
The terminal 20 is a terminal owned by the user, and may be a mobile terminal such as a mobile phone, a smartphone, or a tablet terminal, or an information terminal such as a personal computer. The terminal 20 includes an input section 21 , a storage section 22 , a control section 23 , a transmitting/receiving section 24 , and a display section 25 .

For the input unit 21, an input device such as a keyboard or a mouse can be used. The input unit 21 can upload a video file to the video analysis device 10 and input a tag to be searched.

The storage unit 22 is a storage device such as a semiconductor memory or a hard disk. The storage unit 22 stores data such as a video file to be uploaded to the video analysis device 10 and a URL indicating a storage location of the video file to be uploaded. Furthermore, the storage unit 22 may store a program for controlling the terminal 20.

The control unit 23 controls the operation of the terminal 20 using a program stored in the storage unit 22. A processor such as a CPU can be used for the control unit 23.

The transmitter/receiver 24 is connected to the communication network 30 and can transmit and receive data to and from the video analysis device 10 . As the terminal 20, not only a physical terminal as described above but also a virtual terminal may be used. By using a virtual terminal, applications and data can be aggregated in the video analysis device 10, and no data can be left on the terminal 20, so information leakage can be prevented.

For the display section 25, a display device such as a liquid crystal display device or an organic EL display device can be used.

(How to create video analysis data)
Next, a method of creating video analysis data using a video analysis device according to an embodiment of the present disclosure will be described. FIG. 2 shows a flowchart for explaining the procedure of a video analysis method according to an embodiment of the present disclosure.

First, in step S101, a user uploads a video file using the terminal 20, and the acquisition unit 1 of the video analysis device 10 acquires the uploaded video file.

FIG. 3 shows an example of a video upload screen 200 that is a terminal display screen when uploading a video file from the terminal 20 to the video analysis device 10 according to an embodiment of the present disclosure. First, in order to select a video file to be uploaded, the file name of the video file is input into the video file input field 201. Alternatively, a video file stored in the storage unit 22 may be selected by clicking on the video file input field 201.

Further, in the video file input field 201, a URL indicating where to obtain the video file may be input.

If there is only one video to be uploaded, after selecting the video file to be uploaded (video file (1)), the upload button 202 is pressed to execute the upload of the video file to the video analysis device 10.

Furthermore, when adding a video to be uploaded, an add button 203 is pressed. Then, a video file input field 204 for uploading the second video file, video file (2), is displayed.

After selecting the video file to be uploaded by clicking on the video file input field 204, the upload to the video analysis device 10 can be executed by pressing the upload button 202. Thereafter, three or more video files may be similarly uploaded to the video analysis device 10.

Note that when uploading a video file, the file name of the video file input in the video file input field (201, 204) becomes the video identification information. Therefore, if the video file name that has been uploaded has already been uploaded to the video analysis device 10 and a video file with the same file name exists in the storage unit 13 of the video analysis device 10, the video analysis device 10 The terminal 20 may be prompted to change the file name.

Next, in step S102, the camera work detection unit 2 detects a predetermined camera work included in the video file based on the movement of the image, and extracts the start time and end time of the scene including the camera work. The procedure by which the camera work detection unit 2 detects camera work from a video file will be described later.

Next, as shown in FIG. 2, in step S103, the tagging unit 3 refers to a database that associates the type of camerawork with nuances corresponding to the camerawork, and identifies the detected camerawork for each scene. Assign a tag (impression tag) that represents the corresponding nuance.

FIG. 4 shows an example of a tagging result screen 300 based on camera work detected using the video analysis device according to an embodiment of the present disclosure. In the example shown in FIG. 4, the detected camera work and the detection result 301 of the impression tag corresponding to the camera work are displayed on the upper side of the tagging result screen 300, and the detected camera work (CW1 to CW4) is displayed on the lower side. Scenes including the above are displayed together with thumbnails (SN1 to SN3).

For example, assume that a video is played back, camera work CW1 is detected at time (00:00) [mm:ss], and camera work CW1 is "pan". In this case, the tagging unit 3 refers to the database described later and extracts the nuances "impressive" and "dynamic" corresponding to the camerawork CW1 "pan" as impression tags (1) and (2), respectively. .

Assume that the video is further played back and the next camera work CW2 is detected at time (00:10). Then, a video whose start time is time (00:00) and whose end time is time (00:10) constitutes one scene. Therefore, this scene is defined as the first scene. As shown in FIG. 4, the tagging result screen 300 may display the thumbnail SN1 of the first scene together with the time axis. Next, the tagging unit 3 adds "impression" to the first scene as an impression tag (1) and "dynamic" as an impression tag (2).

Next, assume that the camera work CW2 detected at time (00:10) is "Character Dolly". In this case, the tagging unit 3 refers to the database described later, and sets the nuances "three-dimensional effect" and "powerful" corresponding to the camera work CW2 "character dolly" as impression tags (1) and (2), respectively. Extract.

Assume that the video is further played back and the next camera work CW3 is detected at time (00:20). Then, a video whose start time is time (00:10) and whose end time is time (00:20) constitutes one scene. Therefore, this scene is defined as the second scene. As shown in FIG. 4, the tagging result screen 300 may display the thumbnail SN2 of the second scene together with the time axis. Next, the tagging unit 3 adds "three-dimensional effect" to the second scene as an impression tag (1), and "power" as an impression tag (2).

Next, assume that the camera work CW3 detected at time (00:20) is "tilt". In this case, the tagging unit 3 refers to the database and assigns impression tags (1) to ( Extract as 3).

Assume that the video is further played back and the next camera work CW4 is detected at time (00:30). Then, a video whose start time is time (00:20) and whose end time is time (00:30) constitutes one scene. Therefore, this scene is designated as the third scene. As shown in FIG. 4, the tagging result screen 300 may display the thumbnail SN3 of the third scene together with the time axis. Next, the tagging unit 3 adds "sadness" as an impression tag (1) to the third scene, "feeling of loneliness" as an impression tag (2), and "worries" as an impression tag (3). Grant.

As described above, the tagging result screen 300 displays each of the first to third scenes divided by the detected camerawork CW1 to CW4 as the detection result 301 of the impression tag corresponding to the camerawork. Impression tags (1) to (3) may be displayed.

Next, in step S105, the output unit 14 outputs the video analysis data to the user's terminal 20. Specifically, first, the output unit 14 of the video analysis device 10 outputs video analysis data to the transmission/reception unit 12. Next, the transmitter/receiver 12 of the video analysis device 10 transmits the video analysis data to the transmitter/receiver 24 of the terminal 20 via the communication network 30 . The transmitting/receiving unit 24 of the terminal 20 outputs the received video analysis data to the display unit 25.

FIG. 5 shows an example of video analysis data created using the video analysis device according to an embodiment of the present disclosure. The display unit 25 of the terminal 20 may display a video analysis data display screen 400 that includes video analysis data 401. Video analysis data 401 includes video ID (B1), scene B2, scene start time B3, scene end time B4, scene duration B5, detected camera work B6, impression tag (1) B7, and impression tag ( 2) B8, including impression tag (3) B9. However, it is not limited to such an example.

The video ID (B1) is video identification information. For example, the video file name input by the user on the video upload screen 200 may be used as the video ID.

Scene B2 is information for identifying a scene included in the video file. For example, names including consecutive numbers such as first scene, second scene, etc. may be given. However, the name of the scene is not limited to this example, and may be any name, for example, a name corresponding to an impression tag.

Start time B3 is the time when camera work is detected in the scene. For example, if camera work is detected at time (00:10), time (00:10) becomes the start time of the second scene.

End time B4 is the time when the next camera work is detected in a certain scene. For example, if the next camera work is detected at the time (00:20) of the second scene, the time (00:20) becomes the end time of the second scene.

The duration B5 is the time from the start time to the end time in a certain scene. For example, if the start time of the second scene is (00:10) and the end time is (00:20), the duration is 10 [sec].

Camera work B6 is camera work detected at the beginning of a certain scene. For example, if the camera work "Character Dolly" is detected at the start time (00:10) of the second scene, the camera work of the second scene becomes "Character Dolly".

Impression tags (1) to (3) (B7 to B9) are tags representing nuances corresponding to the detected camera scene, and are extracted with reference to a database described later. For example, if camera work "tilt" is detected in the third scene of the video, impression tags of "sadness", "loneliness", and "worries" are extracted.

(Method of detecting camera work)
Next, a method of detecting camera work from a video file using the video analysis device 10 according to an embodiment of the present disclosure will be described. FIG. 6 shows a flowchart for explaining a procedure for detecting camera work from a video file using the video analysis device 10 according to an embodiment of the present disclosure. Here, as a method for detecting camera work from video files, we will introduce "Atsuo Yoshitaka, Ryoji Matsui, So Hiramatsu, "Extraction of emotional information using camera work," Information Processing Society of Japan Transactions, which is shown in Non-Patent Document 1. Vol. 47, No. 6, p. 1696-1707'' will be explained by taking as an example a method for analyzing a spatio-temporal projection image. A spatio-temporal projection image is an image in which images on a straight line or diagonal line horizontally, vertically, or diagonally to the frame at a certain position within the frame are extracted from each frame and arranged in the temporal direction.

First, in step S201, the camera work detection unit 2 generates a spatiotemporal projection image from the video file acquired from the acquisition unit 1. Specifically, in a certain moving image, a straight line L (fixed line) within a certain frame is determined, the straight lines L are extracted from each frame, and the straight lines L are arranged along the time axis to generate a spatiotemporal projection image.

Next, in step S202, edges are detected from the spatiotemporal projection image. When camera work is applied to a video, a straight line component appears in a spatiotemporal projection image in a portion where a stationary object on a fixed line has a steep change in brightness. Therefore, straight edges are detected in the binarized spatiotemporal projection image.

Next, in step S203, a section to be referred to for camera work detection is determined. For example, in consideration of actual video work, there are two types of reference sections: narrow range and wide range, and the narrow range is included for a predetermined period of time (for example, 2 seconds) from the beginning of the shot, and the wide range is included in the narrow range judgment interval. This is the continuous section from the minimum start point time to 80% of the maximum end point time among the straight edge groups.

Next, in step S204, the camerawork detection unit 2 determines the type of camerawork from the edge pattern that appears due to the camerawork in the generated spatiotemporal projection image. The camera work to be detected is (1) pan, (2) tilt (including tilt up and tilt down), (3) track, (4) zoom in, (5) character dolly, (6) zoom out, (7) There are seven types of pullbacks.

For example, if the camera work involves panning, tilting, tracking, or zooming, use a device such as a tripod or moving vehicle to move the camera back and forth to smoothly change the camera's position and orientation, or the focal length of the lens. change. Therefore, when the camera work is panning, tilting, tracking, or zooming, the edge that appears in response to the movement of the camera work is an edge whose direction is approximately constant and whose length is proportional to the camera operation time. That is, the type of camera work is determined by determining which linear component appears for which of the spatio-temporal projection images in which horizontal lines, vertical lines, and diagonal lines are fixed lines. Note that for the relationship between specific edge patterns and camera work detection in the method of analyzing spatio-temporal projection images, please refer to Non-Patent Document 1.

Based on the camerawork detected as described above, in order to link the nuances expressed by the camerawork to the video as tags (impression tags), a database is prepared in advance that records the relationship between camerawork and nuances. It is preferable to leave it there.

(Relationship between camera work and nuance)
Next, the relationship between camera work and the nuances expressed by camera work will be explained. FIG. 7 shows an example of a database representing the correspondence between camera work and nuances (impression tags) obtained from the camera work. The database 500 may include camera work A1, camera/lens movement A2, and impression tags (1) to (5) (A3 to A7). Camera work A1 includes, but is not limited to, "pan", "tilt", "track", "zoom in", "character dolly", "zoom out", "pull back", etc.

Camera work "Pan" refers to the operation of rotating the camera in the horizontal direction. If a video file contains a scene captured using "pan" camera work, that scene is considered to express nuances such as "impressive" or "dynamic," and these nuances are used as impression tags. (1) and (2) may be used.

Camera work "Tilt" refers to the operation of rotating the camera in the vertical direction. If a video file contains a scene captured using "tilt" camera work, that scene is likely to express nuances such as "sadness," "loneliness," or "worries." , these nuances may be used as impression tags (1) to (3). Note that the camera work "tilt" includes "tilt down" and "tilt up." In the example shown in FIG. 7, nuances (impression tags) corresponding to "tilt down" are shown. When "tilt" is "tilt up," the corresponding nuance may be, for example, "hope," "admiration," "forward," or "expectation."

Camera work "track" refers to the operation of moving the camera horizontally or vertically. If a video file contains a scene captured by the camera work "track", that scene is considered to express nuances such as "powerful" or "objective", and these nuances are used to give an impression. It may be used as tags (1) and (2).

Camera work "Zooming in" refers to an enlargement operation that changes the focal length of the camera. If a video file contains a scene captured by the camera work "zoom in", that scene is considered to express a nuance such as "feeling of tension", and this nuance is used as an impression tag (1). good.

Camera work "Character dolly" refers to an enlargement operation that brings the camera closer to the subject. If a video file contains a scene captured using the camera work "Character Dolly," that scene is likely to express nuances such as "three-dimensional effect" or "powerful force," and these nuances are It may be used as impression tags (1) and (2).

Camera work "Zooming out" refers to a reduction operation that changes the focal length of the camera. If the video file contains a scene captured by the camera work "Zoom Out", the scene may be, for example, "Sadness", "Worship", "Loneliness", "Feeling of freedom", or "Relaxation". These nuances may be used as impression tags (1) to (5).

Camera work "pullback" refers to a reduction operation that moves the camera away from the subject. If the video file contains a scene captured by the camera work "pullback", the scene has nuances such as "breakup", "sadness", "feeling of loss", "loneliness", or "feeling of freedom". These nuances can be used as impression tags (1) to (5).

The camera work is not limited to that shown in FIG. 7, and may include other photographing methods. Furthermore, when associating camera work with nuances corresponding to the camera work, a machine learning model based on artificial intelligence (AI) may be used.

Further, the nuances corresponding to various camera works included in the database 500 may be changed as appropriate. The impression tags included in the database 500 may be modified by the user or by the administrator of the video analysis device 10.

Further, when one camera work includes impression tags representing a plurality of nuances, these impression tags may be prioritized. For example, the priority order of impression tags corresponding to one camera work may be set in descending order from impression tags (1) to (5).

It is also conceivable that there are multiple camera movements corresponding to one nuance (impression tag). For example, as shown in FIG. 7, camera work corresponding to the impression tag "powerful" includes "track" and "character dolly." In this case, as the camera work corresponding to the impression tag "Impact", you may select the camera work "Track" for which "Impact" corresponds to the impression tag (1) with the highest priority, or "Track". , "Character Dolly" in order of priority.

(How to search video files)
Next, a method of searching for a video file to which a target tag has been added based on the created video analysis data will be described. FIG. 8 shows an example of a video search screen 600 that is a terminal display screen when searching a video file from a terminal for a video analysis device according to an embodiment of the present disclosure.

First, on the video search screen 600, a search tag (1) for searching a video is input into the search tag (1) input field 601. In the example shown in FIG. 8, the word "impression" is input as the search tag (1).

Note that instead of directly inputting the search tag word in the search tag (1) input field 601, a pull-down menu may be displayed and a desired search tag may be selected from among the displayed search tags.

If there is only one search tag, after inputting search tag (1), the search start button 602 is pressed to execute a search for video files.

Furthermore, if a search tag is to be added, an add tag button 603 is pressed. Then, a search tag (2) input field 604 for inputting the second search tag, search tag (2), is displayed. In the example shown in FIG. 8, the word "sadness" is input as the search tag (2).

After inputting the search tags (1) and (2), by pressing the search start button 602, a search for the video files associated with the search tags (1) and (2) can be executed. Hereinafter, similarly, a search for video files may be executed using three or more search tags.

When the search start button 602 is pressed, information regarding the search tag is input to the input unit 21 (FIG. 1) of the terminal 20 and output to the transmitting/receiving unit 24. Information regarding the search tag is transmitted from the transmitter/receiver 24 to the transmitter/receiver 12 of the video analysis device 10 via the communication network 30 .

The transmitting/receiving unit 12 of the video analysis device 10 outputs the received information regarding the search tag to the search information acquisition unit 5. The search information acquisition unit 5 acquires information regarding search tags for searching for a desired video, and outputs the information to the video search unit 6.

The video search unit 6 refers to the video analysis data and searches for a video corresponding to the search tag. The video search unit 6 refers to the video analysis data 401 shown in FIG. It is detected that the scene is included in three scenes.

The video search section 6 outputs the search results to the transmitting/receiving section 12, and the transmitting/receiving section 12 transmits information regarding the search results to the terminal 20 via the communication network 30. The transmitting/receiving unit 24 of the terminal 20 receives information regarding the search results and outputs it to the display unit 25.

Information regarding the search results may be displayed by the display unit 25 in a search result display field 605 of the video search screen 600, as shown in FIG. The search result display field 605 may display the searched video file name C1, the scene C2 with the searched tag, and the evaluation C3.

The searched video file name C1 is, for example, the file name of a video containing the search tag (1) "impression" and the search tag (2) "sadness", and in the example shown in FIG. .mp4", "MP2022.wmv", and "DG7777.avi" are detected.

The searched tagged scene C2 is the scene number tagged with the search tag (1) "emotion" and the search tag (2) "sadness." For example, for the video file "1001.mp4", it indicates that at least one of search tags (1) and (2) is attached to the first scene SN001 and the third scene SN0003. There is.

Although the search result display field 605 shown in FIG. 8 shows an example in which the number of detected video files is three, the number of detected video files is not limited to this example. It may be less than three or more than four.

Furthermore, if a plurality of video files are detected, they may be displayed in descending order of priority. The prioritization may be in order of the duration of the scenes that include the search tags. Alternatively, the prioritization may be in order of the number of times scenes containing the search tag have been detected.

The display unit 16 may refer to the video analysis data and display an image at the start time of the searched scene. For example, instead of displaying the number of the first scene shown in the search result display field 605 in FIG. 8, the image at the start time of the first scene may be displayed as a thumbnail.

Furthermore, by clicking on the scene number (for example, "SN001") or its thumbnail displayed in the search result display field 605, the moving image of that scene may be played. For example, since the start time of the first scene is time (00:00), the display unit 16 refers to the video analysis data 401 (FIG. 5) and indicates that "SN001", which is the number of the first scene, has been clicked. In this case, the video of the first scene may be played.

After selecting the number or thumbnail displayed in the scene C2 and playing back the video of that scene, the user may input in the evaluation input field C3 whether or not the scene was the intended scene. The evaluation input in the evaluation input field C3 may be, for example, in five stages. The example shown in FIG. 8 shows that the evaluation for the search results for scenes SN001 and SN003 of the video file "1001.mp4" was "4" out of five. The evaluation input in the evaluation input field C3 is not limited to five levels, and any evaluation method may be used.

The evaluation results are output to the video analysis data creation unit 4, and the video analysis data creation unit 4 may modify a database that records the correspondence between camera work and impression tags representing nuances based on the evaluation results. For example, if a certain scene is given a low evaluation, the relationship between the tag given to that scene and the corresponding camera work may be corrected. When modifying a database, a machine learning model may be used to perform the modification.

In the manner described above, camera work can be detected from a video file and the nuances expressed by the camera work can be tagged. This allows the user to search for a video that includes the desired nuance based on the tag, and improves the search performance for finding the target scene that the user intended.

(Modified example)
Next, a video analysis system according to a modification of the embodiment of the present disclosure will be described. FIG. 9 shows a configuration block diagram of a video analysis system 102 according to a modification of an embodiment of the present disclosure. A video analysis system 102 according to a modification of an embodiment of the present disclosure differs from the video analysis system 101 shown in FIG. Furthermore, the tagging unit 3 adds a tag based on an object to the video analysis data. The rest of the configuration of the video analysis system 102 according to the modification is the same as the configuration of the video analysis system 101 shown in FIG. 1, so a detailed explanation will be omitted.

As described above, it is possible to search for a video containing a target scene using impression tags tagged to videos based on camera work included in the video. On the other hand, by adding objects included in the video to search tags, it is possible to further improve the searchability of the target video. Therefore, in the video analysis system 102 according to the modified example, the video is searched by adding information regarding the object included in the video as additional information to the search tag.

FIG. 10 shows an example of an additional information extraction result screen 700 extracted from a scene including camera work detected using a video analysis device according to a modification of an embodiment of the present disclosure. The object recognition unit 7 recognizes an object in an image included in a video file by image recognition. For example, in the first scene SN1, objects P1 to P4 are detected, and a "window", "chair", "desk", and "bag" are respectively recognized by image recognition. Similarly, in the second scene SN2, objects P5 and P6 are detected, and "woman" and "dog" are respectively recognized by image recognition.

The video analysis data creation unit 4 may create additional information analysis data 800 shown in FIG. 11 as additional information based on information regarding the object recognized by the object recognition unit 7. The additional information analysis data 800 may include a video ID (D1), a scene D2, a detection time D3, an additional information type D4, and tags (1) to (4) (D5 to D8).

The detection time D3 is the time when the object was detected. For example, FIG. 11 shows that objects P1 to P4 were detected at time (00:05) in the first scene.

The video analysis data creation unit 4 may combine the additional information analysis data 800 and the video analysis data 401 shown in FIG. 5 to create video analysis data including additional information. FIG. 12 shows an example of video analysis data 900 including additional information created using a video analysis device according to a modification of an embodiment of the present disclosure.

The video analysis data 900 including additional information may include items E1 to E13. E1 to E9 are the same as B1 to B9 of the moving image analysis data 401 shown in FIG. The video analysis data 900 including additional information may include tags (1) to (4) (E10 to E13) in addition to the video analysis data 401.

Tags (1) to (4) (E10 to E13) are tags corresponding to information regarding the object recognized by the object recognition unit 7. For example, the object P1 detected in the first scene is recognized as a "window", and this is stored as the tag (1) E10 in the video analysis data 900 including additional information. Similarly, objects P2 to P4 detected in the first scene are recognized as "chair," "desk," and "bag," respectively, and are assigned tags (2) to (4) (E11 to E13), respectively. It is stored in the video analysis data 900 including additional information.

By searching for videos using tags (1) to (4) in addition to the impression tags (1) to (3) shown in Figure 12, it is possible to more accurately search for the desired video. . For example, in the video search screen 600 shown in FIG. 8, a tag related to an object can be entered in search tag (1) or (2) to execute a search.

In the above example, an example was described in which an object included in a video is recognized and tagged as additional information, but the additional information is not limited to information regarding the object, but may be information regarding audio included in the video. Therefore, as shown in FIG. 9, the video analysis device 10 according to the modified example further includes a voice recognition unit 8 that recognizes the audio included in the video file, and the tagging unit 3 adds tags based on the audio to the video analysis device. May be added to data.

The voices recognized by the voice recognition unit 8 may include sounds included in moving images, such as human conversations and remarks, animal sounds, environmental sounds (sounds of waves, wind, thunder, etc.), BGM, and the like.

For example, as shown in FIG. 10, when a voice S1 such as "Do you want that that much?" is detected in the third scene, it is converted into text by voice recognition. The converted character information is stored in the additional information analysis data 800 as the tag (1) D5 detected at the time (00:25) of the third scene, and is stored in the video analysis data 900 including additional information.

Furthermore, in the above example, an example was explained in which objects and sounds included in a video are recognized and tagged as additional information. However, additional information is not limited to information about objects and sounds, and information about characters included in a video is also tagged as additional information. may be included. Therefore, as shown in FIG. 9, the video analysis device 10 according to the modification further includes a character recognition unit 9 that recognizes characters included in the video file, and the tagging unit 3 analyzes tags based on characters in the video file. May be added to data.

For example, as shown in FIG. 10, when characters P7 "KEEP AND GROW" and characters P8 "SOUP" are detected in the second scene, they are converted into text by character recognition. The converted character information is stored in the additional information analysis data 800 as tag (1) D5 and tag (2) D6 detected at the time of the second scene (00:17), and is stored in the video analysis data including additional information. 900.

Note that when detecting text information from a video file, it may be possible to recognize not only characters displayed on the background or objects, but also characters such as subtitles and subtitles added by editing the video.

As mentioned above, by not only tagging video files with impression tags related to the nuances extracted from camerawork, but also tagging information about objects, sounds, text, etc. included in the video, users can accurately achieve their goals. You can search for videos.

1 Acquisition unit 2 Camera work detection unit 3 Tagging unit 4 Video analysis data creation unit 5 Search information acquisition unit 6 Video search unit 7 Object recognition unit 8 Voice recognition unit 9 Character recognition unit 10 Video analysis device 11 Control unit 12 Transmission/reception unit 13 Storage unit 14 Output unit 15 Time measurement unit 16 Display unit 17 Internal bus 20 Terminal 21 Input unit 22 Storage unit 23 Control unit 24 Transmission/reception unit 25 Display unit 30 Communication network 101, 102 Video analysis system

Claims (9)

an acquisition unit that acquires video identification information and video files;
a camerawork detection unit that detects predetermined camerawork included in the video file based on image movement and extracts a start time and an end time of a scene including the camerawork;
a tagging unit that refers to a database that associates types of camerawork with nuances corresponding to the camerawork, and adds a tag representing the nuance corresponding to the detected camerawork to the scene;
a video analysis data creation unit that creates video analysis data that includes video identification information and associates information regarding the camera work, the start time and end time, and the tag for each scene;
an output unit that outputs the video analysis data;
A video analysis device comprising: further comprising an object recognition unit that recognizes an object in an image included in the video file,
The tagging unit adds an object-based tag to the video analysis data.
The video analysis device according to claim 1. further comprising a voice recognition unit that recognizes voice included in the video file,
The tagging unit adds a tag based on audio to the video analysis data.
The video analysis device according to claim 1 or 2. further comprising a character recognition unit that recognizes characters in images included in the video file,
The tagging unit adds a text-based tag to the video analysis data.
The video analysis device according to claim 1 or 2. a search information acquisition unit that acquires information regarding a search tag for searching for a desired video;
a video search unit that searches for a video corresponding to the search tag by referring to the video analysis data;
The video analysis device according to claim 1 or 2, further comprising: The video analysis device according to claim 1 or 2, further comprising a display unit that displays an image at a scene start time with reference to the video analysis data. The video analysis device according to claim 1 or 2, wherein the camera work includes at least one of panning, tilting, tracking, zooming in, character dolly, zooming out, and pullback. The acquisition unit acquires the video identification information and the video file,
a camerawork detection unit detects a predetermined camerawork included in the video file based on the movement of the image, and extracts a start time and an end time of a scene including the camerawork,
The tagging unit refers to a database that associates types of camerawork with nuances corresponding to the camerawork, and adds a tag representing the nuance corresponding to the detected camerawork to each scene,
a video analysis data creation unit creates video analysis data that includes the video identification information and associates information regarding the camera work, the start time and end time, and the tag for each scene;
an output unit outputs the video analysis data;
A video analysis method characterized by: to the processor,
Obtain video identification information and video file,
Detecting a predetermined camera work included in the video file based on the movement of the image, extracting the start time and end time of the scene including the camera work,
Referring to a database that associates the type of camera work with the nuance corresponding to the camera work, assigning a tag representing the nuance corresponding to the detected camera work to each scene,
creating video analysis data that includes the video identification information and associates information regarding the camera work, the start time and end time, and the tag for each scene;
outputting the video analysis data;
A video analysis program that executes each step.

Images