JPWO2014013690A1 - Comment information generating apparatus and comment information generating method - Google Patents

Abstract

The comment information generation device (100) displays a moving image acquisition unit (101) that receives a moving image, position information of an object in the moving image received by the moving image acquisition unit (101), and displays the object following the specific timing. A comment input receiving unit (102) that receives an input of a comment, and a target time determining unit that determines a target time that is a target value of a time length during which the comment is continuously displayed based on the comment received by the comment input receiving unit (102). (106) and object movement determination for determining the object movement locus so that the object movement locus for displaying the comment on the object indicated by the position information is a movement locus having a time length sufficiently close to the target time. Object (107), comment, and object determined by the object motion determination unit (107) Comprising output comment information generator for generating an output comment information including the moving locus of the bets and (104), an output unit that outputs an output comment information and (105).

Description

  The present invention relates to a comment information generating apparatus and a comment information generating method for generating comment information for displaying a comment superimposed on an object in a moving image.

  In recent years, due to the development of networks and the spread of mobile terminals, it has become commonplace to enjoy communication through a video content or a still image content with a remote partner via a network. For example, with respect to existing content, there is a service in which a user can write an individual's preferences and thoughts by sentences at a certain timing on the content time axis. Such communication through content is expected to increase more and more with the development of display devices and communication technology.

  In order to realize such communication, Patent Document 1 discloses a technique for generating a moving image in which a comment given by a user is synthesized on a moving image and distributing it on the Internet.

JP 2008-148071 A JP 2010-244437 A International Publication No. 2010/116820 JP 2004-128614 A JP 2009-81592 A JP 2003-130447 A Japanese Patent No. 4994525

P. Anandan, “A Computational Framework and an Algorithm for the Measurement of Visual Motion”, International Journal of Computer Vision, Vol. 2, pp. 283-310, 1989 Vladimir Kolmogorov and Ramin Zabih, “Computing Visual Correspondence with Occlusions via Graph Cuts”, International Conference on Computer Vision, 2001 Jianbo Shi and Carlo Tomasi “Good Features to Track”, IEEE Conference on Computer Vision and Pattern Recognition, pp 593-600, 1994 Pedro F.M. Felzenszwalb and Daniel P. Huttenlocher “Efficient Graph-Based Image Segmentation”, International Journal of Computer Vision, Vol. 59, no. 2, pp. 167-181, Sept, 2004

  However, according to the conventional method for superimposing a comment (or generating a moving image with a comment), the comment is displayed at a position unrelated to the object in the moving image. Also, the comment display time is fixed. For this reason, it is difficult for viewers of moving images to understand which object each comment contributor gives a comment to. Also, in the case of a very long comment, it is impossible to finish reading the comment within the fixed comment display time. For this reason, there exists a subject that the visibility of the comment by a viewer falls.

  The present invention has been made to solve the above-described problem, and an object of the present invention is to provide a comment information generation device and a comment information generation method for generating output comment information that can improve the visibility of a comment. To do.

  In order to achieve the above object, a comment information generation device according to an aspect of the present invention includes a moving image acquisition unit that receives a moving image, position information of an object in the moving image that the moving image acquisition unit receives, and the object A comment input receiving unit that receives an input of a comment to be displayed following from a specific timing, and a target time that is a target value of a time length for continuously displaying the comment based on the comment received by the comment input receiving unit A target time determination unit for determining, and the movement trajectory of the object for following and displaying the comment on the object indicated by the position information is a movement trajectory of the time length of the target time. An object motion determining unit for determining a movement locus, the comment, and the object motion determining unit; And an output comment information generator for generating an output comment information including the moving locus of boss was the object, and an output unit which outputs the output comment information said output comment information generating unit has generated.

  These general or specific aspects may be realized by a system, a method, an integrated circuit, a computer program, or a recording medium such as a computer-readable CD-ROM. The system, method, integrated circuit, computer program And any combination of recording media.

  According to the present invention, it is possible to generate output comment information that can improve comment visibility.

FIG. 1 is a diagram illustrating a configuration of a commented moving image distribution system targeted by the embodiment. FIG. 2A is a diagram illustrating a temporal flow of adding a comment to a moving image. FIG. 2B is a diagram illustrating a temporal flow of adding a comment to a moving image and viewing the moving image. FIG. 3 is a block diagram illustrating a functional configuration of the comment information generation device according to the embodiment. FIG. 4A is a diagram illustrating a hardware configuration of a comment information generation device configured by a computer. FIG. 4B is a diagram illustrating a hardware configuration of a comment information generation device configured by a computer. FIG. 5 is a flowchart showing an operation procedure of the comment information generating apparatus. FIG. 6A is a diagram illustrating an example of a moving image. FIG. 6B is a diagram illustrating a subject area in a moving image. FIG. 7A is a diagram for explaining a procedure for a user to input a comment on a moving image. FIG. 7B is a diagram illustrating a data example of input comment information. FIG. 8A is a diagram for explaining a method of calculating a movement trajectory by the object motion determination unit. FIG. 8B is a diagram for explaining a method of calculating a movement trajectory by the object motion determination unit. FIG. 9A is a diagram showing a movement trajectory when the tolerance of noise or model error is lowered. FIG. 9B is a diagram illustrating a movement trajectory when the tolerance of noise or model error is increased. FIG. 10A is a diagram for describing motion estimation processing in the case where occlusion occurs. FIG. 10B is a diagram for describing motion estimation processing when a scene change occurs. FIG. 11A is a diagram for explaining the extension process of the movement trajectory by tracing back the comment start time. FIG. 11B is a diagram for describing an example of calculating a movement locus that satisfies a target time from within a predetermined pixel range. FIG. 12A is a diagram for describing an example of calculating a movement trajectory that satisfies a target time from within a region designated by a user. FIG. 12B is a diagram for describing an example of calculating a movement trajectory that satisfies a target time from the same region divided by the region dividing unit. FIG. 13A is a diagram illustrating an example of a movement trajectory of an area when rough area division is performed. FIG. 13B is a diagram illustrating an example of a movement trajectory of an area when fine area division is performed. FIG. 14 is a diagram illustrating an example of a database. FIG. 15 is a block diagram illustrating a functional configuration of a comment information generation device including a target time correction unit.

(Knowledge that became the basis of the present invention)
The present inventors have found that the following problems arise with respect to the conventional comment superimposing method described in the “Background Art” column.

  The system described in Patent Document 1 is a system including a moving image server and a comment storage / delivery server, and all comments written by each user via the network are stored on the comment storage / delivery server, and comments are written. The reproduction time on the moving image is superimposed and distributed on the moving image using the reproduction time as a reference point (hereinafter, such a moving image is referred to as “moving image with a comment”). A comment newly written by the user each time a moving image or a moving image with a comment is distributed and played back is managed on the comment accumulation / delivery server in association with the time on the moving image time axis. Regardless of whether new or old, future distribution will be based on the time on the video time axis. Each comment is displayed so as to flow on the moving image or fixed at a certain position on the moving image regardless of the object to which the user has posted the comment.

  Patent Documents 2 and 3 disclose movie devices that add character information to moving images without using the Internet. Patent Documents 2 and 3 particularly indicate which object the user really wanted to give additional data by displaying the additional data as a balloon and changing the balloon according to the movement of the object in the moving image. We propose a method that makes it easy to see if there is any.

  Patent Document 4 discloses a chat system that displays balloons. The generated balloon is displayed on the face image of the user synthesized at a specific position on the common background so as not to hide the face image and other balloons.

  Patent Document 5 discloses a technique for inputting character information or picture information to a video recording apparatus from a mobile phone or a mobile terminal. The mobile phone or the mobile terminal includes a comment tracking menu, and tracks the tracking frame specified by the user using a motion vector.

  In general, it is said that the number of characters that a human can recognize per second is limited. For this reason, for example, for subtitles such as movies, voluntary guidelines such as the number of display characters per second and the number of display words per minute (WPM: Words Per Minute) are established, and the visibility when viewing subtitles is set. The device that is kept is made.

  As described above, conventionally, a method for inserting a comment into a moving image has been proposed. As described above, in the moving image distribution site with comments using the technique disclosed in Patent Document 1, the number of characters that can be displayed in one line from one end to the other end of the screen is limited. For this reason, for example, the setting that the comment is scroll-displayed from one end of the screen to the other end in 3 seconds is applied as a default. On the other hand, it is difficult to understand which object the user really wanted to give additional data.

  Therefore, an object of the present disclosure is to realize a “comment that moves following an object”. As a result, it is easy to understand which object the user really wanted to give additional data, so that the visibility of the user can be improved and communication can be deepened. However, in order to obtain a “comment that moves following an object”, it is necessary to obtain the coordinates of the movement trajectory calculated by the object motion estimation as the coordinates of the “comment that moves following the object”. For example, if the object motion estimation fails early, the comment display time is also shortened, which may cause a problem that the user does not finish reading the comment while the comment is displayed.

  Conversely, it is not possible to simply control the comment display time according to the number of characters in the comment, such as subtitles.

  Many of the moving images distributed by content distributors and personally owned content include a plurality of objects in a scene. In particular, when the size and color of the motion of each object are different, the object motion estimation result described above depends on the object selected by the user and the parameters used for the object motion estimation process in the process of calculating the trajectory of the object in the moving image. The length of time is often different.

  Generally, in the object motion estimation process, a motion is obtained by discriminating the same object region between time-series images. For example, an object in a certain frame is used as a model, an object on another frame that satisfies the model most (the brightness difference from the model is closest to 0) is determined as the same object, and motion is calculated from a change in position between the objects. .

  In addition, there are cases where motion estimation is impossible (cases to be interrupted), such as when an object no longer appears in a moving image. The luminance difference from the above model can also be used as a criterion for the interruption determination.

  In the motion estimation process based on the brightness difference with the model, there is a trade-off relationship between how much noise or the brightness difference with the model is tolerated (ie, the accuracy of motion estimation) and the object tracking time. is there. These trade-offs are not uniformly present in the image. For example, there are many situations in which motion estimation is interrupted earlier than the desired tracking time for one object on a moving image, while motion estimation is not interrupted for another object in the same moving image. Yeah. In other words, if uniform motion estimation conditions are used, the time until tracking ends and the accuracy of motion estimation vary depending on the coordinate position specified by the user with the intention of giving a comment and the time on the moving image. There is a problem of end.

  In order to solve such a problem, a comment information generation device according to an aspect of the present invention includes a moving image acquisition unit that receives a moving image, position information of an object in the moving image that the moving image acquisition unit receives, A comment input receiving unit that receives an input of a comment to be displayed following the object from a specific timing, and a target time that is a target value of a length of time for which the comment is continuously displayed are displayed in the comment received by the comment input receiving unit. A target time determination unit that is determined based on the object, and the object trajectory for displaying the comment following the object indicated by the position information is a trajectory of a time length of the target time. An object motion determination unit for determining a movement locus of the object, the comment, and the object motion determination. And an output comment information generator section to generate output comment information including the moving locus of the determined object, and an output unit which outputs the output comment information said output comment information generating unit has generated.

  According to this configuration, the target time for displaying the comment is determined based on the comment, and the movement trajectory of the object is determined so that the continuous time of the movement trajectory approaches the target time.

  Note that “to be a movement trajectory of the target time length” broadly means that the length of the movement trajectory from when the object motion estimation is started until it is interrupted is the target when the comment is displayed. It refers to a state corresponding to a length of time sufficiently close to time. In a narrow sense, it indicates a state in which the length of the movement trajectory from when the motion estimation is started until it is interrupted is a length corresponding to the target time when the comment is displayed. Hereinafter, when “target time” is used, it means the above “target time for displaying a comment”. How much difference time can be allowed to be “close enough” may vary depending on the frame rate of the moving image to be displayed, the coefficient for determining the target time, or the user. For this reason, it may be determined by an experiment in advance or may be selected by the user in advance. For example, at ± 0.25 seconds corresponding to the display time per character, it can be said that there is no sense of incongruity and “close enough”, so this result may be used.

  The determined movement trajectory of the object can be used as a movement trajectory for displaying the comment following. For this reason, the user can read the comment within the display time when the moving image with the comment is displayed, and can determine to which object the comment is given. Therefore, it is possible to generate output comment information that can improve comment visibility.

  For example, the target time determination unit calculates the target time so that the target time becomes longer as the length of the comment received by the comment input reception unit is longer, and the output unit receives the comment When a longer comment is input to the input receiving unit, a movement trajectory having a longer time may be output.

  The target time determination unit calculates a value obtained by multiplying a unit display time, which is a predetermined display time per character, and the number of characters of the comment received by the comment input reception unit as the target time. The output unit may output a movement trajectory having a length obtained by multiplying the number of characters of the comment by the unit display time when the comment is input to the comment input receiving unit.

  The target time determination unit further calculates the visual recognition time as the target time when the calculated target time does not satisfy a predetermined visual recognition time for visual recognition of characters, When the comment is input to the comment input reception unit, the output unit multiplies the number of characters of the comment by a predetermined unit display time which is a display time per character, and the visual recognition time. It is also possible to output a movement trajectory having a length longer than the visual recognition time and output a movement trajectory having a length equal to or longer than the visual recognition time, regardless of how short the comment is input.

  Here, the visual recognition time is the minimum time necessary for recognizing characters regardless of the number of characters.

  In addition, even if the plurality of comments received by the comment input receiving unit are the plurality of comments given to the same position in the same frame, the output unit may mutually Different movement trajectories may be output.

  Further, the object motion determination unit is the object indicated by the position information received by the comment input reception unit using each of a plurality of motion estimation methods or each of a plurality of motion estimation parameters, and the moving image The movement trajectory of the object in the image may be calculated, and the movement trajectory of the object may be determined by selecting a movement trajectory having a length closest to the target time among the calculated movement trajectories of the object.

  According to this configuration, it is easy to estimate a movement trajectory continuous for a target time by calculating the movement trajectory using a plurality of motion estimation methods or a plurality of motion estimation parameters.

  For example, the object motion determination unit, as each of the plurality of motion estimation parameters, (1) affects the ease of following the object, each of a plurality of error tolerance parameters having different values, and (2) each other. The object indicated by the position information received by the comment input receiving unit based on any of a plurality of search window regions having different sizes or (3) each of a plurality of feature amounts having different values. The movement trajectory of the object in the moving image is calculated, and the movement trajectory of the object is selected by selecting a movement trajectory having a length closest to the target time from the calculated movement trajectories of the object. You may decide.

  If the error is allowed, the movement trajectory becomes long, but the motion estimation accuracy deteriorates. On the contrary, if the error is not allowed, the movement trajectory is shortened but the motion estimation accuracy is improved. Further, if the size of the search window area is reduced, the movement trajectory becomes longer, but the motion estimation accuracy deteriorates. Conversely, if the size of the search window area is increased, the movement trajectory is shortened but the motion estimation accuracy is improved. Further, if the number of feature amounts is reduced, the movement trajectory becomes longer, but the motion estimation accuracy becomes worse. Conversely, if the number of feature quantities is increased, the movement trajectory is shortened, but the motion estimation accuracy is improved.

  In addition, the object motion determination unit is further unable to determine the movement trajectory of the object having a length closest to the target time using each of a plurality of motion estimation methods or each of a plurality of motion estimation parameters. When a state that cannot be estimated occurs, it may be determined whether the cause of the state is due to occlusion or a scene change, and the object motion determination method may be switched based on the determination result.

  In addition, when the object motion determination unit determines that the cause of the “state where motion cannot be estimated” is caused by occlusion, the object movement determination unit displays the movement trajectory of the object in frames after the frame where occlusion occurs. The object movement trajectory having the length closest to the target time may be determined by extrapolation based on the movement trajectory of the object up to the frame where the occurrence of the object occurs.

  According to this configuration, it is possible to extrapolate the movement trajectory of the frame where the occlusion occurred based on the movement trajectory of the frame where the occlusion did not occur.

  In addition, the object motion determination unit outputs a movement trajectory of the object up to the frame in which the scene change occurs when it is determined that the cause of the “state where motion cannot be estimated” is caused by a scene change. You may determine as a movement locus.

  When a scene change occurs, it is very difficult to obtain an accurate movement trajectory of the object across the scene change, and there is a high possibility that an incorrect movement trajectory will be obtained after the scene change. Therefore, in order to improve the visibility of comments, it may be better to determine that the movement trajectory after the scene change is not estimated as a result.

  In addition, the object motion determination unit may determine that the cause of the “state incapable of motion estimation” is caused by a scene change when a change amount of a luminance histogram between frames constituting the moving image is equal to or greater than a predetermined threshold. If the amount of change in the luminance histogram is less than the predetermined threshold, it may be determined that the cause of the “state where motion cannot be estimated” is caused by shielding.

  In addition, the object motion determination unit may include the comment when the movement trajectory of the object obtained using each of the plurality of motion estimation methods or each of the plurality of motion estimation parameters is shorter than the target time by a certain time or more. Estimated by going back to the previous frame on the time axis from the frame where the input receiving unit received the position information and the input of the comment, and the position of the object indicated by the position information received by the comment input receiving unit The movement locus of the object having the length closest to the target time may be determined by connecting the movement locus in front of the movement locus of the object.

  According to this configuration, when the movement trajectory of the object indicated by the position information does not reach the target time, the result of tracking one of the areas obtained by dividing the area is used to determine the object closest to the target time. The movement trajectory can be determined.

  In addition, the object motion determination unit may be configured when the time length of the movement trajectory of the object obtained using each of the plurality of motion estimation methods or each of the plurality of motion estimation parameters is shorter than the target time by a certain time or more. Among the movement trajectories of the object based on the position within a certain distance range from the position indicated by the position information of the object received by the comment input reception unit, the movement trajectory whose time length is the closest to the target time May be determined as the movement trajectory of the object indicated by the position information received by the comment input receiving unit.

  In addition, the object motion determination unit may be configured when the time length of the movement trajectory of the object obtained using each of the plurality of motion estimation methods or each of the plurality of motion estimation parameters is shorter than the target time by a certain time or more. Is the closest to the target time among the movement trajectories of the object based on the position within the range specified by the user, including the position indicated by the position information of the object received by the comment input receiving unit The moving locus of the length may be determined as the moving locus of the object indicated by the position information received by the comment input receiving unit.

  According to this configuration, when the movement trajectory of the object indicated by the position information is shorter than the target time by a certain time or more, based on the result of tracking the object using another movement trajectory based on the vicinity of the movement trajectory. It is possible to estimate the movement trajectory of the object for the target time. The same processing may be performed not only on the spatial neighborhood coordinates but also on the neighborhood in the time direction.

  In addition, the object motion determination unit may be configured when the time length of the movement trajectory of the object obtained using each of the plurality of motion estimation methods or each of the plurality of motion estimation parameters is shorter than the target time by a certain time or more. May divide the object into a plurality of areas, and determine a movement trajectory of an area having a length closest to the target time among the areas obtained by the division as the movement trajectory of the object.

  According to this configuration, even when the movement trajectory of the object indicated by the position information is shorter than the target time by a certain time or more, by using another movement trajectory based on the vicinity of the movement trajectory, the object that satisfies the target time The movement trajectory can be determined.

  In addition, the object movement determination unit further determines a movement locus having a length closest to a target time for the center of gravity of the object indicated by the position information received by the comment input reception unit, and the comment input reception unit receives Further, based on the relative positional relationship between the position where the comment is given and the center of gravity of the object, the movement trajectory of the determined object is as if it were a movement trajectory from the position where the comment is given. It may be corrected and output as described above.

  According to this configuration, for example, the movement trajectory of the object can be corrected so as to maintain the relative positional relationship between the coordinates of the center of gravity of the object and the coordinates in the frame to which the comment is given, which is indicated by the position information. it can.

  When the time length of the movement trajectory of the object obtained using each of a plurality of motion estimation methods or each of a plurality of motion estimation parameters is longer than the target time by a certain time or more, Only the movement trajectory in the range from the start frame to the target time with respect to the movement trajectory having the highest accuracy and the shortest time length is the object of the object indicated by the position information received by the comment input reception unit. By determining the movement trajectory and discarding the movement trajectory corresponding to the frame after the target time, a movement trajectory having a length closest to the target time can be easily obtained.

  Further, the comment information generation device described above further sets the target time based on the movement trajectory of the object determined by the object movement determination unit so that the target time becomes longer as the moving speed of the object increases. A target time correcting unit for correcting the object, and the object movement determining unit further corrects the movement trajectory of the object for causing the object indicated by the position information to follow the comment by the target time correcting unit. The movement trajectory of the object may be determined again so as to be the length of the target time after being performed.

  These general or specific aspects may be realized by a system, a method, an integrated circuit, a computer program, or a recording medium such as a computer-readable CD-ROM. The system, method, integrated circuit, computer program Alternatively, it may be realized by any combination of recording media.

  Hereinafter, a comment information generation device according to an aspect of the present invention will be specifically described with reference to the drawings.

  Note that each of the embodiments described below shows a specific example of the present invention. The numerical values, shapes, materials, constituent elements, arrangement positions and connecting forms of the constituent elements, steps, order of steps, and the like shown in the following embodiments are merely examples, and are not intended to limit the present invention. In addition, among the constituent elements in the following embodiments, constituent elements that are not described in the independent claims indicating the highest concept are described as optional constituent elements.

  FIG. 1 shows the configuration of a commented moving image distribution system targeted by the present embodiment. Similar to Patent Document 1, the commented moving image distribution system includes a moving image server and a comment accumulation and distribution server. Each server may be managed on the same PC (Personal Computer), or may be managed on another PC.

  Different users A to D shown in FIG. 1 browse moving images distributed to terminals (smartphones, PCs, tablet PCs, and the like) held by the users through a network. Here, an example in which user A and user B add comments to a moving image through a keyboard, software keyboard, or the like provided in the terminal is shown. Here, the comment is character information, and is given in association with a temporal position (that is, desired time or frame number) and a spatial position (that is, coordinates) in the moving image specified by the user. In the following description, the term “moving image” refers to moving image data or a moving image indicated by moving image data.

  The comment given by the user includes the moving image to which the comment is attached, the user ID of the user, the time and coordinates (in the moving image) associated with the comment, the actual time when the comment is posted, The information is stored in the comment storage / delivery server as needed. When a user A and a user B view a moving image after giving a comment to a moving image, the user A and the user B, like other comments associated with the moving image. The comments of the user A and the user B are distributed from the comment accumulation / delivery server, and the comments are synthesized on the moving image based on the information associated with the comment (the time and coordinates on the moving image that are the elapsed time in the moving image). Will be displayed.

  2A and 2B show a temporal flow of adding a comment to a moving image.

  First, FIG. 2A shows a relationship between a flow of time in a moving image and a displayed comment. The comment is stored in the server in association with the in-video time (moving image playback time). Each comment is desirably displayed for a length of time suitable for viewing the comment. The calculation of the time length will be described later in detail.

  According to FIG. 2A, the comment of the user A is displayed between the time SA seconds to EA seconds in the video, and the comment of the user B is between SB seconds to EB seconds (may be obtained in a frame instead of seconds). Is displayed. In the following, “second” is used as the unit of time, but “number of frames” may be used instead of second.

  FIG. 2B shows the actual date and time flow. As illustrated in FIG. 2B, when the user C or the user D views the commented moving image after the user A and the user B give (write) the comment, the user C includes the commented moving image. When viewing a moving image whose video time is between SB seconds and EA seconds, see the moving image with a comment in which the comment of user B is superimposed on the comment of user A as shown in FIG. 2A. Can do. On the other hand, it is assumed that the user D is viewing a moving image whose in-video time is between EA seconds and EB seconds. The user D can see a moving image to which only the comment of the user B is given as shown in FIG. 2A.

  The above is the conceptual description of the commented moving image targeted by this embodiment.

  FIG. 3 is a block diagram illustrating a functional configuration of the comment information generation device 100 according to the embodiment. As illustrated in FIG. 3, the comment information generation device 100 includes a moving image acquisition unit 101, a comment input reception unit 102, an object motion generation unit 103, an output comment information generation unit 104, and an output unit 105.

  The comment information generation device 100 receives the moving image 110 and input comment information 111 indicating the content of the comment (including the comment coordinates 112 that are coordinate values on the moving image corresponding to the comment) and is indicated by the comment coordinates 112. Then, the motion of the pixel or subject in the moving image 110 specified by the user is calculated, output comment information (object following comment) is generated, and output to the storage device 120.

  The moving image acquisition unit 101 receives an input of a moving image or a plurality of pictures (also referred to as “images”) constituting the moving image. For example, the moving image acquisition unit 101 may be an interface that reads a moving image stored in a storage device such as a moving image server directly or via a communication path.

  The comment input receiving unit 102 receives input comment information 111 (including comment coordinates 112) input by the user. For example, the comment input receiving unit 102 may be an interface that reads the time and coordinates on a moving image specified by a user clicking a mouse or touching a touch panel, directly or via a communication path. good.

  Based on the moving image 110 received by the moving image acquisition unit 101 and the input comment information 111 (including the comment coordinates 112) received by the comment input receiving unit 102, the object motion generation unit 103 determines the length and visibility of the comment character string. An object motion estimation result for the dependent time length is generated.

  The output comment information generation unit 104 outputs the object motion generated by the object motion generation unit 103 (movement trajectory that is a coordinate value along a series of time axes for displaying a follow-up comment) and the input comment information 111. Generate comment information.

  The output unit 105 outputs the output comment information generated by the output comment information generation unit 104 to the storage device 120 via wired or wireless. Here, the output comment information includes information about comments such as character information of comments given to moving images, coordinates to which comments are given, comment posting time, comment display time, and the like. Furthermore, the output comment information may include the shape or size of the comment when the comment is displayed.

  The object motion generation unit 103 includes a target time determination unit 106 and an object motion determination unit 107.

  Based on the input comment information 111 received by the comment input receiving unit 102, the target time determining unit 106 is suitable for displaying a target time for displaying a comment following an object, that is, a comment included in the input comment information 111. Determine the target time. The target time can be determined, for example, based on the number of characters in the comment.

  The object motion determination unit 107 is based on the moving image 110 received by the moving image acquisition unit 101, the input comment information 111 (including the comment coordinates 112) received by the comment input reception unit 102, and the target time determined by the target time determination unit 106. Then, from a plurality of motion estimation methods or a plurality of motion estimation parameters, determine an object motion that is sufficiently close to the target time, is longer than the target time, and does not degrade the estimation accuracy more than necessary. . The object motion determination unit 107 selects a motion estimation method or a motion estimation parameter such that the time length of the tracking result of the object from the comment coordinates 112 is larger than the target time and sufficiently close to the target time, and the designated comment The movement (movement locus) of the pixel (object) from the coordinate 112 is determined.

  Note that each component (the moving image acquisition unit 101, the comment input reception unit 102, the object motion generation unit 103, the output comment information generation unit 104, and the output unit 105) constituting the comment information generation device 100 is executed on a computer. It may be realized by software such as a program, or may be realized by hardware such as an electronic circuit or an integrated circuit. 4A and 4B are diagrams showing a hardware configuration of the comment information generating apparatus according to the present embodiment configured by a computer.

  4A and 4B, the storage device 210a outputs the stored moving image 110 to an I / F (interface) 201a. The input device 210b that receives input from the user outputs the input comment information 111 to the I / F 201b. The computer 200 acquires the moving image 110 and the input comment information 111, performs an object following comment generation process, and generates output comment information of the object following comment.

  The storage device 220 acquires and stores output comment information generated by the computer 200.

  The computer 200 includes I / Fs 201a and 201b, a CPU 202, a ROM 203, a RAM 204, an HDD 205, and an I / F 206. A program for operating the computer 200 is stored in advance in the ROM 203 or the HDD 205. The program is read out from the ROM 203 or HDD 205 to the RAM 204 and expanded by the CPU 202 as a processor. The CPU 202 executes each coded instruction in the program expanded in the RAM 204. The I / Fs 201a and 201b fetch the moving image 110 and the input comment information 111 into the RAM 204 in accordance with the execution of the program. The I / F 206 outputs the output comment information of the object following comment generated by executing the program and stores it in the storage device 220.

  The computer program is not limited to the semiconductor ROM 203 or HDD 205, and may be stored in, for example, a CD-ROM. Alternatively, the data may be transmitted via a wired or wireless network, broadcasting, or the like and taken into the RAM 204 of the computer.

  Hereinafter, the operation of the comment information generating apparatus 100 in the present embodiment will be described with reference to FIG. FIG. 5 is a flowchart showing the operation of the comment information generating apparatus 100 in the present embodiment.

  In FIG. 5, seven steps S301 to S307 correspond to the respective processing units 101 to 107 in FIG. That is, the video acquisition unit 101 has a video acquisition step S301, the comment input reception unit 102 has a comment input reception step S302, the object motion generation unit 103 has an object motion generation step S303, and the output comment information generation unit 104 has an output comment information generation step S304. The output unit 105 executes each operation of the output step S305. The object motion generation step S303 includes two steps: a target time determination step S306 and an object motion estimation step S307. The target time determination unit 106 executes each operation of target time determination step S306, and the object motion determination unit 107 executes each operation of object motion estimation step S307.

  First, the moving image acquisition unit 101 executes the moving image acquisition step S301. The moving image acquisition unit 101 acquires the moving image 110.

  In the present embodiment, the moving image 110 acquired by the moving image acquisition unit 101 is assumed to be a broadcast moving image, various moving images such as a moving image taken by a user, and a plurality of pictures (images) constituting the moving image. Yes. These moving images 110 are stored in a moving image server or the like, and the moving image acquisition unit 101 acquires the moving images 110 via a wired or wireless network, broadcasting, or the like. In the present embodiment, the moving image is 30 frames / second.

  An example of the acquired moving image is shown in FIG. 6A. Here, an example of music-related moving image content is shown, but the moving image targeted by the present invention is not limited to this. Hereinafter, for simplification of description and drawings, a diagram in which subject areas are simply displayed as shown in FIG. 6B is used for the moving image example shown here.

  With reference to FIG. 5, the comment input receiving step S <b> 302 is executed by the comment input receiving unit 102. The comment input receiving unit 102 acquires input comment information 111. An example of comment input by the user and an example of acquired input comment information 111 are shown in FIGS. 7A and 7B, respectively.

  As shown in FIG. 7B, the input comment information 111 includes at least three pieces of information including a comment time (time), a comment target coordinate (position), and a comment character string (comment) that is input sentence information. The comment time is information indicating the elapsed time (reproduction time) in the moving image 110, and may be information that can specify the timing at which the comment is given by the user in the moving image 110, such as a frame number, instead of the time information. Other information may be used. The comment target coordinates are also information that can specify the spatial position where the comment is given by the user in the frame, such as coordinate values normalized so that the vertical and horizontal value ranges of the screen are 1 instead of the pixel coordinate values. Other information may be used as long as it exists.

  Note that these comment inputs may be performed through a user interface on a device such as a PC or a portable terminal provided with the comment information generation device 100. Alternatively, the position of the object to be displayed by following the comment and the comment via a communication line from an operation performed through a user interface on a device such as a general PC or a portable terminal that does not include the comment information generation device 100 A comment may be input by receiving information.

  Note that the information included in the input comment information 111 is only two pieces of information, that is, a comment time (time) and a comment character string (comment) that is input text information, and is additionally provided with a comment target coordinate (position). A configuration in which the position is estimated by the position estimation unit may be used.

  A comment displayed on a certain frame is considered to have a characteristic in content for each object to be commented. That is, it can be said that the comment text in a certain frame is related to the position information of the comment. Accordingly, among the comments accumulated so far, it is possible to estimate the comment target coordinates of a comment to be newly added from the position where a similar comment passes through the frame.

  In addition, for example, when the moving image 110 is reproduced on the terminal being browsed by the user and the user's mouse input, key input, or touch input is detected on the terminal side, the comment is automatically posted. Is performed in a state where playback of the video is paused. Alternatively, the posting of a comment is executed in a state where playback of the moving image 110 is paused by a user operation on the screen.

  For example, in the display screen shown in FIG. 7A, the user designates certain coordinates on the screen, preferably while the reproduction of the target moving image 110 is paused (for ease of operation) (see FIG. 7A). 7A (a)). In response to this designation, a pop-up screen is superimposed on the comment information generating apparatus 100 ((b) in FIG. 7A). The user posts a comment by inputting the comment in the pop-up screen. Here, the coordinates on the screen can be specified by, for example, clicking the mouse or touching the screen of the touch panel display directly with the mouse pointer on the coordinates where the user wants to give a comment on the PC. Done. If the position estimation unit is separately provided as described above, there is no need for the user to specify the position. Therefore, there is no problem in ease of use even if the moving image is not temporarily stopped when a comment is input.

  Alternatively, the comment may be posted by voice input. In this case, a voice analysis unit may be provided, and the voice analysis unit may convert the input voice into a comment sentence. However, the input speech is usually a spoken language and is strictly different from the written language. Therefore, it is desirable that the voice analysis unit converts the input voice into a comment sentence as written words. For example, Patent Document 6 discloses a method for converting spoken language input by voice into written language. To the written language, a morphological analysis is performed on the input spoken word data, a written word candidate is obtained using a table that associates the spoken word with the written word, and a word order and words that are likely to appear in the written word database are selected. The conversion is done. According to the above mechanism, expressions peculiar to spoken words, homonyms, etc. can be appropriately converted into written sentences as written words.

  Referring to FIG. 5, the object motion generation step S303 is executed by the object motion generation unit 103.

  The target time determination unit 106 executes the target time determination step S306 based on the input comment information 111 received in the comment input reception step S302, and determines the target time. Subsequently, based on the target time determined by the target time determination unit 106, the moving image 110 acquired in the moving image acquisition step S301, and the input comment information 111 received in the comment input reception step S302, the object movement determination unit 107 The object movement determination step S307 is executed to determine a movement trajectory that is a coordinate value along a series of time axes for displaying a follow-up comment. The above processing will be described in detail below.

  The target time determination unit 106 determines a target time which is a time required for the user who has input the comment or another user to visually recognize the input comment.

  For example, for subtitles such as movies, there is a guideline of “display 12 characters per second” in English. Further, the target time may be obtained in units of words. For example, a unit called WPM (Words per minute: the number of words that can be read per minute) may be used as the speed of reading a sentence. Since it is said that an adult American usually reads a magazine, newspaper or book at 250 to 300 WPM, this standard can be applied when calculating the target time.

  For example, for readability, if the target is set to 200 WPM, which is a little later, the number of words W can be obtained by detecting a space in the input comment sentence. Therefore, the target time Nw (seconds) can be obtained by calculation such that the target time Nw = W * 60/200.

  It is said that it takes about 0.5 seconds for a person to perform visual recognition. This time is called visual recognition time. In consideration of the visual recognition time, it is desirable to obtain the target time so as to be a predetermined length (for example, 0.5 seconds) or more.

  In addition, when the user inputs a comment in a frame in which another comment is already displayed, the time in which both the past comment character information displayed and the comment character information newly input by the user are visible The target time of a newly input comment may be calculated to be longer so that The target time may be calculated after adding a predetermined number of characters or words to the number of characters or words of a new comment. By doing in this way, the comment sentence displayed simultaneously can be visually recognized more certainly for the user who browses a movie with a comment.

  In addition, when a comment made of the same character string has already been given on the frame to which the user wants to give a comment, redundancy due to duplication of comment contents may give an uncomfortable feeling. Therefore, as the number of comments having the same content existing on the same frame increases, the target time for newly added comments having the same content may be shortened to zero. In order to shorten the target time, for example, the target time obtained from the number of characters can be changed to a value divided by the number of the same comments. Even if the same comment is not assigned to the same frame, the target time may be similarly shortened to zero for a comment that appears multiple times in the same moving image.

  In the present disclosure, the target time determination unit 106 estimates the target time each time from the number of input comment characters, but the operation of the target time determination unit 106 is not limited to this. For example, a relationship between the number of characters in the comment and the target time is obtained in advance and stored as a database or table in a storage device provided inside or outside the comment information generation device. When receiving a comment input, the target time determination unit 106 may obtain the target time by referring to the database or table via wired or wireless instead of estimating the target time from the number of comment characters. By estimating the target time in advance, the target time determination process can be performed at a higher speed when a comment is input.

  The object motion generation unit 103, the output comment information generation unit 104, and the output unit 105 may be provided on each terminal side that views a comment-added moving image.

  Nowadays, automatic translation functions between different languages have become commonplace. Accordingly, it can be said that it is possible to view a comment-added video with a comment translated into a language different from the language of the comment originally given to the video. In that case, it is desirable to select which language should be translated for each terminal on the viewing side. That is, most simply, the viewing terminal translates the comment into another language based on the language information of the terminal, and the object motion generation unit 103 and the output comment information generation unit 104 for the translated comment. The processing of the output unit 105 may be performed.

  Alternatively, the object motion generation unit 103, the output comment information generation unit 104, and the output unit 105 may be provided in a server that translates and distributes a moving image with a comment in a specific language area, and the same processing may be performed.

  Subsequently, the object motion determination unit 107 uses coordinate values along a series of time axes for displaying a follow-up comment based on the target time determined by the target time determination unit 106, input comment information, and the moving image 110. A certain movement trajectory is determined.

  Specifically, the object motion determination unit 107 receives a plurality of pictures from the moving image acquisition unit 101, detects corresponding points between the pictures, generates a movement locus, and outputs it. Hereinafter, a motion of a pixel or a block including a pixel between two temporally adjacent pictures constituting the moving image 110 is detected, and the detected motion is connected to a series of time axes connected to the plurality of pictures. The coordinate value along is called a movement trajectory.

As illustrated in FIG. 8A, the object motion determination unit 107 uses the motion vector information 502 between the two pictures calculated based on the pixel i 503a and the pixel j 503b of the input picture 501 at time t to generate the pixel i 503a, The movement of the pixel j503b is tracked, and corresponding points of the pixel i503a and the pixel j503b are obtained. At this time, the object motion determination unit 107 determines the coordinate value (x ₁ ⁱ , y ₁ ⁱ ) of a pixel i on a picture of one frame and the pixel coordinate value (x _t ⁱ , since y _t ⁱ⁾ and calculates the movement trajectory ^{x i} as equation 1.

In this embodiment, the movement trajectory x ⁱ is assumed to be the corresponding points over between T sheets of pictures from one frame to T frame.

FIG. 8B shows an example of the movement trajectory. The moving image 110 input to the object motion determination unit 107 is composed of T pictures 504. At this time, the movement trajectories x ⁱ 506a and x ^j 506b are a collection of corresponding points on the picture from the second frame to the T frame corresponding to the pixel i 505a and the pixel j 505b in one frame, respectively. The movement trajectories x ⁱ 506a and x ^j 506b are represented by vectors having the picture coordinate values of each picture as elements. Here, the corresponding pixels on (T−1) pictures from 2 frames to T frames are obtained on the basis of all the pixels (I) on one frame picture.

  Note that when the corresponding point between pictures is obtained by the object motion determination unit 107, the corresponding point is obtained for each of a plurality of adjacent pixels (blocks) in the picture, instead of obtaining the corresponding point for every pixel of the picture. Also good. In the present embodiment, processing in units of pixels will be described. However, when processing is performed in units of blocks (regions) composed of a plurality of pixels, (i) pixel values are summed in the block, or (ii) blocks By calculating the average of the pixel values in the block, or (iii) determining the median value of the pixel values in the block, the data (representative value) corresponding to the block is determined, and the obtained representative value is used as a pixel unit. What is necessary is just to process like the process of. In the present embodiment, it is not distinguished whether a certain corresponding point is obtained for each pixel or whether one corresponding point is obtained for a plurality of pixels. Also, the corresponding point of another picture corresponding to the pixel i of a certain picture and the corresponding point of another picture corresponding to the block i of a certain picture are both called the movement locus of the pixel i. The movement trajectory obtained by the motion estimation procedure as described above is used as the basis of the motion of the object following comment. Further, when obtaining corresponding points between pictures, the frames do not necessarily have to be continuous. For example, the movement trajectory may be obtained from two pictures input at time t and time t + n. However, n is an integer of 1 or more.

  As a specific method for calculating the corresponding points between the plurality of pictures described above, a method disclosed in Non-Patent Document 1, Non-Patent Document 2, or the like may be used. Both are methods for calculating a motion vector by calculating an optical flow. In Non-Patent Document 1, an optical flow is calculated based on hierarchical block matching. Since the smoothness between pixels is set as a constraint, an optical flow in which a motion vector smoothly changes between adjacent optical flows can be obtained. An efficient and accurate corresponding point is required especially when there is no steep movement or occlusion. Further, since the reliability of the estimation can be calculated, as will be described later, by removing the corresponding points whose reliability is lower than a certain threshold value from the subsequent processing, it is possible to reduce the ratio of erroneous motion vectors to all motion vectors.

  In contrast, Non-Patent Document 2 discloses a graph cut-based optical flow calculation method. Although this method has a high calculation cost, accurate corresponding points can be obtained densely on a picture. In this method, a bidirectional search is performed, and a corresponding point having a correlation lower than a threshold is estimated to be a pixel in the shielding area. For this reason, the corresponding point located in the shielding area can be excluded from the subsequent processing. It is possible to reduce the ratio of erroneous motion vectors to all motion vectors.

  At this time, motion information may be obtained for all pixels. In order to perform processing at a higher speed, the motion information may be obtained only for pixels on the grid at a fixed interval by dividing the picture into grids. You may also ask for motion information.

  In this case, it is possible to use a method of calculating a motion vector on the assumption that the block is translated. In particular, for a moving object, it is possible to estimate the pixel motion with higher accuracy by using the method assuming affine deformation disclosed in Non-Patent Document 3, rather than assuming translational movement.

  In addition, when calculating a motion vector using the technique disclosed in Non-Patent Document 1, the reliability can be calculated. For this reason, only pixels having highly reliable motion information may be used. Further, when the motion vector is calculated using the technology disclosed in Non-Patent Document 2, since the shielding can be estimated, only the motion information of pixels that are not shielded may be used.

  Here, in general, a motion estimation method that can be used to generate an object following comment is processing based on some model for discriminating the same object region between time-series images. For example, this process includes an assumption that the luminance difference between the same object areas is zero. Furthermore, since the actual moving image 110 includes measurement noise and an error with the model, the regions that “satisfy the model” are determined to be the same object, and the movement is calculated from the temporal position change between the same objects. To do.

  On the other hand, there are actually cases where an object is not shown in the moving image 110, and such cases belong to cases where motion estimation is inherently impossible. In this case, even if areas that “satisfy the model most” are determined according to the above-described criteria, there is no actual movement between the areas, and thus an incorrect movement is obtained. However, information indicating whether or not the arbitrary comment coordinates at the arbitrary comment time included in the input comment information 111 received by the comment input receiving unit 102 is a case where motion estimation is possible or impossible is obtained. In order to do this, information is manually given in advance to all pixels in all frames of the moving image 110 (a person checks the movement and determines whether the movement can be estimated), and the information is stored in a server or the like. It is necessary to keep. However, manually providing information to the moving image 110 having a large number of frames and pixels is not practical. Accordingly, the object motion determination unit 107 determines whether or not motion estimation is possible for each comment coordinate at a certain comment time included in the input comment information 111 received by the comment input reception unit 102 when performing motion estimation. judge. As a determination criterion, another threshold is set with respect to the criterion of “the model is satisfied more than a certain level” for estimating the motion described above. That is, it is common to separately set a criterion that “the model is not satisfied more than a certain level”.

  Therefore, the object motion determination unit 107 determines whether or not motion estimation is possible based on the first determination criterion “whether or not a model used for motion estimation satisfies a certain level”. Further, when the motion estimation is possible, the object motion determination unit 107 determines that the regions satisfying the second determination criterion “whether the model is satisfied most” are the same object, and the same object. The motion is calculated from the temporal position change between the objects.

  On the other hand, there is a kind of trade-off in the motion estimation method having the above mechanism.

  If a criterion for determining whether or not motion estimation is possible is set more strictly, a state where “motion estimation is impossible” can be detected even if the error from the model is relatively small. By setting a criterion for determining whether or not motion estimation is possible, that is, by reducing the error tolerance, more “motion estimation impossible” cases can be detected correctly. On the other hand, it is actually "motion estimation is possible", for example, even when measurement noise or an error that occurs between the model and the actual image has an effect, motion estimation is impossible. Is erroneously determined. For this reason, there is a possibility that the motion estimation process that should not be interrupted is interrupted and the obtained movement trajectory is shortened. That is, the ease of following is reduced.

  Conversely, in order to increase the ease of tracking, it is necessary to enable motion estimation even when measurement noise or an error generated between the model and the actual image is relatively large. In this way, when a criterion that excessively allows noise or model error is adopted, that is, when the error tolerance is increased, contrary to the previous case, the object in the moving image 110 does not appear in the frame out or the like. Even if motion estimation becomes impossible, the motion estimation process that should be interrupted is continued. For this reason, there is a possibility that a movement trajectory including an erroneous motion estimation result (non-robust to noise or model error) is generated.

  As described above, in the motion estimation process, between robust noise estimation and model error and accurate determination of “motion estimation impossible” caused by occlusion and scene change. Has a trade-off relationship as shown in FIGS. 9A and 9B.

  9A and 9B, the solid arrow indicates a movement trajectory estimated by the general motion estimation method, and the dotted arrow indicates an actual (correct) movement trajectory. As shown in FIG. 9A, when the error tolerance parameter is set low, it is possible to more accurately determine the “case in which motion estimation is not possible” and to suppress noise included in the movement trajectory, but the movement trajectory is shorter than the actual movement. It is easy to become, and it is easy to shorten time to follow a comment. On the other hand, when the error tolerance parameter is set high as shown in FIG. 9B, the comment tracking time becomes longer and a moving trajectory longer than that in FIG. 9A is obtained, but the motion estimation result including a motion different from the correct motion is obtained. There is a possibility. In order to improve the visibility of the comment, it is desirable that the movement trajectory is longer as shown in FIG. 9B. However, from the viewpoint of the comment following the object, a more accurate movement trajectory can be obtained as shown in FIG. 9A. Is preferable.

  Based on these trade-offs, the minimum time required for the user to visually recognize the comment string is the time required to obtain the movement trajectory with the best time length and position accuracy for comment display. It is necessary to determine the length and obtain a more accurate movement trajectory while minimizing the time length of the necessary movement trajectory.

  That is, for each arbitrary comment coordinate included in the input comment information 111, the target time determined by the target time determination unit 106 is set as the minimum time required for the movement trajectory, and is equal to the time length. It is desirable to obtain the closest movement locus.

  Therefore, the object motion determination unit 107 prepares in advance a plurality of determination criterion threshold values “satisfying the model more than a certain level” for noise or luminance difference from the model at the time of motion estimation, and uses each threshold value. A plurality of movement trajectory calculations are performed, and among the plurality of results, the coordinates of the movement trajectory having the time length closest to the target time determined by the target time determination unit 106 are used as the coordinates of the object following comment. The trade-off shall be resolved.

  It should be noted that the parameter affecting the “ease of following” by adjusting the trade-off relationship in general motion estimation is not limited to the “degree of satisfying the model more than a certain level (error tolerance)”.

  As another example, the size of the window region used for motion estimation can be cited. In other words, if the size of the window area is increased, motion estimation can be performed more robustly against partial luminance changes and deformations, while `` motion estimation is impossible '' when the specified location is blocked. There is a trade-off that it is difficult to detect. For this reason, the size of the window region can be used as a parameter for adjusting the time length of the movement locus. For example, the object motion determination unit 107 performs a movement trajectory calculation using each of a plurality of size window regions, and among these results, a time length equal to or closest to the target time determined by the target time determination unit 106 By using the coordinates of the movement trajectory of the object as the coordinates of the object following comment, it is possible to obtain a motion estimation result that has the time length required for comment display and is robust to noise or model errors. .

  As another example, the number of feature quantities (image feature quantities) used for motion estimation can be given. That is, if the feature amount is increased, the change in the feature amount is relatively robust. For this reason, the movement trajectory is calculated under the condition of the different number of features, and the coordinate of the movement trajectory having the time length closest to the target time determined by the target time determination unit 106 among the plurality of results is obtained as the object following comment. By using the coordinates, it is possible to obtain a motion estimation result that has a time length necessary for displaying a comment and is robust to the noise or model error.

  Note that as the parameter of “size of window region”, the size of the window region in Non-Patent Document 1, the size of a block assuming affine deformation in Non-Patent Document 3, and the like can be similarly used. Therefore, adjusting these means adjusting the time length of the movement trajectory based on the “window region size” described above.

  Further, as a parameter of “error tolerance”, a feature amount such as a confidence range (confidence measurement range k3) of Non-Patent Document 1, a shielding weight value (Occlusion Penalty) of Non-Patent Document 2, or a texture of Non-Patent Document 3 is used. Similarity (feature dissimilarity) or the like can be used as well. If the value of the reliability range is 0, a longer movement trajectory can be easily obtained. If the value of the feature quantity similarity is increased, a longer movement trajectory can be easily obtained. Therefore, adjusting these means adjusting the time length of the movement trajectory based on the “error tolerance” described above.

  Further, as the parameter of “number of feature quantities”, the number of feature points used for estimation in Non-Patent Document 3 can be similarly used. By adjusting these, it is possible to adjust the time length of the movement trajectory based on the “number of feature amounts”. Of course, parameters other than those given here may be used. The time length of the movement trajectory can also be adjusted using the search range (a range in which pixel motion is assumed) in Non-Patent Document 2. The search range is related to the trade-off between the time length of the movement trajectory and the motion estimation accuracy (estimation accuracy of the movement trajectory), as with the other motion parameters described above. On the other hand, there is a possibility of a motion estimation result including motion different from the correct motion. It should be noted that parameters related to the time length of the movement trajectory and the motion estimation accuracy also exist in known motion detection methods other than the motion detection methods described so far and can be used in the same manner.

  As described above, the parameter such as the model error threshold, the size of the window area, or the number of features may be adjusted using only one parameter, or the time length of the movement trajectory may be adjusted by combining a plurality of parameters. The time length may be adjusted. For example, in the simplest example, two parameters are preset for each of the model error threshold value, the window area size, and the number of feature values, and the target time is obtained from the results obtained in all eight combinations. The coordinates of the movement trajectory having the time length closest to the target time determined by the determination unit 106 may be used as the coordinates of the object following comment.

  In addition, when the movement of the target to be tracked is complex and difficult to assume a specific model, it is difficult to assume a specific model. Of the results, the coordinates of the movement trajectory having the time length closest to the target time determined by the target time determination unit 106 may be used as the coordinates of the object following comment. As a result, a movement trajectory closer to the target time can be obtained with respect to various movements.

  Up to this point, a method for obtaining a movement locus based on a target time, which is a desired comment display time, has been described for a case in which motion estimation is essentially possible.

  However, in particular, in the case of a general moving image 110 such as a TV program video or a video recorded in a movie, a specified pixel or a specified area is captured by a scene change, occlusion by another object, self-occlusion, or movement of the object or camera. Moving out of range is likely to occur. When the designated pixel or the designated area does not appear in the moving image, there are many cases in which the subsequent motion estimation becomes impossible at a certain time point less than the target time. In such a case, even if the parameters are adjusted, only a short movement trajectory can be obtained, or the estimation accuracy of the obtained movement trajectory is significantly reduced. In order to generate object follow-up comments that are easy for the user to see, the above-mentioned cases where “motion estimation is impossible” are correctly determined, and processing according to the factors that caused the “motion estimation impossible” state has been introduced. It is desirable to do.

  That is, the object motion determination unit 107 is in a state that “motion estimation is not possible” even when the movement trajectory has a time length closest to the target time and is shorter than the target time (a certain value or more). Of the generated factors, it is determined whether “shielding” or “scene change” has occurred, and the process is switched according to the determination result.

  Whether the occlusion has occurred or the scene change has occurred can be determined from, for example, a temporal change in the luminance histogram of the entire image. That is, when the luminance histogram changes greatly between frames (for example, when the amount of change in the luminance histogram is equal to or greater than a predetermined threshold), it can be determined that a scene change has occurred, and in other cases Can be determined to be occluded. Alternatively, in the case where scene change time information is attached to the moving image 110 as metadata, it is possible to determine whether or not a scene change has occurred using this information.

  A process performed by the object motion determination unit 107 when it is determined as “occlusion” will be described with reference to FIG. 10A. If the motion estimation is interrupted due to occlusion, the object will not be visible, but it is likely that the object will move behind the occlusion with the same movement as before, that is, it is likely that the object still exists in the video screen. . Accordingly, the object motion determination unit 107 estimates a region where the object is likely to move, and extends the movement trajectory by, for example, extrapolating the motion of the object that is obtained, and generates a comment tracking trajectory. For extrapolation, linear interpolation or the like can be used.

  A process performed by the object motion determination unit 107 when “scene change” is determined will be described with reference to FIG. 10B. In the case of a scene change, there is a high possibility that the object to be tracked is outside the camera or is present in a different location from the previous one even if it exists in the camera screen. Therefore, as in the case where it is determined as “blocking”, if the movement is extended, there is a high possibility that a comment tracking locus that gives a sense of incongruity to the user is generated. Therefore, if it is determined as “scene change”, the movement trajectory is not expanded. In other words, exceptionally, the time length of the movement trajectory calculated by the object motion determination unit 107 may be shorter than the target time, and the movement trajectory is up to the frame where the scene change has occurred. Similarly, a “scene change” also includes a case where motion cannot be estimated because the object has moved to the edge of the image. That is, the movement after the frame is not expanded, and the movement trajectory to the frame is output.

  When it is determined as “occlusion”, the object motion estimation determining unit 107 may extend the movement locus by the following procedure. Corresponding point information is defined between two frames. For this reason, the movement trajectory can be generated by going back the time axis. That is, as shown in FIG. 11A, instead of linear interpolation, when the movement trajectory can be extended retroactively from the comment start time, the object motion determination unit 107 moves the target time by moving the comment start time forward. A trajectory may be calculated. Since a long movement trajectory is obtained in the direction of the time zone in which the object is more visible, it is possible to obtain comment following coordinates that realize a comment display that is comfortable for the user.

  Further, when it is determined as “occlusion”, the object motion estimation determining unit 107 may extend the movement locus by the following procedure. In other words, it can be generally assumed that the pixels in a region close to each other on the image move in a similar manner. Therefore, the object motion determination unit 107 exists within a predetermined pixel range R from the comment coordinates at a certain comment time included in the input comment information 111 received by the comment input reception unit 102 as shown in FIG. 11B. Similarly, a plurality of movement trajectories are obtained for the pixels having the coordinates. The object movement determination unit 107 may select a movement locus having a time length closest to the target time from the plurality of movement loci, and calculate the movement of the selected movement locus as the movement of the designated pixel. By using peripheral information, it is possible to obtain comment tracking coordinates that are more robust against noise and the like.

  As another example, the comment coordinates 112 received by the comment input receiving unit 102 may be specified as an area as shown in FIG. 12A. In this case, a plurality of comment coordinates 112 corresponding to the designated area at a certain comment time included in the received input comment information 111 can be used instead of the predetermined pixel range R.

  As yet another example, the object motion determination unit 107 includes a region dividing unit, and the region dividing unit performs region division of a picture. As shown in FIG. 12B, the object motion determination unit 107 determines a region including the comment coordinates at a certain comment time included in the input comment information 111 among the regions divided by the region dividing unit. It may be used instead of R.

  For example, the region dividing unit divides the picture into a plurality of regions based on the color similarity of the pixel or block to which each movement locus belongs. Also, a method of dividing a picture into a plurality of regions called “superpixel” based on the color similarity of pixels may be used. As a method for calculating the superpixel, a graph-based method or the like can be used. A detailed description of the processing procedure is described in Non-Patent Document 4 and the like, and will be omitted. This method divides the picture into small regions while maintaining efficient and global features by estimating the boundary between regions based on the graphical representation of the picture, and is more robust because it is more resistant to occlusion. Can be divided.

  Especially in a scene where the color of the moving object and the background color are different, there is a high possibility that each area is composed of only the moving object or only the background (the probability that each area will be separated into different subclasses). Therefore, the picture can be more correctly divided into regions with similar motion.

  Further, a region division method based on motion may be applied. Specifically, a technique such as that disclosed in Patent Document 7 may be used. By doing as described above, even when the color of the moving object and the background is similar, the picture can be divided more correctly for each subject area that moves similarly.

  As yet another example, the object motion determination unit 107 includes an area dividing unit, and instead of obtaining a plurality of movement trajectories, a plurality of area division results are prepared as shown in FIGS. 13A and 13B, and the movement of each area is determined. The tracking time length is obtained as the average of the movement trajectories included in the region and the minimum time length of the movement trajectories. The object motion determination unit 107 may select a region whose tracking time length is closest to the target time from among regions obtained by dividing the region designated by the user, and calculate the motion of the region as the motion of the designated pixel. . In general, there is a trade-off relationship between the division granularity and the tracking time length. For example, in the case of coarse area division as shown in FIG. 13A, even if the follow-up time length of the designated area is shorter than the target time, the follow-up time length can be set as a target by performing fine area division as shown in FIG. 13B. Can approach time. As shown in FIG. 13B, the head region has a shorter follow-up time length, but the body region specified by the user may result in longer follow-up. In this case, the result of FIG. 13B is used. Since information on not only one pixel but also surrounding pixels is used, it is possible to obtain object tracking comment coordinates that are more robust against noise and the like. As the region dividing method, the method based on the color similarity as described above or the method based on motion can be used.

  Note that, as shown in FIGS. 13A and 13B, even when the position designated by the user is the end of the divided area, in order to make a comment that matches the position intended by the user, The center-of-gravity coordinate position of the divided area is obtained for each frame. In addition, the relative position of the user instruction coordinate position in the instruction start frame from the center of gravity coordinate position of the divided region is held, and thereby the follow-up result is always corrected. In addition, when the size of the divided area is changed, there is a high possibility that the coordinates after the follow-up result correction are out of the actual divided area. Therefore, in such a case, the center of gravity is obtained in the divided rectangular area obtained by surrounding the outer edge of the divided area with a rectangle, etc., and based on the size fluctuation of the rectangular area or the fluctuation of the number of moving tracks in the rectangular area. Thus, it is preferable to correct the positional relationship between the user-indicated coordinate position and the barycentric coordinate position of the divided region, and to correct the follow-up result using the corrected positional relationship. Instead of obtaining the centroid coordinate position of the divided area in each frame, the centroid coordinate position of the divided area is obtained in the instruction start frame, and the movement locus is calculated from the centroid coordinate position of the divided area in the instruction start frame. Similarly, the movement trajectory from the user designated position may be determined based on the relative position.

  In the present disclosure, the object motion determination unit 107 estimates the motion of the object from the moving image, but the operation of the object motion determination unit 107 is not limited to this. For example, the motion between frames is estimated in advance separately from a moving image using a plurality of parameters, and is stored as a database or table in a storage device provided inside or outside the comment information generation device. When the object motion determination unit 107 receives a comment input, the object motion determination unit 107 may obtain the movement locus by referring to the database or table via a wired or wireless method instead of estimating the motion from the moving image. FIG. 14 is a diagram illustrating an example of a database. The database includes information on the movement trajectory for each time length. By performing motion estimation in advance, the object motion determination process can be performed at a higher speed when a comment is input.

  As illustrated in FIG. 15, the object motion generation unit 103 further includes a target time correction unit 108, and the target time correction unit 108 is based on the movement trajectory calculated by the object motion determination unit 107. The target time determined by 106 may be corrected.

  For example, even if a comment has exactly the same comment text, a comment displayed following an object that moves faster than a comment displayed following a slower moving object or a stationary object The sentence becomes difficult to read and it takes time to finish reading. Therefore, it is more desirable to correct the target time according to the speed of the object. That is, for a comment given to an object that moves a longer distance in the same time, the target time is corrected so that the target time becomes longer.

  Specifically, the target time correction unit 108 receives the movement locus determined by the object movement determination unit 107, and calculates the speed of the object (movement distance per unit time). The speed of the object is most simply expressed by an average value of the lengths of the motion vectors for each frame.

  If the calculated speed of the object is zero, the target time may not be corrected because the object is stationary.

  In other cases, it is desirable to correct the target time to be longer based on the speed of the object. For example, when the target time is doubled for the maximum possible object speed, the target time corresponding to the calculated object speed is obtained linearly from the target time when the speed is zero. It may be a thing. A value other than double may be adopted depending on the size of the image. Further, as the maximum possible value of the speed of the object, for example, the length of a motion vector when moving on the diagonal line of the image from one end to the other in one frame may be used. If the scene type is known in advance, the length of the motion vector when moving from the center of the image to the corner in one frame may be used according to the type of the scene.

  Further, the reference may be set based on a person's viewing angle. It is said that the region (central vision) that is clearly visible when a person is gazing at a certain part on the screen is about 2 degrees in view angle. Therefore, as the speed of the object, since the central vision area cannot be deviated until the speed corresponding to the viewing angle, if the comment can be read at a speed different from that when the object is stationary, the speed of the object is the center speed. It can be easily obtained from the viewing angle of view, the distance between the screen during viewing and the user's eyes, and the display size and resolution during viewing.

  For example, assuming that the distance between the screen during viewing and the user's eyes is approximately 40 cm, the display size and resolution during viewing are assumed to be 9.7 inches (19.71 cm × 14.78 cm) and 2048 × 1536 pixels. The speed of the object is about 145 pixels. Therefore, the target time may be made longer when the speed of the object exceeds 145 pixels. For example, the speed of the object is set to the speed of the object corresponding to the central view (the above-described speed) with respect to the target time so that the speed of the object becomes a number of pixels corresponding to the viewing angle of the central view. In the example, a value calculated by multiplying by 145 pixels) may be obtained as the corrected target time.

  The object motion determination unit 107 receives the target time corrected as described above, and based on the corrected target time, the object motion determination unit 107 calculates the movement locus again by the method described above. In this way, it is possible to generate a movement trajectory that does not decrease the visibility even for a comment given to a fast-moving subject.

  Note that the target time correction processing loop between the target time correction unit 108 and the object motion determination unit 107 described above may be performed only once, or may be performed a plurality of times in order to improve accuracy.

  Referring to FIG. 5, output comment information generation step S304 is executed by output comment information generation unit 104. The output comment information generation unit 104 generates output comment information from the object motion estimation result generated by the object motion generation unit 103 and the input comment information 111.

  The output comment information includes, as minimum information, comment time, comment target coordinates, and text information of the input comment information 111, and object follow comment coordinate values for a plurality of frames generated by the object motion generation unit 103. . Furthermore, as for the output comment information, when the input comment information 111 includes a comment color, a comment display (balloon) shape, a comment character string font, and the like, these information may be included in the output comment information. . Thereby, output comment information that can be displayed more reflecting the user's intention can be generated.

  Finally, the output step S305 is executed by the output unit 105. The output unit 105 outputs the output comment information generated by the output comment information generation unit 104 to the storage device 120 through a wired or wireless communication path. More specifically, the output unit 105 displays the posted comment character string included in the output comment information and the position information of the object to be displayed following the comment in a comment database provided in the comment accumulation / delivery server. Remember me.

  When the comment information generating apparatus 100 performs processing on the input comment information 111 that is continuously input, the operation of steps S301 to S307 described above may be repeatedly performed each time the input comment information 111 is input. good.

  Since the comment information generation device 100 generates different movement trajectories according to the length of the comment, even if the comments are attached to the same frame and coordinate position, Thus, when a moving image with a comment is displayed, a movement trajectory can be generated such that a comment that moves differently is displayed.

  As described above, the comment information generating apparatus 100 according to the present embodiment uses the coordinates of the moving trajectory having the time length equal to or closest to the target time determined by the target time determining unit 106 as the coordinates of the object following comment. Thus, the motion estimation result having the time length necessary for displaying the comment and maximally robust against noise or model error is obtained. Thereby, as a result of temporally tracking an object designated with the intention of giving a comment, it is possible to generate coordinates for an object following comment without a sense of incongruity.

  That is, the comment information generating apparatus 100 determines a target time for displaying a comment based on the comment, and estimates the movement trajectory of the object so that the continuous time of the movement trajectory becomes the target time. The estimated movement trajectory of the object can be used as a movement trajectory for displaying the comment following. For this reason, the user can read the comment within the display time when the moving image with the comment is displayed, and can determine to which object the comment is given. Therefore, it is possible to generate output comment information that can improve comment visibility.

  INDUSTRIAL APPLICABILITY The present invention can be used for a comment information generating apparatus that generates output comment information of a comment that follows an object specified by a user during communication through a moving image on a network. For example, the present invention can be used as a comment information generation device incorporated in an AV device such as a tablet PC, a smartphone, a PC, a video camera, or a TV that can acquire or view a moving image through a network connection.

DESCRIPTION OF SYMBOLS 100 Comment information generation apparatus 101 Movie acquisition part 102 Comment input reception part 103 Object motion generation part 104 Output comment information generation part 105 Output part 106 Target time determination part 107 Object motion determination part 108 Target time correction part 110 Movie 111 Input comment information 112 Comment coordinates 120 Storage device 200 Computer 201a, 201b, 206 I / F
202 CPU
203 ROM
204 RAM
205 HDD
210a, 220 storage device 210b input device

Claims (19)

A video acquisition unit that accepts moving images;
A comment input receiving unit that receives position information of the object in the moving image received by the moving image acquisition unit and a comment that causes the object to follow-up display from a specific timing;
A target time determination unit that determines a target time that is a target value of a time length for continuing to display the comment based on the comment received by the comment input reception unit;
Object movement determination for determining the movement trajectory of the object so that the movement trajectory of the object for displaying the comment following the object indicated by the position information becomes a movement trajectory of the time length of the target time. And
An output comment information generation unit that generates output comment information including the comment and the movement trajectory of the object determined by the object movement determination unit;
A comment information generating apparatus comprising: an output unit that outputs the output comment information generated by the output comment information generating unit. The target time determination unit calculates the target time so that the longer the length of the comment received by the comment input reception unit, the longer the target time,
The comment information generation device according to claim 1, wherein the output unit outputs a movement trajectory having a longer time length when a longer comment is input to the comment input receiving unit. The target time determination unit calculates, as the target time, a value obtained by multiplying a unit display time that is a predetermined display time per character and the number of characters of the comment received by the comment input reception unit,
The comment information generation device according to claim 2, wherein when the comment is input to the comment input reception unit, the output unit outputs a movement trajectory having a length obtained by multiplying the number of characters of the comment by the unit display time. . The target time determination unit further calculates the visual recognition time as the target time when the calculated visual recognition time required for visual recognition of characters does not satisfy the calculated target time,
When the comment is input to the comment input reception unit, the output unit multiplies the number of characters of the comment by a predetermined unit display time which is a display time per character, and the visual recognition time. 3. The comment trajectory generation according to claim 2, wherein a movement trajectory having a length longer than the visual recognition time is output and a movement trajectory having a length equal to or longer than the visual recognition time is output regardless of how short the comment is input. apparatus. The output unit may move differently if the plurality of comments received by the comment input receiving unit are a plurality of the comments assigned to the same position in the same frame but have different numbers of characters. The comment information generating apparatus according to claim 1, wherein a comment is output. The object motion determination unit is the object indicated by the position information received by the comment input reception unit using each of a plurality of motion estimation methods or each of a plurality of motion estimation parameters, and is included in the moving image The movement trajectory of the object is calculated, and the movement trajectory of the object is determined by selecting a movement trajectory having a length closest to the target time among the calculated movement trajectories of the object. The comment information generation device according to any one of the above items. The object motion determination unit, as each of the plurality of motion estimation parameters, (1) influences the ease of tracking of the object, each of a plurality of error tolerance parameters having different values, and (2) the size of each of the plurality of motion estimation parameters. The object indicated by the position information received by the comment input receiving unit based on either one of a plurality of different search window regions or (3) each of a plurality of feature amounts having different values. The movement trajectory of the object in the moving image is calculated, and the movement trajectory of the object is determined by selecting the movement trajectory having the length closest to the target time among the calculated movement trajectories of the object. The comment information generation device according to claim 6. The object motion determination unit may further determine a movement trajectory of the object having a length closest to the target time by using each of a plurality of motion estimation methods or each of a plurality of motion estimation parameters. The comment information generation device according to claim 6 or 7, wherein when a "possible state" occurs, it is determined whether the cause of the state is due to occlusion or a scene change, and the object motion determination method is switched based on the determination result. When the object motion determination unit determines that the cause of the “state where motion cannot be estimated” is caused by occlusion, the occlusion occurs in the movement trajectory of the object in frames after the occlusion frame. The comment information generation device according to claim 8, wherein the movement trajectory of the object having a length closest to the target time is determined by extrapolation based on the movement trajectory of the object up to a frame. When the object motion determination unit determines that the cause of the “state where motion cannot be estimated” is caused by a scene change, the object motion determination unit outputs a movement track of the object up to the frame where the scene change has occurred. The comment information generating device according to claim 8, wherein the comment information generating device is determined as follows. The object motion determination unit determines that the cause of the “state where motion cannot be estimated” is caused by a scene change when a change amount of a luminance histogram between frames constituting the moving image is equal to or greater than a predetermined threshold. The comment information generation device according to claim 8, wherein when the change amount of the luminance histogram is less than the predetermined threshold value, it is determined that the cause of the “state incapable of motion estimation” is due to occlusion. The object motion determination unit receives the comment input when a movement trajectory of the object obtained using each of the plurality of motion estimation methods or each of the plurality of motion estimation parameters is shorter than the target time by a certain time or more. The movement path estimated from the position of the object indicated by the position information and the comment and the position of the object indicated by the position information received by the comment input reception part to the previous frame on the time axis The comment information generation device according to claim 6, wherein a movement trajectory of the object having a length closest to the target time is determined by connecting the front of the movement trajectory of the object. When the time length of the movement trajectory of the object obtained using each of the plurality of motion estimation methods or each of the plurality of motion estimation parameters is shorter than the target time by a certain time or more, Among the movement trajectories of the object based on the position within a certain distance range from the position indicated by the position information of the object received by the comment input reception unit, a movement trajectory whose time length is closest to the target time, The comment information generation device according to any one of claims 6 to 11, wherein the comment input reception unit determines the movement locus of the object indicated by the position information received. When the time length of the movement trajectory of the object obtained using each of the plurality of motion estimation methods or each of the plurality of motion estimation parameters is shorter than the target time by a certain time or more, Among the movement trajectories of the object based on the position within the range specified by the user, including the position indicated by the position information of the object received by the comment input reception unit, the time length is the length closest to the target time The comment information generation device according to any one of claims 6 to 11, wherein the movement locus is determined as a movement locus of the object indicated by the position information received by the comment input reception unit. When the time length of the movement trajectory of the object obtained using each of the plurality of motion estimation methods or each of the plurality of motion estimation parameters is shorter than the target time by a certain time or more, The area of the object is divided into a plurality of areas, and a movement trajectory of an area having a length closest to the target time among the areas obtained by the division is determined as a movement trajectory of the object. The comment information generation device according to any one of the above items. The object movement determination unit further determines a movement locus having a length closest to a target time for the center of gravity of the object indicated by the position information received by the comment input reception unit, and the comment input reception unit receives Based on the relative positional relationship between the position where the comment is given and the center of gravity of the object, the determined movement trajectory of the object is as if it were a movement trajectory from the position where the comment is given. It correct | amends and outputs The comment information generation apparatus of any one of Claims 1-11. further,
Based on the movement trajectory of the object determined by the object movement determination unit, a target time correction unit that corrects the target time so that the target time becomes longer as the moving speed of the object increases.
The object motion determination unit further includes a time of the target time after the movement trajectory of the object is corrected by the target time correction unit for displaying the comment following the object indicated by the position information. The comment information generation device according to claim 1, wherein the movement trajectory of the object is re-determined so as to be long. A video acquisition step for accepting a moving image as input;
A comment input receiving step for receiving input of position information of the object in the moving image received in the moving image acquisition step and a comment to be displayed following the object;
A target time determining step for determining a target time that is a target value of a time length for displaying the comment based on the comment received in the comment input receiving step;
An object movement determination step of determining the movement trajectory of the object so that the movement trajectory of the object for causing the object indicated by the position information to follow and display the comment has a length sufficiently close to the target time;
An output comment information generation step for generating output comment information including the comment and the movement trajectory of the object determined in the object movement determination step;
An output step of outputting the output comment information generated in the output comment information generation step.   A program for causing a computer to execute the comment information generation method according to claim 18.

Images