WO2016092933A1

WO2016092933A1 - Information processing device, information processing method, and program

Info

Publication number: WO2016092933A1
Application number: PCT/JP2015/076633
Authority: WO
Inventors: 卓青木; 亮向山; 田村　錬志; 賢一山浦
Original assignee: ソニー株式会社
Priority date: 2014-12-08
Filing date: 2015-09-18
Publication date: 2016-06-16

Abstract

Provided is an information processing device provided with: an evaluation information acquisition unit (205) for acquiring action evaluation information that evaluates an action of an actor on the basis of an action image including the action; a situation information acquisition unit (207) for acquiring user situation information indicating the situation of a user viewing the action image; and an additional information generation unit (209) for generating, according to the user situation information, additional information that is presented to the user with the action image and expresses at least part of the action evaluation information.

Description

Information processing apparatus, information processing method, and program

This disclosure relates to an information processing apparatus, an information processing method, and a program.

In recent years, services for enjoying sports played by users themselves in a variety of ways have been proposed by using sensors such as high-performance and low-cost acceleration sensors. For example, in Patent Document 1, in order to more accurately grasp the state of sports play by a user, the occurrence of a play event in sports is detected based on sensor information, and a play image is captured according to the occurrence of the play event. Techniques for controlling the are described.

JP 2013-188426 A

However, the way of enjoying sports is not necessarily limited to being played by the users themselves. Watching professional sports and sports competitions via television etc. is also a very common way to enjoy sports. However, the technique as described in Patent Document 1 is intended to more accurately grasp the state of sports play, and does not necessarily provide a better experience for users who exclusively watch play. It is not intended.

Therefore, in the present disclosure, a new and improved information processing apparatus and information processing capable of providing more appropriate information reflecting an evaluation of an action based on an image to a user who views the image including the action Propose methods and programs.

According to the present disclosure, an evaluation information acquisition unit that acquires action evaluation information obtained by evaluating the action based on an action image including an action of an actor, and user situation information indicating a situation of a user who views the action image are acquired. An information processing apparatus comprising: a situation information acquisition unit; and an additional information generation unit that generates additional information that is presented to the user together with the action image and expresses at least part of the action evaluation information according to the user situation information Is provided.

Further, according to the present disclosure, acquiring action evaluation information obtained by evaluating the action based on an action image including an actor's action, and acquiring user situation information indicating a situation of a user viewing the action image And generating additional information that is presented to the user together with the action image and expresses at least a part of the action evaluation information according to the user situation information.

Moreover, according to this indication, the function which acquires the action evaluation information which evaluated the said action based on the action image containing an action of an actor, and the function which acquires the user condition information which shows the condition of the user who views the said action image And a program for causing the computer to realize the function of generating additional information that is presented to the user together with the action image and expresses at least part of the action evaluation information according to the user situation information. .

Providing information that evaluates an action based on an image together with the action image helps the user understand the action because the provided information is linked to the content of the image. However, since what kind of information is appropriate may vary depending on the user's situation, more appropriate information is provided to the user by generating additional information based on action evaluation based on the user's situation. can do.

As described above, according to the present disclosure, it is possible to provide more appropriate information reflecting the evaluation of the action based on the image to the user who views the image including the action.

Note that the above effects are not necessarily limited, and any of the effects shown in the present specification, or other effects that can be grasped from the present specification, together with or in place of the above effects. May be played.

It is a figure showing an example of composition of a system concerning one embodiment of this indication. It is a figure showing an example of functional composition of an information processor concerning one embodiment of this indication. It is a figure showing the 1st example of additional information concerning one embodiment of this indication. It is a figure showing the 2nd example of additional information concerning one embodiment of this indication. It is a figure which shows the example of additional information with little information content in one Embodiment of this indication. It is a figure which shows the example of the additional information with medium amount of information in one Embodiment of this indication. It is a figure which shows the example of additional information with much information amount in one Embodiment of this indication. 14 is a flowchart illustrating an example of processing for controlling the amount of additional information according to an embodiment of the present disclosure. FIG. 5 is a diagram illustrating an example of additional information for a user who views an action image alone in an embodiment of the present disclosure. FIG. 4 is a diagram illustrating an example of additional information for a user who views a plurality of action images in an embodiment of the present disclosure. It is a figure showing the 1st example of other additional information in one embodiment of this indication. It is a figure showing the 2nd example of other additional information in one embodiment of this indication. It is a figure showing the 3rd example of other additional information in one embodiment of this indication. FIG. 3 is a block diagram illustrating a hardware configuration example of an information processing apparatus according to an embodiment of the present disclosure.

Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In the present specification and drawings, components having substantially the same functional configuration are denoted by the same reference numerals, and redundant description is omitted.

The description will be made in the following order.
1. System configuration 2. Functional configuration of information processing apparatus Example of visual additional information 4. 4. Example of control of information amount of additional information 5. Example of control of content of additional information 6. Examples of other additional information Hardware configuration Supplement

(1. System configuration)
FIG. 1 is a diagram illustrating a configuration example of a system according to an embodiment of the present disclosure. Referring to FIG. 1, a system 10 includes a camera 100, a server 200, and an output device 300 (a television 300a and a recorder 300b).

The camera 100 captures an action image including an actor's action. Here, an actor is a person or thing that performs some action. Actions can include all kinds of actions that can be evaluated, such as sports play, performances such as dance, and professional skills. For example, if the action is sport play, the actor is also called a player. In addition, actors such as performers, dancers, and testers are called in various ways depending on the type of action. The camera 100 can be, for example, a digital video camera that can acquire a still image or a moving image. The camera 100 has at least one image sensor and records an image including an actor as a subject as electronic data. Furthermore, the camera 100 may include an image processing circuit that performs processing for converting electronic image data into a format suitable for the server 200 and the output device 300. In addition, the camera 100 may include an information processing circuit that realizes part or all of the functions of the information processing apparatus described later. The information processing circuit can be configured by applying a hardware configuration of an information processing apparatus to be described later, for example.

The server 200 receives an action image from the camera 100 (S11), and generates or receives action evaluation information that evaluates an action based on the action image. Further, in an example, the server 200 receives user status information indicating the status of the user viewing the action image from the output device 300 (S13a), and adds the additional information generated according to the user status information together with the action image. It transmits to the output device 300 (S12). Here, the additional information includes at least a part of the action evaluation information. Alternatively, the server 200 transmits the action image together with the action evaluation information to the output device 300 as it is (S12). In this case, the generation of additional information using the action evaluation information is performed by the output device 300 (S13b). The server 200 can be realized by one or a plurality of server devices on a network. Each server device can be configured by applying a hardware configuration of an information processing device to be described later, for example.

The output device 300 receives an action image from the server 200 (S12), outputs the action image together with additional information, and presents it to the user. As described above, the additional information may be generated by the server 200 based on the user situation information (S13a) transmitted by the output device 300, and transmitted from the server 200 to the output device 300 together with the action image (S12). Alternatively, the output device 300 receives the action evaluation information together with the action image from the server 200 (S12), generates additional information from the action evaluation information based on the user situation information (S13b), and then displays the action image and additional information. It may be output. In the illustrated example, the output device 300 is illustrated as a television 300a and a recorder 300b. For example, an action image transmitted from the server 200 by a broadcast wave is received, recorded, and saved by the recorder 300b, and later played and output via the television 300a. Alternatively, the action image may be received and output directly by the television 300a without using the recorder 300b. In such a case, the television 300a or the recorder 300b may include an information processing circuit that performs information processing related to generation of additional information and the like. The information processing circuit can be configured by applying a hardware configuration of an information processing apparatus to be described later, for example.

Note that the example of the output device 300 is not limited to the combination of the television 300a and the recorder 300b in the illustrated example. For example, the television 300a may be used alone as the output device 300. In addition, any device having a function of outputting a received image can be used as the output device 300. For example, a smartphone, a tablet, a personal computer, or the like may be used as the output device 300.

(2. Functional configuration of information processing apparatus)
FIG. 2 is a diagram illustrating a functional configuration example of the information processing apparatus according to an embodiment of the present disclosure. Referring to FIG. 2, the information processing apparatus includes an image acquisition unit 201, an evaluation information generation unit 203, an evaluation information acquisition unit 205, a situation information acquisition unit 207, an additional information generation unit 209, and a screen generation unit 211. The functional configuration of the information processing apparatus is realized by the camera 100, the server 200, or the output device 300 in the system 10 as described above. These functional configurations may be realized by being integrated in a single device, or may be realized by being distributed to a plurality of devices. The functional configuration described below is realized by, for example, a processor such as a CPU of one or a plurality of devices operating according to a program stored in a memory or storage.

The image acquisition unit 201 acquires an action image taken by the camera 100. As described above, the action image is an image including an action by an actor. The image acquisition unit 201 may be realized by, for example, a communication device that receives image data from the camera 100 connected via a network or the like by wired or wireless communication. The image acquisition unit 201 may be a software interface such as a driver program that acquires data from such a communication device. For example, when a screen generation unit 211 described later is realized in the camera 100, the image acquisition unit 201 may acquire an action image via an internal communication path such as a bus.

The evaluation information generation unit 203 generates action evaluation information that evaluates an action based on the action image. In the illustrated example, the evaluation information generation unit 203 acquires an action image from the camera 100. Similar to the image acquisition unit 201 described above, the action image may be received by communication between apparatuses, or may be acquired via an internal communication path. For example, the evaluation information generation unit 203 performs image analysis on the action image, and estimates the displacement, rotation, posture, and the like of the actor. Furthermore, the evaluation information generation unit 203 evaluates the estimated information. At this time, the evaluation information generation unit 203 may evaluate the action by applying an action determination or scoring standard. Judgment or scoring standards are determined for each type of action (for example, sports, performance, professional skills, etc.). Such criteria may already be used by referees or judges, or may be newly introduced for the evaluation of actions in the system 10. As will be described later, action evaluation information or additional information based thereon may be provided to a referee or a judge. If necessary, the evaluation information generation unit 203 may generate action evaluation information based on information other than the action image. For example, the evaluation information generation unit 203 may generate action evaluation information based on a detection value obtained from a sensor such as an inertial sensor attached to the actor in addition to the action image.

The evaluation information acquisition unit 205 acquires the action evaluation information generated by the evaluation information generation unit 203. For example, in the system 10, when the evaluation information generation unit 203 and the additional information generation unit 209 included in the server 200 are realized by different server devices, the evaluation information generation unit 203 is realized by the camera 100, or the system When the action evaluation information is generated by an external server that is not included in 10, the evaluation information acquisition unit 205 may be realized by a communication device that receives the action evaluation information through communication between the devices. It may be a software interface for acquiring data from such a communication device. Alternatively, when the evaluation information generation unit 203 and the additional information generation unit 209 are realized by the same server device (or the camera 100 or the output device 300), the evaluation information acquisition unit 205 uses an internal communication path such as a bus. It can be realized as a software interface for acquiring action evaluation information through the interface.

The status information acquisition unit 207 acquires user status information indicating the status of the user who views the action image via the output device 300. The user status information indicates, for example, the time for the user to refer to the additional information presented together with the action image, the user's familiarity regarding the action included in the action image, or the number of users. Can contain information. Similar to the image acquisition unit 201 described above, the situation information acquisition unit 207 also receives user situation information through communication between devices when the additional information generation unit 209 is realized in a device different from the output device 300. It is realized by a device or a software interface for acquiring data from such a communication device. In addition, when the additional information generation unit 209 is realized by the output device 300, the status information acquisition unit 207 is a software interface that acquires user status information via an internal communication path such as a bus. Can be realized.

Here, some specific examples of user status information will be described. For example, the number of users who view an action image is detected by analyzing an image including a user in front of the display acquired by a camera attached to the display in the output device 300. Further, for example, a user who views an action image recognizes the user's face from an image acquired by a camera attached to a display in the output device 300 or when the output device 300 accepts a user login. Is specified by information of the logged-in user. For each identified user, by referring to the profile and schedule registered in the service, content usage history, etc., the depth of the user's knowledge about the action, the time to watch the action image, etc. Can be estimated.

The additional information generation unit 209 generates additional information that represents at least a part of the action evaluation information acquired by the evaluation information acquisition unit 205 based on the user situation information acquired by the situation information acquisition unit 207. As described above, the additional information is presented to the user together with the action image in the output device 300. Since the additional information expresses action evaluation information, it does not necessarily include the information included in the action evaluation information as it is. The additional information generation unit 209 may generate additional information by adding some processing to the action evaluation information. In the present embodiment, the additional information includes visual information displayed on the screen together with the action image. More specifically, for example, the additional information includes text or graphics. The text or graphic may be superimposed on the action image or displayed in the vicinity of the action image. A specific example of additional information will be described later. When generating additional information from at least a part of the action evaluation information, the additional information generation unit 209 changes the amount of information and the type of information expressed by the additional information based on the user situation information.

In the above example, the additional information generation unit 209 selects information included in the additional information from the action evaluation information generated by the evaluation information generation unit 203. However, the embodiment of the present disclosure is such an example. Is not limited. In another example, even if the evaluation information generation unit 203 generates action evaluation information according to the amount of information or the type of information that the additional information generation unit 209 has decided to generate according to the user situation information. Good. That is, in the embodiment of the present disclosure, the additional information may be generated by extracting part or all of the action evaluation information generated regardless of the user's situation, or generated according to the situation by the user. It may be generated based on the action evaluation information.

The screen generation unit 211 generates a screen including the action image acquired by the image acquisition unit 201 and the additional information generated by the additional information generation unit 209. As described above, the additional information generation unit 209 includes text or graphics generated based on the action evaluation information. The screen generation unit 211 arranges these texts or graphics so as to be superimposed on the action image or displayed in the vicinity of the action image. For example, the screen generation unit 211 may draw a screen including an action image and additional information, and provide the image data of the drawn screen for display by the output device 300. Alternatively, the screen generation unit 211 generates image data for drawing an action image on the screen and image data for drawing additional information, respectively, and the output device 300 generates an action image and an image based on these image data. Additional information may be displayed on the screen. The screen generation unit 211 may be realized, for example, in the server 200 or in the output device 300.

Note that the functional configuration of the information processing apparatus illustrated in FIG. 2 is described corresponding to an example in which the information processing apparatus includes a processor that mainly implements the additional information generation unit 209. In this case, the information processing apparatus includes, in addition to the additional information generation unit 209, an evaluation information acquisition unit 205 that is an interface for acquiring action evaluation information, and a situation information acquisition unit 207 that is an interface for acquiring user situation information. . In another example, the processor of the information processing apparatus may further realize the evaluation information generation unit 203. In this case, the evaluation information acquisition unit 205 can be an internal program interface. Further, the information processing apparatus may include an image acquisition unit 201, and the processor of the information processing apparatus may realize the screen generation unit 211. In still another example, the information processing apparatus is included in the camera 100 or the output apparatus 300, and the functional configuration of the information processing apparatus illustrated in FIG. 2 may be implemented internally in the camera 100 or the output apparatus 300. .

(3. Examples of additional visual information)
FIG. 3 is a diagram illustrating a first example of additional information according to an embodiment of the present disclosure. FIG. 3 shows an action image 1000 and a graphic display of a rotation axis 1010, a rotation axis blur 1020, a rotation speed 1030, and a rotation axis shift 1040 transparently superimposed on the action image 1000. In the action image 1000, the spin of figure skating is captured as an action. For example, the evaluation information generation unit 203 evaluates the spin based on the attitude of the skater estimated from the action image 1000. For example, the evaluation information generation unit 203 may estimate the posture by extracting a skater image from the action image 1000 that is an RGB image by using an image recognition technique. Alternatively, the evaluation information generation unit 203 may extract a skater image from the action image 1000 in which the depth information is added to the RGB image, using image recognition and space recognition techniques. Instead of or together with the depth information, RGB images taken simultaneously from a plurality of positions may be used. Furthermore, the evaluation information generation unit 203 may use a detection value of an inertial sensor attached to the skater in order to estimate the attitude of the skater.

For example, by analyzing the change in the attitude of the skater estimated as described above in a time series, the evaluation information generating unit 203 can evaluate the spin executed by the skater. In the illustrated example, the evaluation information generation unit 203 evaluates the time series change of the spin rotation axis (represented by a straight line in space) and the spin rotation speed. Based on the action evaluation information including such evaluation, the additional information generation unit 209 generates additional information to be presented together with the action image. For example, the additional information generation unit 209 expresses, on the action image 1000, the rotation of the axis relative to the foot of the skater (the ground contact point of the skate) along with the rotation axis 1010 of the spin as the rotation axis blur 1020. At this time, the additional information generation unit 209 determines the position of the skater in the action image 1000 and the posture of the skater estimated based on the action image 1000 in order to identify the position of the skater's foot or the position of the rotation axis. Use the information. Further, the additional information generation unit 209 displays the rotation speed 1030 according to the position of the specified rotation axis. Further, in the illustrated example, the additional information generation unit 209 represents the transition of the position of the skater's foot on the action image 1000 as a rotation axis shift 1040. In addition to this, in skating, for example, graphics indicating the jump height and distance, edge depth, and the like may be superimposed on the action image 1000 and displayed.

FIG. 4 is a diagram illustrating a second example of additional information according to an embodiment of the present disclosure. FIG. 4 shows an action image 1000 and a rotation speed graph 1050 displayed together with the action image 1000. In the illustrated example, the additional information generation unit 209 draws the graph 1050 based on the rotation speed included in the action evaluation information generated by the evaluation information generation unit 203. At this time, the additional information generation unit 209 reads, for example, an ideal change in the rotation speed recorded in the internal storage of the information processing apparatus and compares it with the rotation speed of the skater's spin shown in the action image 1000. The graph 1050 may be drawn. Note that information on the ideal rotation speed may be read from a storage external to the information processing apparatus. Here, the graph 1050 is an example of additional information that does not necessarily need to be superimposed on the action image 1000. The rotating shaft 1010, the rotating shaft blur 1020, the rotating speed 1030, and the rotating shaft shift 1040 described above with reference to FIG. 3 are basically transparently superimposed on the action image 1000 and reflected in the action image 1000. While the additional information is visible along with the skater image, the graph 1050 may be transparently superimposed on the action image 1000, but may be opaquely superimposed as illustrated. Further, it may be displayed separately in the vicinity thereof without being superimposed on the action image 1000.

(4. Example of controlling the amount of additional information)
Next, an example of controlling the amount of additional information according to an embodiment of the present disclosure will be described. In the present embodiment, as described above, the additional information is presented together with the action image. At this time, as the amount of information included in the additional information increases, the user can obtain more enriched information regarding the action included in the action image (in the above example, the action in figure skating). On the other hand, for example, when the user is watching an action image in a live broadcast, or when the user does not have much time even when watching a recorded action image, the amount of information is too large. It becomes an obstacle. Therefore, the present embodiment may include a configuration in which the additional information generation unit 209 controls the information amount of the additional information according to the user situation when generating the additional information based on the user situation information. Hereinafter, an example of such a configuration will be described.

FIG. 5 is a diagram illustrating an example of additional information with a small amount of information according to an embodiment of the present disclosure. FIG. 5 shows an action image 1100 and a jump locus 1110 that is transparently superimposed on the action image 1100. In the action image 1100, a figure skating jump is captured as an action. For example, the evaluation information generation unit 203 evaluates the jump based on the skater posture estimated from the action image 1100. The estimation of the skater posture by the evaluation information generation unit 203 can be performed, for example, in the same manner as the spin example described above with reference to FIG. By analyzing the pose change of the skater estimated in this way in time series, the evaluation information generating unit 203 can evaluate the jump executed by the skater. In the illustrated example, the evaluation information generation unit 203 performs skating in the action image 1100 at the time of jump crossing (the moment when the skates are separated from the ice surface) and at the time of landing (the moment when the skates are touching the ice surface). Identify shoe coordinates. The additional information generation unit 209 draws the trajectory of the skate shoes in the action image 1100 between the jump crossing and landing as the jump trajectory 1110.

Here, the jump trajectory 1110 shown in FIG. 5 includes the skater's jump trajectory 1111 currently shown in the action image 1100 and the last year's jump trajectory 1112 by the same skater. The additional information generation unit 209 draws the last year's jump trajectory 1112 based on, for example, information read from the storage. Information for drawing last year's jump trajectory 1112 is stored in, for example, a storage inside or outside the information processing apparatus as reference information. In the vicinity of the

jump trajectories

1111 and 1112, texts “TODAY” and “LAST YEAR” and texts indicating the jump distances of the respective jumps are displayed. In the additional information displayed together with the action image 1100 in the example shown in FIG. 5, the jump-related information captured in the action image 1100 is simply expressed by the graphic and text of the jump trajectory 1110.

FIG. 6 is a diagram illustrating an example of additional information having a medium information amount according to an embodiment of the present disclosure. FIG. 6 shows an action image 1100, a jump trajectory 1110, and jump information 1120. In the example shown in FIG. 6, the jump trajectory 1110 includes the current jump trajectory 1111 of the skater (assumed to be Jane Smith) shown in the action image 1100, the jump trajectory 1112 last year of Jane, and the current winner ( Unfortunately not Jane) jump trajectory 1113. Furthermore, the jump information 1120 is displayed in association with each jump trajectory 1110 and includes text indicating the jump flight distance, height, and rotation speed during the jump. In the additional information displayed together with the action image 1100 in the example shown in FIG. 6, more complete information regarding the jump captured in the action image 1100 is expressed by the graphic of the jump trajectory 1110 and the text of the jump information 1120. Yes.

FIG. 7 is a diagram illustrating an example of additional information with a large amount of information according to an embodiment of the present disclosure. FIG. 7 shows an action image 1100, a jump locus 1110, jump information 1120, and a graph 1130. In the example shown in FIG. 7, jump trajectory 1110 and jump information 1120 are displayed so that they can be selected from Jane's current jump, Jane's last year's jump, and current winner's jump. For example, the user can display any set of jump trajectory 1110 and jump information 1120 by selecting arbitrary jump information 1120 using a pointing device such as a touch panel or a mouse on the output device 300. A graph 1130 corresponds to the displayed set of jump trajectory 1110 and jump information 1120, and displays a change in rotational speed in the jump as a graph. The graph 1130 is folded (wound up) at the start of display, and the non-display portion 1131 is expanded when the user selects the icon of the graph 1130 with a pointing device, for example.

In the example shown in FIG. 7 above, the additional information displayed together with the action image 1100 includes more information than the examples shown in FIGS. In addition, unlike the examples shown in FIGS. 5 and 6, in the example shown in FIG. 7, additional information that allows the user viewing the action image 1100 to select the content is generated. When the amount of information to be provided is large, visibility can be improved, for example, by displaying the same type of information according to the user's selection, rather than displaying all information at once. Accordingly, it may be useful to generate selectable additional information even if it is not necessarily the case of additional information having the largest amount of information (for example, in the case of the example shown in FIG. 6). However, since it takes a certain amount of time to accept a user operation for selection and change the content of additional information according to the operation, selectable additional information is generated when the user views the action image 1100. When the time taken to do this is long, more specifically, it is desirable that the user temporarily stops the action image 1100 in the jump section or reproduces the action image 1100 repeatedly. Therefore, in the present embodiment, the additional information generation unit 209 displays selectable additional information, for example, when it is estimated that the user has a long time for referring to the action image to some extent.

FIG. 8 is a flowchart illustrating an example of processing for controlling the amount of additional information according to an embodiment of the present disclosure. Referring to FIG. 8, first, the evaluation information acquisition unit 205 and the situation information acquisition unit 207 acquire action evaluation information and user situation information (S101). Next, the additional information generation unit 209 estimates the time for the user to refer to the additional information provided together with the action image from the user situation information.

Here, the holding time may be estimated based on, for example, whether the user is viewing the action image live or viewing the recorded action image. For example, when the difference between the time when the actor performed the action included in the action image and the time when the user views the action image is below the threshold, that is, when the user is viewing the action image in a state close to live broadcast The time for the user to refer to the additional information can be estimated to be short. Conversely, if the difference between the time when the actor performs the action and the time when the user views the action image exceeds the threshold, that is, if the user is viewing the recorded action image afterwards, the user will receive additional information. It can be estimated that the holding time for referring to is long.

Alternatively, when the user views the recorded action image, the time for the user to refer to the additional information may be estimated based on the user's schedule, the playback time set for viewing the action image, or the like. Good. In addition, for example, in the output device 300, information indicating the time for the user to refer to the additional information or the amount of additional information that the user considers appropriate for viewing the action image may be input by the user.

The additional information generating unit 209 that has estimated the holding time as described above determines whether or not the estimated holding time is less than the first threshold th1 (S103). Here, when the possession time is less than the threshold th1 (YES), the additional information generation unit 209 generates simple additional information with a small amount of information, such as the example illustrated in FIG. 5 (S105). Alternatively, when it is determined that the holding time is shorter as in S105, the additional information generation unit 209 may determine not to generate additional information.

If it is determined in step S103 that the time for the user to refer to the additional information does not fall below the threshold th1, the additional information generation unit 209 further determines whether or not the estimated time remaining exceeds the second threshold th2. Is determined (S107). Here, when the holding time exceeds the threshold th2 (YES), the additional information generation unit 209 generates selectable additional information with a larger amount of information, such as the example illustrated in FIG. 7 (S109). . In other cases (NO in S107), the additional information generation unit 209 generates additional information with a medium amount of information, such as the example illustrated in FIG. 6 (S111). In the illustrated example, the additional information in this case is normal additional information, but the normal additional information may not be defined.

According to the control of the information amount of the additional information as described above, an appropriate amount of information can be presented according to, for example, the time for the user to refer to the additional information. The user may view the action image live, or may view the recorded action image. The appropriate amount of additional information may vary depending on the user's situation such as a schedule. In the embodiment, it is possible to flexibly cope with such a change and provide an appropriate amount of information to the user.

(5. Example of control of content of additional information)
Next, an example of control of the content of additional information according to an embodiment of the present disclosure will be described. In the present embodiment, as described above, additional information is presented together with action information. At this time, what kind of information the user needs as additional information depends on the user's situation, more specifically, the depth of the user's knowledge about the action included in the action image and the action image. It may vary depending on the number of users. Therefore, the present embodiment may include a configuration in which the additional information generation unit 209 controls the content of the additional information in accordance with the user situation when generating the additional information based on the user situation information. Hereinafter, an example of such a configuration will be described.

FIG. 9 is a diagram illustrating an example of additional information for a user who views an action image alone in an embodiment of the present disclosure. FIG. 9 shows an action image 1100, a jump trajectory 1110 that is transparently superimposed on the action image 1100, and jump information 1120. In the example shown in FIG. 9, the jump trajectory 1110 and the jump information 1120 are assumed to be a jump of a skater (Jane Smith) shown in the action image 1100 and a user who is watching the action image 1100 jumps temporarily. Jumps (labeled “YOU”), assumed jumps of the user's same age (30's), and assumed jumps of the user's youth (YOU IN YOUTH). Such jump trajectory 1110 and jump information 1120 are drawn based on, for example, action evaluation information related to Jane Smith's jump and average basic physical strength data for each user's basic physical strength data, age, and age.

FIG. 10 is a diagram illustrating an example of additional information for a user who views a plurality of action images according to an embodiment of the present disclosure. In the example illustrated in FIG. 10, the additional information includes a jump locus 1110 that is transparently superimposed on the action image 1100, jump information 1120, and a ranking 1140 related to the jump. In the example shown in FIG. 10, the jump trajectory 1110 and the jump information 1120 are a skater (Jane Smith) jump shown in the action image 1100 and a plurality of users (JOHNNY and DAD) who are watching the action image 1100. It is displayed about the assumed jump when each jumps temporarily. Such jump trajectory 1110 and jump information 1120 are drawn based on, for example, action evaluation information regarding Jane Smith's jump and basic physical strength data of each user. The ranking 1140 displays the assumed flying distance, rotational speed, and overall ranking for the assumed jumps when JOHNNY and DAD each jump.

In the example shown in FIGS. 9 and 10 above, the additional information generation unit 209 controls the content of the additional information according to the number of users viewing the action image. More specifically, when one user who is viewing an action image is detected by the output device 300, the additional information generation unit 209, in addition to the skater (Jane Smith), Additional information including virtual jump information of the user himself, the same age, and young age is generated. On the other hand, when a plurality of users viewing an action image is detected by the output device 300, the additional information generation unit 209 compares virtual jumps of a plurality of users in addition to Jane Smith, Generate additional information to rank. As a result, the user who is watching the action image alone can feel the jump level of Jane Smith by providing a lot of familiar information about himself. In addition, users who are viewing action images in a plurality are provided with virtual jump information about each user, and further provided with information such as ranking that allows them to compete, for example, between each other. You can get a topic at.

It should be noted that the additional information such as the example shown in FIGS. 9 and 10 can be additional information for a user who is not deeply familiar with the action (figure skating in the illustrated example), for example. For example, when it is estimated that the user's knowledge about figure skating is deep from the user's profile, viewing history of content such as images, the additional information generation unit 209 is, for example, the example described above with reference to FIG. As described above, additional information that enables comparison with other skaters or past records may be provided. In this case, it is assumed that users who are familiar with figure skating are interested in comparison with various skaters and past records as competitions. On the other hand, users who are not very proficient are assumed to be interested in realizing a high level of skater based on comparison with themselves and nearby users. Note that such an assumption is merely an example, and additional information can be generated based on various assumptions.

(6. Examples of other additional information)
FIG. 11 is a diagram illustrating a first example of other additional information according to an embodiment of the present disclosure. FIG. 11 shows an action image 1200 and a composite image 1210 displayed on the action image 1200. In the action image 1200, the performance of synchronized swimming is captured as an action. For example, the evaluation information generation unit 203 evaluates the performance of synchronized swimming based on the posture of the swimmer estimated from the action image 1200. More specifically, the evaluation information generation unit 203 evaluates the performance according to how much the movements of the respective swimmers are synchronized. For example, the evaluation information generation unit 203 compares the movements of the respective swimmers in time series, and compares the movement start timing, the magnitude of movement, the speed of movement, and the like.

On the other hand, the additional information generation unit 209 transparently superimposes the swimmer images extracted from the action image 1200 to generate a composite image 1210. More specifically, the additional information generation unit 209 combines the positions of the images 1211 to 1214 of the respective swimmers based on the postures of the swimmers estimated by the evaluation information generation unit 203 and transparently superimposes them. An image 1210 is generated. Thus, the user who views the action image 1200 can easily grasp how much the movements of the respective swimmers are synchronized.

FIG. 12 is a diagram illustrating a second example of other additional information according to an embodiment of the present disclosure. FIG. 12 shows an action image 1300 and a punch icon 1310 displayed on the action image 1300. In the action image 1300, a punch in boxing is captured as an action. For example, the evaluation information generation unit 203 estimates the strength of the punch based on the detection value of the impact sensor built in the glove. Furthermore, the evaluation information generation unit 203 estimates the contact position between the glove and the boxer body at the time of punching, that is, the position where the punch hits, based on the posture of the boxer estimated from the action image 1300. For example, the evaluation information generation unit 203 determines whether or not the punch is an effective hit based on the position where the punch hits and the strength of the punch. Further, the evaluation information generation unit 203 may evaluate the aggressiveness of the boxer from the number of punches including those that are not effective hits and the tendency of the hit positions.

On the other hand, the additional information generation unit 209 displays a punch icon 1310 on the action image 1300 based on the punch hit position estimated by the evaluation information generation unit 203 and the punch strength. More specifically, the additional information generation unit 209 determines the display position of the punch icon 1310 according to the estimated position where the punch hits, and determines the size of the punch icon 1310 based on the estimated punch strength. Determine the color. Further, the additional information generation unit 209 may change the color of the punch icon 1310 depending on which punch of the fighting boxers is punched. As a result, the user who views the action image 1300 can intuitively understand where the punches emitted by the respective boxers hit and how strong they were.

FIG. 13 is a diagram illustrating a third example of other additional information according to an embodiment of the present disclosure. FIG. 13 shows an action image 1400 and a ground area 1410 displayed on the action image 1400. In the action image 1400, a throwing technique in judo is captured as an action. For example, the evaluation information generation unit 203 evaluates the effectiveness of the throwing technique based on the relationship between the player posture estimated from the action image 1400 and the floor surface recognized in the action image 1400. For example, the effectiveness of throwing techniques varies depending on whether the thrown athlete fell from the shoulder to the floor or from the back. In this case, the evaluation information generation unit 203 also refers to the athlete's skeleton model to determine whether the contact position between the thrown athlete's body and the floor is the shoulder or the back. Thereby, the effectiveness of the throwing technique and the points given based on it can be determined.

On the other hand, the additional information generation unit 209 causes the action image 1400 to display the ground contact area 1410 based on the contact position between the body of the thrown player and the floor estimated by the evaluation information generation unit 203. More specifically, the additional information generation unit 209 displays a graphic of a predetermined color in an area corresponding to the estimated contact position. Further, the additional information generation unit 209 may change the color or the like of the ground contact area 1410 depending on, for example, whether the contact position is the shoulder or the back. Accordingly, the user who views the action image 1400 can intuitively understand why the points are given or not given to the throwing technique.

For example, the additional information as described above with reference to FIGS. 11 to 13 and the action evaluation information based on the additional information include information related to action determination or scoring (synchronization of swimmer movements in synchronized swimming). , The hit position and strength of the punch in boxing, and the grounded position of the thrown athlete in judo). Therefore, in the above example, the action evaluation information includes information on determining or scoring the action, and the additional information can be said to indicate the progress or result of the determination or scoring. In the above example, the additional information indicates the progress of scoring, but the result (eg, given points) may be indicated in the same manner. Such information is not limited to users who watch action images as entertainment, for example, but is provided to assist users who actually evaluate actions, such as referees or judges. Also good.

As an example of the relationship between the other additional information and the user situation as described above, the additional information generation unit 209 determines whether or not to generate the additional information as described above based on the action evaluation information. The determination may be made according to the depth of the user's knowledge about the competition captured in the action image. For example, if the user has a deep knowledge of the game, the additional information as in the above example may rather interfere with watching the game. On the other hand, if the user is not familiar with the game, the additional information as in the above example is useful for the user to understand why the points given by the judgment of the referee were given, for example. sell. Therefore, the additional information generation unit 209 generates simple additional information that does not generate additional information when the knowledge is deep, or displays only the determination result, according to the depth of knowledge about the user's competition, Otherwise, easy-to-understand additional information with graphics as in the above example may be generated.

As another example, the additional information generation unit 209 uses the additional information as described above with reference to FIGS. 5 to 7 and the additional information as described with reference to FIGS. It may be selectively generated according to the depth of the slag. For example, if the user has a deep knowledge of figure skating, additional information regarding the content of the competition, as described with reference to FIGS. 5 to 7, may be appropriate. On the other hand, if the user is not familiar with figure skating, additional information for comparing a familiar example such as the user himself with the skater performance shown in the action image as described with reference to FIGS. 9 and 10 is appropriate. It can be. As described above, the additional information generation unit 209 may change the content of the additional information according to the depth of the user's knowledge about the action.

(7. Hardware configuration)
Next, a hardware configuration of the information processing apparatus according to the embodiment of the present disclosure will be described with reference to FIG. FIG. 14 is a block diagram illustrating a hardware configuration example of the information processing apparatus according to the embodiment of the present disclosure. The illustrated information processing apparatus 900 can realize, for example, the server apparatus, camera, or output apparatus in the above-described embodiment.

The information processing apparatus 900 includes a CPU (Central Processing unit) 901, a ROM (Read Only Memory) 903, and a RAM (Random Access Memory) 905. The information processing apparatus 900 may include a host bus 907, a bridge 909, an external bus 911, an interface 913, an input device 915, an output device 917, a storage device 919, a drive 921, a connection port 923, and a communication device 925. Furthermore, the information processing apparatus 900 may include an imaging device 933 and a sensor 935 as necessary. The information processing apparatus 900 may include a processing circuit such as a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), or an FPGA (Field-Programmable Gate Array) instead of or in addition to the CPU 901.

The CPU 901 functions as an arithmetic processing device and a control device, and controls all or a part of the operation in the information processing device 900 according to various programs recorded in the ROM 903, the RAM 905, the storage device 919, or the removable recording medium 927. The ROM 903 stores programs and calculation parameters used by the CPU 901. The RAM 905 primarily stores programs used in the execution of the CPU 901, parameters that change as appropriate during the execution, and the like. The CPU 901, the ROM 903, and the RAM 905 are connected to each other by a host bus 907 configured by an internal bus such as a CPU bus. Further, the host bus 907 is connected to an external bus 911 such as a PCI (Peripheral Component Interconnect / Interface) bus via a bridge 909.

The input device 915 is a device operated by the user, such as a mouse, a keyboard, a touch panel, a button, a switch, and a lever. The input device 915 may be, for example, a remote control device that uses infrared rays or other radio waves, or may be an external connection device 929 such as a mobile phone that supports the operation of the information processing device 900. The input device 915 includes an input control circuit that generates an input signal based on information input by the user and outputs the input signal to the CPU 901. The user operates the input device 915 to input various data and instruct processing operations to the information processing device 900.

The output device 917 is configured by a device capable of notifying the acquired information to the user using a sense such as vision, hearing, or touch. The output device 917 can be, for example, a display device such as an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) display, an audio output device such as a speaker or headphones, or a vibrator. The output device 917 outputs the result obtained by the processing of the information processing device 900 as video such as text or image, sound such as sound or sound, or vibration.

The storage device 919 is a data storage device configured as an example of a storage unit of the information processing device 900. The storage device 919 includes, for example, a magnetic storage device such as an HDD (Hard Disk Drive), a semiconductor storage device, an optical storage device, or a magneto-optical storage device. The storage device 919 stores, for example, programs executed by the CPU 901 and various data, and various data acquired from the outside.

The drive 921 is a reader / writer for a removable recording medium 927 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, and is built in or externally attached to the information processing apparatus 900. The drive 921 reads information recorded on the attached removable recording medium 927 and outputs the information to the RAM 905. In addition, the drive 921 writes a record in the attached removable recording medium 927.

The connection port 923 is a port for connecting a device to the information processing apparatus 900. The connection port 923 can be, for example, a USB (Universal Serial Bus) port, an IEEE 1394 port, a SCSI (Small Computer System Interface) port, or the like. The connection port 923 may be an RS-232C port, an optical audio terminal, an HDMI (registered trademark) (High-Definition Multimedia Interface) port, or the like. By connecting the external connection device 929 to the connection port 923, various types of data can be exchanged between the information processing apparatus 900 and the external connection device 929.

The communication device 925 is a communication interface configured with, for example, a communication device for connecting to the communication network 931. The communication device 925 can be, for example, a communication card for LAN (Local Area Network), Bluetooth (registered trademark), Wi-Fi, or WUSB (Wireless USB). The communication device 925 may be a router for optical communication, a router for ADSL (Asymmetric Digital Subscriber Line), or a modem for various communication. The communication device 925 transmits and receives signals and the like using a predetermined protocol such as TCP / IP with the Internet and other communication devices, for example. The communication network 931 connected to the communication device 925 is a network connected by wire or wireless, and may include, for example, the Internet, a home LAN, infrared communication, radio wave communication, satellite communication, or the like.

The imaging device 933 uses various members such as an imaging element such as a CMOS (Complementary Metal Oxide Semiconductor) or a CCD (Charge Coupled Device), and a lens for controlling the formation of a subject image on the imaging element. It is an apparatus that images a real space and generates a captured image. The imaging device 933 may capture a still image or may capture a moving image.

The sensor 935 is various sensors such as an acceleration sensor, an angular velocity sensor, a geomagnetic sensor, an illuminance sensor, a temperature sensor, an atmospheric pressure sensor, or a sound sensor (microphone). The sensor 935 acquires information about the state of the information processing apparatus 900 itself, such as the posture of the information processing apparatus 900, and information about the surrounding environment of the information processing apparatus 900, such as brightness and noise around the information processing apparatus 900, for example. To do. The sensor 935 may include a GPS receiver that receives a GPS (Global Positioning System) signal and measures the latitude, longitude, and altitude of the device.

Heretofore, an example of the hardware configuration of the information processing apparatus 900 has been shown. Each component described above may be configured using a general-purpose member, or may be configured by hardware specialized for the function of each component. Such a configuration can be appropriately changed according to the technical level at the time of implementation.

(8. Supplement)
In the embodiment of the present disclosure, for example, an information processing apparatus, a system, an information processing method executed by the information processing apparatus or system, a program for causing the information processing apparatus to function, and a program are recorded. It may include tangible media that is not temporary.

The preferred embodiments of the present disclosure have been described in detail above with reference to the accompanying drawings, but the technical scope of the present disclosure is not limited to such examples. It is obvious that a person having ordinary knowledge in the technical field of the present disclosure can come up with various changes or modifications within the scope of the technical idea described in the claims. Of course, it is understood that it belongs to the technical scope of the present disclosure.

In addition, the effects described in this specification are merely illustrative or illustrative, and are not limited. That is, the technology according to the present disclosure can exhibit other effects that are apparent to those skilled in the art from the description of the present specification in addition to or instead of the above effects.

The following configurations also belong to the technical scope of the present disclosure.
(1) An evaluation information acquisition unit that acquires action evaluation information obtained by evaluating the action based on an action image including an action of an actor;
A situation information acquisition unit that acquires user situation information indicating a situation of a user who views the action image;
An information processing apparatus comprising: an additional information generation unit that generates additional information that is presented to the user together with the action image and expresses at least part of the action evaluation information according to the user situation information.
(2) The information processing apparatus according to (1), wherein the additional information includes visual information.
(3) The information processing apparatus according to (2), wherein the additional information includes text or graphics that are superimposed on the action image or displayed in the vicinity of the action image.
(4) The action evaluation information includes information indicating coordinates of a body part of the actor in the action image,
The information processing apparatus according to (3), wherein the additional information includes a graphic representing a locus of the portion.
(5) The action evaluation information includes information for determining or scoring the action,
The information processing apparatus according to (3) or (4), wherein the additional information includes text or graphics indicating a progress or result of the determination or scoring.
(6) The user status information includes information for estimating the length of time the user has for referring to the additional information,
6. The information processing apparatus according to any one of (1) to (5), wherein the additional information generation unit generates the additional information according to the length of time.
(7) The information processing apparatus according to (6), wherein the additional information generation unit increases the information amount of the additional information as the holding time increases.
(8) The information processing apparatus according to (6) or (7), wherein the additional information generation unit generates the additional information from which the user can select contents when the holding time is long.
(9) The user status information includes information indicating a relationship between a first time when the actor performs the action and a second time when the user views the action image,
The additional information generation unit according to any one of (6) to (8), wherein the time is determined to be short when a difference between the first time and the second time is less than a threshold value. Information processing device.
(10) The user situation information includes information for estimating a depth of knowledge about the action of the user,
The information processing apparatus according to any one of (1) to (9), wherein the additional information generation unit generates the additional information according to a depth of the sculpting.
(11) The user status information includes information indicating the number of users,
The information processing apparatus according to any one of (1) to (10), wherein the additional information generation unit generates the additional information according to the number of users.
(12) The information processing apparatus according to any one of (1) to (11), wherein the action evaluation information includes information obtained by evaluating the posture of the actor estimated based on the action image.
(13) In any one of (1) to (12), the action evaluation information includes information obtained by evaluating a contact position between the actor and another object estimated based on the action image. The information processing apparatus described.
(14) The information processing apparatus according to any one of (1) to (13), further including an evaluation information generation unit that generates the action evaluation information based on the action image.
(15) obtaining action evaluation information obtained by evaluating the action based on an action image including an actor's action;
Obtaining user situation information indicating a situation of a user viewing the action image;
Generating additional information that is presented to the user together with the action image and expresses at least a part of the action evaluation information according to the user situation information.
(16) a function of acquiring action evaluation information obtained by evaluating the action based on an action image including an action of an actor;
A function of acquiring user situation information indicating a situation of a user who views the action image;
A program for causing a computer to realize a function of generating, according to the user situation information, additional information that is presented to the user together with the action image and expresses at least a part of the action evaluation information.

DESCRIPTION OF SYMBOLS 10 System 100 Camera 200 Server 300 Output device 300a Television 300b Recorder 201 Image acquisition part 203 Evaluation information generation part 205 Evaluation information acquisition part 207 Status information acquisition part 209 Additional information generation part 211 Screen generation part

Claims

An evaluation information acquisition unit for acquiring action evaluation information for evaluating the action based on an action image including an action of an actor;
A situation information acquisition unit that acquires user situation information indicating a situation of a user who views the action image;
An information processing apparatus comprising: an additional information generation unit that generates additional information that is presented to the user together with the action image and expresses at least part of the action evaluation information according to the user situation information.
The information processing apparatus according to claim 1, wherein the additional information includes visual information.
The information processing apparatus according to claim 2, wherein the additional information includes a text or a graphic superimposed on the action image or displayed in the vicinity of the action image.
The action evaluation information includes information indicating the coordinates of the body part of the actor in the action image,
The information processing apparatus according to claim 3, wherein the additional information includes a graphic representing a locus of the part.
The action evaluation information includes information for determining or scoring the action,
The information processing apparatus according to claim 3, wherein the additional information includes text or a graphic indicating a progress or result of the determination or scoring.
The user status information includes information for estimating the length of time the user has for referring to the additional information,
The information processing apparatus according to claim 1, wherein the additional information generation unit generates the additional information in accordance with the length of the possession time.
The information processing apparatus according to claim 6, wherein the additional information generation unit increases the information amount of the additional information as the holding time increases.
The information processing apparatus according to claim 6, wherein the additional information generation unit generates the additional information from which the user can select contents when the holding time is long.
The user status information includes information indicating a relationship between a first time when the actor performs the action and a second time when the user views the action image,
The information processing apparatus according to claim 6, wherein the additional information generation unit determines that the time is short when a difference between the first time and the second time is less than a threshold value.
The user situation information includes information for estimating a depth of knowledge about the action of the user,
The information processing apparatus according to claim 1, wherein the additional information generation unit generates the additional information according to a depth of the sculpting.
The user status information includes information indicating the number of users,
The information processing apparatus according to claim 1, wherein the additional information generation unit generates the additional information according to the number of users.
The information processing apparatus according to claim 1, wherein the action evaluation information includes information obtained by evaluating the posture of the actor estimated based on the action image.
2. The information processing apparatus according to claim 1, wherein the action evaluation information includes information obtained by evaluating a contact position between the actor and another object estimated based on the action image.
The information processing apparatus according to claim 1, further comprising an evaluation information generation unit that generates the action evaluation information based on the action image.
Obtaining action evaluation information evaluating the action based on an action image including an actor's action;
Obtaining user situation information indicating a situation of a user viewing the action image;
Generating additional information that is presented to the user together with the action image and expresses at least a part of the action evaluation information according to the user situation information.
A function of acquiring action evaluation information obtained by evaluating the action based on an action image including an actor's action;
A function of acquiring user situation information indicating a situation of a user who views the action image;
A program for causing a computer to realize a function of generating, according to the user situation information, additional information that is presented to the user together with the action image and expresses at least a part of the action evaluation information.