WO2019235350A1

WO2019235350A1 - Information processing system, information processing method, and storage medium

Info

Publication number: WO2019235350A1
Application number: PCT/JP2019/021505
Authority: WO
Inventors: 典孝志村
Original assignee: 日本電気株式会社
Priority date: 2018-06-06
Filing date: 2019-05-30
Publication date: 2019-12-12
Also published as: JPWO2019235350A1; JP6939999B2

Abstract

Provided is an information processing system comprising: an identification information acquisition unit for acquiring, from an image, identification information for a person specified by means of image recognition; an attention information acquisition unit for acquiring, from a facial image of the person, attention information indicating an attention state of the person; and a behavior prediction unit for predicting a behavior of the person on the basis of the identification information and the attention information.

Description

Information processing system, information processing method, and storage medium

The present invention relates to an information processing system, an information processing method, and a storage medium.

Patent Document 1 discloses a navigation device that detects a driver's line of sight and detects a gaze target based on the movement of the line of sight. The navigation device has a function of predicting driving behavior based on a gaze behavior pattern.

JP 2008-232912 A

In the navigation apparatus exemplified in Patent Document 1, there is a case in which it is not possible to sufficiently consider that the behavior tendency differs depending on the target person in the behavior prediction based on the gaze target.

The present invention has been made in view of the above-described problems, and an object of the present invention is to provide an information processing system, an information processing method, and a storage medium that can perform behavior prediction in consideration of different behavior trends for each person. And

According to an aspect of the present invention, an identification information acquisition unit that acquires identification information of a person specified by image recognition from an image, and attention information indicating the attention state of the person is acquired from the face image of the person. There is provided an information processing system comprising an attention information acquisition unit that performs an action prediction unit that predicts the behavior of the person based on the identification information and the attention information.

According to another aspect of the present invention, the step of acquiring identification information of a person specified by image recognition from an image, and acquiring attention information indicating the attention state of the person from the face image of the person. There is provided an information processing method comprising: a step; and predicting the behavior of the person based on the identification information and the attention information.

According to another aspect of the present invention, a step of acquiring identification information of a person specified by image recognition from an image in a computer, and attention information indicating the attention state of the person from the face image of the person A storage medium storing a program for executing an information processing method comprising: a step of acquiring an information processing method; and a step of predicting the behavior of the person based on the identification information and the attention information Is provided.

According to the present invention, it is possible to provide an information processing system, an information processing method, and a storage medium that can perform behavior prediction considering that the tendency of behavior differs for each person.

It is a block diagram which shows the hardware structural example of the information processing system which concerns on 1st Embodiment. It is a functional block diagram of the information processing system concerning a 1st embodiment. It is a flowchart which shows the outline of the process performed by the information processing system which concerns on 1st Embodiment. It is a figure which shows typically the example of a person's face detection from the video of a game. It is a table | surface which shows the structural example of a player database. It is a figure which shows typically the example of a gaze detection from a player's face image. It is a figure which shows the example of a display of an estimated image. It is a functional block diagram of the information processing system concerning a 2nd embodiment.

Hereinafter, exemplary embodiments of the present invention will be described with reference to the drawings. In the drawings, similar or corresponding elements are denoted by the same reference numerals, and the description thereof may be omitted or simplified.

[First Embodiment]
The information processing system according to the present embodiment will be described with reference to FIGS. The information processing system of this embodiment is a system used for predicting the behavior of a person who appears in an image. This image is, for example, a video of a sports game, and the person is a sports player. By displaying the player's behavior prediction, it is possible to visualize the player's psychology, strategy, etc., and improve the satisfaction of the viewer watching the sport.

In addition, in this specification, an image is a concept including a still image, a moving image, and a frame image included in the moving image. If the image is a moving image, it may further include audio data. In addition, in this specification, sports include physical sports involving physical exercise, but is not limited thereto, and may include competitions that have few physical movement elements and sports that do not involve physical movements. That is, sports may include motor sports such as car racing, motorcycle racing, and motor boat racing, e-sports that play computer games (Electronic Sports), go sports, shogi, chess, mahjong, poker, backgammon, and other mind sports. .

In the embodiment described below, as an example, an example in which a player's line of sight appearing in a video of a volleyball game is detected to predict the player's behavior and the prediction result is displayed will be described, but the present invention is not limited thereto. is not.

FIG. 1 is a block diagram illustrating a hardware configuration example of the information processing system 100. The information processing system 100 may be a computer such as a desktop PC (Personal Computer), a notebook PC, or a tablet PC.

The information processing system 100 includes a CPU (Central Processing Unit) 151, a RAM (Random Access Memory) 152, a ROM (Read Only Memory) 153, and an HDD (Hard Disk Drive) 154 as computers that perform calculation, control, and storage. The information processing system 100 includes a communication I / F (interface) 155, a display device 156, and an input device 157. The CPU 151, RAM 152, ROM 153, HDD 154, communication I / F 155, display device 156, and input device 157 are connected to each other via a bus 158. Note that the display device 156 and the input device 157 may be connected to the bus 158 via a driving device (not shown) for driving these devices.

In FIG. 1, each unit configuring the information processing system 100 is illustrated as an integrated device, but some of these functions may be provided by an external device. For example, the display device 156 and the input device 157 may be external devices that are different from the parts that constitute the functions of the computer including the CPU 151 and the like.

The CPU 151 is a processor that performs a predetermined operation according to a program stored in the ROM 153, the HDD 154, and the like and also has a function of controlling each part of the information processing system 100. The RAM 152 is composed of a volatile storage medium and provides a temporary memory area necessary for the operation of the CPU 151. The ROM 153 is composed of a nonvolatile storage medium and stores necessary information such as a program used for the operation of the information processing system 100. The HDD 154 is a storage device that includes a nonvolatile storage medium and stores data necessary for processing, an operation program for the information processing system 100, and the like.

The communication I / F 155 is a communication interface based on standards such as Ethernet (registered trademark), Wi-Fi (registered trademark), and 4G, and is a module for performing communication with other devices. The display device 156 is a liquid crystal display, an OLED display, or the like, and is used for displaying images, characters, interfaces, and the like. The input device 157 is a keyboard, a pointing device, or the like, and is used for a user to operate the information processing system 100. Examples of pointing devices include a mouse, a trackball, a touch panel, and a pen tablet. The display device 156 and the input device 157 may be integrally formed as a touch panel.

Note that the hardware configuration shown in FIG. 1 is an exemplification, and other devices may be added, or some devices may not be provided. Some devices may be replaced with another device having the same function. Furthermore, a part of the functions of the present embodiment may be provided by another device via a network, and the functions of the present embodiment may be realized by being distributed to a plurality of devices. For example, the HDD 154 may be replaced with an SSD (Solid State Drive) using a semiconductor memory, or may be replaced with a cloud storage.

FIG. 2 is a functional block diagram of the information processing system 100 according to the present embodiment. The information processing system 100 includes an image acquisition unit 101, a face detection unit 102, a feature amount extraction unit 103, a collation unit 104, a state determination unit 105, an identification information acquisition unit 106, an attention information acquisition unit 107, a behavior prediction unit 108, and an image generation Part 109 and storage part 110.

The CPU 151 loads a program stored in the ROM 153, HDD 154, etc. into the RAM 152 and executes it. Thereby, the CPU 151 causes the image acquisition unit 101, the face detection unit 102, the feature amount extraction unit 103, the collation unit 104, the state determination unit 105, the identification information acquisition unit 106, the attention information acquisition unit 107, the behavior prediction unit 108, and the image generation. The function of the unit 109 is realized. Processing performed in each of these units will be described later. The CPU 151 realizes the function of the storage unit 110 by controlling the HDD 154.

FIG. 3 is a flowchart showing an outline of processing performed by the information processing system 100 according to the present embodiment. The behavior prediction process performed by the information processing system 100 will be described with reference to FIG. In the following description, the information processing system 100 detects a player (server) trying to hit a serve from a video of a volleyball game, predicts a direction in which the ball will fly, and displays a prediction result. Shall be performed. This processing is desirably performed in real time during live broadcasting from the viewpoint of improving the satisfaction of the viewer by providing a behavior prediction of the player. This is because live broadcasts have stronger needs for providing behavior predictions than recorded broadcasts. In this specification, live broadcasting refers not only to broadcasting moving images taken at substantially the same time as shooting, but also to broadcasting with a delay of several seconds to several minutes from shooting using a delayed sending system. Including.

In step S101, the image acquisition unit 101 acquires video data of a volleyball game (frame image data constituting the video). The acquisition of the moving image data may be, for example, received from a device external to the information processing system 100 such as a video camera, or may be read out the moving image data temporarily stored in the storage unit 110. .

In step S102, the face detection unit 102 detects an area including a human face from the frame image data included in the moving image data. FIG. 4 is a diagram schematically illustrating an example of detecting a human face from video data of a volleyball game. In the frame image data 301, a player 302 holding a ball to hit a serve is displayed. The face detection unit 102 automatically searches for a human face from the frame image data 301 and detects the face. Thereby, as shown in FIG. 4, a rectangular face region R1 including the face of the player 302 is extracted.

In step S103, the feature amount extraction unit 103 extracts a feature amount from the image in the face area R1 extracted by the face detection unit 102. The feature amount may be an amount indicating a facial feature such as a position of a characteristic part such as a pupil, a nose, or a mouth end.

In step S104, the collation unit 104 collates the feature amount extracted in the process of step S103 with the feature amount to be collated included in the player database (player DB), and determines whether there is a matching combination. Perform face matching. If there is a feature quantity that matches the feature quantity extracted by the feature quantity extraction unit 103 in the player database (YES in step S104), it is determined that the detected person is a player participating in the game, and the process is as follows. The process proceeds to step S105. If there is no feature quantity that matches the feature quantity extracted by the feature quantity extraction unit 103 in the database (NO in step S104), the detected person is a person different from the player participating in the game such as a manager. It is determined that there is, and the process ends.

Here, the configuration of the player database will be described with reference to FIG. FIG. 5 is a table showing a configuration example of the player database. As shown in FIG. 5, the player database items include names of players who can participate in the game, player IDs (Identifiers), player face image features, player spin numbers, player teams and players. Positions are included. In the player database, these items are associated with each other. FIG. 5 shows a feature amount ID for identifying the feature amount, but actually, a feature amount corresponding to the feature amount ID is also included in the player database.

In addition to the one shown in FIG. 5, the player database may include the face image itself used for extracting the feature amount. In addition, since the processing of this embodiment is possible if the feature amount and information for identifying the player are associated with each other, the player name, spine number, team, position, etc., among the items shown in FIG. May be omitted.

The player database may be stored in the storage unit 110 in the information processing system 100, or may be stored in a device external to the information processing system 100 such as a data server. The player database may be owned by the broadcasting station or the like to which the user of the information processing system 100 belongs. However, the player database may be provided in a cloud environment so that a plurality of broadcasting stations and the like can be used jointly.

In step S105, the state determination unit 105 determines whether or not the player extracted by the face detection unit 102 is a server. If the player is a server (YES in step S105), the process proceeds to step S106. If the player is not a server (NO in step S105), the process ends.

Here, the determination algorithm in the state determination unit 105 may be based on, for example, the position of the player on the court, the posture of the player, whether or not the player has the ball. The algorithm of the state determination unit 105 uses a learning model generated by machine learning using as input data learning data in which a player image is associated with information indicating whether or not the player is a server. It may be a thing.

Also, the determination algorithm in the state determination unit 105 may refer to the player database and determine that the player is not a server when the player's position is Libero. This is because Libero cannot serve according to the rules, so if the extracted player is Libero, that player is not a server.

The determination algorithm in the state determination unit 105 may be determined with reference to the court position information. The state determination unit 105 can specify the server by calculating the change (rotation) of the court position accompanying the movement of the serve right.

It is not essential that the timing at which the process of step S105 is performed is after step S104. For example, it may be after step S102, after step S103, or the like.

In step S106, the identification information acquisition unit 106 acquires the identification information of the person specified as the server. This identification information may be a player ID in the player database. However, the identification information may be information other than the player ID as long as the information can be used for identifying the player, such as a feature amount ID, a player name, a spine number, and a team name.

In step S107, the attention information acquisition unit 107 detects the direction of the player's line of sight from the face image in the player's face area R1. FIG. 6 is a diagram schematically illustrating a gaze detection example from the player's face image. As illustrated in FIG. 6, the attention information acquisition unit 107 detects the orientation of the player's line of sight 303 based on the face image. This line-of-sight detection algorithm can acquire the direction of the line of sight by, for example, acquiring the relative positions of the iris, pupil, and the like from the image with the eyes and the corners of the eyes as reference points. Alternatively, the algorithm may acquire the direction of the line of sight by acquiring the position of the pupil from the image based on the corneal reflection of the light emitted from the light source. Since the direction of the player's line of sight 303 indicates the direction in which the player is paying attention, it can be said that this is typical information indicating the attention state of the player. In this way, the attention information acquisition unit 107 acquires attention information indicating the attention state of the player.

Note that the attention information acquisition unit 107 may be one that detects the orientation of the player's face, or may be a combination of detection of the direction of the line of sight and detection of the orientation of the face. In other words, the attention information may include at least one of a gaze direction and a face direction. Since the face has a larger area than the eyes, there is an advantage that the orientation can be acquired with relatively high accuracy even when the resolution of the image is low. Other examples that can be included in the attention information include the player's facial expression, the player's mouth shape, and the like.

In step S108, the behavior prediction unit 108 predicts the player's behavior based on the identification information acquired by the identification information acquisition unit 106 and the attention information acquired by the attention information acquisition unit 107. In addition, the prediction result obtained by this process may be called action prediction information.

An example of an algorithm for behavior prediction performed by the behavior prediction unit 108 will be described. In general, there is a correlation between attention information such as line of sight and the action of the player. For example, in the serve example, the player's line of sight often points in the direction he is aiming to hit the serve. However, some players intentionally or unconsciously gaze at a direction different from the direction in which they are trying to hit in order to avoid being read by the opponent. In this way, there is an individual difference in the correlation between the attention information such as the player's line of sight and the actual behavior of the player. Therefore, the prediction accuracy is improved by performing behavior prediction using a different model for each player. Can be made.

Therefore, since the behavior prediction unit 108 of the present embodiment performs prediction using a different model for each player, the player is identified using the identification information, and the behavior of the player is determined from the attention information using the identified player model. Predict. Thereby, prediction in consideration of individual differences in correlation between attention information and behavior is possible, and prediction accuracy is improved.

The individual difference in the correlation between this attention information and action can be obtained by analyzing each player's past game information. Specifically, a learning model for predicting behavior different for each player can be obtained by machine learning using as input learning data in which the relationship between attention information and behavior is stored in a database for each player. By incorporating this learning model into the behavior prediction unit 108, behavior prediction that takes into consideration that the tendency of behavior differs for each player is realized. By using machine learning to generate a learning model, labor is reduced compared to creating a model manually, so a highly accurate learning model using a large amount of data can be easily constructed. .

Note that the model that can be used in the behavior prediction unit 108 is not limited to that obtained by machine learning, and other models may be used. For example, the data analyst may classify the direction of the line of sight included in the attention information and manually construct the behavior prediction model by deriving the most likely behavior for each classification from past data. By creating a model manually, it is possible to construct a learning model with high interpretability. In this case, the process performed by the behavior prediction unit 108 may refer to a database such as a table in which attention information and a player's behavior are associated with each other.

In step S109, the image generation unit 109 generates a predicted image including the behavior prediction information predicted by the behavior prediction unit 108. This predicted image is provided to viewers, for example, by being incorporated in a broadcast image of a volleyball game. FIG. 7 is a diagram illustrating a display example of a predicted image. In the example shown in FIG. 7, the prediction image 305 is generated by superimposing the behavior prediction information 304 on the frame image of the volleyball game. A broken line in the behavior prediction information 304 indicates a trajectory of the ball hit by the server. The viewer can visually understand that the server is aiming at the lower left corner of the court by looking at the predicted image 305.

Note that the display method of the behavior prediction information 304 does not have to be based on a figure such as an arrow shown in FIG. 7, and may be a display based on a change in color, a display based on text, or the like. In addition, it is not essential to superimpose the behavior prediction information 304 on the game image. For example, the prediction image 305 may be a schematic diagram in which a serve course is displayed on a picture of a volleyball court.

The result of behavior prediction may be used for purposes other than the purpose of providing it to viewers by incorporating it into a broadcast image, for example, may be provided to a person involved in a sports team such as a director for strategic planning. . In this case, the behavior prediction result can be transmitted from the terminal of the information processing system 100 to the tablet terminal owned by the director, for example. The director can formulate a strategy using the behavior prediction result and send an instruction to the player.

As described above, in the present embodiment, a person is identified by image recognition, and behavior prediction is performed based on identification information of the identified person and attention information of the person. Thereby, the information processing system 100 which can perform the behavior prediction considering that the tendency of the behavior is different for each person is provided.

Note that the behavior prediction performed by the behavior prediction unit 108 in the above-described embodiment is performed based on uncertain information such as the player's line of sight, and may not necessarily predict the player's behavior reliably. Absent. For example, when a player consciously performs an action having a different tendency from the conventional one, the action prediction may be missed. However, viewers are required to ensure that the behavior prediction matches the actual behavior in order to view the behavior prediction results for the purpose of enjoying the game while predicting whether the behavior prediction matches the actual behavior. I don't mean. Therefore, in the information processing system 100 of this embodiment, it is not essential that the accuracy of behavior prediction is high.

The system described in the above embodiment can also be configured as in the following second embodiment.

[Second Embodiment]
FIG. 8 is a functional block diagram of the information processing system 200 according to the second embodiment. The information processing system 200 includes an identification information acquisition unit 201, an attention information acquisition unit 202, and an action prediction unit 203. The identification information acquisition unit 201 acquires identification information of a person specified by image recognition from an image. The attention information acquisition unit 202 acquires attention information indicating the attention state of the person from the face image of the person. The behavior prediction unit 203 predicts the behavior of the person based on the identification information and the attention information.

According to the present embodiment, an information processing system 200 capable of performing behavior prediction considering that the tendency of behavior differs for each person is provided.

[Modified Embodiment]
The present invention is not limited to the above-described embodiment, and can be appropriately changed without departing from the spirit of the present invention.

In the above-described embodiment, an example is shown in which the direction of the serve is predicted from the line of sight of the server in a volleyball game, but the behavior prediction of this embodiment is applicable to other scenes. For example, the direction in which the setter raises the toss, the type of toss, the position where the attack is hit, and the like may be predicted based on the direction of the line of sight of the setter or the attacker.

Also, the behavior prediction of this embodiment can be applied to behavior prediction of sports other than volleyball. For example, the present embodiment can be applied to a soccer free kick or penalty kick scene. In this case, the direction in which the ball is kicked, the direction in which the keeper flies, and the like can be predicted from the direction of the line of sight of the kicker or the line of sight of the goalkeeper.

Also, this embodiment can be applied to baseball. In this case, the ball type, course, speed, and the like of the ball thrown by the pitcher can be predicted from the direction of the line of sight of the pitcher or the direction of the line of sight of the catcher. Similarly, the direction in which the batter is about to strike can be predicted from the direction of the line of sight of the batter.

Also, this embodiment can be applied to scenes other than sports. For example, the behavior prediction of this embodiment may be applied to a person shown in an image taken with a security camera.

In the above-described embodiment, the behavior prediction unit 108 uses identification information for identifying a person and attention information as input information for behavior prediction, but performs behavior prediction using further information other than this. May be. Examples of such information are listed as examples.

For example, a history of past actions in the game may be used for action prediction. In the example of the volleyball serve, the server may determine the direction in which the server is struck in consideration of the past serve direction and its success or failure. If there is a success in the left corner (if you have scored), the server will tend to hit the next serve to the left corner. Also, if the server continues to serve in the left corner but fails in the previous serve (if it conceded), the server tends to change the direction of the next strike and hit the right corner. Alternatively, the direction of the server's line of sight is not consistent with the direction of the serve, and if this is successful, the server will also serve in the direction that does not match the line of sight in the next serve as well. Tend. On the other hand, if the direction of the server's line of sight does not match the direction of the serve, but if the previous serve fails, the server will change the direction of the next serve and the direction of the line of sight and the direction of the serve Tend to match. Thus, since the past behavior history in the game may affect the next behavior, the past behavior history can be used for behavior prediction, so that the accuracy of behavior prediction can be further improved. In addition, the above-mentioned tendency is an example, and there exists a player with another tendency. Therefore, in the behavior prediction, the way of considering the past history may be changed depending on the player.

Also, a combination of players involved in cooperative play may be taken into account in action prediction. When there is cooperation between players, such as toss and attack of volleyball, there are cases where there is a tendency for behavior depending on the combination of players. For example, there is a tendency that the right corner is often aimed at a combination of a setter and an attacker, and the left corner is often aimed at another certain combination. Also, in the case of a combination with a setter and an attacker, the direction of the line of sight often coincides with the direction of the attack, and in the case of another combination, the direction of the line of sight and the direction of the attack are often different. Such a tendency may exist. Thus, when there exists a tendency depending on the combination of players, it is desirable to perform behavior prediction in consideration of these tendencies. Therefore, by using a combination of players involved in cooperative play for behavior prediction, the accuracy of behavior prediction can be further improved.

Also, the result of facial expression analysis based on the movement of the mouth corner or the like may be used for behavior prediction. When the expression is normal, there is often room for a strategic action such as hitting a serve with a different direction from the line of sight. However, when the facial expression is not normal, such as when fatigue is accumulated, there is often no room for such strategic action. For this reason, when the facial expression is normal, it is often aimed at a different direction from the line of sight, and when a difficult expression is seen from the corner of the mouth, etc., it is often aimed at the same direction as the line of sight. Behavior may be correlated. Therefore, by using the result of facial expression analysis for behavior prediction, the accuracy of behavior prediction can be further improved.

Moreover, you may use the information regarding the situation of a game for action prediction. In the example of volleyball, the situation of the game includes the situation of a player different from the player whose behavior is to be predicted, such as the scores of both teams, the positions of the setters or ace attackers of both teams, and the positional relationship between the players. For example, in order to make it difficult for an ace attacker to attack, a strategy of hitting a serve with an ace attacker is known. Therefore, the prediction accuracy of the direction of a serve can be improved by performing action prediction in consideration of the position of the ace attacker. Further, when the score is disadvantageous to the own team, there is a case where there is a correlation between the score and the behavior such as the direction of the line of sight and the behavior are easily matched because the mental margin is lost. That is, if the score of your team is winning, you often aim for a different direction from the line of sight, and if you lose your score, you often aim for the same direction as your line of sight. May be correlated. Therefore, prediction accuracy can be improved by performing behavior prediction in consideration of the game situation such as score.

Moreover, you may use the past game information of the player of the other party's team specified in the action prediction. If there is a player who has a good feint in the opponent team, it becomes easier to read the faint of the own team, so there may be a strategy such as matching the direction of the line of sight with the direction of hitting the ball without intentionally performing the feint. In other words, when there is a player who is good at the opponent in the opponent team, it is often aimed at the same direction as the line of sight, and when there is no player who is good at the faint, it is often aimed at a direction different from the line of sight. A strategy may be determined in consideration of past game information of team players. Therefore, the accuracy of the behavior prediction can be further improved by using the past game information of the player of the opponent team for the behavior prediction.

As described above, the behavior prediction unit 108 performs behavior prediction by further using identification information for identifying a person as input information for behavior prediction, attention information, and other information, thereby improving the accuracy of behavior prediction. Can be improved. Note that some of the above examples may be combined and used for behavior prediction.

In the above-described embodiment, an example is shown in which a person is specified by face matching, but other techniques may be adopted as long as the person can be identified by image recognition. For example, a person may be specified by acquiring a character number, a name, and the like written on a player's uniform by character recognition.

The installation position of an imaging device (camera) that acquires an image for performing face detection, line-of-sight detection, and the like is not particularly limited, but is preferably close to the player in order to improve the accuracy of face detection, line-of-sight detection, and the like. For example, in the case of volleyball, it may be desirable to use an image taken with a camera installed on the net rather than a camera outside the court.

A processing method for recording a program for operating the configuration of the embodiment to realize the functions of the above-described embodiment on a storage medium, reading the program recorded on the storage medium as a code, and executing the program on a computer is also included in each embodiment. Included in the category. That is, a computer-readable storage medium is also included in the scope of each embodiment. In addition to the storage medium on which the above-described program is recorded, the program itself is included in each embodiment. In addition, one or more components included in the above-described embodiments are circuits such as an ASIC (Application Specific Integrated Circuit) and an FPGA (Field Programmable Gate Array) configured to realize the function of each component. There may be.

As the storage medium, for example, a floppy (registered trademark) disk, hard disk, optical disk, magneto-optical disk, CD (Compact Disk) -ROM, magnetic tape, nonvolatile memory card, ROM can be used. In addition to a program executed in a single program recorded in the storage medium, the program runs on an OS (Operating System) in cooperation with other software and expansion board functions. Are also included in the category of each embodiment.

The service realized by the functions of the above-described embodiments can be provided to the user in the form of SaaS (Software as a Service).

It should be noted that the above-described embodiments are merely examples of implementation in carrying out the present invention, and the technical scope of the present invention should not be interpreted in a limited manner. That is, the present invention can be implemented in various forms without departing from the technical idea or the main features thereof.

Some or all of the above-described embodiments can be described as in the following supplementary notes, but are not limited thereto.

(Appendix 1)
An identification information acquisition unit for acquiring identification information of a person identified by image recognition from the image;
An attention information acquisition unit that acquires attention information indicating the attention state of the person from the face image of the person;
An action prediction unit that predicts the action of the person based on the identification information and the attention information;
An information processing system comprising:

(Appendix 2)
The information processing system according to appendix 1, wherein the attention information includes information extracted from features of the person's face or eyes.

(Appendix 3)
The information processing system according to

appendix

1 or 2, wherein the attention information includes at least one of a gaze direction of the person and a face direction of the person.

(Appendix 4)
The information processing system according to any one of appendices 1 to 3, wherein the image recognition includes at least one of face matching and character recognition.

(Appendix 5)
The behavior prediction unit predicts the behavior of the person using a learning model generated by machine learning that receives learning data including the attention information and the behavior of the person. The information processing system according to any one of the above.

(Appendix 6)
The information processing according to any one of appendices 1 to 4, wherein the behavior prediction unit predicts the behavior of the person using a database in which the attention information and the behavior of the person are associated with each other. system.

(Appendix 7)
The information processing system according to any one of appendices 1 to 6, further comprising an image generation unit that generates a predicted image including behavior prediction information including the behavior of the person predicted by the behavior prediction unit.

(Appendix 8)
The information processing system according to appendix 7, wherein the image generation unit generates the predicted image by superimposing the behavior prediction information on the image.

(Appendix 9)
The image is a video of a sports game,
The information processing system according to any one of appendices 1 to 8, wherein the person is a player of the sport.

(Appendix 10)
The information processing system according to appendix 9, wherein the behavior prediction unit further predicts the behavior of the person based on the situation of the game.

(Appendix 11)
The information processing system according to appendix 9 or 10, wherein the behavior prediction unit further predicts the behavior of the person based on a situation of a player other than the person.

(Appendix 12)
The information processing system according to any one of appendices 9 to 11, wherein the behavior prediction unit further predicts the behavior of the person based on a history of the behavior of the person in the game.

(Appendix 13)
The information according to any one of appendices 9 to 12, wherein the behavior prediction unit further predicts the behavior of the person based on a combination of the person and a player who cooperates with the person. Processing system.

(Appendix 14)
The information processing system according to any one of appendices 9 to 13, wherein the behavior prediction unit further predicts the behavior of the person based on the facial expression of the person.

(Appendix 15)
The information processing system according to any one of appendices 9 to 14, wherein the behavior prediction unit further predicts the behavior of the person based on the score of the game.

(Appendix 16)
16. The information according to any one of appendices 9 to 15, wherein the behavior predicting unit further predicts the behavior of the person based on past game information of players of the opponent team of the person. Processing system.

(Appendix 17)
Obtaining identification information of a person identified by image recognition from an image;
Obtaining attention information indicating the attention state of the person from the face image of the person;
Predicting the behavior of the person based on the identification information and the attention information;
An information processing method comprising:

(Appendix 18)
On the computer,
Obtaining identification information of a person identified by image recognition from an image;
Obtaining attention information indicating the attention state of the person from the face image of the person;
Predicting the behavior of the person based on the identification information and the attention information;
A storage medium storing a program for executing an information processing method.

This application claims priority based on Japanese Patent Application No. 2018-108332 filed on June 6, 2018, the entire disclosure of which is incorporated herein.

100, 200 Information processing system 101 Image acquisition unit 102 Face detection unit 103 Feature amount extraction unit 104 Collation unit 105

State determination unit

106, 201 Identification

information acquisition unit

107, 202 Attention

information acquisition unit

108, 203 Action prediction unit 109 Image generation unit 110 Storage unit 151 CPU
152 RAM
153 ROM
154 HDD
155 Communication I / F
156 Display device 157 Input device 158 Bus 301 Frame image data 302 Player 303 Line of sight 304 Behavior prediction information 305 Predicted image R1 Facial region

Claims

An identification information acquisition unit for acquiring identification information of a person identified by image recognition from the image;
An attention information acquisition unit that acquires attention information indicating the attention state of the person from the face image of the person;
An action prediction unit that predicts the action of the person based on the identification information and the attention information;
An information processing system comprising:
The information processing system according to claim 1, wherein the attention information includes information extracted from characteristics of the face or eyes of the person.
The information processing system according to claim 1, wherein the attention information includes at least one of a gaze direction of the person and a face direction of the person.
The information processing system according to any one of claims 1 to 3, wherein the image recognition includes at least one of face matching and character recognition.
The behavior predicting unit predicts the behavior of the person using a learning model generated by machine learning using learning data including the attention information and the behavior of the person as inputs. 5. The information processing system according to any one of 4.
The information according to any one of claims 1 to 4, wherein the behavior prediction unit predicts the behavior of the person using a database in which the attention information and the behavior of the person are associated with each other. Processing system.
The information processing system according to any one of claims 1 to 6, further comprising: an image generation unit that generates a predicted image including behavior prediction information including the behavior of the person predicted by the behavior prediction unit. .
The information processing system according to claim 7, wherein the image generation unit generates the predicted image by superimposing the behavior prediction information on the image.
The image is a video of a sports game,
The information processing system according to any one of claims 1 to 8, wherein the person is a player of the sport.
The information processing system according to claim 9, wherein the behavior prediction unit further predicts the behavior of the person based on the situation of the game.
The information processing system according to claim 9 or 10, wherein the behavior prediction unit further predicts the behavior of the person based on a situation of a player other than the person.
The information processing system according to any one of claims 9 to 11, wherein the behavior prediction unit further predicts the behavior of the person based on a history of the behavior of the person in the game.
The behavior predicting unit further predicts the behavior of the person based on a combination of the person and a player who cooperates with the person. Information processing system.
The information processing system according to claim 9, wherein the behavior prediction unit further predicts the behavior of the person based on the facial expression of the person.
The information processing system according to claim 9, wherein the behavior prediction unit further predicts the behavior of the person based on the score of the game.
The said action prediction part further predicts the action of the said person based on the past game information of the player of the team of the other party of the said person. The Claim 1 characterized by the above-mentioned. Information processing system.
Obtaining identification information of a person identified by image recognition from an image;
Obtaining attention information indicating the attention state of the person from the face image of the person;
Predicting the behavior of the person based on the identification information and the attention information;
An information processing method comprising:
On the computer,
Obtaining identification information of a person identified by image recognition from an image;
Obtaining attention information indicating the attention state of the person from the face image of the person;
Predicting the behavior of the person based on the identification information and the attention information;
A storage medium storing a program for executing an information processing method.