JP2017022727A - Information processing device, information processing system, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To further precisely determine a state of play in sport by a user.SOLUTION: An information processing device comprises: a sensor information acquisition section for acquiring sensor information which shows a behavior of a user; an event information generation section for detecting occurrence of an event in response to the sensor information and generating event information corresponding to the event; and a control section for extracting a scene in response to the event information from an image obtained by imaging the user.SELECTED DRAWING: Figure 3

Description

  The present disclosure relates to an information processing apparatus, an information processing system, and a program.

  To improve sports, it is important to analyze your own play objectively and play with a sense of improvement. For this reason, for example, it is widely performed to record a play as a still image or a moving image, and to view the still image or the moving image after the play to grasp improvement points. Such still images and moving images may be taken by other people, but are often so-called self-portraits taken by the user himself / herself.

  In such a case, for example, if a part where a specific user is shown or a part where the user is playing is extracted from the content of a still image or video, the play can be looked back and improved in a short time. The point can be grasped. In other words, if the user's desired scene can be automatically detected from the content, the searchability of the content can be improved, and the content can be efficiently used to improve play.

  Techniques for automatically detecting a user's desired scene from content have already been proposed for commercial content such as professional sports broadcasts. For example, Patent Literature 1 describes a technique for automatically detecting a scene (highlight scene) that is a target of user's interest by learning based on content images and audio feature quantities.

JP 2011-223287 A

  However, it is not easy to apply the technique as described in Patent Document 1 to content that the user has photographed to improve his / her own sport. With the above technology, for example, in the case of content of a soccer game, image feature quantities indicating field lines and ball trajectories, and voice feature quantities indicating whistles and cheers are extracted and learned to detect highlight scenes. To do. However, in the case of content photographed by the user, there are various places to play, and there are often no whistles or cheers. Furthermore, obviously unnecessary scenes are included in the content, for example, while the player has been picking up the ball from the court. Therefore, it is difficult and inefficient to learn the feature amount as in commercial content.

  Therefore, there has been a demand for a technique for detecting a user's desired scene by using a method different from the technique described above, for example, for content that the user has taken to improve his / her own sport. Therefore, the present disclosure proposes a new and improved information processing apparatus, information processing system, and program capable of more accurately grasping the state of sports play by a user.

  According to the present disclosure, a sensor information acquisition unit that acquires sensor information indicating user behavior, an event information generation unit that detects the occurrence of an event based on the sensor information, and generates event information corresponding to the event, And a control unit that extracts a scene based on the event information from an image of the user.

  According to the present disclosure, a sensor that acquires sensor information indicating a user's behavior, a sensor device that includes a communication unit that transmits the sensor information, a communication unit that receives the sensor information from the sensor device, and the sensor Information processing comprising: an event information generation unit that detects occurrence of an event based on information and generates event information corresponding to the event; and a control unit that extracts a scene based on the event information from an image of the user An information processing system including the apparatus is provided.

  Further, according to the present disclosure, a function of acquiring sensor information indicating a user's behavior, a function of detecting occurrence of an event based on the sensor information, and generating event information corresponding to the event, A program for causing a computer to realize a function of extracting a scene from a captured image based on the event information is provided.

  In the present technology, a play event is defined as a unit of action that has some meaning in sports play. As will be described in detail later, for example, in the case of tennis, play events such as “movement in the court”, “swing”, “jump”, and “hit the ball” can be defined. By detecting the occurrence of such a play event and generating play event information corresponding to the detected play event, it is possible to grasp the play state of the sport by the user, for example, by an operation having some meaning for the user. Is possible.

  As described above, according to the present disclosure, it is possible to more accurately grasp the state of sports play by the user.

It is a figure showing an outline of an information processing system concerning a 1st embodiment of this indication. It is a figure for demonstrating the example of installation of the imaging device in 1st Embodiment of this indication. 2 is a block diagram illustrating a schematic functional configuration of a sensor device and an imaging device according to a first embodiment of the present disclosure. FIG. It is a figure showing an example of play event detection in a 1st embodiment of this indication. It is a figure showing an example of play event information generated in a 1st embodiment of this indication. It is a figure which shows the example of the detailed information of the play event produced | generated in 1st Embodiment of this indication. 6 is a diagram for describing an example in which a swing type is set as detailed information of a play event generated in the first embodiment of the present disclosure. FIG. It is a figure showing an example of use of play event information in a 1st embodiment of this indication. FIG. 10 is a diagram illustrating an example of a processing flow for capturing a play image in a predetermined play event section in the first embodiment of the present disclosure. In the first embodiment of the present disclosure, it is a diagram illustrating an example of a processing flow for setting a chapter in a captured moving image. It is a figure showing an outline of a system concerning a 2nd embodiment of this indication. FIG. 6 is a block diagram illustrating a schematic functional configuration of a sensor device and a notification device according to a second embodiment of the present disclosure. It is a figure showing an example of notice DB set up in a 2nd embodiment of this indication. FIG. 10 is a diagram illustrating an example of a processing flow for providing notification information to a user in the second embodiment of the present disclosure. It is a figure showing an example of hardware constitutions of a sensor device concerning an embodiment of this indication. It is a figure which shows the example of the hardware constitutions of the information processing apparatus which concerns on embodiment of this indication.

  Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In the present specification and drawings, components having substantially the same functional configuration are denoted by the same reference numerals, and redundant description is omitted.

The description will be made in the following order.
1. 1. First embodiment 1-1. System configuration 1-2. Device configuration 1-3. Example of play event detection 1-4. Example of play event information 1-5. Example of use of play event information 1-6. Example of processing flow 1-7. Summary of Embodiments 2. Second embodiment 2-1. System configuration 2-2. Device configuration 2-3. Example of notification DB 2-4. Example of processing flow 2-5. Summary of Embodiments 2. 3. Third embodiment Other Embodiments 5. Hardware configuration Supplement

(1. First embodiment)
First, a first embodiment of the present disclosure will be described with reference to FIGS. In the first embodiment of the present disclosure, play event information is used for capturing a play image and setting additional information.

(1-1. System configuration)
FIG. 1 is a diagram illustrating an overview of an information processing system according to the first embodiment of the present disclosure. The information processing system 10 includes a sensor device 100, an imaging device 200, and a server 300.

(Sensor device)
The sensor device 100 is attached to a user who is playing sports or sports equipment used by the user. When the sensor device 100 is worn by the user, for example, the sensor device 100 may be shaped like a bracelet as shown in the figure and directly worn on the user's body. In addition, when the sensor device 100 is mounted on a sports equipment (for example, in the case of tennis, racket, wear, shoes, wristband, etc.) Moreover, you may incorporate in a tool previously.

  Here, the sensor device 100 acquires sensor information indicating the behavior of a user who plays sports. The user's behavior indicated by the sensor information can be, for example, a physical motion (position, velocity, acceleration, etc.) of the user or tool. In order to acquire such sensor information, for example, the sensor device 100 includes at least one sensor. The sensor detects, for example, acceleration, angular velocity, vibration, temperature, time, or position (for example, a position on the ground surface represented by latitude and longitude, or a position relative to a court). For example, the sensor device 100 transmits the sensor information acquired in this way to the imaging device 200.

(Imaging device)
The imaging device 200 captures a sport play image. A play image is an image in which a user who is playing sports is shown at least in part, and can be a still image or a moving image. The imaging device 200 is installed around a court where a user is playing sports, for example, and captures a play image. At this time, the imaging device 200 detects the occurrence of a play event in the sport in which the user is playing based on the sensor information received from the sensor device 100, and generates play event information corresponding to the play event. Based on the event information, the shooting of the play image is controlled. The imaging device 200 may upload the captured play image to the server 300.

  Here, for example, the imaging apparatus 200 determines at least one of imaging start or imaging end of the play image based on the play event information. Thereby, for example, a play image including only a scene of a play event designated in advance by the user can be generated. Further, for example, the imaging apparatus 200 sets additional information such as chapter information in the play image based on the play event information. Accordingly, for example, when a user views a play image, it is easy to search for a scene of a desired play event.

  In addition, a plurality of imaging devices 200 may be installed. In the illustrated example, three imaging devices 200a to 200c are installed. In such a case, the play images captured by the imaging devices 200a to 200c may be selected based on the play event information. More specifically, for example, each of the imaging devices 200a to 200c may receive sensor information from the sensor device 100 to generate play event information, and control each imaging based on the information. In addition, any of the imaging devices 200a to 200c may receive sensor information from the sensor device 100, and the device may serve as a host to control imaging with other devices. Thereby, for example, a play image is picked up by the image pickup apparatus 200 having the most appropriate angle for each play event. Alternatively, an image having the most appropriate angle for each play event is selected from the play images captured by the respective imaging devices 200.

  Furthermore, the imaging device 200 may include a device that takes a bird's-eye view of a court where sports are played, such as the imaging device 200a in the illustrated example. In this case, the imaging device 200a can also function as a sensor device. For example, the imaging device 200a may provide the captured images or position information in the user's court acquired by analyzing the images as sensor information to the imaging devices 200b and 200c.

  In the above example, three imaging devices 200 are installed. However, in other embodiments, the imaging device 200 may be one, two, or four or more. Also good.

(Server device)
The server 300 can acquire a play image captured by the imaging apparatus 200 as recorded content. The server 300 may store the acquired play image and provide it for download by the client 400. In this way, the user can view his / her play image at home after play. Note that, among the functions of the imaging apparatus 200 described above, the server 300 may perform the functions other than the function of actually capturing a play image. That is, the server may communicate not only with the imaging device 200 but also with the sensor device 100 and control the imaging device 200 based on sensor information received from the sensor device 100.

(Example of how to install the imaging device)
FIG. 2 is a diagram for describing an installation example of the imaging device according to the first embodiment of the present disclosure. As described above, in the present embodiment, the imaging device 200 that captures an image of a user who plays sports is provided. Such an imaging apparatus 200 may be a digital camera fixed to a cradle, for example, as illustrated in the imaging apparatus 200a1. Further, the imaging apparatus 200 may be a mobile phone (smartphone) having an imaging function fixed to a cradle like the illustrated imaging apparatus 200a2. Furthermore, the imaging apparatus 200 may be an omnidirectional camera module, as illustrated in the imaging apparatus 200a3.

  For example, in the case of tennis, such an imaging device 200 may be attached to a court post as shown. Since the relative position of the support with respect to the court is determined, it is relatively easy to identify the position of the player in the court from the image of the imaging device 200 attached to the support. When the imaging apparatus 200 uses a wide-angle lens or is an omnidirectional camera such as the imaging apparatus 200a3, it is possible to look down on the entire coat surface by attaching it to a support.

  As described above, by appropriately arranging the imaging device 200, it is possible to determine the result of the hit ball by analyzing the moving image and to determine the movement of the player. The image acquired by the imaging apparatus 200 or information on the analysis result of the image may also be used by the imaging apparatus 200 that detects a play event as sensor information indicating the user's behavior.

(1-2. Device configuration)
FIG. 3 is a block diagram illustrating a schematic functional configuration of the sensor device and the imaging device according to the first embodiment of the present disclosure.

(Sensor device)
The sensor device 100 includes a sensor 110, a preprocessing unit 120, and a communication unit 130. A hardware configuration for realizing these functions will be described later.

  The sensor 110 detects, for example, acceleration, angular velocity, vibration, temperature, time, or position. More specifically, the sensor 110 may include, for example, an acceleration sensor, an angular velocity sensor, a vibration sensor, a temperature sensor, a clock, or a GPS (Global Positioning System) receiver. Further, when the imaging device 200 also functions as a sensor device as described above, the sensor 110 may be an imaging unit that captures an image for acquiring position information in a user's court or the like.

  The preprocessing unit 120 performs preprocessing of data detected by the sensor 110. The preprocessing can be, for example, amplification of detected data, filtering of data below a threshold value, or the like. Further, when the sensor 110 is an imaging unit, the preprocessing unit 120 may execute image processing for specifying the position of the user. Depending on the type of sensor 110, data may be supplied to the communication unit 130 without necessarily passing through the preprocessing unit 120.

  The communication unit 130 is a communication module that transmits data provided from the preprocessing unit 120 (or the sensor 110) to the imaging apparatus 200 as sensor information. For example, wireless communication is used for transmitting the sensor information. The communication method is not particularly limited. For example, when the imaging device 200 is near the sensor device 100, Bluetooth (registered trademark), a wireless LAN (Local Area Network), or the like can be used. Further, when the imaging apparatus 200 is away from the sensor apparatus 100 or transmits sensor information to the server 300, the Internet or the like may be used.

(Imaging device)
The imaging apparatus 200 includes a communication unit 210, an analysis unit 220, a control unit 230, and an imaging unit 240. A hardware configuration for realizing these functions will be described later.

  The communication unit 210 is a communication module that receives sensor information transmitted from the sensor device 100. As described for the sensor device 100, short-range wireless communication such as Bluetooth (registered trademark) or wireless LAN may be used for transmission and reception of sensor information. The communication unit 210 provides the received sensor information to the analysis unit 220. In the present embodiment, the communication unit 210 corresponds to a sensor information acquisition unit that acquires sensor information. As another example, the sensor information acquisition unit may be an interface that acquires sensor information internally from a separately provided communication device or from another part of the imaging device 200.

  The analysis unit 220 analyzes the sensor information provided from the communication unit 210 and provides the analysis result to the control unit 230. The analysis unit 220 includes a play event information generation unit 221. The play event information generation unit 221 detects the occurrence of a play event in sports based on the sensor information, and generates play event information corresponding to the play event.

  Here, as described above, a play event is defined as a unit of action having some meaning in sports play. For example, in the case of tennis, play events such as “movement on the court”, “swing”, “jump”, and “hit the ball” can be defined. The play event information generation unit 221 detects the occurrence of a predefined play event by analyzing sensor information using machine learning or pattern recognition. Further, the play event information generation unit 221 generates play event information by adding detailed information regarding the play event, for example, by further referring to the sensor information as necessary for the play event detected to have occurred. To do. A specific example of play event detection and play event information will be described later.

  The control unit 230 controls the operation of the imaging apparatus 200 based on play event information that is an analysis result provided from the analysis unit 220. In the present embodiment, the control unit 230 includes an imaging control unit 231 and a chapter setting unit 232.

  The imaging control unit 231 controls the imaging of the play image by the imaging unit 240 based on the play event information. For example, the imaging control unit 231 determines at least one of imaging start and imaging end of the play image, which is a moving image, by the imaging unit 240 based on the play event information. More specifically, when the occurrence of a predetermined play event is detected, the imaging control unit 231 controls the imaging unit 240 so that a play image is captured for the play event section.

  Further, the imaging control unit 231 may select play images captured by a plurality of imaging devices based on play event information. As described above, when a play image is picked up by a plurality of image pickup devices 200, the image pickup control unit 231 of each image pickup device 200 picks up an image when the angle of the own device is appropriate for the detected play event. You may make the part 240 image a play image. Further, the imaging control unit 231 of any imaging device 200 controls the imaging unit 240 of another imaging device 200 so that a play image is captured by the imaging device 200 having the most appropriate angle for the play event. Also good. Alternatively, the image control unit 231 of any of the image capturing apparatuses 200 may select an image with the most appropriate angle for each play event from the play images captured by the respective image capturing apparatuses 200.

  On the other hand, the chapter setting unit 232 sets chapter information corresponding to the play event in the play image captured by the imaging unit 240 based on the play event information. The chapter information is information that defines a playback section of a moving image based on, for example, a playback time when the play image is a moving image. By setting the chapter information, the user can skip reproduction or return in units of chapters when viewing the play image. Therefore, it is possible to jump to a desired play event portion or repeatedly watch the play event portion.

  In the present embodiment, the chapter setting unit 232 corresponds to an additional information setting unit that sets additional information for a play image. As another example, the additional information setting unit may add, to the play image, information for displaying a caption indicating the play event in a section corresponding to a specific play event of the play image, or in a section before that. . For example, when the play image is a still image, the additional information setting unit may add caption information indicating a play event to the still image.

  The imaging unit 240 includes an imaging element and a lens for capturing an image, a drive circuit thereof, and the like. The imaging unit 240 captures a play image under the control of the control unit 230. The captured play image data may be transmitted from the communication unit 210 to the server 300 (not shown in FIG. 2), for example. Alternatively, the captured play image data may be stored in the storage device of the imaging device 200.

(1-3. Example of play event detection)
FIG. 4 is a diagram illustrating an example of play event detection according to the first embodiment of the present disclosure. In the illustrated example, the play event information generation unit 221 of the imaging device 200 detects a tennis swing using vibration of a racket used by the user as sensor information.

  In the illustrated example, the swing is made up of takeback, forward swing, impact, post impact, and follow-through operations. In the present embodiment, a “swing” play event is defined as a series of actions from takeback to follow-through. As shown in the exemplary vibration waveform, the racket vibration at this time occurs during the impact operation and attenuates during the post-impact operation.

  In this case, the play event information generation unit 221 first detects the vibration of the racket generated in the impact section based on the sensor information by the vibration sensor (sensor 110) included in the sensor device 100 attached to the racket. The play event information generation unit 221 specifies that this vibration represents a swing in consideration of the subsequent attenuation waveform of the vibration. That is, here, the play event information generation unit 221 detects that a play event “swing” has occurred.

  Further, the play event information generation unit 221 sets the vibration occurrence time as the start time of impact, and sets the section of 3 to 5 milliseconds as impact, and takes back, forward swing, post impact, and follow-through before and after that Define the interval. The whole of these sections is the section of the play event “swing”. The play event information generation unit 221 may include the start and end times of these sections in play event information described later.

  Here, the length of each section of takeback, forward swing, impact, post impact, and follow-through may be set in advance from, for example, general swing data. The set value may be a fixed value, for example, or may be a relative value according to the amplitude or length of vibration.

(1-4. Example of play event information)
FIG. 5 is a diagram illustrating an example of play event information generated in the first embodiment of the present disclosure. In the illustrated example, the play event information includes a sensor device ID, a start time, an end time, a play event type, and detailed information.

  The sensor device ID is information for identifying the sensor device 100 that has transmitted the sensor information that is the basis for detecting the play event. For example, when a plurality of players wear the sensor device 100 and sensor information indicating the behavior of a plurality of users is received by the communication unit 210, the play event is identified by identifying these sensor devices 100 by the sensor device ID. Can be identified. For example, a MAC (Media Access Control) address of the sensor device 100 may be used as the ID information. In addition, when there is only one sensor device 100, information on the sensor device ID may be omitted.

  The start time and end time information specifies the start and end times of the play event. For example, in the case of the “swing” play event shown in FIG. 4, the times of the start point of the takeback and the end point of the follow-through can be recorded as the start time and the end time, respectively. As in this example, since a play event can occur in units of seconds or milliseconds, it is desirable that the start time and end time be recorded at least in units of seconds, preferably in units of milliseconds. The start time and the end time may be specified based on time information added when sensor information is transmitted on the sensor device 100 side, or added when sensor information is received on the imaging device 200 side, for example. It may be specified based on the time information.

  The play event type is information indicating the type of play event. For example, in the case of tennis, types of play events such as “movement on the court”, “swing”, “jump”, “hit the ball”,. In this case, for example, a plurality of types of play events may be set for overlapping sections, such as “swing” and “hit the ball” while “jumping”. Note that the type of play event can be set in various ways depending on the sport event and the preference of the player.

  The detailed information is information describing the details of each play event in more detail. An example of detailed information will be further described with reference to FIG.

  FIG. 6 is a diagram illustrating an example of detailed information of a play event generated in the first embodiment of the present disclosure. In the example shown in the figure, detailed information that can be set in the play event information for four types of play events generated by tennis, such as “movement in the court”, “swing”, “jump”, and “hit the ball”. It is shown.

  For example, in the case of a play event of “move on the court”, GPS information (latitude and longitude) is recorded as detailed information. For example, the GPS information may be recorded as information indicating a start point and an end point of movement. Such information can be acquired practically if the accuracy of GPS is improved by using, for example, a quasi-zenith satellite.

  In the “swing” play event, the type of swing is recorded as detailed information. The detection of the type of swing will be described later with reference to FIG. In the “jump” play event, GPS information (latitude and longitude) is recorded as information on the position where the jump was made. In the “hit the ball” play event, the position where the ball hits the racket is recorded as detailed information. The sweet spot rate (the rate at which the ball hits the sweet spot (such as the position where the racket vibration is minimized when the ball is hit)) is calculated from the position hit by the ball and may be recorded as detailed information. . The position where the ball has hit can be detected from, for example, racket vibration information.

  In this way, by setting detailed information for each play event, for example, when shooting of a play image is controlled using the play event information, fine control according to the user's desire is possible. In addition, when setting additional information related to a play event in a play image, it is possible to add not only the type of event but also various information, which can be useful for improving the user's sport, for example.

  For example, an unnecessary play event may be set as a play image, such as “I went to pick up a ball outside the court”, “Break”, “Chat”, etc. Or that part can be skipped during playback of the play image). For example, in such a play event, detailed information may not be set.

  FIG. 7 is a diagram for describing an example in which a swing type is set as detailed information of a play event generated in the first embodiment of the present disclosure. In the illustrated example, the sensor device 100a is attached to the shaft portion of the racket. The sensor device 100a includes a three-axis acceleration sensor that can detect acceleration in each of the x-axis, y-axis, and z-axis directions as the sensor 110.

  In this case, when the user swings the racket, a waveform specific to the type of swing appears in the time series data of acceleration. For example, the features of the forehand stroke and the serve appear as shown (note that the waveform shown in the figure is not necessarily observed depending on the mounting position of the sensor device 100a and the performance of the acceleration sensor). ). It is possible to identify the type of swing by vectorizing such time series data of acceleration and collating it with an identification dictionary registered in advance. A general machine learning method may be used for identification. For example, a k-NN method or a support vector machine can be used. Examples of types of swing include forehand stroke, forehand volley, forehand slice, backhand stroke, backhand slice, backhand volley, serve, and smash.

  In this way, it is possible to set the swing type in the detailed information of the play event “swing”. In addition to this, the detailed information may include information acquired using various known techniques according to the user's request.

(1-5. Example of using play event information)
FIG. 8 is a diagram illustrating an example of use of play event information according to the first embodiment of the present disclosure. In the illustrated example, tennis is played by user A and user B. Both user A and user B wear the sensor device 100. One imaging device 200 captures a play pattern as a play image.

  In this case, for example, the imaging apparatus 200 can generate play event information for each swing opportunity for the user A and the user B. In the figure, the generated play event information is shown in time series for each of user A and user B. What is described in the play event is the type of swing specified as detailed information.

  Thus, by setting the play event information for each user, for example, it is possible to extract or search the play of the user A or the user B from the play image in which the play of the user A and the user B is shown alternately. It is. Also, by setting the type of swing as the detailed information, for example, it is possible to extract or search a scene showing a stroke (may be volley, serve, etc.) regardless of user A or user B. . When other detailed information such as the position where the ball hits is set in the play event “swing”, a scene of the play image may be extracted or searched based on the detailed information.

(1-6. Example of processing flow)
(Imaging control)
FIG. 9 is a diagram illustrating an example of a processing flow for capturing a play image in a predetermined play event section according to the first embodiment of the present disclosure.

  First, in the sensor device 100, the sensor 110 detects a user's behavior (step S101). Here, the detected value may be, for example, a measurement value such as an acceleration sensor, an angular velocity sensor, or a vibration sensor, or position information (latitude, longitude) received by GPS.

  Next, the preprocessing unit 120 determines whether or not the detection value detected by the sensor 110 is equal to or greater than a predetermined threshold (step S103). For example, when the detected value is a measured value such as an acceleration sensor, an angular velocity sensor, or a vibration sensor, the preprocessing unit 120 may determine whether the magnitude or amplitude of the detected value is equal to or greater than a predetermined threshold value. Good. Note that the detection value used for the determination may have undergone a predetermined amplification process. On the other hand, for example, when the detected value is GPS position information, the preprocessing unit 120 compares the previous position information with the newly detected position information, and determines whether or not the distance therebetween is a predetermined threshold value or more. May be determined.

  In step S <b> 103, when the detected value is greater than or equal to a predetermined threshold value, the communication unit 130 transmits sensor information to the imaging device 200. In addition to the detection value detected in step S101, the detection value and time information by other sensors may be added to the sensor information transmitted here. For example, when the acceleration sensor that is the sensor 110 detects a value that is equal to or greater than a predetermined threshold value, information on the position of the user acquired by the GPS receiver and the time at which the detection value is acquired is added together with the detection value of the acceleration sensor. The information may be transmitted from the communication unit 130 to the imaging device 200.

  On the other hand, if the detected value is not greater than or equal to the predetermined threshold value in step S103, the communication unit 130 does not transmit the sensor information, and the sensor device 100 returns to detecting the user's behavior again (step S101). As described above, when the detected value is not equal to or greater than the predetermined threshold value, the power consumption of the sensor device 100 associated with communication can be saved by not transmitting the sensor information. Further, by not transmitting unnecessary sensor information, it is possible to prevent the analysis processing load in the imaging apparatus 200 from becoming unnecessarily high.

  Next, in the imaging device 200, the communication unit 210 receives the sensor information transmitted from the sensor device 100 (step S107). The communication unit 210 provides the received sensor information to the analysis unit 220. When the time information is not added to the sensor information on the sensor device 100 side, the time information may be added at this point.

  Next, the play event information generation unit 221 of the analysis unit 220 determines whether a valid play event has occurred from the received sensor information (step S109). As described above, in the present embodiment, the play event information generation unit 221 recognizes the characteristics of sensor information for a predefined play event, and detects the occurrence of the play event from the sensor information based on the feature.

  The “valid play event” referred to here may not be a play event defined in advance, but may mean a play event designated as valid in the play event. For example, in the case of tennis, a play event “going to pick up a ball outside the court” may be defined, but this is not a play event that is effective for a user who views a captured image, for example. Therefore, the play event information generation unit 221 detects the occurrence of a predefined play event “going to pick up the ball outside the court” from the characteristics of the sensor information, but the event is not a valid play event. You may judge.

  If it is determined in step S109 that a valid play event has occurred, the imaging control unit 231 of the control unit 230 starts imaging the play image by the imaging unit 240 from the start time of the play event (step S111). Here, a time difference from the start of the actual play event until the occurrence of the play event is detected may be considered. In that case, the imaging unit 240 continuously performs imaging and temporarily records the captured image in the buffer memory, and the imaging control unit 231 selects a play event from the images stored in the buffer memory. Images after the start time may be recorded as a play image.

  For example, it is assumed that the play event “swing” is detected at the timing of “impact” in the example of FIG. In this case, the imaging control unit 231 records the image stored in the buffer memory retroactively to the “forward swing” and “takeback” sections defined before the “impact” time as a reference. .

  Thereafter, the imaging control unit 231 ends the imaging of the play image by the imaging unit 240 at the end time of the play event (step S113). The end of the play event may be determined from the sensor information, for example, in the same manner as in steps S101 to S109 described above, and after a predetermined time from the start of the play event is specified as the end of the play event depending on the type of the play event. May be.

(Chapter setting)
FIG. 10 is a diagram illustrating an example of a processing flow for setting a chapter in a captured moving image according to the first embodiment of the present disclosure.

  In this example, the steps for detecting a play event (steps S101 to S109) are the same as the example described above with reference to FIG.

  If it is determined in step S109 that a valid play event has occurred, the chapter setting unit 232 of the control unit 230 has a play event at a position corresponding to the play event start time of the play image captured by the imaging unit 240. The chapter attribute indicating the start of is recorded (step S211).

  Thereafter, the chapter setting unit 232 records a chapter attribute indicating the end of the play event at a position corresponding to the play event end time of the captured play image (step S213). The end of the play event may be determined from the sensor information, for example, in the same manner as in steps S101 to S109 described above, and after a predetermined time from the start of the play event is specified as the end of the play event depending on the type of the play event. May be.

(1-7. Summary of Embodiment)
Heretofore, the first embodiment of the present disclosure has been described. In the present embodiment, for example, by controlling the shooting of the play image by the imaging device 200 according to the play event information generated based on the sensor information, the user can control the shooting in the case of so-called selfie. It is possible to take a desired play image without bothering the operation. Further, if a play event to be looked back in advance is set in advance as an effective play event, it is possible to automatically generate a play image in which only a desired portion is extracted in advance. Even without such setting, scenes that are clearly unnecessary, such as picking up a ball, can be automatically excluded from the play image. As a result, the capacity of the entire play image is also reduced. For example, when the play image is uploaded from the imaging device 200 to the server 300, the communication time can be shortened.

  Further, for example, by setting a chapter or the like in the image captured by the imaging device 200 according to the information of the play event detected from the sensor information, the searchability in the generated play image is remarkably improved. Further, since the chapter is set according to the play event, it is easy for the user to reproduce only a desired part in his / her play when viewing the play image.

  In the above description of the present embodiment, the example of tennis has been mainly described, but the sporting event is not limited to tennis. This embodiment can also be applied to baseball, for example. In the case of baseball, the sensor device 100 can be mounted on, for example, the user himself or a tool such as a bat or a glove. A play image is captured or a chapter is set corresponding to the detected play event, as in the above example of tennis. In the case of baseball, whether it is an attack or a defensive, the opportunity for one user to play is limited. Therefore, applying this embodiment to facilitate the extraction or search of a desired scene has a great advantage. is there.

  Moreover, this embodiment can also be applied to a batting center or a golf driving range. For example, it is conceivable to attach the sensor device 100 to a hitting tool such as a bat or a golf club, identify a shot that is good as a play event and a shot that is not so, extract a video of the good shot, and provide it to the user. .

  Note that the present embodiment is not limited to the above example, and is useful in all aspects of sports in which the user may look back on his or her play with images.

(2. Second Embodiment)
Next, a second embodiment of the present disclosure will be described with reference to FIGS. In the second embodiment of the present disclosure, play event information is used to notify the user of information related to play.

(2-1. System configuration)
FIG. 11 is a diagram illustrating an overview of a system according to the second embodiment of the present disclosure. The information processing system 20 includes a sensor device 500 and a notification device 600.

  The sensor device 500 is worn by a user who is playing sports, like the sensor device 100 in the first embodiment. The sensor device 500 not only acquires sensor information and transmits it to the notification device 600 but also receives notification information from the notification device 600 and outputs it to the user.

  The notification device 600 generates play event information from the sensor information provided from the sensor device 500, generates notification information to the user based on this, and transmits the notification information to the sensor device 500. In addition, about the production | generation of play event information, the notification apparatus 600 is the same as that of the imaging device 200 in 1st Embodiment. The notification device 600 may be any device as long as it has a communication function and an information processing function. In the illustrated example, a mobile phone (smart phone) is used as the notification device 600.

(2-2. Device configuration)
FIG. 12 is a block diagram illustrating a schematic functional configuration of a sensor device and a notification device according to the second embodiment of the present disclosure.

(Sensor device)
The sensor device 500 includes a sensor 110, a preprocessing unit 120, a communication unit 530, and a notification output unit 540. A hardware configuration for realizing these functions will be described later. Further, since the sensor 110 and the preprocessing unit 120 have the same configuration as that of the first embodiment, detailed description thereof is omitted.

  The communication unit 530 has substantially the same configuration as the communication unit 130 of the first embodiment, but is used not only to transmit sensor information to the notification device 600 but also to receive notification information from the notification device 600. The point is different.

  The notification output unit 540 outputs the notification information received by the communication unit 530 toward the user. The notification output unit 540 may include, for example, a speaker 541 and a display device 542. The speaker 541 outputs the notification information to the user by voice. The display device 542 outputs the notification information as an image toward the user. For example, one of the speaker 541 and the display device 542 may be selected and used depending on the type of notification information, or may be used in combination. In addition, the notification output unit 540 may include a lighting unit such as an LED lamp, a vibrator that vibrates the user himself / her and the tool, and the like.

  Note that the notification information does not necessarily have to be output from the sensor device 500 and may be output by a separate display device or speaker. However, as an application scene of the present embodiment, the user is playing sports personally. Considering that many scenes can be considered, outputting notification information from the sensor device 500 is one of the rational methods.

(Notification device)
The notification device 600 includes a communication unit 610, an analysis unit 220, a notification information generation unit 650, a notification DB 660, and a registration unit 670. A hardware configuration for realizing these functions will be described later. Further, the analysis unit 220 has the same configuration as that of the first embodiment, and thus detailed description thereof is omitted.

  The communication unit 610 has substantially the same configuration as the communication unit 210 of the first embodiment, but is used not only to receive sensor information from the sensor device 500 but also to transmit notification information to the sensor device 500. The point is different.

  The notification information generation unit 650 generates notification information for the user based on the play event information that is the analysis result provided from the analysis unit 220. The notification information may be information registered in the notification DB 660 by the user in advance in association with the play event, for example. In this case, the notification information generation unit 650 determines whether or not the generated play event is a notification information generation target by matching the types of play events, for example. When a play event that is a target of notification information generation occurs, the notification information generation unit 650 generates notification information according to information registered in the notification DB 660 and transmits the notification information to the sensor device 500 via the communication unit 610. The notification information generation unit 650 may use sensor information received by the communication unit 610 in addition to the play event information in order to generate notification information. An example of information registered in the notification DB 660 will be described later.

  The registration unit 670 is an operation input unit for adding or correcting information registered in the notification DB 660. For example, the registration unit 670 may use a display unit or an operation unit of the notification device 600 to provide a user with a GUI (Graphical User Interface) for setting registration contents of the notification DB 660.

(2-3. Notification DB example)
FIG. 13 is a diagram illustrating an example of a notification DB set in the second embodiment of the present disclosure. In the illustrated example, information regarding three notifications “remind (serve)”, “remind (break)”, and “success confirmation” is shown.

  For example, the notification “remind (serve)” is executed when the user fails to serve once and performs the next serve. In the notification DB 660, for example, (1) a play event “swing” occurs, and (2) the “swing type” in the detailed information of the “swing” is “serve”. (3) The “position where the ball hits” in the detailed information is other than the sweet spot, and (4) the position of the user indicated by the sensor information continues to be in the vicinity of the service line. When it is determined that the above condition is satisfied based on the play event information and the sensor information, the notification information generation unit 650 detects notification information for causing the speaker 541 to output a voice “raise the toss carefully”. To device 500.

  The notification of “remind (break)” is executed when the user starts playing and continues for 60 minutes. In the notification DB, the conditions for specifying this case are, for example, (1) some play event other than “break” has occurred before, (2) the play event “break” has not occurred, and (3) A predetermined time (60 minutes) has elapsed since the occurrence of the first play event. When it is determined that the above condition is satisfied based on the play event information and the sensor information, the notification information generation unit 650 transmits the notification information for blinking the LED and vibrating the vibrator to the sensor device 500.

  Further, the notification of “success confirmation” is executed when the user hits the ball at the sweet spot of the racket during the stroke. In the notification DB, for example, the conditions for specifying this case are (1) a play event “swing” occurs, and (2) “swing type” in the detailed information of the “swing” is “stroke”. (3) The “position where the ball hits” in the detailed information is set as a sweet spot. When it is determined that the above condition is satisfied based on the play event information, the notification information generation unit 650 transmits notification information for causing the display device 542 to display the characters “Good!” To the sensor device 500.

(2-4. Example of processing flow)
FIG. 14 is a diagram illustrating an example of a processing flow for providing notification information to a user in the second embodiment of the present disclosure.

  In this example, the steps for detecting a play event (steps S101 to S109) are the same as in the example of the first embodiment described above with reference to FIG.

  If it is determined in step S109 that a valid play event has occurred, the notification information generation unit 650 of the notification device 600 matches the generated play event with information registered in the notification DB 660 (step S311). As described above, not only the play event information detected in the immediately preceding step S109 but also the play event information detected so far and the sensor information received in step S107 are used for matching here. Can be.

  When matching with the notification conditions stored in the notification DB 660 occurs in step S311, the notification information generation unit 650 transmits the notification information to the sensor device 500 via the communication unit 610 (step S313). The notification information transmitted here may include, for example, notification device information for specifying a device (speaker 541, display device 542, etc.) used for notification, and notification action information indicating a specific operation of notification.

  Next, in the sensor device 500, the communication unit 530 receives notification information (step S315). The communication unit 530 provides the received notification information to the notification output unit 540. The notification output unit 540 selects an appropriate notification device according to the provided notification information, and executes a notification action (step S317).

(2-5. Summary of Embodiment)
Heretofore, the second embodiment of the present disclosure has been described. In the present embodiment, for example, during sports play, an appropriate notification is provided at a timing intended by the user in advance. Therefore, it is possible to remember the improvement points of the play that the user has recognized before the play even at the time of play during which the user tends to concentrate on the play.

  Also, if you concentrate on playing, it is easy to forget to take a break. For example, by detecting a break as one play event and providing a notification that prompts the user for a break when this has not been detected for a long time, It can also prevent accidents such as heat stroke.

  In this way, the present embodiment realizes improvement of sports or safe play by providing appropriate information to a user who is playing sports. Therefore, the present embodiment is not limited to the example of tennis described above, and can be applied to all types of sports.

(3. Third embodiment)
Next, a third embodiment of the present disclosure will be described. In the third embodiment of the present disclosure, play event information is generated in a sport that goes around a course such as golf.

  In the present embodiment, for example, hitting a ball (shot) is detected as a play event based on sensor information that detects vibration at the time of impact of a tool (such as a golf club) used by the user. In the detailed information of the play event “shot”, position information is set by GPS information, for example.

  With this position information, for example, it is possible to identify where the shot was made. Thus, for example, in the case of golf, it is possible to identify whether the shot is in a hole number, a shot from a specific position such as a tee shot or a bunker shot, or a putt on the green. Based on the result of such identification, for example, as in the first embodiment described above, if shooting of a play image is controlled or chapter information is added to the play image, the user can later Useful when looking back at play.

  It is also possible to calculate the ball flight distance as the distance between successive shot positions. For example, the play image may be classified for each ball flight distance, or whether the shot was successful or unsuccessful may be determined from the relationship between the shot position and the ball flight distance. Further, as the notification information, an image in which the trajectory of the ball connecting the positions of shots in the same hole is mapped to the map of the golf course may be generated and presented to the user.

(4. Other embodiments)
While the above embodiments have been beneficial to users who play sports, such as users looking back on their play with images, etc., or users receiving notifications about their play, the present disclosure The embodiment is not limited to such an example.

  For example, as in the first embodiment described above, a play image in which only a part of a specific play event is cut out or a chapter or the like is set for each play event is used as a user other than a user who plays sports, for example, a child May be provided to parents who can attend a tennis school (of course, a school other than tennis may be used, or a club activity may be used instead of a school). In this case, the parent can watch only the portion of the play image where his / her child is shown, or can easily search for it, and can accurately grasp the state of the child at school in a shorter time. Can do.

(5. Hardware configuration)
Subsequently, an example of a hardware configuration for realizing the sensor device and the information processing device (corresponding to an imaging device, a notification device, and the like) according to the embodiment of the present disclosure will be described with reference to FIGS. 15 and 16. .

(Sensor device)
FIG. 15 is a diagram illustrating an example of a hardware configuration of the sensor device according to the embodiment of the present disclosure. The sensor device 800 may realize the sensor device according to the embodiment of the present disclosure, for example, the sensor device 100 or the sensor device 500 described above.

  The sensor device 800 includes a CPU (Central Processing Unit) 801, a ROM (Read Only Memory) 802, a RAM (Random Access Memory) 803, a vibration sensor / amplifier 804, a user interface 805, an external storage device 806, A communication device 807 and an image / audio output device 808 may be included. These elements are connected to each other by, for example, a bus.

  The CPU 801, the ROM 802, and the RAM 803 realize various functions in software by reading and executing program instructions recorded in the external storage device 806, for example. In the embodiment of the present disclosure, the CPU 801, the ROM 802, and the RAM 803 can realize, for example, control of the entire sensor device 800 and some of the functions of the preprocessing unit 120 in the above example.

  The vibration sensor / amplifier 804 corresponds to the sensor 110 in the functional configuration of the above embodiment. Sensor device 800 may further include an acceleration sensor, an angular velocity sensor, a temperature sensor, a clock, a GPS receiver, or the like.

  The user interface 805 is an input device such as a button or a touch panel that accepts a user operation on the sensor device 800. The user's operation may be, for example, instructing the start or end of transmission of sensor information from the sensor device, or instructing to stop the output when notification information is provided. Also good.

  The external storage device 806 stores various types of information regarding the sensor device 800. The external storage device 806 may store, for example, program instructions for causing the CPU 801, the ROM 802, and the RAM 803 to realize functions in software, and data acquired by a sensor such as the vibration sensor / amplifier 804 is temporarily stored. May be cached automatically. Considering that the sensor device 800 is attached to the user himself / herself or sports equipment, it is desirable to use an external storage device 806 that is resistant to impact, such as a semiconductor memory.

  The communication device 807 corresponds to the communication units 130 and 530 in the functional configuration of the above embodiment. The communication device 807 communicates with an information processing device 900 described later by short-range wireless communication such as Bluetooth (registered trademark) or wireless LAN. When the information processing apparatus 900 is a server apparatus on a network, the communication apparatus 807 may communicate with the information processing apparatus 900 via a network such as the Internet.

  The image / sound output device 808 is configured by a device capable of visually or audibly notifying acquired information to the user. The image / sound output device 808 can be, for example, a display device such as an LCD (Liquid Crystal Display) or a sound output device such as a speaker or headphones. The image / sound output device 808 corresponds to the notification output unit 540 in the functional configuration of the above embodiment. The sensor device 800 may further include a lighting unit such as an LED lamp, a vibrator that vibrates the user himself / herself or a tool, and the like as an output unit.

(Information processing device)
FIG. 16 is a diagram illustrating an example of a hardware configuration of the information processing apparatus according to the embodiment of the present disclosure. The information processing apparatus 900 can realize the information processing apparatus according to the embodiment of the present disclosure, for example, the imaging apparatus 200 and the notification apparatus 600 described above.

  The information processing apparatus 900 can include a CPU 901, a ROM 902, a RAM 903, a user interface 905, an external storage device 906, a communication device 907, and an image / audio output device 908. These elements are connected to each other by, for example, a bus.

  The CPU 901, the ROM 902, and the RAM 903 realize various functions in software by reading and executing program instructions recorded in the external storage device 906, for example. In the embodiment of the present disclosure, the CPU 901, the ROM 902, and the RAM 903 realize, for example, control of the entire information processing apparatus 900 and functions such as the analysis unit 220, the control unit 230, and the notification information generation unit 650 in the above example. Can be done.

  The user interface 905 is an input device such as a button or a touch panel that accepts user operations on the information processing apparatus 900. The user interface 905 can realize, for example, the function of the registration unit 670 in the above example.

  The external storage device 906 stores various types of information regarding the information processing device 900. For example, the CPU 901, the ROM 902, and the RAM 903 may store program instructions for realizing functions in software, and the sensor information received by the communication device 907 is temporarily cached in the external storage device 906. Also good. Also, the external storage device 906 may store a log of play event information and sensor information. Furthermore, the notification DB 660 in the above example may be stored in the external storage device 906.

  The image / sound output device 908 is configured by a device capable of visually or audibly notifying acquired information to the user. The image / audio output device 908 can be, for example, a display device such as an LCD, or an audio output device such as a speaker or headphones. The image / sound output device 908 outputs the result obtained by the processing of the information processing device 900 as a video such as text or image, or as a sound or sound.

(6. Supplement)
The preferred embodiments of the present disclosure have been described in detail above with reference to the accompanying drawings, but the technical scope of the present disclosure is not limited to such examples. It is obvious that a person having ordinary knowledge in the technical field of the present disclosure can come up with various changes or modifications within the scope of the technical idea described in the claims. Of course, it is understood that it belongs to the technical scope of the present disclosure.

  For example, in the above-described embodiment, the information processing system including the sensor device and the information processing device has been described. However, the embodiment of the present disclosure is, for example, a server on a network that realizes at least a part of the functions of the information processing device. (Including those realized as a set of functions of a plurality of devices), a program for causing a computer to realize the functions of these devices, and a recording medium on which such programs are recorded.

The following configurations also belong to the technical scope of the present disclosure.
(1) a sensor information acquisition unit that acquires sensor information indicating the behavior of a user who plays sports;
An information processing apparatus comprising: a play event information generation unit that detects occurrence of a play event in the sport based on the sensor information and generates play event information corresponding to the play event.
(2) The information processing apparatus according to (1), further including a notification information generation unit that generates notification information for the user based on the play event information.
(3) The information processing apparatus according to (2), wherein the notification information is information that the user registers in advance in association with the play event.
(4) The information processing apparatus according to (2) or (3), wherein the notification information generation unit generates the notification information further based on the sensor information.
(5) The information processing apparatus according to any one of (1) to (4), further including an imaging control unit that controls imaging of a play image of the sport based on the play event information.
(6) The information processing apparatus according to (5), wherein the imaging control unit determines at least one of imaging start and imaging end of the play image that is a moving image based on the play event information.
(7) The information processing device according to (5) or (6), wherein the imaging control unit selects the play image captured by a plurality of imaging devices based on the play event information.
(8) The information processing apparatus according to any one of (1) to (7), further including an additional information setting unit that sets additional information to the play image of the sport based on the play event information.
(9) The information processing apparatus according to (8), wherein the additional information setting unit sets chapter information corresponding to the play event in the play image that is a moving image.
(10) The sensor information acquisition unit is any one of the items (1) to (9), which is a communication unit that receives the sensor information from the user or an external device attached to a sports equipment used by the user. The information processing apparatus described in 1.
(11) A notification information generation unit that generates notification information for the user based on the play event information is further provided.
The information processing apparatus according to (10), wherein the communication unit transmits the notification information to the external apparatus.
(12) The sensor information acquisition unit acquires the sensor information indicating behaviors of a plurality of users who play the sport,
The information processing apparatus according to any one of (1) to (11), wherein the play event information generation unit includes information for specifying a user corresponding to the play event.
(13) A sensor device having sensor information indicating behavior of a user who plays sports, and a communication unit that transmits the sensor information;
A communication unit that receives the sensor information from the sensor device; and a play event information generation unit that detects occurrence of a play event in the sport based on the sensor information and generates play event information corresponding to the play event. An information processing system including an information processing device.
(14) The information processing apparatus further includes a notification information generation unit that generates notification information for the user based on the play event information,
The communication unit of the information processing device transmits the notification information to the sensor device,
The communication unit of the sensor device receives the notification information from the information processing device,
The information processing system according to (13), wherein the sensor device further includes an output unit that outputs the notification information to the user, and is attached to the user or a sports equipment used by the user.
(15) a function of acquiring sensor information indicating the behavior of a user who plays sports;
A program for causing a computer to realize a function of detecting occurrence of a play event in the sport based on the sensor information and generating play event information corresponding to the play event.

DESCRIPTION OF SYMBOLS 10,20 Information processing system 100,500 Sensor apparatus 110 Sensor 130,530 Communication part 200 Imaging device 210 Communication part 220 Analysis part 221 Play event information generation part 230 Control part 231 Imaging control part 232 Chapter setting part 240 Imaging part 300 Server 540 Notification output unit 541 Speaker 542 Display device 600 Notification device 610 Communication unit 650 Notification information generation unit 660 Notification DB
670 Registration Department

Claims (13)

A sensor information acquisition unit that acquires sensor information indicating a user's behavior;
An event information generation unit that detects the occurrence of an event based on the sensor information and generates event information corresponding to the event;
A controller that extracts a scene based on the event information from an image of the user,
An information processing apparatus comprising:   The information processing apparatus according to claim 1, further comprising a notification information generation unit that generates notification information for the user based on the event information.   The information processing apparatus according to claim 2, wherein the notification information is information registered in advance by the user in association with the event.   The information processing apparatus according to claim 2, wherein the notification information generation unit generates the notification information further based on the sensor information.   The information processing apparatus according to any one of claims 1 to 4, further comprising an additional information setting unit that sets additional information in the image based on the event information.   The information processing apparatus according to claim 5, wherein the additional information setting unit sets chapter information corresponding to the event in the image that is a moving image. The information processing apparatus according to claim 1, wherein the sensor information acquisition unit is a communication unit that receives the sensor information from an external device attached to the user or a sports equipment used by the user. A notification information generating unit that generates notification information for the user based on the event information;
The information processing apparatus according to claim 7, wherein the communication unit transmits the notification information to the external apparatus. The sensor information acquisition unit acquires the sensor information indicating the behavior of a plurality of users,
The information processing apparatus according to claim 1, wherein the event information generation unit includes information that specifies a user corresponding to the event.   The information processing apparatus according to claim 1, wherein the event is a play event related to sports, and the event information is play event information related to sports.   The information processing apparatus according to claim 10, wherein the scene is a play image extracted from the image of a user who plays sports. A sensor device that has sensor information indicating user behavior, and a communication unit that transmits the sensor information; and
A communication unit for receiving the sensor information from the sensor device;
An event information generation unit that detects the occurrence of an event based on the sensor information and generates event information corresponding to the event; and a control unit that extracts a scene based on the event information from an image of the user,
An information processing system including: A function to acquire sensor information indicating user behavior;
A function of detecting the occurrence of an event based on the sensor information and generating event information corresponding to the event;
A function of extracting a scene based on the event information from an image of the user,
A program to make a computer realize.

Images