JP2009118513A - Information processor, method and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To know the current state of user who is the communication counterpart and carry out processings corresponding to the state. <P>SOLUTION: A sensor is built into a cellular phone 1 and recognizes the behavior of the user having the cellular phone 1, based on sensor data. Behavior such as "walking", "running", "resting" and "riding a car" is recognized as a behavior of the user. A recognition result is included in presence state information and is transmitted to a cellular phone 2. Data on a character taking respective actions is prepared for the cellular phone 2, and an animation, corresponding to a present act of the user of the cellular phone 1 expressed by the transmitted presence state information, is displayed on an image screen of the cellular phone 1. The present invention can be applied to apparatuses communicatable with an apparatus for recognizing the behavior of the user and transmitting information containing the recognized result. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

The present invention relates to an information processing apparatus, method, and program, and more particularly, to an information processing apparatus, method, and program that can know the current state of a communication partner user and perform processing corresponding to the state. .

  "What are you doing now?". This is a phrase often used as an inquiry to the other party of the call immediately after the start of the call on a mobile phone or the like. In response to this inquiry, for example, “Getting on the train.” For example, the person who made the call tells the other party to “call again” and hangs up.

  Patent Document 1 discloses a technique for extracting the next communicable time of a user terminal from schedule information and managing it on a server.

JP-A-2005-57706

  Since this kind of exchange is common on a daily basis, if you can check the current state of the other party before making a call, it is convenient that you don't have to make a negligible exchange as described above. it is conceivable that.

  Before making a call, use the e-mail function of the mobile phone to tell you that you want to make a call, and when you get permission by replying, make a call when the other party is inconvenient This can be prevented, but it is cumbersome to send an email just to get permission.

  Also, for example, when a parent wants to know what their child is doing outside, it is not necessary to make a call or send an e-mail, but the other party (in this case, the child) There are times when I want to know the situation, and even in this case, it is convenient if the current state of the other party can be confirmed.

  According to the technique disclosed in Patent Document 1, the user can confirm the user terminal user's schedule (the time when communication is possible next) by accessing the server. The current status of the terminal user cannot be confirmed in real time.

  The present invention has been made in view of such a situation, and is intended to know the current state of a communication partner user and perform processing according to the state.

  The first information processing apparatus of the present invention recognizes a user's action based on an output of a built-in sensor, and transmits other action information indicating the recognized user's action together with information indicating the current position. In the information processing apparatus connected to the network via the network, the time of arrival at the predetermined destination of the user of the other information processing apparatus or the time until the user arrives at the predetermined destination Predicting means for predicting based on information is provided.

  In the predicting means, the predicted arrival time of the user of the other information processing apparatus at the predetermined destination or the time until the user arrives at the predetermined destination is displayed. It can be displayed with a history of actions.

  The first information processing method of the present invention recognizes a user's behavior based on the output of a built-in sensor, and transmits other behavior information representing the recognized user's behavior together with information representing the current position. In the information processing method of the information processing apparatus connected to the network via the network, the arrival time of the user of the other information processing apparatus at the predetermined destination or the time until the user arrives at the predetermined destination A predicting step of predicting the behavior based on the behavior information.

  The first program of the present invention recognizes the user's behavior based on the output of the built-in sensor, and transmits the behavior information representing the recognized user behavior together with information representing the current position. In a program for causing a computer to execute information processing in an information processing apparatus connected via a network, arrival time at a predetermined destination of a user of the other information processing apparatus or arrival at the predetermined destination The computer is caused to execute a process including a prediction step of predicting the time until the time based on the behavior information.

  The second information processing apparatus according to the present invention recognizes a user's action based on an output of a built-in sensor, and transmits the action information representing the recognized user's action together with a captured image to another information processing apparatus. In an information processing apparatus connected via a network, a character that takes the same action as the user's action of the other information processing apparatus represented by the action information is displayed in the vicinity of the transmitted image Display control means is provided.

  The second information processing method of the present invention recognizes a user's behavior based on the output of a built-in sensor, and transmits the behavior information representing the recognized user's behavior together with a captured image to another information processing device. In the information processing method of the information processing apparatus connected via the network, a character that takes the same action as the user action of the other information processing apparatus represented by the action information A display control step for displaying in the vicinity is included.

  The second program of the present invention recognizes the user's behavior based on the output of the built-in sensor, and transmits the behavior information representing the recognized user's behavior together with the captured image to another information processing apparatus. In a program for causing a computer to execute information processing in an information processing device connected via a network, a character having the same behavior as the user behavior of the other information processing device represented by the behavior information has been transmitted A computer is caused to execute processing including a display control step of displaying in the vicinity of the image.

  In the first information processing apparatus, information processing method, or program of the present invention, the arrival time of the user of another information processing apparatus at the predetermined destination, or the time until the user arrives at the predetermined destination, Predicted based on behavior information.

  In the second information processing apparatus, information processing method, or program of the present invention, a character that takes the same action as the action of the user of another information processing apparatus represented by the action information is located in the vicinity of the transmitted image. Is displayed.

  According to the present invention, it is possible to know the current state of a communication partner user.

It is a figure which shows the structural example of the communication system which concerns on one Embodiment of this invention. It is a figure which shows the example of a character. 6 is a diagram showing an example of a screen displayed on the mobile phone 2. FIG. 6 is a diagram showing an example of a screen displayed on the personal computer 3. FIG. It is a figure which shows the example of a recognition result. 2 is a block diagram illustrating a hardware configuration example of a mobile phone 1. FIG. FIG. 3 is a block diagram illustrating a software configuration example of the mobile phone 1. It is a figure which shows the example of feature data. It is a figure which shows action recognition typically. It is a figure which shows the example of sensor data, a feature-value, and a recognition result in time series. It is a figure which shows the example of a recommendation table. 5 is a flowchart for explaining presence state information transmission processing of the mobile phone 1; It is a flowchart explaining the detail of the action recognition process performed in FIG.12 S11. 5 is a flowchart for explaining communication means selection processing of the mobile phone 2; It is a figure which shows the example of the display screen of a prediction result. 10 is a flowchart for describing prediction processing of the mobile phone 2. It is a figure which shows the example of a presence status display screen. 7 is a flowchart for explaining display processing of the mobile phone 2. And FIG. 16 is a block diagram illustrating a configuration example of a personal computer.

  FIG. 1 is a diagram illustrating a configuration example of a communication system according to an embodiment of the present invention.

  In the communication system of FIG. 1, a mobile phone 1 and a mobile phone 2 are connected via a network 4 so that they can communicate with each other. Further, the cellular phone 1 and the personal computer 3 are also connected via the network 4 so that they can communicate with each other.

  The mobile phone 1 includes sensors such as an acceleration sensor and a gyro sensor, and a user who has the mobile phone 1 based on sensor data (acceleration sensor data, gyro sensor data, etc.) acquired by the built-in sensor. And presence state information including information (behavior information) representing the recognized user behavior is transmitted to the mobile phone 2 and the personal computer 3 via the network 4 respectively.

  For example, in the mobile phone 1, actions such as “walking”, “running”, “stationary”, and “riding in a car” are recognized as actions of the user of the mobile phone 1.

  On the mobile phone 2 and the personal computer 3 that have received the presence status information from the mobile phone 1, animations of characters that take actions such as “walking”, “running”, “still”, “riding in a car”, etc. Is available. When presence status information is transmitted from the mobile phone 1, an animation corresponding to the current behavior of the user of the mobile phone 1 represented by the presence status information is displayed on the respective screens of the mobile phone 2 and the personal computer 3, for example. Is displayed.

  2A to 2D are diagrams showing examples of characters for which data is prepared in the mobile phone 2 and the personal computer 3. The characters in FIGS. 2A to 2D represent “walking”, “running”, “still”, and “riding in a car”, respectively.

  The display of the character is switched as needed every time presence status information is transmitted from the mobile phone 1, for example, when the user of the mobile phone 1 starts running from a walking state, the character display is transmitted at that time. Based on the presence status information, the display of the character is also switched from the screen in FIG. 2A to the screen in FIG. 2B on the screens of the mobile phone 2 and the personal computer 3.

  By viewing such an animation, the user of the mobile phone 2 or the user of the personal computer 3 can know the current state of the user of the mobile phone 1. For example, by confirming the display of animation before making a call and preventing the user from making a call when the user of the mobile phone 1 is in a car (when the animation of FIG. 2D is displayed), The user of the telephone 2 does not have to make a call when the user of the mobile telephone 1 is inconvenient.

  FIG. 3 is a diagram illustrating an example of a screen displayed on the mobile phone 2.

  In the example of FIG. 3, “Murata”, “趙” and “Brian” have their respective devices to the right of the names “Murata”, “趙” and “Brian” registered in the address book. Characters 2A to 2C representing actions confirmed from the presence status information transmitted from are displayed. In the communication system of FIG. 1, a plurality of devices having functions similar to those of the mobile phone 1 are connected to the network 4, and as shown in FIG. 3, based on presence status information transmitted from these devices, Animations representing the actions of multiple people are displayed.

  In the example of FIG. 3, “Murata” is represented by the character 2A that it is currently running. From this display, the user of the mobile phone 2 indicates that “Murata” is busy. You can check, and if you make a call, you can decide to leave it later.

  In the example of FIG. 3, the character 2B indicates that “キ ャ ラ ク タ” is currently “still”, and from this display, the user of the mobile phone 2 has “趙” You can confirm that you are making a call and you can now decide to make a call.

  Similarly, in the example of FIG. 3, “Brian” is represented by the character 2 </ b> C as being “in a car”, and from this display, the user of the mobile phone 2 can read “Brian” "Can be confirmed in the car, and it can be determined that the requirement is not communicated by phone but by email.

  FIG. 4 is a diagram illustrating an example of a screen (window display) displayed on the personal computer 3.

  In the example of FIG. 4, an instant message window is displayed, which is confirmed from the presence status information transmitted from each device possessed by “Tamago”, “Fujio”, and “Akira”. Characters 3A to 3C representing actions are displayed.

  In the example of FIG. 4, “Tamago” is represented by the character 3A as being “in a car”, and “Fujio” is a character representing that it is currently running. It is represented by 3B.

  The character 3C indicates that “Akira” is currently “still”. In the example of FIG. 4, the letters “K-005 Conference Room” are displayed next to the name “Akira”. This is because “Akira” is included in the presence status information from your device. It is displayed based on the information of whereabouts sent.

  In this way, the presence status information can be transmitted to other devices including not only information indicating the user's behavior but also various information.

  In the example of FIG. 4, the setting status of “Tamago”, “Fujio”, and “Akira” is “Not notified of location”, “Not notified”, “Notified all” at the right end of the window. Is displayed, but this is the setting status of what information is to be displayed on the other party (the device that received the presence status information) out of the information included and transmitted in the presence status information. Represents.

  For example, since “Akira” is set to “Notify all”, in the personal computer 3, based on information transmitted from the device “Akira” included in the presence status information, “Akira” Not only the character 3C representing the action but also the whereabouts are displayed. That is, the user of the device that sent the presence status information can set how much information is disclosed on the device that received the presence status information, and the presence status information includes information indicating the setting status. included.

  The functions of the mobile phone 2 and the personal computer 3 for displaying the screens shown in FIGS. 3 and 4 based on the transmitted presence status information as described above are also provided in the mobile phone 1. On the other hand, the function of the mobile phone 1 that recognizes an action based on an output from a built-in sensor and transmits presence state information including information representing the recognized action to another device is the function of the mobile phone 2 or a personal computer. 3 is also available.

  Therefore, the user of the mobile phone 1 can also check the status of the user of the mobile phone 2 or the user of the personal computer 3 from the screen displayed on the mobile phone 1.

  In the above example, it is assumed that the device from which the presence status information is transmitted can recognize four actions: “walking”, “running”, “still”, and “riding in a car”. However, based on sensor data obtained by various sensors such as voice picked up by microphones as well as acceleration sensors and gyro sensors, as well as current time, etc., `` walking '', `` running '', `` stationary '', `` Actions other than “in the car” may be recognized.

  FIG. 5 is a diagram illustrating an example of a recognition result.

For example, as shown on the left side of FIG. 5, it is obtained that the volume of the sound collected by the microphone is “quiet”, and that the current time is obtained from the output of the clock is “daytime”. In the case of the state shown in the status S ₁ obtained from the outputs of the acceleration sensor and the gyro sensor as “no movement”, the current action (state) of the user is determined as “relaxing” and represents it. Presence status information including information is transmitted.

  Since the device that has received the presence status information is “relaxing”, it is determined that the user who sent the presence status information can now chat, and the application that manages the chat is automatically (Without user operation). As will be described later, in a device that has received presence status information, a tool for communicating with the user who transmitted the presence status information is automatically set according to the current status of the user who transmitted the presence status information. It is made to be selected.

Further, as shown in the center of FIG. 5, for example, it is obtained that the sound volume collected by the microphone is “quiet”, and that the current time is obtained from the clock output “night”, Furthermore, in the case of the state shown in the status S ₂ obtained from the outputs of the acceleration sensor and the gyro sensor as “no movement”, the current state of the user is determined as “sleeping”, and information indicating it Presence status information including is transmitted.

  In the device that has received the presence status information, since it is “sleeping”, it is determined that the user who has transmitted the presence status information cannot now chat. In this case, for example, it is determined that mail can be performed, and an application (mailer) that manages mail is started.

Similarly, as shown on the right side of FIG. 5, for example, in the case of the state shown in the status S ₃ that is determined as “dashing rapidly” from the outputs of the acceleration sensor and the gyro sensor, information representing it is included. Presence status information is transmitted.

  Since the device that has received the presence status information is “dashing rapidly”, it is determined that the user who has transmitted the presence status information cannot now chat or make a call. . Also in this case, for example, it is determined that mail can be performed, and the mailer is activated.

  The operation of each device that performs the transmission / reception of presence status information and the processing according to the status of the other user will be described later with reference to flowcharts.

  FIG. 6 is a block diagram illustrating a hardware configuration example of the mobile phone 1 of FIG.

  The sensor unit 11 includes an acceleration sensor, a gyro sensor, a GPS (Global Positioning System) sensor, and the like. The sensor unit 11 performs measurement at a predetermined cycle, and obtains sensor data such as acceleration sensor data, gyro sensor data, and GPS sensor data as measurement results. The data is output to the action recognition unit 12 and the main processing unit 13. The acceleration sensor that constitutes the sensor unit 11 can measure, for example, accelerations in three axial directions orthogonal to each other. The gyro sensor can measure angular velocities around three axes orthogonal to each other, for example.

  As will be described in detail later, the behavior recognition unit 12 extracts features from the sensor data supplied from the sensor unit 11, and based on the extracted time series of the features, user behaviors are prepared in advance. Recognize by referring to (Hidden Markov Model). The behavior recognition unit 12 outputs information representing the recognition result to the main processing unit 13.

  The main processing unit 13 controls the overall operation of the mobile phone 1, for example, generates presence state information including information representing the action recognition result supplied from the action recognition unit 12, and uses the generated presence state information. The communication unit 14 transmits the information to a previously registered device. As the transmission destination, for example, a device registered in the address book or a device registered in an application for managing instant messages is selected.

  Further, the main processing unit 13 receives the presence state information transmitted from the other device at the communication unit 14, and when it is supplied, confirms the behavior of the user of the other device, and confirms the confirmed behavior. The character to be displayed is displayed on the display unit 15. Further, the main processing unit 13 selects a tool (communication means) to be used for communication with the user according to the behavior of the user of the other device represented by the presence state information transmitted from the other device, Start an application that manages the selected communication tool.

  As will be described later, the main processing unit 13 is provided with a recommendation table that indicates correspondence between each action and a communication tool suitable for use in each action. A communication tool suitable for the current behavior of the user of the device is selected.

  The communication unit 14 transmits information supplied from the main processing unit 13 to other devices via the network 4 and receives information transmitted from other devices via the network 4. 13 is output.

  The display unit 15 includes an LCD (Liquid Crystal Display) or the like, and displays a character (screen on which the character is displayed) and the like according to control by the main processing unit 13.

  The storage unit 16 includes an HDD (Hard Disk Drive) or the like, and stores various data such as data for displaying characters.

  FIG. 7 is a block diagram illustrating a software configuration example (functional configuration example) of the mobile phone 1. 7 is realized by executing a predetermined program by the main processing unit 13 in FIG.

  Hereinafter, the state (behavior) of the user of the mobile phone 1 handled in the mobile phone 1 is referred to as “My presence state”, and the user of another device such as the mobile phone 2 or the personal computer 3 who corresponds to the user of the mobile phone 1. Is the Buddy presence state (conversely, for example, when viewed from the mobile phone 2, the user of the mobile phone 2 is in the My presence state and corresponds to the user of the mobile phone 2, such as the mobile phone 1 and the personal computer 3. The user status of the other device is the Buddy presence status).

  As shown in FIG. 7, the mobile phone 1 basically performs a transmission processing unit 21 that transmits presence status information to other devices, and various processes based on presence status information transmitted from other devices. And a reception processing unit 22 to be performed.

  The transmission processing unit 21 includes a feature extraction processing unit 31, an action recognition processing unit 32, an HMM DB 33, a My presence state recognition processing unit 34, and a presence state information transmission unit 35. Among these, the feature extraction processing unit 31, the behavior recognition processing unit 32, and the HMM DB 33 shown by being surrounded by a solid line are realized by the behavior recognition unit 12 of FIG.

  For example, the feature extraction processing unit 31 extracts features from acceleration sensor data supplied from the acceleration sensor and gyro sensor data supplied from the gyro sensor, and the feature recognition processing unit 32 uses the feature data representing the extracted features. Output to.

  Specifically, as shown in FIG. 8A, the feature extraction processing unit 31 obtains a walking frequency (Impulse pitch) as one of the features from the acceleration sensor data. In the sample result of FIG. 8A, the horizontal axis represents time (the number of samples of the acceleration sensor 11), and the vertical axis represents frequency (Hz). The frequency measured by the acceleration sensor when a person walks is typically 2 Hz (2 steps per second), so the HMM refers to the “walk” as the recognition result from the time series of the 2 Hz measurement results. To be obtained.

  Further, as shown in FIG. 8B, the feature extraction processing unit 31 obtains the walking strength and the impact magnitude as one of the features from the acceleration sensor data. In the sample result of FIG. 8B, the horizontal axis represents time and the vertical axis represents strength.

  Further, as shown in FIG. 8C, the feature extraction processing unit 31 obtains the gravity axis from the acceleration sensor data, and obtains the rotation angle (Heading) around the gravity axis obtained from the gyro sensor data as one of the features. When the acceleration sensor is in a state of detecting acceleration in the horizontal direction with respect to the gravity axis, the output is 0. On the other hand, when the acceleration sensor is in a state of detecting acceleration in a direction other than the horizontal direction with respect to the gravity axis, Since the predetermined value is measured, as described above, when it is possible to measure the acceleration in the three-axis direction, the gravity axis can be obtained from the measurement result of the acceleration in each direction. . In the sample result of FIG. 8C, the horizontal axis represents time and the vertical axis represents the rotation angle.

  For example, various features as described above are extracted by the feature extraction processing unit 31 based on the sensor data. The feature data representing the extracted features is sequentially output to the action recognition processing unit 32 every time it is extracted.

  The behavior recognition processing unit 32 accumulates a predetermined amount of feature data supplied from the feature extraction processing unit 31, and prepares the behavior of the user of the mobile phone 1 in the HMM DB 33 based on the time series of the accumulated feature data. Recognize by referring to the HMM.

  The HMM DB 33 is a model used for recognizing “walking” generated based on features extracted from acceleration sensor data and gyro sensor data when a person is walking. A model used to recognize "running" generated based on features extracted from acceleration sensor data and gyro sensor data, and extracted from acceleration sensor data and gyro sensor data when a person is stationary A model used to recognize "stillness" generated based on the features that were generated, based on features extracted from acceleration sensor data and gyro sensor data when a person is in a car, Models and the like used for recognizing “in the car” are prepared in advance.

  Information representing the recognition results of actions such as “walking”, “running”, “still”, and “riding in a car” recognized by the action recognition processing unit 32 is output to the My presence state recognition processing unit 34.

  FIG. 9 is a diagram schematically illustrating behavior recognition by the behavior recognition unit 12.

  The acceleration sensor data and the gyro sensor data as raw data are calibrated, and the above-described walking pitch and walking strength are obtained from the data obtained by the calibration. Gravity and direction of travel are acquired as a low level context. Moreover, based on those acquired feature quantities, the HMM is referred to, and the user's behavior (high level context) is statistically recognized.

  Note that the action recognition algorithm is not limited to that described above, and may be performed by various algorithms.

  FIG. 10 is a diagram showing examples of sensor data, feature amounts, and recognition results in time series.

  In the example of FIG. 10, action recognition results are acquired in the order of “walking”, “running”, “walking”, “running”, “still”, “walking”, and “still”. For example, the measurement time of the sensor data used for obtaining the recognition result is added to each such recognition result, and used as information representing the behavior in the presence state information.

  Returning to the description of FIG. 7, the My presence state recognition processing unit 34 generates presence state information including the recognition result of the behavior supplied from the behavior recognition processing unit 32, and uses the generated presence state information as the presence state information transmission unit. 35. The My presence state recognition processing unit 34 includes GPS sensor data and the like in the presence state information as necessary.

  Further, the My presence state recognition processing unit 34 causes the display unit 15 to display an animation representing the state of the user of the mobile phone 1 based on the behavior recognition result supplied from the behavior recognition processing unit 32 as appropriate.

  The presence status information transmission unit 35 controls the communication unit 14 to transmit the presence status information generated and supplied by the My presence status recognition processing unit 34 to other devices via the network 4.

  The reception processing unit 22 includes a presence state information acquisition unit 41, a buddy presence state recognition processing unit 42, a communication means selection processing unit 43, a table DB 44, and a behavior prediction processing unit 45.

  The presence status information acquisition unit 41 receives presence status information transmitted from another device via the network 4 and outputs it to the Buddy presence status recognition processing unit 42.

  The Buddy presence status recognition processing unit 42 recognizes the Buddy presence status (the status of the user of another device) based on the presence status information supplied from the presence status information acquisition unit 41, and displays an animation representing the Buddy presence status. Display on the unit 15.

  Further, the Buddy presence state recognition processing unit 42 outputs information representing the recognized Buddy presence state to the communication means selection processing unit 43 and the behavior prediction processing unit 45. When the presence status information transmitted from the other device includes GPS sensor data representing the current position of the user of the other device, the Buddy presence status recognition processing unit 42 also includes the GPS sensor data as a behavior prediction processing unit. Output to 45. As will be described later, in the behavior prediction processing unit 45, based on the Buddy presence status and GPS sensor data, the estimated time to the destination of the user of the device that transmitted the GPS sensor data and the like included in the presence status information, etc. Is required.

  Further, in the Buddy presence status recognition processing unit 42, a tool used for communication with the user of the device that has transmitted the presence status information is selected by the communication means selection processing unit 43, and information indicating the selected tool has been supplied. When appropriate, a message indicating which communication tool is selected is displayed on the display unit 15 and presented to the user of the mobile phone 1.

  Based on the Buddy presence status recognized by the Buddy presence status recognition processing unit 42, the communication means selection processing unit 43 creates a recommendation table stored in the table DB 44 that is suitable as a communication tool with users of other devices. Browse and select to launch an application that manages the selected communication tool.

  FIG. 11 is a diagram illustrating an example of a recommendation table stored in the table DB 44.

  In the example of the recommendation table in FIG. 11, the behavior of each time zone (weekday day / night, holiday day / night) corresponds to a communication tool suitable when the user of another device is taking the action. It is attached. In the example of FIG. 11, as communication tools prepared for the mobile phone 1, telephone, e-mail (e-mail reply), SMS (Short Message Service) (SMS reply), and chat are shown. .

  In FIG. 11, “◯” indicates that the communication tool is suitable, and “X” indicates that it is not suitable. “Δ” represents an intermediate point.

  For example, when the day is a weekday daytime and the recognized Buddy presence status is walking ("walking"), other than chat, telephone, e-mail (email reply), SMS (SMS reply) Is considered a suitable communication tool.

  Therefore, when the time zone is daytime on weekdays and the recognized Buddy presence status is recognized based on the presence status information transmitted from another device, the communication means selection is performed. The processing unit 43 selects a communication tool of telephone, e-mail (reply of e-mail), or SMS (reply of SMS), and starts an application for managing it. Thus, when it is determined that a plurality of communication tools are suitable, for example, a tool with a high priority preset by the user is selected.

  Such a recommendation table is referred to, and the communication means selection processing unit 43 selects a tool corresponding to the current state of the user of the other device as the communication partner. In addition, in the example of FIG. 11, as a user's action of another apparatus, while walking ("walking" mentioned above), running ("running" mentioned above), sitting, standing, vehicle (described above) "Getting on a car") is shown.

  In addition, as information representing the correspondence between the behavior and the communication tool, various information such as a probability model in which the behavior is input and the communication tool corresponding to the behavior is output is not the table-like information illustrated in FIG. It may be prepared.

  Returning to the description of FIG. 7, the behavior prediction processing unit 45 is based on the Buddy presence status and the GPS sensor data until it arrives at the destination of the user of the device that has transmitted the GPS sensor data and the like included in the presence status information. The predicted time and the predicted time of arrival at the destination are obtained. For example, the obtained predicted time and time are displayed on the map together with the action history of the user.

  6 and 7 includes not only the mobile phone 1 but also the mobile phone 2 and the personal computer 3. In the following, the configurations shown in FIGS. 6 and 7 will be described with reference to the configurations of the mobile phone 2 and the personal computer 3 as appropriate.

  Next, operations of the mobile phone 1 and the mobile phone 2 will be described with reference to flowcharts.

  In the following, a case where the mobile phone 1 is a device on the presence status information transmission side and the mobile phone 2 is a device on the presence status information reception side will be described. This is also performed in the mobile phone 2 and the personal computer 3. Further, the processing performed by the mobile phone 2 is also performed by the mobile phone 1 and the personal computer 3 as appropriate.

  First, processing of the mobile phone 1 that transmits presence status information will be described with reference to the flowchart of FIG.

  Measurement is performed by various sensors in accordance with control by the main processing unit 13 of the mobile phone 1, and when sensor data as measurement results is supplied, behavior recognition processing is performed by the behavior recognition unit 12 in step S1. Details of the action recognition process will be described later with reference to the flowchart of FIG. 13, but the current action of the user of the mobile phone 1 is recognized by this process. Information representing the recognized behavior is output from the behavior recognition processing unit 32 to the My presence state recognition processing unit 34.

  In step S2, the My presence state recognition processing unit 34 generates presence state information including information representing the recognition result of the behavior supplied from the behavior recognition processing unit 32 and GPS sensor data, and the generated presence state information. Is output to the presence status information transmission unit 35.

  In step S3, the presence status information transmission unit 35 transmits the presence status information generated by the My presence status recognition processing unit 34 to the mobile phone 2 via the network 4 and ends the process.

  By repeatedly performing such processing at a predetermined cycle, the user of the mobile phone 2 that has received the presence status information transmitted from the mobile phone 1 can check the status of the user of the mobile phone 1 in real time. it can.

  Next, details of the action recognition process performed in step S1 of FIG. 12 will be described with reference to the flowchart of FIG.

  In step S11, the feature extraction processing unit 31 of the action recognition unit 12 determines the walking pitch and the walking strength as described above from the acceleration sensor data that is the output of the acceleration sensor and the gyro sensor data that is the output of the gyro sensor. Extract features such as gravity and direction of travel. Feature data representing features extracted by the feature extraction processing unit 31 is output to the behavior recognition processing unit 32.

  In step S12, the behavior recognition processing unit 32 accumulates a predetermined amount of feature data supplied from the feature extraction processing unit 31, and based on the time series of the accumulated feature data, the behavior recognition processing unit 32 Recognize by referring to the HMM prepared in the HMM DB 33. The action recognition processing unit 32 outputs the recognition result to the My presence state recognition processing unit 34 and ends the process. Thereafter, the processing returns to step S1 in FIG. 12, and the subsequent processing is performed.

  Next, processing of the mobile phone 2 for selecting a communication tool will be described with reference to the flowchart of FIG. This process is a process performed corresponding to the process of FIG.

  In step S21, the presence status information acquisition unit 41 of the mobile phone 2 determines whether or not presence status information has been transmitted, and waits until it is determined that it has been transmitted.

  If the presence status information acquisition unit 41 determines in step S21 that the presence status information has been transmitted, the presence status information acquisition unit 41 proceeds to step S22. Presence status information transmitted from the mobile phone 1 is output from the presence status information acquisition unit 41 to the Buddy presence status recognition processing unit 42.

  In step S22, the Buddy presence status recognition processing unit 42 recognizes the Buddy presence status (the status of the user of the mobile phone 1) based on the presence status information supplied from the presence status information acquisition unit 41, and recognizes the recognized Buddy presence. Information representing the state is output to the communication means selection processing unit 43.

  In step S23, the communication means selection processing unit 43 stores in the table DB 44 what is suitable as a communication tool with the user of the mobile phone 1 based on the Buddy presence status recognized by the Buddy presence status recognition processing unit 42. Select and refer to the recommendation table.

  In step S24, the communication means selection processing unit 43 determines whether or not an automatic selection mode for automatically selecting a communication tool is set. If it is determined that the automatic selection mode is set, the process proceeds to step S25. move on.

  In step S25, the communication means selection processing unit 43 uses the communication tool selected in step S23 when the user of the mobile phone 2 communicates by exchanging messages with the user of the mobile phone 1. Set as a tool. If it is determined in step S24 that the automatic selection mode is not set, the process of step S25 is skipped.

  In step S26, the Buddy presence state recognition processing unit 42 displays an animation representing the state of the user of the mobile phone 1 recognized in step S22. For example, a Buddy presence state display screen as shown in FIG. Display.

  When the Buddy presence status display screen is displayed, for example, when the user of the mobile phone 2 instructs to start exchanging messages with the user of the mobile phone 1, in step S27, the communication means The selection processing unit 43 activates an application that manages the communication tool set in step S25.

  Thereby, the user of the mobile phone 2 can communicate with the user of the mobile phone 1 with a tool suitable for the current state of the user of the mobile phone 1.

  In the above, the process of selecting the communication tool based on the Buddy presence status that is the user status of the mobile phone 1 obtained based on the presence status information has been mainly described. Next, based on the Buddy presence status, A process for predicting the time when the user of the mobile phone 1 arrives at the predetermined destination or the time until the user arrives at the predetermined destination will be described.

  FIG. 15 is a diagram illustrating an example of a screen displayed on the mobile phone 2 based on the prediction result. FIG. 15 shows an example in which the time until the user of the mobile phone 1 arrives at the destination is predicted.

  In the example of FIG. 15, a map in a predetermined range is displayed and the destination P is set. The destination P is selected from the map by the user of the mobile phone 2, for example.

  In the example of FIG. 15, characters representing actions taken by the user of the mobile phone 1 at each place are displayed at each place on the map. The display position of the character on the map is a position obtained based on the GPS sensor data transmitted from the mobile phone 1 included in the presence state information, and the character is represented by the presence state information as described above. The mobile phone 1 is selected based on the state of the user.

  The character 61 displayed on the screen of FIG. 15 represents that the user of the mobile phone 1 was in the car when the user of the mobile phone 1 was at the place where the character 61 was displayed. This indicates that when the user of the mobile phone 1 is at the place where it is displayed, the user of the mobile phone 1 is stationary. Characters 63, 64, and 65 represent that the user of the mobile phone 1 was walking when the user of the mobile phone 1 was at the place where each character was displayed. This indicates that the user of the mobile phone 1 was running when the user of the mobile phone 1 was present at the place where it was placed.

  The character 67 represents the current position of the user of the mobile phone 1 and the action thereof. Under the character 67, a message “currently: moving by car” indicating the current action of the user of the mobile phone 1 is displayed. Yes.

  Further, in the example of FIG. 15, “arrival” indicating the time until the user of the mobile phone 1 arrives at the destination P predicted by the mobile phone 2 on the destination P set on the map. The message “Schedule: 10 minutes later” is displayed. This predicted time is, for example, the distance from the current position of the user of the mobile phone 1 to the destination P and the speed toward the destination P (for example, GPS that is sequentially transmitted from the mobile phone 1 in the presence status information) It is obtained from the speed obtained from the change amount of the position represented by the sensor data or the expected speed of the car).

  For example, when the user of the mobile phone 1 does not come at the time of waiting even though the user of the mobile phone 1 is waiting at the destination, the user of the mobile phone 2 displays such a screen on the mobile phone 2. By using and confirming, it is possible to confirm that the user of the mobile phone 1 is heading to the destination and the time until arrival. That is, the user of the mobile phone 2 does not have to make an inquiry about the current state to the user of the mobile phone 1 by making a call from the mobile phone 2.

  Here, with reference to the flowchart of FIG. 16, the process of the mobile phone 2 for displaying a screen as shown in FIG. 15 will be described. In the mobile phone 1, processing similar to the processing in FIG. 12 is performed, and presence status information including GPS sensor data is repeatedly transmitted to the mobile phone 2.

  In step S41, the presence status information acquisition unit 41 of the mobile phone 2 determines whether or not presence status information has been transmitted, and waits until it is determined that it has been transmitted.

  If the presence status information acquisition unit 41 determines in step S41 that the presence status information has been transmitted, the process proceeds to step S42. Presence status information transmitted from the mobile phone 1 is output from the presence status information acquisition unit 41 to the Buddy presence status recognition processing unit 42.

  In step S42, the Buddy presence status recognition processing unit 42 recognizes the Buddy presence status (the status of the user of the mobile phone 1) based on the presence status information supplied from the presence status information acquisition unit 41, and recognizes the recognized Buddy presence. Information indicating the state is included in the presence state information and output to the behavior prediction processing unit 45 together with the GPS sensor data transmitted from the mobile phone 1.

  In step S43, the behavior prediction processing unit 45 calculates, for example, a cellular phone from the behavior of the user of the cellular phone 1 recognized by the Buddy presence state recognition processing unit 42 and the current location of the user of the cellular phone 1 obtained from the GPS sensor data. The arrival time of the user of the mobile phone 1 at the destination set by the user 2 or the time until arrival at the destination is predicted by calculation. For predicting the arrival time, in addition to the distance from the current position of the user of the mobile phone 1 to the destination P and the speed toward the destination P, which is used when obtaining the time until arrival, The current time is also used.

  In step S44, the behavior prediction processing unit 45 causes the display unit 15 to display the arrival time of the user of the mobile phone 1 obtained in step S43 or the time until the user arrives at the destination. Present to the user. This presentation is performed, for example, by a screen display as shown in FIG.

  In step S45, the Buddy presence state recognition processing unit 42 causes the display unit 15 to display a Buddy presence state display screen on which an animation representing the state of the user of the mobile phone 1 recognized in step S42 is displayed.

  For example, when the presence of the presence status information display screen is displayed and the user of the mobile phone 2 instructs to communicate with the user of the mobile phone 1, the communication means selection processing unit 43 is instructed. Then, an application for managing the communication tool is activated to enable communication with the user of the mobile phone 1. When the automatic selection mode in which the mobile phone 2 automatically selects a communication tool is set, communication with the user of the mobile phone 1 is automatically performed by the process described with reference to FIG. A state that can be taken may be set.

  Next, processing of the mobile phone 2 that displays an animation representing the state of the user of the mobile phone 1 side by side with an image (moving image) that is captured by the mobile phone 1 and transmitted together with presence status information will be described.

  In this example, the sensor unit 11 of the mobile phone 2 is also provided with a camera, and an image captured thereby is displayed side by side with an animation representing the user's own behavior of the mobile phone 2 recognized by the mobile phone 2. It is made to be done. That is, from the displayed screen, the user of the mobile phone 2 confirms the My presence state, which is his / her own presence state, and the Buddy presence state of the user of the mobile phone 1, etc., and images taken by the respective devices. Can be confirmed.

  FIG. 17 is a diagram illustrating an example of a presence status display screen displayed on the mobile phone 2.

  In the presence status display screen of FIG. 17, a user is selected from an image photographed by the camera of the mobile phone 2 and an image photographed by the camera of another device including the mobile phone 1 displayed on the right side. A Stream View 81 for displaying the image selected by is displayed.

  In addition, the presence status display screen includes a My presence status display unit 82 that displays information (actions and captured images) related to the user of the mobile phone 2, and a Buddy list 83 that displays a list of information related to other devices. It is displayed.

  Among these, the My presence state display unit 82 includes an animation display unit 82A that displays an animation representing the user's behavior of the mobile phone 2 recognized by the mobile phone 2, and an image captured by the camera of the mobile phone 2. The image display unit 82B is displayed. The animation displayed on the animation display unit 82A is displayed according to the recognition result of the action of the user of the mobile phone 2 performed at a predetermined cycle.

  In the example of FIG. 17, the Buddy list 83 displays a Buddy presence status display unit 83-1 for displaying information about the user of the mobile phone 1 and information about users of devices other than the mobile phone 1. A Buddy presence status display section 83-2 is displayed. That is, in the example of FIG. 17, the mobile phone 2 communicates with devices other than the mobile phone 1 in addition to the mobile phone 1 and is displayed based on presence status information transmitted from these devices. An example of the screen is shown.

  The Buddy presence status display unit 83-1 is an image displayed by an animation display unit 83-1A on which an animation representing the behavior of the user of the mobile phone 1 recognized by the mobile phone 1 is displayed, and an image taken by the camera of the mobile phone 1. Is displayed on the image display unit 83-1B. The Buddy presence status display unit 83-2 is shot by an animation display unit 83-2A that displays an animation representing the user's behavior of the device recognized by the other device, and a camera of the other device. The image display unit 83-2B displays an image. The animations displayed on the animation display units 83-1A and 83-2A are also displayed according to the recognition result of the user's behavior of each device, which is represented by the presence state information transmitted from each device.

  The user of the mobile phone 2 selects, for example, an image displayed on the image display unit 83-1B of the Buddy presence status display unit 83-1, so that the same image as the image displayed there is displayed in the Stream View 81. It can be enlarged and displayed. In the example of FIG. 17, the same image as that captured by the mobile phone 2 and displayed on the image display unit 82 </ b> B is enlarged and displayed on the Stream View 81.

  On a screen that allows users to view images that have been shot remotely and sent over a network, the user who is viewing the screen cannot usually confirm what action the photographer is taking. In this way, the animation representing the action of the photographer based on the presence status information transmitted from the device that is shooting and the image being shot are displayed side by side (close to each other), thereby displaying the screen. The viewing user can confirm not only the image being photographed but also the action of the photographer.

  Here, with reference to the flowchart of FIG. 18, the process of the mobile phone 2 for displaying a screen as shown in FIG. 17 will be described. Note that the mobile phone 1 performs the same processing as the processing in FIG. 12, and the presence status information is repeatedly transmitted to the mobile phone 2 together with the photographed image.

  In step S61, the presence status information acquisition unit 41 of the mobile phone 2 determines whether or not presence status information has been transmitted, and waits until it is determined that it has been transmitted.

  If the presence status information acquisition unit 41 determines in step S61 that the presence status information has been transmitted, the presence status information acquisition unit 41 proceeds to step S62. For example, presence status information transmitted from the mobile phone 1 is output from the presence status information acquisition unit 41 to the Buddy presence status recognition processing unit 42. In addition, an image taken by another device such as the mobile phone 1 and transmitted together with the presence state information is also supplied to the Buddy presence state recognition processing unit 42.

  In step S 62, the Buddy presence status recognition processing unit 42 recognizes the Buddy presence status (the status of the user of the mobile phone 1) based on the presence status information supplied from the presence status information acquisition unit 41.

  In step S63, measurement is performed by various sensors constituting the sensor unit 11, and when sensor data as a measurement result is supplied, the behavior recognition unit 12 of the mobile phone 2 performs a behavior recognition process. That is, processing similar to that described with reference to FIG. 13 is performed in the mobile phone 2, and the recognition result (My presence state) is output to the My presence state recognition processing unit 34.

  In step S <b> 64, the My presence state recognition processing unit 34 displays an animation representing the My presence state at a predetermined position on the display unit 15 and displays an image taken by the mobile phone 2 itself. Similarly, the Buddy presence state recognition processing unit 42 displays an animation representing the Buddy presence state at a predetermined position on the display unit 15 and also displays an image captured by another device. As a result, a presence status display screen as shown in FIG. 17 is displayed.

  By exchanging messages with the user of the mobile phone 1 while the presence status display screen is displayed, the user of the mobile phone 2 can, for example, take actions of the user of the mobile phone 1 with respect to the shooting target of the user of the mobile phone 1. You can communicate by message while watching.

  In the above description, it is assumed that the user's action recognition process is performed in the device on the transmission side of the presence status information, and the recognition result is included in the presence status information and transmitted to the device on the reception side. It may be transmitted to the receiving device, and the receiving device may perform a process of recognizing the action based on the sensor data (the action of the user of the transmitting device).

  The series of processes described above can be executed by hardware, but can also be executed by software. In this case, the apparatus for executing the software is constituted by a personal computer as shown in FIG. 19, for example.

  In FIG. 19, a CPU (Central Processing Unit) 101 performs various processes according to a program stored in a ROM (Read Only Memory) 102 or a program loaded from a storage unit 108 to a RAM (Random Access Memory) 103. Execute. The RAM 103 also stores data necessary for the CPU 101 to execute various processes as appropriate.

  The CPU 101, ROM 102, and RAM 103 are connected to each other via a bus 104. An input / output interface 105 is also connected to the bus 104.

  The input / output interface 105 includes an input unit 106 including a keyboard and a mouse, a display including an LCD (Liquid Crystal Display), an output unit 107 including a speaker, a storage unit 108 including a hard disk, and a network. A communication unit 109 that performs all communication processes is connected.

  A drive 110 is also connected to the input / output interface 105 as necessary. A removable medium 111 including a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory is appropriately installed in the drive 110, and a computer program read therefrom is installed in the storage unit 108 as necessary.

  When a series of processing is executed by software, a program constituting the software may execute various functions by installing a computer incorporated in dedicated hardware or various programs. For example, it is installed from a network or a recording medium into a general-purpose personal computer or the like.

  As shown in FIG. 19, this recording medium is distributed to provide a program to the user separately from the main body of the apparatus, and includes a magnetic disk (including a flexible disk) on which the program is recorded, an optical disk (CD- Removable media 111 composed of ROM (Compact Disk-Read Only Memory), DVD (Digital Versatile Disk), magneto-optical disk (including MD (registered trademark) (Mini-Disk)), or semiconductor memory In addition, it is configured by a ROM 102 on which a program is recorded and a hard disk included in the storage unit 108 provided to the user in a state of being incorporated in the apparatus main body in advance.

  In the present specification, each step includes not only processing performed in time series according to the described order but also processing executed in parallel or individually, although not necessarily performed in time series.

  Further, in this specification, the system represents the entire apparatus constituted by a plurality of apparatuses.

  1, 2 mobile phone, 3 personal computer, 11 sensor unit, 12 action recognition unit, 13 main processing unit, 14 communication unit, 15 display unit, 16 storage unit, 31 feature extraction processing unit, 32 action recognition processing unit, 33 HMM DB, 34 My presence status recognition processing unit, 35 presence status information transmission unit, 41 presence status information acquisition unit, 42 Buddy presence status recognition processing unit, 43 communication means selection processing unit, 44 table DB, 45 behavior prediction processing unit

Claims (7)

A user's behavior is recognized based on the output of the built-in sensor, and the behavior information indicating the recognized user's behavior is connected to another information processing apparatus that transmits the information indicating the current position together with information via the network. In an information processing device,
An information processing apparatus comprising: a prediction unit that predicts an arrival time of a user of the other information processing apparatus at a predetermined destination or a time until the user arrives at the predetermined destination based on the behavior information. The prediction means calculates the predicted arrival time of the user of the other information processing apparatus at the predetermined destination or the time until the user arrives at the predetermined destination. The information processing apparatus according to claim 1, wherein the information processing apparatus is displayed together with the history of the information processing apparatus. A user's behavior is recognized based on the output of the built-in sensor, and the behavior information indicating the recognized user's behavior is connected to another information processing apparatus that transmits the information indicating the current position together with information via the network. In the information processing method of the information processing apparatus,
An information processing method including a prediction step of predicting an arrival time of a user of the other information processing apparatus at a predetermined destination or a time until the user arrives at the predetermined destination based on the behavior information. A user's behavior is recognized based on the output of the built-in sensor, and the behavior information indicating the recognized user's behavior is connected to another information processing apparatus that transmits the information indicating the current position together with information via the network. In a program for causing a computer to execute information processing in an information processing apparatus,
Causing the computer to execute a process including a prediction step of predicting, based on the behavior information, the arrival time of the user of the other information processing apparatus at the predetermined destination or the time until the user arrives at the predetermined destination program. Information processing connected to another information processing apparatus that recognizes user behavior based on the output of the built-in sensor and transmits behavior information representing the recognized user behavior together with a captured image via a network In the device
An information processing apparatus comprising: display control means for displaying a character that takes the same action as a user action of the other information processing apparatus represented by the action information in the vicinity of the transmitted image. Information processing connected to another information processing apparatus that recognizes user behavior based on the output of the built-in sensor and transmits behavior information representing the recognized user behavior together with a captured image via a network In the information processing method of the device,
An information processing method including a display control step of displaying a character that takes the same action as a user of the other information processing apparatus represented by the action information in the vicinity of the transmitted image. Information processing connected to another information processing apparatus that recognizes user behavior based on the output of the built-in sensor and transmits behavior information representing the recognized user behavior together with a captured image via a network In a program for causing a computer to execute information processing in a device,
A program that causes a computer to execute a process including a display control step of displaying a character that takes the same action as the action of the user of the other information processing apparatus represented by the action information in the vicinity of the transmitted image.

Images