JP2016181234A - Information processing apparatus, program, portable terminal, and information processing system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an information processing apparatus that can estimate a relative position of a user to a display part without using a measurement result from a dedicated measurement instrument, a program, a portable terminal, and an information processing system.SOLUTION: There is provided an information processing apparatus comprising: a cooperation processing part that performs determination related to establishment of information cooperation between a portable terminal and a display part; and a position estimation part that estimates a relative position of a user carrying the portable terminal to the display part on the basis of a change in position of the portable terminal detected by the portable terminal and a contact position of the portable terminal with the display part detected by the display part when the establishment of information cooperation is determined.SELECTED DRAWING: Figure 2

Description

  The present invention relates to an information processing device, a program, a portable terminal, and an information processing system.

  Conventionally, a digital signage that provides information to a user using a display is installed in a facility such as a station, an airport, or a shopping mall. For example, Patent Literature 1 describes a technique for displaying content related to a selected content on a display when any of a plurality of contents displayed on the display is selected by a user.

  A technique for enabling intuitive interaction between content displayed on a display and a user has also been proposed. For example, in Patent Literature 2, when an approach of a user's hand to any one of a plurality of pieces of outline information displayed on the display is detected by an ultrasonic sensor, the information is associated with the outline information. A technique for displaying detailed information on a display is described.

Japanese Patent Laid-Open No. 2015-15005 JP 2006-301534 A

  However, the technique described in Patent Document 2 complicates the system configuration. For example, in the technique described in Patent Document 2, a dedicated measuring device is required for each display in order to estimate the relative position of the user with respect to the display.

  Therefore, the present invention has been made in view of the above problems, and an object of the present invention is to estimate the relative position of the user with respect to the display unit without using a measurement result obtained by a dedicated measuring device. It is an object of the present invention to provide a new and improved information processing apparatus, program, portable terminal, and information processing system.

  In order to solve the above-described problem, according to an aspect of the present invention, there is provided a cooperation processing unit that performs determination related to establishment of information cooperation between a mobile terminal and a display unit, and the mobile terminal detected by the mobile terminal. Relative to the display unit of the user carrying the mobile terminal based on a change in position and a contact position of the mobile terminal in the display unit detected by the display unit when it is determined that the information cooperation is established An information processing apparatus including a position estimation unit that estimates a position is provided.

  The change in the position of the portable terminal includes the position of the portable terminal when the portable terminal is in contact with the display unit, and the position of the portable terminal before or after the portable terminal is contacted with the display unit. It may be a change between the position of the mobile terminal.

  The position change of the mobile terminal is performed when the user's posture shifts from the first posture to a touch posture that is a posture when the user brings the mobile terminal into contact with the display unit, or from the touch posture. It may be a change in the position of the mobile terminal when shifting to the first posture.

  The posture of the user may be determined based on an acceleration of the mobile terminal detected by the mobile terminal or an angular acceleration of the mobile terminal.

  The position estimation unit may further estimate the relative position of the user with respect to the display unit based on the human body model information stored in the storage unit.

  The information linkage may be established when the mobile terminal is in contact with the display unit while the mobile terminal is held by the user.

  When the relative position of the user is estimated by the position estimation unit, the information processing apparatus changes a display mode of content displayed on the display unit based on the estimated relative position of the user. A display control unit may be further provided.

  The display control unit may change the display position of the displayed content based on the relative position of the user.

  The display control unit may move and display the content displayed around the projection position, which is the position where the relative position of the user is projected on the display unit, above the projection position.

  Based on the contact position of the mobile terminal, the inclination of the mobile terminal with respect to the display unit when the mobile terminal is in contact with the display unit, and the relative position of the user, It further includes a posture estimation unit that estimates whether the gripping arm that grips the mobile terminal is the left arm or the right arm, and the display control unit is further displayed based on the estimated gripping arm of the user. You may change the display mode of the content.

  The posture estimation unit includes an inclination threshold value indicated by a gripping arm determination condition registered in a database in association with a positional relationship between the contact position of the mobile terminal and the estimated relative position of the user, and the mobile terminal By comparing the inclination, it may be estimated whether the gripping arm is the left arm or the right arm.

  When the portable terminal is in contact with the display unit, the information processing apparatus is configured to detect the gripping arm estimated by the posture estimation unit, the estimated relative position of the user, and the contact position of the portable terminal. The display unit further includes a shielding region estimation unit that estimates a shielding region that is a region shielded from the user's field of view by the gripping arm, and the display control unit is displayed in the estimated shielding region. The display position of the existing content may be changed.

  The display control unit may display content displayed in the shielding area in an area outside the shielding area.

  In order to solve the above-described problem, according to another aspect of the present invention, a computer is detected by a cooperation processing unit that performs determination regarding establishment of information cooperation between a portable terminal and a display unit, and the portable terminal. A user who carries the portable terminal based on the position change of the portable terminal and the contact position of the portable terminal in the display unit detected by the display unit when it is determined that the information cooperation is established A program is provided for functioning as a position estimation unit that estimates a relative position of the display unit.

  Moreover, in order to solve the said subject, according to another viewpoint of this invention, when it is a portable terminal and the said portable terminal is contacted with the display part in the state hold | gripped by the user, Control unit that causes the information processing apparatus to transmit a request for establishment of information cooperation between the mobile terminal, the position of the mobile terminal when the mobile terminal is in contact with the display unit, and the display unit An operation control unit that detects a change in position between the mobile terminal and the information processing apparatus before or after moving away from the display unit, and transmits the detected change in position to the information processing apparatus. A mobile terminal is provided.

  Moreover, in order to solve the said subject, according to another viewpoint of this invention, the determination regarding establishment of the information cooperation between a portable terminal, the display part which displays a content, and the said portable terminal and the said display part is performed. A linkage processing unit to perform, a position change of the portable terminal detected by the portable terminal, and a contact position of the portable terminal in the display unit detected by the display unit when the establishment of the information linkage is determined And a position estimation unit that estimates a relative position of the user carrying the mobile terminal with respect to the display unit.

  As described above, according to the present invention, it is possible to estimate the relative position of the user with respect to the display unit without using the measurement result of the dedicated measuring device.

It is explanatory drawing which showed the structural example of the information processing system by the 1st Embodiment of this invention. It is explanatory drawing which showed the example of the base attitude | position and the touch attitude | position. It is the functional block diagram which showed the structure of the portable terminal 20 by the embodiment. FIG. 6 is an explanatory diagram showing an example of a coordinate system set in the mobile terminal 20. It is explanatory drawing which showed the direction of the gravitational acceleration concerning the portable terminal 20 in the scene where the portable terminal 20 was inclined. It is explanatory drawing which showed the example of the one part determination method regarding a base attitude | position for every display direction of the portable terminal 20 by the embodiment. 6 is an explanatory diagram showing a state in which a change direction of acceleration detected by the mobile terminal 20 is projected on a display unit 326. FIG. 4 is a functional block diagram showing a configuration of a display device 30 according to the embodiment. FIG. It is the functional block diagram which showed the structure of the server 10-1 by the embodiment. FIG. 6 is a sequence diagram showing a part of the operation according to the embodiment. FIG. 6 is a sequence diagram showing a part of the operation according to the embodiment. It is the functional block diagram which showed the structure of the server 10-2 by the 2nd Embodiment of this invention. FIG. 6 is an explanatory diagram showing the degree of freedom of rotation of the grip arm of the mobile terminal 20. It is explanatory drawing which showed the example of the unreasonable attitude | position of the palm part of the holding arm of the portable terminal 20 at the time of a touch attitude | position. It is explanatory drawing which showed the example of a relationship between the holding arm of the portable terminal 20 at the time of a touch attitude | position, and the inclination of the portable terminal 20 with respect to a display part. It is explanatory drawing which showed the example of a relationship between the holding arm of the portable terminal 20 at the time of a touch attitude | position, and the inclination of the portable terminal 20 with respect to a display part. It is explanatory drawing which showed the example of a relationship between the holding arm of the portable terminal 20 at the time of a touch attitude | position, and the inclination of the portable terminal 20 with respect to a display part. It is explanatory drawing which showed the example of a division | segmentation of the area | region where the determination conditions of a holding | grip arm differ by the embodiment. It is explanatory drawing which showed the example of the relationship between a user's relative position and a touch position. It is explanatory drawing which showed the example of the relationship between the positional relationship of a user's relative position and a touch position, the combination of a holding arm, and a shielding area. It is explanatory drawing which showed the example of the relationship between a shielding area | region and the area | region where a content is hidden. FIG. 6 is a sequence diagram showing a part of the operation according to the embodiment. It is explanatory drawing which showed the hardware constitutions of the server 10 common to each embodiment.

  Exemplary embodiments of the present invention will be described below in detail with reference to the accompanying drawings. In addition, in this specification and drawing, about the component which has the substantially same function structure, duplication description is abbreviate | omitted by attaching | subjecting the same code | symbol.

  In the present specification and drawings, a plurality of components having substantially the same functional configuration may be distinguished by adding different alphabets after the same reference numeral. For example, a plurality of configurations having substantially the same functional configuration are distinguished as necessary, such as the communication network 32a and the communication network 32b. However, when it is not necessary to particularly distinguish each of a plurality of constituent elements having substantially the same functional configuration, only the same reference numerals are given. For example, when it is not necessary to distinguish between the communication network 32a and the communication network 32b, they are simply referred to as the communication network 32.

Further, the “DETAILED DESCRIPTION OF THE INVENTION” will be described according to the following item order.
1. 1. Basic configuration of information processing system 1. First embodiment Second Embodiment 4. 4. Hardware configuration Modified example

  In the present specification and drawings, the server 10-1 according to the first embodiment and the server 10-2 according to the second embodiment are collectively referred to as the server 10 when it is not necessary to distinguish between them. There is a case.

<< 1. Basic configuration of information processing system >>
First, a basic configuration of an information processing system common to each embodiment will be described with reference to FIG. As illustrated in FIG. 1, the information processing system common to the embodiments includes a server 10, a mobile terminal 20, a display device 30, and a communication network 32.

[1-1-1. Server 10]
The server 10 is an example of an information processing apparatus according to the present invention. The server 10 is a device for performing information cooperation (pairing) establishment processing between the mobile terminal 20 and the display device 30 when the mobile terminal 20 is touched on the display device 30 by a user, for example. In addition, the server 10 can perform display control on the display device 30.

[1-1-2. Mobile terminal 20]
The mobile terminal 20 is a communication terminal carried by the user. The mobile terminal 20 is a mobile phone such as a smartphone.

  In addition, the mobile terminal 20 can transmit and receive various types of information to and from the server 10 via the communication network 32a. Note that the mobile terminal 20 and the display device 30 are not specified as communication partners in the initial state, and communication is impossible.

[1-1-3. Display device 30]
The display device 30 includes a display including a capacitive touch panel or an infrared shielding touch panel, for example. The display device 30 is, for example, a device for digital signage.

  The display device 30 displays a plurality of contents such as images in accordance with display control information received from the server 10. Further, the display device 30 is configured such that, for example, as shown in the right diagram of FIG. 2, when the user brings the mobile terminal 20 into contact with the display of the display device 30 (at the touch position), the touch position of the mobile terminal 20 on the display Can be detected.

[1-1-4. Communication network 32]
The communication network 32 is a wired or wireless transmission path for information transmitted from a device connected to the communication network 32. For example, the communication network 32 may include a public line network such as a telephone line network, the Internet, a satellite communication network, various local area networks (LANs) including the Ethernet (registered trademark), a wide area network (WAN), and the like. . The communication network 32 may include a dedicated line network such as an IP-VPN (Internet Protocol-Virtual Private Network).

<1-2. Organizing issues>
By the way, it is desired that an intuitive interaction is possible between the content displayed on the display device 30 and the user. For example, as shown in the right diagram of FIG. 2, when the user touches the mobile terminal 20 on the display, the display mode of the displayed content can be changed according to the touch position of the mobile terminal 20. desirable.

  In a known technique, a technique for measuring the positional relationship and posture relationship between a user and a display at the time of interaction by using a dedicated measuring device that is used only for measuring the state of the user has been proposed. However, this technique requires a dedicated measuring instrument, which complicates the system configuration and increases the system introduction cost.

  Therefore, the server 10 according to each embodiment has been created by focusing on the above circumstances. The server 10 according to each embodiment can estimate the relative position of the user with respect to the display only from the measurement results obtained by the mobile terminal 20 and the display device 30. Further, the server 10 can change the display mode of the content displayed on the display device 30 according to the estimated relative position of the user.

<< 2. First Embodiment >>
<2-1. Configuration>
[2-1-1. Mobile terminal 20]
Next, the configuration according to the first embodiment will be described in detail. FIG. 3 is a functional block diagram showing the configuration of the mobile terminal 20 according to the first embodiment. As illustrated in FIG. 3, the mobile terminal 20 includes a control unit 200, an input unit 220, a sensor unit 222, a communication unit 224, a storage unit 226, and a display unit 228.

(2-1-1-1. Control unit 200)
The control unit 200 generally performs operations of the mobile terminal 20 using hardware such as a CPU (Central Processing Unit), a RAM (Random Access Memory), or a ROM (Read Only Memory) built in the mobile terminal 20. To control. As illustrated in FIG. 3, the control unit 200 includes an operation processing unit 202, an operation processing unit 204, a display control unit 206, and a cooperation control unit 208.

(2-1-1-2. Operation processing unit 202)
The operation processing unit 202 performs processing of a user input operation detected by the input unit 220 described later.

(2-1-1-3. Operation processing unit 204)
-Derivation of operation The operation processing unit 204 performs a process of deriving operation information indicating the operation of the user or the operation of the mobile terminal 20 held by the user based on the detection result by the sensor unit 222 described later.

-Determination of user posture (base posture) For example, the motion processing unit 204 determines whether or not the user posture is the base posture based on the detection result of acceleration or angular acceleration by the sensor unit 222. Here, the base posture is an example of a first posture in the present invention. The base posture is an arbitrary posture that can identify the posture switching between the touch posture. Further, the touch posture is a posture when the user touches the display of the display device 30 with the mobile terminal 20 as illustrated in the right diagram of FIG.

  For example, as shown in the left diagram of FIG. 2, the base posture may be a posture in which the user holds the mobile terminal 20 tilted forward of the chest and is looking at the display screen of the mobile terminal 20. Alternatively, the base posture may be a posture in which the user is facing the display device 30 and holding his hand down while holding the mobile terminal 20 in his / her hand. Alternatively, the base posture may be a posture when the user is walking toward the display device 30.

  For example, the motion processing unit 204 determines that the user's posture is the base posture when both of the following two conditions (condition A and condition B) are satisfied, and satisfies at least one of the two conditions. If not, it may be determined that the posture is not the base posture. Here, the condition A is that the mobile terminal 20 is in a stationary state. The condition B is that the posture state of the mobile terminal 20 is a specific posture state.

  Here, the above contents will be described in more detail. Assume that the coordinate system of the mobile terminal 20 (that is, the coordinate system of a three-axis acceleration sensor or gyro sensor included in the sensor unit 222 described later) is set as an xyz coordinate system as shown in FIG. In this case, the condition A is, for example, that accelerations in the three-axis directions of x, y, and z measured by a three-axis acceleration sensor converge to a certain value (or fall within a certain range), respectively. In addition, the values of the angular accelerations (angular velocities) about the three axes are all “0” (or are within a certain range with “0” as a base point).

  By the way, if the base posture is determined as a posture in which the user is looking at the display screen of the mobile terminal 20 as shown in the left diagram of FIG. It is tilted by the user in the direction as shown in FIG. For this reason, the gravitational acceleration G (gx, gy, gz) is applied to the mobile terminal 20 in the direction indicated by the solid line arrow shown in FIG. Therefore, when the base posture is determined to be the posture shown in the left diagram of FIG. 2, the above condition B includes both of the following two conditions (condition B-1 and condition B-2). It can be determined that the condition is satisfied.

  The condition B-1 is a condition for determining that the display unit 228 of the mobile terminal 20 is facing upward. The condition B-2 is a condition for determining that the display unit 228 is tilted toward the chest of the user (so that the user can easily see the display unit 228).

  As a modification, when the mobile terminal 20 has a function that can change the display direction of the display screen in four directions depending on the attitude of the mobile terminal 20 in space, instead of the condition B-2, A determination formula defined for each display direction of the display screen as shown in FIG. 6 may be used. For example, when the mobile terminal 20 is tilted so that the negative direction of the Y axis in the display unit 228 matches the upward direction of the display screen, the following determination formula is used instead of the condition B-2. It is done.

--Identification of position change of portable terminal 20 In addition, the operation processing unit 204 identifies a change in position of the portable terminal 20 based on a change in acceleration detected by the sensor unit 222. For example, the motion processing unit 204 specifies that the mobile terminal 20 has moved in a direction opposite to the direction of the detected acceleration change.

  Here, the above content will be described in more detail with reference to FIG. FIG. 7 is an explanatory diagram illustrating a state in which the change direction of acceleration detected by the sensor unit 222 is projected on a display unit 326 described later (in the display device 30) when the mobile terminal 20 is moved by the user. It is. 7A shows an example of projection when it is detected that the acceleration has changed in the “upper left” direction as indicated by the arrow when the mobile terminal 20 is touched on the display unit 326. Show. Similarly, (b) in FIG. 7 is when the acceleration is detected to change in the “up” direction, (c) is when the acceleration is detected to change in the “lower left” direction, and (d) Respectively show projection examples when it is detected that the acceleration has changed in the “right” direction.

  In the example of FIG. 7A, the motion processing unit 204 determines that the moving direction of the mobile terminal 20 is “lower right” (which is a direction opposite to the acceleration changing direction). Similarly, the motion processing unit 204 sets the movement direction of the mobile terminal 20 to “down” in the example of FIG. 7B, “upper right” in the example of FIG. 7C, and (d) of FIG. In the example of), it is determined as “left”.

-Transmission of processing result The operation processing unit 204 also transmits the determination result of the posture of the user and the position change information of the mobile terminal 20 to the server 10-1 to the communication unit 224 described later.

(2-1-1-4. Display control unit 206)
The display control unit 206 causes the display unit 228 to display various display information such as GUI (Graphical User Interface).

(2-1-1-5. Coordination control unit 208)
The cooperation control unit 208 performs a request process related to establishment of information cooperation with the display device 30 based on a user input to the input unit 220 described later. For example, the cooperation control unit 208 sends a first pairing request including a request signal for establishing information cooperation with the display device 30 and a processing time, which is input by the user to the input unit 220 and the processing time. -1 is transmitted to the communication unit 224. Here, the first pairing request may further include a triaxial acceleration component measured by the triaxial acceleration sensor included in the sensor unit 222 (for example, when the movement of the mobile terminal 20 stops).

(2-1-1-6. Input unit 220)
The input unit 220 has a function of detecting a user operation on the mobile terminal 20. For example, the input unit 220 may include a touch panel that can detect a user's operation on the GUI displayed on the display unit 228. The input unit 220 may include a camera that can detect a user's gesture. The input unit 220 may include a microphone that can detect a user's utterance.

(2-1-1-7. Sensor unit 222)
The sensor unit 222 has a function of detecting the acceleration or angular acceleration of the mobile terminal 20. For example, the sensor unit 222 includes a triaxial acceleration sensor and detects the triaxial acceleration applied to the mobile terminal 20. Note that the three axes (coordinate system) of x, y, and z in the mobile terminal 20 can be set, for example, as shown in FIG. The sensor unit 222 includes a gyro sensor and detects angular acceleration around the three axes in the mobile terminal 20.

  The sensor unit 222 is basically provided inside the mobile terminal 20. However, the sensor unit 222 may be provided outside the mobile terminal 20 without being limited to such an example.

(2-1-1-8. Communication unit 224)
The communication unit 224 transmits / receives information to / from other devices connected to the communication network 32a. For example, the communication unit 224 transmits the position change information of the mobile terminal 20 to the server 10-1 according to the control of the operation processing unit 204.

  Note that when the information link between the portable terminal 20 and the display device 30 is established by the server 10-1, the communication unit 224 can also transmit and receive information to and from the display device 30. .

(2-1-1-9. Storage Unit 226)
The storage unit 226 stores, for example, software and data related to the operation of the control unit 200.

(2-1-1-10. Display unit 228)
The display unit 228 is configured by a liquid crystal display, for example. The display unit 228 displays a display screen according to the control of the display control unit 206.

[2-1-2. Display device 30]
FIG. 8 is a functional block diagram showing the configuration of the display device 30 according to the first embodiment. As illustrated in FIG. 8, the display device 30 includes a control unit 300, an input unit 320, a communication unit 322, a storage unit 324, and a display unit 326.

(2-1-2-1. Control unit 300)
The controller 300 generally controls the operation of the display device 30 using hardware such as a CPU, RAM, or ROM built in the display device 30. As illustrated in FIG. 8, the control unit 300 includes an operation processing unit 302, a display control unit 304, and a cooperation control unit 306.

(2-1-2-2. Operation processing unit 302)
The operation processing unit 302 performs user input operation processing detected by the input unit 320 described later. For example, the operation processing unit 302 specifies the user's selection operation for the content displayed on the display unit 326.

(2-1-2-3. Display control unit 304)
The display control unit 304 causes the display unit 326 to display display information such as content in accordance with display control information received from the server 10-1, for example. The display control unit 304 controls the display mode of content.

(2-1-2-4. Cooperation control unit 306)
When the touch input to the display unit 326 in a state where the mobile terminal 20 is held with one hand by the user is detected by the input unit 320 described later, the cooperation control unit 306 performs information with the mobile terminal 20. Request processing related to establishment of cooperation.

-Capacitive touch panel For example, in the above case, and when the display unit 326 described later includes a capacitive touch panel, the cooperation control unit 306 first detects the two touch points detected by the input unit 320. Is specified as a contact point with respect to the display unit 326 (if there are a plurality of sets of touch points, the center of gravity of the center point of each set of touch points). Then, the cooperation control unit 306 causes the communication unit 322 to transmit the second pairing request including the identified contact point and the processing time to the server 10-1.

-Infrared shielding type touch panel In the above case and when the display unit 326 includes an infrared shielding type touch panel, the cooperation control unit 306 first includes the area of the mobile terminal 20 touched by the display unit 326. The center of the smallest rectangular area is specified as a contact point with respect to the display unit 326. Next, the cooperation control unit 306 acquires a triaxial component of the gravitational acceleration applied to the display unit 326, which is measured by a triaxial acceleration sensor (not shown) built in the display device 30. Next, the cooperation control unit 306 calculates the inclination of the mobile terminal 20 with respect to the display unit 326. Then, the cooperation control unit 306 communicates the second pairing request including the specified contact point, the acquired measurement information of the triaxial acceleration, the calculated inclination of the mobile terminal 20, and the processing time to the server 10-1. 322 is transmitted.

(2-1-2-5. Input unit 320)
-Input example 1
The input unit 320 has a function of detecting a user operation on the display device 30. For example, the input unit 320 detects a user's touch input on the display unit 326.

--Capacitance type touch panel For example, when the display unit 326 includes a capacitance type touch panel, the touch panel has a capacitance type configured with a panel shape (for example, electrodes and electrodes arranged in a grid shape). A so-called projected capacitance method that utilizes the principle that the capacitance of the capacitor formed at the intersection of the first and second electrodes changes when a conductive object such as a human body approaches or comes into contact therewith. In this case, the input unit 320 measures the capacitance that changes due to the approach or contact of the conductive object with respect to the touch panel, and based on the measured information, determines whether or not the display unit 326 is touched and the touch position. To detect.

--Infrared shielding type touch panel Or, when the display unit 326 includes an infrared shielding type touch panel, the touch panel has an infrared LED (Light Emitting Diode) on one side of the long side and a short side of the bezel, and an opposite side. Infrared sensors are respectively laid, and an infrared matrix is formed on the surface of the touch panel by sequentially emitting infrared LEDs. In this case, the input unit 320 detects the presence / absence of a touch on the display unit 326 and the touch position based on the light reception level of the infrared sensor laid on the touch panel. For example, when the touch panel is touched with a finger or the like, the infrared ray passing through the touched portion is blocked, and the light receiving level of the infrared sensor at the corresponding location is lowered, so that the input unit 320 detects the touch position. To do. The input unit 320 can also detect the size (vertical x horizontal) of the smallest rectangular area including the area covered by the arranged object when an object (for example, the mobile terminal 20) is arranged on the touch panel. is there.

-Input example 2
In addition, when information cooperation is established between the mobile terminal 20 and the display device 30, the input unit 320 can accept any input method within a range where the information cooperation can be realized.

(2-1-2-6. Communication unit 322)
The communication unit 322 transmits / receives information to / from other devices connected to the communication network 32b. For example, the communication unit 322 receives display control information for causing the display unit 326 to display content from the server 10-1.

  In addition, when information cooperation is established between the mobile terminal 20 and the display device 30, the communication unit 322 can transmit and receive information to and from the mobile terminal 20.

(2-1-2-7. Storage Unit 324)
The storage unit 324 stores, for example, software and data related to the operation of the control unit 300.

(2-1-2-8. Display unit 326)
The display unit 326 is configured by a liquid crystal display including a touch panel, for example. The display unit 326 displays a display screen according to the control of the display control unit 304.

[2-1-3. Server 10-1]
FIG. 9 is a functional block diagram showing the configuration of the server 10-1 according to the first embodiment. As shown in FIG. 9, the server 10-1 includes a control unit 100-1, a communication unit 120, and a storage unit 122-1.

(2-1-3-1. Control unit 100-1)
The control unit 100-1 generally controls the operation of the server 10-1 using hardware such as a CPU 150, a ROM 152, and a RAM 154, which will be described later, built in the server 10-1. Further, as illustrated in FIG. 9, the control unit 100-1 includes a cooperation processing unit 102, a person position estimation unit 104, and a display control unit 106.

(2-1-3-2. Cooperation processing unit 102)
-Process for Establishing Information Cooperation The cooperation processing unit 102 is based on a comparison between the first pairing request received from the portable terminal 20 and the second pairing request received from the display device 30. And control related to establishment of information linkage between the display device 30 and the display device 30.

--Capacitive touch panel For example, when the display unit 326 includes a capacitive touch panel, the cooperation processing unit 102 receives the processing time included in the received first pairing request and the received first When the difference from the processing time included in the pairing request 2 falls within a predetermined time, information cooperation (pairing) is established between the mobile terminal 20 and the display device 30.

--Infrared shielding type touch panel When the display unit 326 includes an infrared shielding type touch panel, the cooperation processing unit 102 can perform the portable terminal 20 only when the following two conditions (condition C and condition D) are satisfied. And information display are established between the display device 30 and the display device 30. Here, the condition C is that the difference between the processing time included in the received first pairing request and the processing time included in the received second pairing request falls within a predetermined time. . In addition, the condition D includes the posture information of the mobile terminal 20 included in the first pairing request (acceleration component in three axes direction) and the posture information of the display device 30 included in the second pairing request (three axes direction). Of the mobile terminal 20 on the display unit 326 calculated by the cooperation processing unit 102 based on the acceleration component) and the display unit 326 calculated by the display device 30 included in the second pairing request. That is, the difference between the inclination of the portable terminal 20 (presumed object) falls within a predetermined threshold. In this way, not only the condition C but also the condition D is added for the determination, so that the pairing accuracy can be improved.

  Note that, for example, when the display device 30 is a table-type device and the display unit 326 is a table top, or when the display device 30 is mounted on a wall, the posture of the display unit 326 in the space varies. possible. However, in the condition D, the determination can be performed without depending on the posture of the display unit 326.

-Touch Posture Determination Processing Further, the cooperation processing unit 102 determines whether or not the user posture is the touch posture based on the position change information of the mobile terminal 20 received from the mobile terminal 20.

  For example, the cooperation processing unit 102 determines that the user's posture is a touch posture when both of the following two conditions (condition E and condition F) are satisfied, and satisfies at least one of the two conditions. If not, it is determined that the touch posture is not set. Here, the condition E is that the position change information of the mobile terminal 20 received from the mobile terminal 20 indicates that the mobile terminal 20 is in a stationary state. The condition F is received from the display device 30 that information cooperation between the mobile terminal 20 and the display device 30 has already been established, and that the contact of the mobile terminal 20 with the display unit 326 is detected. It is that.

(2-1-3-3. Person position estimation unit 104)
The person position estimation unit 104 receives the position change information received from the mobile terminal 20 and the touch position of the mobile terminal 20 received from the display device 30 when the information processing unit 102 establishes (or after) information cooperation. Based on this, the user's relative position with respect to the display unit 326 is estimated. Here, the target position change information includes, for example, the position of the mobile terminal 20 immediately before the mobile terminal 20 is touched on the display unit 326 or immediately after the mobile terminal 20 is released from the display unit 326, and the display unit 326. The information which shows the position change between the touch positions of the portable terminal 20 may be sufficient. Alternatively, the position change information may be information indicating a position change of the mobile terminal 20 when the user's posture shifts from the base posture to the touch posture, or when the user's posture changes from the touch posture to the base posture.

  For example, the person position estimation unit 104 further estimates the relative position of the user with respect to the display unit 326 based on the human body model information such as the arm length stored in the human body model DB 124. Note that the relative position of the user may be a relative position of the chest of the user with respect to the display unit 326, or may be a relative position of the front of the user's face or eyes.

-Human model DB124
The human body model DB 124 is a database that stores average data such as the height of the human body, the length of each part, and the positional relationship between the parts. The human body model DB 124 is stored in the storage unit 122-1, for example.

(2-1-3-4. Display control unit 106)
The display control unit 106 displays display control information for changing the display mode of the content displayed on the display device 30 (the display unit 326) based on the relative position of the user estimated by the person position estimation unit 104. The display device 30 is caused to transmit to the communication unit 120 described later.

  For example, the display control unit 106 changes the display position of the content currently displayed on the display unit 326 based on the estimated relative position of the user, or displays control information for newly displaying the content. Is transmitted to the communication unit 120 to the display device 30. As an example, the display control unit 106 moves the content displayed around the chest projection position, which is a position where the estimated relative position of the user's chest is projected on the display unit 326, to a position higher than the chest projection position. To display. Further, when the display control unit 106 newly displays content on the display unit 326, the display control unit 106 displays the content above the chest projection position. According to these display examples, it becomes easier for the user to see the content. For example, since the user's eyes are positioned above the user's chest, the user can view the content without lowering his line of sight.

  Alternatively, the display control unit 106 displays, in the display unit 326, the content that the user has interacted among the plurality of contents displayed on the display unit 326 and the content related to the content on the display unit 326. Are moved around the projected position or area (hereinafter also referred to as face projection position / area), or are newly displayed at the face projection position / area.

  Alternatively, the display control unit 106 moves the display position of the content so that the density of the content displayed in the face projection position / region on the display unit 326 is higher than the other regions, or newly displays the content. Is displayed.

(2-1-3-5. Communication unit 120)
The communication unit 120 transmits and receives information to and from other devices via the communication network 32, for example. For example, the communication unit 120 receives the position change information of the mobile terminal 20 and the determination result of the user's base posture from the mobile terminal 20. In addition, the communication unit 120 transmits display control information to the display device 30 according to the control of the display control unit 106.

(2-1-3-6. Storage Unit 122-1)
The storage unit 122-1 stores various data such as the human body model DB 124, for example.

  The configuration of the server 10-1 according to the first embodiment is not limited to the configuration described above. For example, instead of being stored in the storage unit 122-1, the human body model DB 124 can be stored in another device (not shown) that can communicate with the server 10-1.

<2-2. Operation>
The configuration according to the first embodiment has been described above. Subsequently, an operation according to the first embodiment will be described. FIG. 10 is a sequence diagram showing a part of the operation according to the first embodiment. As shown in FIG. 10, first, the communication unit 120 of the server 10-1 transmits display control information for displaying one or more contents to the display device 30 according to the control of the display control unit 106 (S101). . Then, the display control unit 304 of the display device 30 displays a display screen on the display unit 326 according to the received display control information (S103).

  Further, the operation processing unit 204 of the mobile terminal 20 periodically determines whether or not the user's posture is the base posture based on the detection result by the sensor unit 222, for example. For example, when the motion processing unit 204 determines that the posture is the base posture, the operation processing unit 204 causes the server 10-1 to transmit the determination result to the communication unit 224 (S105).

  Thereafter, when the user's posture shifts from the base posture to the touch posture, the motion processing unit 204 of the mobile terminal 20 detects a change in the position of the mobile terminal 20 based on the detection result by the sensor unit 222 (S107). .

  Thereafter, the user touches the display unit 326 (of the display device 30) with the hand holding the mobile terminal 20, and presses a GUI button displayed on the mobile terminal 20, for example, with the display device 30. A request for establishment of the information linkage is input to the input unit 220 (S109). Thereafter, the input unit 320 of the display device 30 detects the touch position of the mobile terminal 20 with respect to the display unit 326 (S111).

  Further, the communication unit 224 of the portable terminal 20 transmits the request signal input in S109 and the first pairing request including the processing time to the server 10-1 according to the control of the cooperation control unit 208 (S113). .

  Thereafter, the communication unit 322 of the display device 30 transmits the second pairing request including the touch position detected in S111 and the processing time to the server 10-1 according to the control of the cooperation control unit 306 (S115). .

  Thereafter, the cooperation processing unit 102 of the server 10-1 determines whether the portable terminal 20 and the display device are based on the comparison between the first pairing request received in S113 and the second pairing request received in S115. Control relating to establishment of information cooperation with 30 is performed (S117).

  When the information linkage is established in S117, the communication unit 224 of the mobile terminal 20 transmits the position change information of the mobile terminal 20 detected in S107 to the server 10-1 according to the control of the operation processing unit 204 ( S119).

  Here, the operation after S119 will be described with reference to FIG. As shown in FIG. 11, after S119, the person position estimation unit 104 of the server 10-1 stores the position change information received in S119, the touch position received in S115, and the human body model DB 124. The relative position of the user with respect to the display unit 326 is estimated based on the human body model information (S121).

  Subsequently, the display control unit 106 generates display control information for changing the display position of the content currently displayed on the display unit 326 based on the relative position of the user estimated in S121 (S123). And the communication part 120 transmits the produced | generated display control information to the display apparatus 30 according to control of the display control part 106 (S125).

  Thereafter, the display control unit 304 of the display device 30 updates the display content of the display screen according to the received display control information, and causes the display unit 326 to display the updated display screen (S127).

<2-3. Effect>
[2-3-1. Effect 1]
As described above, for example, as described with reference to FIGS. 9 to 11, the server 10-1 according to the first embodiment includes the position change of the mobile terminal 20 detected by the mobile terminal 20 and the mobile terminal 20. The relative position of the user with respect to the display unit 326 is estimated based on the touch position of the mobile terminal 20 detected by the display unit 326 when the information cooperation with the display device 30 is established. For this reason, according to the first embodiment, the relative position of the user with respect to the display unit 326 can be estimated only from the measurement results obtained by the mobile terminal 20 and the display device 30. That is, since a dedicated measuring device for measuring the user's state is not required, the information processing system can be configured simply and inexpensively.

[2-3-2. Effect 2]
In addition, the server 10-1 can change the display position of the content displayed on the display unit 326 according to the estimated relative position of the user. For example, the server 10-1 moves the currently displayed content to the face projection position / area and displays it, or displays the content at the face projection position / area when newly displaying the content. Let For this reason, the user can view the content without lowering the line of sight, and the content becomes easy to see.

<2-4. Modification>
[2-4-1. Modification 1]
Note that the first embodiment is not limited to the above description. For example, as a first modification, after changing the display position of the content based on the estimated relative position of the user, for example, when the user's next input is not performed for a certain time, the display control unit 106 The display position may be returned to the original position. Here, an example of the next input by the user may be a touch operation (for example, a content selection operation) with the finger on the display unit 326, or a new touch posture is made by the user (that is, The user may newly touch the mobile terminal 20 with respect to the display unit 326). When information cooperation is established between the mobile terminal 20 and the display device 30, the user's next input is, for example, a mobile terminal having contents related to display information displayed on the display unit 326. 20 may be input.

[2-4-2. Modification 2]
In the above description, the example in which the mobile terminal 20 determines whether or not the user's posture is the base posture has been described. However, the present invention is not limited to this example, and the server 10-1 can also perform the determination. It is. Specifically, the mobile terminal 20 transmits information such as measurement results of acceleration and angular acceleration measured by the sensor unit 222 to the server 10-1, and the server 10-1 transmits the received information to the received information. Based on this, it may be determined whether or not the user's posture is the base posture.

[2-4-3. Modification 3]
For example, when the accuracy of the triaxial acceleration sensor and the gyro sensor included in the sensor unit 222 of the mobile terminal 20 and the accuracy of the calculation process of the position change and the posture change are high, the mobile terminal 20 It is possible to calculate the moving distance of the portable terminal 20 (not only the moving direction of the portable terminal 20) using the integrated value of Then, the server 10-1 can estimate the relative position of the user based on the calculated moving distance, and can change the display mode of the content based on the estimated relative position.

[2-4-4. Modification 4]
Further, the mobile terminal 20 has an in-camera function, and the base posture is determined to be a posture in which the user is looking at the display screen of the mobile terminal 20 as shown in the left diagram of FIG. In this case, a condition that the user's face can be recognized within the field of view of the in-camera (condition G) may be added as a condition for determining the base posture. That is, the operation processing unit 204 of the mobile terminal 20 determines that the user's posture is the base posture when all three conditions (condition A, condition B, and condition G) are satisfied, and the three conditions If at least one of them is not satisfied, it may be determined that the posture is not the base posture. According to this modification, when the base posture is determined as the posture shown in the left diagram of FIG. 2, the base posture can be determined with high accuracy.

[2-4-5. Modification 5]
If the base posture is determined to be the posture shown in the left diagram of FIG. 2, the mobile terminal 20 uses the in-camera function to make the user's face in the base posture state. Alternatively, the positional relationship between the face portion and the mobile terminal 20 can be estimated. For example, when the user's face can be recognized within the field of view of the in-camera, the mobile terminal 20 carries the user's face or eyes and the mobile phone based on the size of the user's face area that occupies the field of view of the in-camera. The positional relationship (or distance) with the terminal 20 may be calculated.

<< 3. Second Embodiment >>
The first embodiment has been described above. Next, a second embodiment will be described. First, the background that led to the creation of the second embodiment will be described.

  In the touch posture described above, a part of the display area of the display unit 326 can be shielded from the user's view by the user's arm holding the mobile terminal 20. For this reason, there is a problem that the user cannot see the content displayed in the shielded area (hereinafter referred to as a shielded area) or is difficult to see.

  As will be described later, the server 10-2 according to the second embodiment can estimate the shielding area and display the content displayed in the shielding area outside the shielding area.

<3-1. Configuration>
[3-1-1. Mobile terminal 20, display device 30]
The configurations of the mobile terminal 20 and the display device 30 according to the second embodiment are the same as those in the first embodiment.

[3-1-2. Server 10-2]
FIG. 12 is a functional block diagram showing the configuration of the server 10-2 according to the second embodiment. As shown in FIG. 12, the server 10-2 is different from the first embodiment in that instead of the control unit 100-1 and the storage unit 122-1, the control unit 100-2 and the storage unit 122-2 are used. Have In addition, below, description is abbreviate | omitted about the content which overlaps with 1st Embodiment.

(3-1-2-1. Control unit 100-2)
As illustrated in FIG. 12, the control unit 100-2 further includes a person posture estimation unit 108 and a shielding area estimation unit 110, as compared with the control unit 100-1.

(3-1-2-2. Person Posture Estimation Unit 108)
Based on the touch position of the mobile terminal 20, the inclination of the mobile terminal 20 with respect to the display unit 326, and the relative position of the user estimated by the person position estimation unit 104, the person posture estimation unit 108 holds the mobile terminal 20. It is estimated whether the holding arm to be operated is the left hand or the right hand. For example, the person posture estimation unit 108 associates with the positional relationship between the touch position of the mobile terminal 20 and the relative position of the user, based on the gripping arm determination condition stored in the gripping arm determination condition DB 126 described later. Guess whether the arm is left hand or right hand.

  More specifically, the person posture estimation unit 108 first stores the gripping arm determination condition DB 126 in association with the positional relationship between the touch position of the mobile terminal 20 and the estimated relative position of the user. The classification of the arm determination area is extracted from the gripping arm determination condition DB 126. Then, based on the gripping arm determination condition stored in the gripping arm determination condition DB 126 in association with the extracted category and the inclination of the mobile terminal 20 with respect to the display unit 326, the person posture estimation unit 108 determines that the gripping arm is in the left hand. Guess which is the right hand or the right hand.

  Here, the above functions will be described in more detail with reference to FIGS. Generally, the operating range (rotatable range) of the hand holding the mobile terminal 20 is limited to a certain range due to the structure of the human body. For example, the degree of freedom of the wrist of the grip arm of the mobile terminal 20 can be around three axes Y, R, and P that are orthogonal as shown in FIG.

  For this reason, the natural posture of the palm of the grip arm of the mobile terminal 20 in the touch posture is as shown in FIG. 14, for example. As shown in FIG. 14, when the gripping arm and the touch position are determined, the inclination of the mobile terminal 20 with respect to the display unit 326 can be specified to some extent. Note that FIG. 14 shows an example in which the portable terminal 20 is held with the right hand, and the case where the portable terminal 20 is held with the left hand is a diagram in contrast to FIG.

  15 to 17 are explanatory diagrams illustrating an example of the relationship between the gripping arm of the mobile terminal 20 and the inclination of the mobile terminal 20 in the touch posture. 15 to 17 are examples of the positional relationship between the touch position of the mobile terminal 20 and the relative position of the user and the inclination of the mobile terminal 20 with respect to the display unit 326 for each gripping arm (left hand, right hand). Is shown. Here, the inclination of the mobile terminal 20 is the Y-axis direction of the mobile terminal 20 on the display unit 326. In addition, in (a), (b), (c) in FIG. 15, (d), (e), (f) in FIG. 16, and (g), (h), (i) in FIG. The relative position of the user with respect to the touch position 40 of the mobile terminal 20 is “upper left”, “up”, “upper right”, “left”, “same”, “right”, “lower left”, “lower”, or “lower right”, respectively. The example in the case of being located is shown.

As shown by the arrows in FIGS. 15 to 17, the inclination of the mobile terminal 20 with respect to the display unit 326 is normally set to be positive when counterclockwise is set to a positive direction.
Tilt when holding with right hand> Tilt when holding with left hand.

-Grasping arm determination condition DB 126
The gripping arm determination condition DB 126 is a database in which the display screen classification method based on the positional relationship between the relative position of the user and the touch position and the determination condition of the gripping arm in each section are defined. FIG. 18 is an explanatory diagram showing an example of a method of dividing the display screen into nine regions based on the user's relative position 42 stored in the gripping arm determination condition DB 126.

  In the example illustrated in FIG. 18, an area where the distance between the touch position and the user's relative position 42 is equal to or less than a predetermined value is the section E, and areas other than the section E are centered on the user's relative position 42. Are divided into 8 and are defined as sections A to D and F to I, respectively.

Further, in the gripping arm determination condition DB 126, for example, X (degree) is set as a threshold value in the area of the section A shown in FIG.
When the inclination of the mobile terminal 20 is equal to or greater than X, the gripping arm is the right hand,
When the inclination of the mobile terminal 20 <X, the gripping arm is the left hand,
Are determined to be determined.

  For example, the developer of the server 10-2 can determine an appropriate value by executing a test for a plurality of users in advance.

  FIG. 18 is a diagram when the user's relative position 42 is the center, whereas FIGS. 15 to 17 described above are diagrams when the touch position of the mobile terminal 20 is the center. That is, the correspondence between FIG. 18 and FIGS. 15 to 17 is as follows: section A is (i) in FIG. 17, section B is (h) in FIG. 17, section C is (g) in FIG. 16 (f), section E is FIG. 16 (e), section F is FIG. 16 (d), section G is FIG. 15 (c), section H is FIG. 15 (b), and section I is the figure. This corresponds to (a) of 15 respectively.

--- Modification 1
In the above description, one threshold value is set for each section in the gripping arm determination condition DB 126, and an example of a method for determining a gripping arm using this threshold value is shown, but the present invention is not limited to this example. For example, instead of the threshold value, an angle range for determining whether the gripping arm is the right hand or the left hand may be determined for each section. As an example, in the area of section A shown in FIG. 18, the angle range (hereinafter also referred to as range A) for determining that the gripping arm is the left hand is [a1, a2], and the angle for determining that the gripping arm is the right hand. May be defined as [b1, b2] and a2 <b1. That means
If b1 ≦ tilt of mobile terminal 20 ≦ b2, the gripping arm is the right hand,
If a1 ≦ tilt of mobile terminal 20 ≦ a2, the gripping arm is the left hand,
It may be determined that each of these is determined.

  When the inclination of the mobile terminal 20 is out of the range A or the range B, the person posture estimation unit 108 determines whether the holding arm is the right hand or the left hand according to a predetermined criterion. Is possible. Here, the predetermined reference may be, for example, a determination history of the gripping arm when the user touches the display unit 326 in the past.

--- Modification 2
FIG. 18 shows an example of a method in which the display screen is divided into nine areas. However, the method is not limited to this example, and the display screen can be divided into an arbitrary number of areas.

(3-1-2-3. Shielding region estimation unit 110)
The occlusion area estimation unit 110 includes a user's relative position estimated by the person position estimation unit 104 while the mobile terminal 20 is touching the display unit 326, a gripping arm estimated by the person posture estimation unit 108, and Based on the touch position of the mobile terminal 20 received from the display device 30, the shielding area is estimated. For example, the shielding area estimation unit 110 shields the area stored in the shielding area DB 128 described later in association with the positional relationship between the estimated relative position of the user and the touch position and the estimated gripping arm. Guess as an area. Alternatively, the shielding area estimation unit 110 may include the positional relationship between the estimated relative position of the user and the touch position, and arm information (arm shape) stored in the human body model DB 124 in association with the estimated gripping arm. May be extracted from the human body model DB 124, and the shielding area may be calculated using the extracted arm information.

  Here, the above functions will be described in more detail with reference to FIGS. 19 and 20. FIG. 19 is a diagram illustrating four examples of the positional relationship between the relative position 42 of the user and the touch position 40 of the mobile terminal 20. FIG. 19A shows an example in which the direction of the touch position 40 with respect to the relative position 42 of the user is “upper left”. Similarly, the direction of the touch position 40 relative to the relative position 42 of the user is “up” in FIG. 19B, “lower left” in FIG. 19 and “right” in FIG. 19D. Each example is shown.

  20 shows the shielding area 50 for each gripping arm estimated by the shielding area estimation unit 110 regarding the four types of positional relationships between the relative position 42 of the user and the touch position 40 of the mobile terminal 20 shown in FIG. It is explanatory drawing which each showed. In FIGS. 20A to 20D, the shielding area 50a is a shielding area when the gripping arm is the right hand, and the shielding area 50b is a shielding area when the gripping arm is the left hand. Yes.

  For example, when the estimated positional relationship between the relative position 42 of the user and the touch position 40 is the positional relationship shown in FIG. 19A and the estimated gripping arm is the right hand, The estimation unit 110 estimates the shielding area 50a illustrated in FIG. 20A as a shielding area.

-Shielding area DB128
The shielding area DB 128 is a database in which a shielding area template is stored in association with the positional relationship between the relative position of the user and the touch position, and the grip arm (right hand or left hand). Note that the shielding area template stored in the shielding area DB 128 may be a rectangular area as shown in FIG. 20, for example.

(3-1-2-4. Display control unit 106)
-Display example 1
The display control unit 106 according to the second embodiment displays content on the display unit 326 based on the relative position of the user estimated by the person position estimation unit 104 and the shielding area estimated by the shielding region estimation unit 110. Display control information for controlling the position is transmitted to the communication device 120 to the display device 30. For example, when the mobile terminal 20 is touched on the display unit 326, the display control unit 106 moves the content displayed in the shielding area estimated by the shielding area estimation unit 110 to the outside of the shielding area and displays the content. . Alternatively, when the display unit 326 newly displays content on the display unit 326 while the mobile terminal 20 is touching the display unit 326, the display control unit 106 displays the content in an area outside the shielding area.

  Here, the above function will be described in more detail with reference to FIG. (Α) and (β) in FIG. 21 are examples of the relationship between the shielding area 50 estimated by the shielding area estimation unit 110 and the area where the content is hidden by the display control unit 106 (non-display area 52). It is explanatory drawing which showed. For example, as illustrated in (α) of FIG. 21, the display control unit 106 sets an area such as an ellipse including the entire shielding area 50 as the non-display area 52. Then, the display control unit 106 moves the content displayed in the non-display area 52 to the outside of the non-display area 52 and displays it.

  Alternatively, as illustrated in (β) of FIG. 21, the display control unit 106 is on the opposite side of the user's face (or eye) with a straight line connecting the midpoints of the short sides of the shielding region 50 as the boundary. This area is referred to as a non-display area 52. In FIG. 21, (β) shows an example in which the non-display area 52 is a triangle, but the invention is not limited to this example, and a trapezoid may be used. In this trapezoid, the upper point of the midpoints of the two short sides included in the shielding region 50 is one vertex of the upper base, and the upper base is parallel to the horizontal line of the display unit 326. Such a trapezoid may be used.

  According to these display examples, content is not displayed in a region that is difficult for the user including the shielded region to view, so that the user can easily view all the content displayed on the display unit 326.

-Display example 2
In addition, when the mobile terminal 20 is separated from the display unit 326, the display control unit 106 displays the content whose display position has been moved back to the original position (that is, in the area that was the shielding area). May be. Alternatively, in the above case, the display control unit 106 may move and display a part of the content displayed outside the shielding area when the mobile terminal 20 is touched within the area that was the shielding area. According to these display examples, when the mobile terminal 20 is separated from the display unit 326, the content can be displayed so that the area that was the shielding area does not become a blank area. For this reason, it can be avoided that the blank area is seen by the user and seems unnatural to the user.

(3-1-2-5. Storage unit 122-2)
The storage unit 122-2 can further store a gripping arm determination condition DB 126 and a shielding region DB 128 compared to the storage unit 122-1.

  Note that functions of other components included in the server 10-2 are the same as those in the first embodiment.

  Note that the configuration of the server 10-2 according to the second embodiment is not limited to the configuration described above. For example, the gripping arm determination condition DB 126 and the shielding area DB 128 may be stored in another device that can communicate with the server 10-2 instead of being stored in the storage unit 122-2.

<3-2. Operation>
The configuration according to the second embodiment has been described above. Next, the operation according to the second embodiment will be described with reference to FIG. Note that the operations in S101 to S119 according to the first embodiment shown in FIG. 10 are substantially the same in the second embodiment. However, in S115 according to the second embodiment, the communication unit 322 of the display device 30 transmits a second pairing request further including the inclination of the mobile terminal 20 with respect to the display unit 326 detected in S111 to the server 10-2. This is different from the first embodiment.

  Further, the operation of S221 shown in FIG. 22 is the same as that of S121 according to the first embodiment shown in FIG. Hereinafter, the operation after S221 will be described.

  As shown in FIG. 22, after S221, the human posture estimation unit 108 of the server 10-2 performs the touch position of the mobile terminal 20 included in the second pairing request received in S115 and the mobile unit for the display unit 326. Based on the inclination of the terminal 20 and the relative position of the user estimated in S221, it is estimated whether the user's gripping arm is the left hand or the right hand (S223).

  Subsequently, the shielding area estimation unit 110 estimates the shielding area based on the touch position of the mobile terminal 20 received in S115, the relative position of the user estimated in S221, and the gripping arm estimated in S223. (S225).

  Subsequently, the display control unit 106 generates display control information for moving the content displayed in the shielding area estimated in S225 to the outside of the shielding area (S227). And the communication part 120 transmits the produced | generated display control information to the display apparatus 30 according to control of the display control part 106 (S229).

  Thereafter, the display control unit 304 of the display device 30 updates the display content of the display screen according to the received display control information, and causes the display unit 326 to display the updated display screen (S231).

<3-3. Effect>
As described above, for example, as described with reference to FIGS. 12 and 22, the server 10-2 according to the second embodiment allows the user's estimated user terminal 20 to touch the display unit 326 while the mobile terminal 20 is touched. The shielding area is estimated based on the relative position, the estimated user's gripping arm, and the touch position of the mobile terminal 20, and the display position of the content is controlled so that the content is not displayed in the estimated shielding area. For this reason, according to the second embodiment, no content is displayed in the shielding area, so that the user can easily see all the content displayed on the display unit 326.

<3-4. Modification>
Note that the second embodiment is not limited to the above description. For example, when the human posture estimation unit 108 once estimates the user's gripping arm, the display control unit 106 subsequently performs the interaction by holding the mobile terminal 20 by the user with the gripping arm (that is, the mobile terminal 20 It may be determined that it is not gripped by the other arm. Then, the display control unit 106 can display the content at a position where the shielding area generated when the mobile terminal 20 is touched by the grip arm on the display unit 326 is reduced.

<< 4. Hardware configuration >>
Next, a hardware configuration of the server 10 common to each embodiment will be described with reference to FIG. As illustrated in FIG. 23, the server 10 includes a CPU 150, a ROM 152, a RAM 154, an internal bus 156, an input / output interface 158, an HDD (Hard Disk Drive) 160, and a network interface 162.

<4-1. CPU 150>
The CPU 150 functions as an arithmetic processing device and a control device, and controls the overall operation in the server 10 according to various programs. In addition, the CPU 150 realizes the function of the control unit 100-1 or 100-2 in the server 10. The CPU 150 is configured by a processor such as a microprocessor.

<4-2. ROM152>
The ROM 152 stores programs and calculation parameters used by the CPU 150.

<4-3. RAM154>
The RAM 154 temporarily stores programs used in the execution of the CPU 150, parameters that change as appropriate during the execution, and the like.

<4-4. Internal bus 156>
The internal bus 156 is composed of a CPU bus and the like. The internal bus 156 connects the CPU 150, the ROM 152, and the RAM 154 to each other.

<4-5. Input / output interface 158>
The input / output interface 158 connects the HDD 160 and the network interface 162 to the internal bus 156. For example, the HDD 160 exchanges data with the RAM 154 or the like via the input / output interface 158 and the internal bus 156.

<4-6. HDD 160>
The HDD 160 is a data storage device that functions as the storage unit 122-1 or the storage unit 122-2. The HDD 160 includes, for example, a storage medium, a recording device that records data on the storage medium, a reading device that reads data from the storage medium, and a deletion device that deletes data recorded on the storage medium. The HDD 160 stores programs executed by the CPU 150 and various data.

<4-7. Network Interface 162>
The network interface 162 is a communication interface including a communication device for connecting to a communication network such as the Internet. This network interface 162 functions as the communication unit 120. The network interface 162 may be a wireless LAN compatible communication device, a LTE (Long Term Evolution) compatible communication device, or a wire communication device that performs wired communication.

<< 5. Modification >>
The preferred embodiments of the present invention have been described in detail above with reference to the accompanying drawings, but the present invention is not limited to such examples. It is obvious that a person having ordinary knowledge in the technical field to which the present invention pertains can come up with various changes or modifications within the scope of the technical idea described in the claims. Of course, it is understood that these also belong to the technical scope of the present invention.

<5-1. Modification 1>
For example, in each embodiment described above, an example in which the server 10 performs display control has been described. However, the present invention is not limited to such an example, and the display device 30 may perform display control. In the case of this modification, for example, the server 10-1 or the server 10-2 transmits the estimated relative position of the user (and the estimated shielding area) to the display device 30. Then, the display control unit 304 of the display device 30 changes the display position of the content displayed on the display unit 326 based on the relative position (and shielding area) of the user received from the server 10.

<5-2. Modification 2>
Moreover, although each embodiment mentioned above demonstrated the example in which a user is one person, it is not limited to this example. The present invention can also be applied to a scene where a plurality of users having the mobile terminal 20 interact with the display device 30 at the same time. For example, when there are two users, the server 10 allocates an area for the user in the display area of the display unit 326 at the stage of estimating the position of one user, and within the allocated area The display control of the content for the user may be performed. According to this modification, it is possible to suppress an interaction performed by one user from affecting other users' interactions.

<5-3. Modification 3>
Moreover, although each embodiment mentioned above demonstrated the example whose information processing apparatus in this invention is the server 10, it is not limited to this example. For example, when all the components included in the control unit 100-1 and the storage unit 122-1 (or the control unit 100-2 and the storage unit 122-2) are included in the mobile terminal 20, the information processing apparatus The portable terminal 20 may be sufficient. Further, when all the components are included in the display device 30, the information processing device may be the display device 30. In the third modification, the server 10 may not be installed.

<5-4. Modification 4>
Moreover, although each embodiment mentioned above demonstrated centering on the example whose portable terminal 20 is a mobile telephone, it is not limited to this example. For example, the portable terminal 20 may be a wearable computer such as a wristwatch computer or a clothing computer.

<5-5. Modification 5>
Further, according to each embodiment, it is also possible to provide a computer program for causing hardware such as the CPU 150, the ROM 152, and the RAM 154 to perform the same function as the configuration of the server 10 according to each embodiment described above. A recording medium on which the computer program is recorded is also provided.

10-1, 10-2 Server 20 Mobile terminal 30 Display device 32 Communication network 100-1, 100-2, 200, 300 Control unit 102 Cooperation processing unit 104 Human position estimation unit 106, 206, 304 Display control unit 108 Human posture Estimating unit 110 Shielded region estimating unit 120, 224, 322 Communication unit 122-1, 122-2, 226, 324 Storage unit 124 Human body model DB
126 Grasping arm determination condition DB
128 Shielding area DB
202, 302 Operation processing unit 204 Operation processing unit 208, 306 Cooperation control unit 220, 320 Input unit 222 Sensor unit 228, 326 Display unit

Claims (16)

A cooperation processing unit that performs a determination on establishment of information cooperation between the mobile terminal and the display unit;
Based on a change in position of the portable terminal detected by the portable terminal and a contact position of the portable terminal in the display unit detected by the display unit when it is determined that the information cooperation is established. A position estimation unit that estimates a relative position of the user carrying the terminal with respect to the display unit;
An information processing apparatus comprising:   The change in the position of the portable terminal includes the position of the portable terminal when the portable terminal is in contact with the display unit, and the position of the portable terminal before or after the portable terminal is contacted with the display unit. The information processing apparatus according to claim 1, wherein the information processing apparatus is a change between the position of the mobile terminal.   The position change of the mobile terminal is performed when the user's posture shifts from the first posture to a touch posture that is a posture when the user brings the mobile terminal into contact with the display unit, or from the touch posture. The information processing apparatus according to claim 2, wherein the information processing apparatus is a change in the position of the mobile terminal when shifting to the first posture.   The information processing apparatus according to claim 3, wherein the posture of the user is determined based on an acceleration of the mobile terminal detected by the mobile terminal or an angular acceleration of the mobile terminal.   The information processing according to any one of claims 1 to 4, wherein the position estimation unit further estimates a relative position of the user with respect to the display unit based on human body model information stored in a storage unit. apparatus.   The information processing apparatus according to any one of claims 1 to 5, wherein the information cooperation is established when the mobile terminal is in contact with the display unit while the mobile terminal is held by the user. .   When the relative position of the user is estimated by the position estimation unit, the information processing apparatus changes a display mode of content displayed on the display unit based on the estimated relative position of the user. The information processing apparatus according to any one of claims 1 to 6, further comprising a display control unit.   The information processing apparatus according to claim 7, wherein the display control unit changes a display position of the displayed content based on the relative position of the user.   9. The display control unit according to claim 8, wherein the display unit moves the content displayed around the projection position, which is a position where the relative position of the user is projected on the display unit, to the upper side of the projection position and displays the content. The information processing apparatus described. Based on the contact position of the mobile terminal, the inclination of the mobile terminal with respect to the display unit when the mobile terminal is in contact with the display unit, and the relative position of the user, A posture estimation unit that estimates whether a gripping arm that grips the mobile terminal is a left arm or a right arm;
The information processing apparatus according to claim 7, wherein the display control unit further changes a display mode of the displayed content based on the estimated gripping arm of the user.   The posture estimation unit includes an inclination threshold value indicated by a gripping arm determination condition registered in a database in association with a positional relationship between the contact position of the mobile terminal and the estimated relative position of the user, and the mobile terminal The information processing apparatus according to claim 10, wherein the information processing apparatus estimates whether the gripping arm is a left arm or a right arm by comparing with an inclination. When the portable terminal is in contact with the display unit, the information processing apparatus is configured to detect the gripping arm estimated by the posture estimation unit, the estimated relative position of the user, and the contact position of the portable terminal. Based on the above, the display unit further includes a shielding region estimation unit that estimates a shielding region that is a region shielded from the user's field of view by the gripping arm,
The information processing apparatus according to claim 10 or 11, wherein the display control unit changes a display position of content displayed in the estimated shielding area.   The information processing apparatus according to claim 12, wherein the display control unit displays content displayed in the shielding area in an area outside the shielding area. Computer
A cooperation processing unit that performs a determination on establishment of information cooperation between the mobile terminal and the display unit;
Based on a change in position of the portable terminal detected by the portable terminal and a contact position of the portable terminal in the display unit detected by the display unit when it is determined that the information cooperation is established. A position estimation unit that estimates a relative position of the user carrying the terminal with respect to the display unit;
Program to function as A mobile device,
A linkage control unit that transmits a request for establishment of information linkage with the display unit to the information processing apparatus when the portable terminal is in contact with the display unit while being held by a user;
Detects a change in position between the position of the portable terminal when the portable terminal is in contact with the display unit and the position of the portable terminal before being contacted with the display unit or after being separated from the display unit An operation control unit that causes the information processing apparatus to transmit the detected position change to the transmission unit;
A portable terminal. A mobile device,
A display for displaying content;
A cooperation processing unit that performs a determination on establishment of information cooperation between the mobile terminal and the display unit;
Based on a change in position of the portable terminal detected by the portable terminal and a contact position of the portable terminal in the display unit detected by the display unit when it is determined that the information cooperation is established. A position estimation unit that estimates a relative position of the user carrying the terminal with respect to the display unit;
An information processing system comprising:

Images